[Bug]: load timeout

[ChaoXuaan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.2.9
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 
- OS(Ubuntu or CentOS): centos
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

milvus cluster load collection fail, we found that querynode load timeout and will panic. error logs as follows

[2024/07/26 14:10:58.269 +08:00] [DEBUG] [querynode/flow_graph_service_time_node.go:76] ["update tSafe:"] [collectionID=448059328615196981] [tSafe=451405219876569089] [tSafe_p=2024/07/26 14:10:57.952 +08:00] [tsLag=317.86284ms] [channel=mcv-rootcoord-dml_9_448059328615196981v0]
[2024/07/26 14:10:58.961 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds\n"]
[2024/07/26 14:10:58.961 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5513 milliseconds\n"]
[2024/07/26 14:10:58.967 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds\n"]
[2024/07/26 14:10:58.967 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5513 milliseconds\n"]
[2024/07/26 14:10:58.968 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5513 milliseconds\n"]
[2024/07/26 14:10:58.969 +08:00] [WARN] [querynode/cgo_helper.go:56] ["LoadFieldData failed, C Runtime Exception: [UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds\n"]
[2024/07/26 14:10:58.973 +08:00] [INFO] [gc/gc_tuner.go:84] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=29] ["total memory"=117] ["next GC"=63] ["new GOGC"=200] [gc-pause=84.789µs] [gc-pause-end=1721974258972366444]
[2024/07/26 14:10:58.975 +08:00] [ERROR] [querynode/segment_loader.go:205] ["load segment failed when load data into memory"] [collectionID=448059328615207941] [segmentType=Sealed] [partitionID=448059328615207942] [segmentID=448059328627448762] [error="[UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds"] [stack="github.com/milvus-io/milvus/internal/querynode.(*segmentLoader).LoadSegment.func3\n\t/go/src/github.com/milvus-io/milvus/internal/querynode/segment_loader.go:205\ngithub.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:83"]
[2024/07/26 14:10:58.975 +08:00] [ERROR] [funcutil/parallel.go:85] [loadSegmentFunc] [error="[UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds"] [idx=0] [stack="github.com/milvus-io/milvus/internal/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/parallel.go:85"]
[2024/07/26 14:10:58.975 +08:00] [DEBUG] [funcutil/parallel.go:51] [loadSegmentFunc] [total=1] ["time cost"=1m22.174494008s]
[2024/07/26 14:10:58.975 +08:00] [INFO] [querynode/segment.go:236] ["delete segment from memory"] [collectionID=448059328615207941] [partitionID=448059328615207942] [segmentID=448059328627448762] [segmentType=Sealed]
[2024/07/26 14:10:58.978 +08:00] [INFO] [gc/gc_tuner.go:84] ["GC Tune done"] ["previous GOGC"=200] ["heapuse "=23] ["total memory"=114] ["next GC"=52] ["new GOGC"=200] [gc-pause=55.434µs] [gc-pause-end=1721974258977661756]
[2024/07/26 14:10:58.979 +08:00] [WARN] [querynode/load_segment_task.go:125] ["failed to load segment"] [collectionID=448059328615207941] [replicaID=451402767679356932] [error="[UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds"]
[2024/07/26 14:10:58.979 +08:00] [WARN] [querynode/task_scheduler.go:110] ["[UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds"]
[2024/07/26 14:10:58.979 +08:00] [INFO] [querynode/load_segment_task.go:40] ["LoadSegmentTask PreExecute start"] [msgID=268]
[2024/07/26 14:10:58.979 +08:00] [INFO] [querynode/load_segment_task.go:66] ["LoadSegmentTask PreExecute done"] [msgID=268]
[2024/07/26 14:10:58.979 +08:00] [INFO] [querynode/load_segment_task.go:71] ["LoadSegmentTask Execute start"] [msgID=268]
[2024/07/26 14:10:58.980 +08:00] [INFO] [querynode/segment_loader.go:125] ["segmentLoader start loading..."] [collectionID=448059328617569901] [segmentType=Sealed] [segmentNum=1] [msgID=268]
[2024/07/26 14:10:58.980 +08:00] [WARN] [querynode/impl.go:522] ["[UnexpectedError] Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5512 milliseconds"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

Error:GetObjectSize:ClientError:200028 Timeout was reached.Resolving timed out after 5513 milliseconds\

[xiaofan-luan]

this seems to to be a timeout .

1. upgrade to at least 2.2.17 or above, recommended 2.3.20
2. check your config. make sure you can right connection to your S3.
   3

[yanliang567]

/assign @ChaoXuaan
/unassign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.