Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: After reinstalling Milvus, the QueryNode restarted during the testing process. #34988

Closed
1 task done
zhuwenxing opened this issue Jul 25, 2024 · 2 comments
Closed
1 task done

[Bug]: After reinstalling Milvus, the QueryNode restarted during the testing process. #34988

zhuwenxing opened this issue Jul 25, 2024 · 2 comments
Assignees
@weiliu1031
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20240724-c5da4fa0-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

testcases/test_action_second_deployment.py::TestActionSecondDeployment::test_check[deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_not_string_indexed_replica_number_1_is_deleted_is_deleted_data_size_3000] - pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 29: stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:208 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).SearchSegments

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/querynodev2/cluster/worker.go:107 github.com/milvus-io/milvus/internal/querynodev2/cluster.(*remoteWorker).SearchSegments

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator.go:254 github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).search.func3

[2024-07-24T10:20:25.189Z] /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/delegator.go:652 github.com/milvus-io/milvus/internal/querynodev2/delegator.executeSubTasks[...].func1

[2024-07-24T10:20:25.189Z] /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Unknown desc = node not match[expectedNodeID=32][actualNodeID=39])>

the search failed due to querynode restarting
image

[2024/07/24 10:17:56.142 +00:00] [INFO] [segments/manager.go:788] ["remove segment from cache"] [traceID=c6cdf95e582dd0b99b979b14924677e3] [segmentID=451363515462980985]
[2024/07/24 10:17:56.147 +00:00] [INFO] [segments/segment.go:1550] ["delete segment from memory"] [traceID=c6cdf95e582dd0b99b979b14924677e3] [collectionID=451363006576106644] [partitionID=451363006576106645] [segmentID=451363515462980985] [segmentType=Sealed] [insertCount=3000]
[2024/07/24 10:17:56.148 +00:00] [DEBUG] [segments/collection.go:217] ["collection ref decrement"] [collectionID=451363006576106644] [refCount=0]
[2024/07/24 10:17:56.148 +00:00] [INFO] [segments/collection.go:119] ["release collection due to ref count to 0"] [collectionID=451363006576106644]
_Z21_mm512_cmp_epi64_maskDv8_xS_i
	/usr/lib/gcc/x86_64-linux-gnu/12/include/avx512fintrin.h:10073 pc=0x7fd8a4eccee8
_ZN6milvus6bitset6detail3x866avx51216OpCompareValImplIlLNS0_13CompareOpTypeE2EE14op_compare_valEPhPKlmRS8_
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/detail/platform/x86/avx512-impl.h:194 pc=0x7fd8a4eccee8
_ZN6milvus6bitset6detail17VectorizedDynamic14op_compare_valIlLNS0_13CompareOpTypeE2EEEbPhPKT_mRS7_
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/detail/platform/dynamic.h:212 pc=0x7fd8a4968d0e
_ZZN6milvus6bitset6detail33VectorizedElementWiseBitsetPolicyImNS1_17VectorizedDynamicEE14op_compare_valIlLNS0_13CompareOpTypeE2EEEvPmmPKT_mRS9_ENKUlmmmE0_clEmmm
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/detail/element_vectorized.h:249 pc=0x7fd8a4968d0e
_ZN6milvus6bitset6detail33VectorizedElementWiseBitsetPolicyImNS1_17VectorizedDynamicEE7op_funcIZNS4_14op_compare_valIlLNS0_13CompareOpTypeE2EEEvPmmPKT_mRSA_EUlmmmE_ZNS6_IlLS7_2EEEvS8_mSB_mSC_EUlmmmE0_EEvmmS9_T0_
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/detail/element_vectorized.h:426 pc=0x7fd8a4968d0e
_ZN6milvus6bitset6detail33VectorizedElementWiseBitsetPolicyImNS1_17VectorizedDynamicEE14op_compare_valIlLNS0_13CompareOpTypeE2EEEvPmmPKT_mRS9_
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/detail/element_vectorized.h:236 pc=0x7fd8a4968d0e
_ZN6milvus6bitset10BitsetBaseINS0_6detail33VectorizedElementWiseBitsetPolicyImNS2_17VectorizedDynamicEEENS0_10BitsetViewIS5_Lb1EEELb1EE19inplace_compare_valIlLNS0_13CompareOpTypeE2EEEvPKT_mRSC_
	/go/src/github.com/milvus-io/milvus/internal/core/src/bitset/bitset.h:501 pc=0x7fd8a4968d0e
_ZN6milvus4exec16UnaryElementFuncIlLNS_5proto4plan6OpTypeE2EEclEPKlmlNS_6bitset10BitsetViewINS8_6detail33VectorizedElementWiseBitsetPolicyImNSA_17VectorizedDynamicEEELb1EEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/UnaryExpr.h:117 pc=0x7fd8a4968d0e
_ZZN6milvus4exec23PhyUnaryRangeFilterExpr27ExecRangeVisitorImplForDataIlEESt10shared_ptrINS_10BaseVectorEEvENKUlPKliNS_6bitset10BitsetViewINS8_6detail33VectorizedElementWiseBitsetPolicyImNSA_17VectorizedDynamicEEELb1EEElE_clES7_iSE_l
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/UnaryExpr.cpp:786 pc=0x7fd8a4968d0e
_ZN6milvus4exec11SegmentExpr17ProcessDataChunksIlZNS0_23PhyUnaryRangeFilterExpr27ExecRangeVisitorImplForDataIlEESt10shared_ptrINS_10BaseVectorEEvEUlPKliNS_6bitset10BitsetViewINSA_6detail33VectorizedElementWiseBitsetPolicyImNSC_17VectorizedDynamicEEELb1EEElE_JlEEElT0_St8functionIFbRKNS_9SkipIndexEN6fluent9NamedTypeIlNS_4impl10FieldIdTagEJNSN_10ComparableENSN_8HashableEEEEiEESG_DpT1_
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/Expr.h:253 pc=0x7fd8a4968d0e
_ZN6milvus4exec23PhyUnaryRangeFilterExpr27ExecRangeVisitorImplForDataIlEESt10shared_ptrINS_10BaseVectorEEv
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/UnaryExpr.cpp:833 pc=0x7fd8a49698f9
_ZN6milvus4exec23PhyUnaryRangeFilterExpr20ExecRangeVisitorImplIlEESt10shared_ptrINS_10BaseVectorEEv
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/UnaryExpr.cpp:602 pc=0x7fd8a4942c5a
_ZN6milvus4exec23PhyUnaryRangeFilterExpr4EvalERNS0_7EvalCtxERSt10shared_ptrINS_10BaseVectorEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/UnaryExpr.cpp:165 pc=0x7fd8a4942c5a
[2024/07/24 10:17:56.425 +00:00] [INFO] [querynodev2/services.go:385] ["received load partitions request"] [traceID=007c65cfeba7ca9cf2b09ae63d12dac5] [collectionID=451363006576106644] [partitionIDs="[451363006576106645]"]
[2024/07/24 10:17:56.425 +00:00] [INFO] [querynodev2/services.go:397] ["load partitions done"] [traceID=007c65cfeba7ca9cf2b09ae63d12dac5] [collectionID=451363006576106644] [partitionIDs="[451363006576106645]"]
_ZN6milvus4exec7ExprSet4EvalEiibRNS0_7EvalCtxERSt6vectorISt10shared_ptrINS_10BaseVectorEESaIS7_EE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/expression/Expr.cpp:42 pc=0x7fd8a4930dc7
_ZN6milvus4exec10FilterBits9GetOutputEv
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/operator/FilterBits.cpp:67 pc=0x7fd8a492bdc0
_ZN6milvus4exec6Driver11RunInternalERSt10shared_ptrIS1_ERS2_INS0_13BlockingStateEERS2_INS_9RowVectorEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/Driver.cpp:249 pc=0x7fd8a4925893
_ZN6milvus4exec6Driver4NextERSt10shared_ptrINS0_13BlockingStateEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/Driver.cpp:137 pc=0x7fd8a4926236
_ZN6milvus4exec4Task4NextEPN5folly10SemiFutureINS2_4UnitEEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/exec/Task.cpp:199 pc=0x7fd8a491a767
_ZN6milvus5query19ExecPlanNodeVisitor23ExecuteExprNodeInternalERKSt10shared_ptrINS_4plan8PlanNodeEEPKNS_7segcore24SegmentInternalInterfaceElRNS_6bitset6BitsetINSC_6detail33VectorizedElementWiseBitsetPolicyImNSE_17VectorizedDynamicEEEN5folly8fbvectorIhSaIhEEELb1EEERbRSt6vectorIlSaIlEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/query/visitors/ExecPlanNodeVisitor.cpp:98 pc=0x7fd8a4731d74
_ZN6milvus5query19ExecPlanNodeVisitor15ExecuteExprNodeERKSt10shared_ptrINS_4plan8PlanNodeEEPKNS_7segcore24SegmentInternalInterfaceElRNS_6bitset6BitsetINSC_6detail33VectorizedElementWiseBitsetPolicyImNSE_17VectorizedDynamicEEEN5folly8fbvectorIhSaIhEEELb1EEE
	/go/src/github.com/milvus-io/milvus/internal/core/src/query/generated/ExecPlanNodeVisitor.h:122 pc=0x7fd8a4736ac3
_ZN6milvus5query19ExecPlanNodeVisitor17VectorVisitorImplINS_11FloatVectorEEEvRNS0_14VectorPlanNodeE
	/go/src/github.com/milvus-io/milvus/internal/core/src/query/visitors/ExecPlanNodeVisitor.cpp:172 pc=0x7fd8a4736ac3
_ZN6milvus5query19ExecPlanNodeVisitor16get_moved_resultERNS0_8PlanNodeE
	/go/src/github.com/milvus-io/milvus/internal/core/src/query/generated/ExecPlanNodeVisitor.h:61 pc=0x7fd8a46d1411
_ZNK6milvus7segcore24SegmentInternalInterface6SearchEPKNS_5query4PlanEPKNS2_16PlaceholderGroupEm
	/go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentInterface.cpp:77 pc=0x7fd8a46d1411
[2024/07/24 10:17:56.627 +00:00] [DEBUG] [pipeline/stream_pipeline.go:59] ["stream pipeline fetch msg"] [sum=0]
[2024/07/24 10:17:56.643 +00:00] [INFO] [querynodev2/services.go:203] ["received watch channel request"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [currentNodeID=32] [version=1721816276642576372]
[2024/07/24 10:17:56.643 +00:00] [INFO] [segments/collection.go:95] ["put new collection"] [collectionID=451363006576106644] [schema="name:\"deploy_test_index_type_IVF_PQ_is_compacted_is_compacted_segment_status_only_growing_is_string_indexed_not_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000\" fields:<name:\"RowID\" description:\"row id\" data_type:Int64 > fields:<fieldID:100 name:\"int64\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"float\" data_type:Float > fields:<fieldID:102 name:\"varchar\" data_type:VarChar type_params:<key:\"max_length\" value:\"65535\" > > fields:<fieldID:103 name:\"float_vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"128\" > > fields:<fieldID:1 name:\"Timestamp\" description:\"time stamp\" data_type:Int64 > "]
I20240724 10:17:56.643452  2520 Collection.cpp:64] [SERVER][parseIndexMeta][milvus] index meta info: maxRowCount : {161870} FieldId : {103 IndexParams : { index_type : IVF_PQ, m : 16, metric_type : L2, nbits : 8, nlist : 128,  }TypeParams : {dim : 128, }}
[2024/07/24 10:17:56.643 +00:00] [DEBUG] [segments/collection.go:208] ["collection ref increment"] [collectionID=451363006576106644] [refCount=1]
[2024/07/24 10:17:56.643 +00:00] [INFO] [delegator/delegator.go:851] ["Init delete cache with list delete buffer"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [replicaID=451363546099089649] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [version=1721816276642576372] [startTs=451363805456760833] [sizePerBlock=1048576] [startTime=2024/07/24 10:17:54.478 +00:00]
[2024/07/24 10:17:56.643 +00:00] [INFO] [delegator/delegator.go:881] ["finish build new shardDelegator"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [replicaID=451363546099089649] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [version=1721816276642576372] [startTs=451363805456760833]
[2024/07/24 10:17:56.643 +00:00] [INFO] [tsafe/manager.go:69] ["add tSafe done"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [timestamp=2024/07/24 10:17:54.478 +00:00]
[2024/07/24 10:17:56.643 +00:00] [INFO] [pipeline/manager.go:66] ["start create pipeine"] [collectionID=451363006576106644] [channel=by-dev-rootcoord-dml_9_451363006576106644v0]
[2024/07/24 10:17:56.643 +00:00] [INFO] [delegator/delegator_data.go:408] ["loading growing segments..."] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [replicaID=451363546099089649] [segmentIDs="[]"]
[2024/07/24 10:17:56.643 +00:00] [INFO] [segments/segment_loader.go:611] ["no segment to load"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [segmentType=Growing]
[2024/07/24 10:17:56.643 +00:00] [INFO] [delegator/delegator_data.go:440] ["load growing segments done"] [traceID=01674b1a0b278fb0ff09f3f25607043b] [collectionID=451363006576106644] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [replicaID=451363546099089649] [segmentIDs="[]"]
[2024/07/24 10:17:56.643 +00:00] [INFO] [delegator/delegator_data.go:397] ["add growing segments to delegator"] [collectionID=451363006576106644] [channel=by-dev-rootcoord-dml_9_451363006576106644v0] [replicaID=451363546099089649] [segmentIDs="[]"]
[2024/07/24 10:17:56.643 +00:00] [INFO] [msgdispatcher/dispatcher.go:93] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_9] [subName=querynode-32-by-dev-rootcoord-dml_9_451363006576106644v0-false] [isMain=false]
[2024/07/24 10:17:56.643 +00:00] [INFO] [msgstream/mq_msgstream.go:117] ["Msg Stream state"] [can_produce=true]
[2024/07/24 10:17:56.648 +00:00] [INFO] [msgstream/mq_msgstream.go:878] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_9] [MessageID="CIAIEOMoGAAgAA=="]
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/segcore/segment_c.cpp:112 pc=0x7fd8a45d57e4
operator()
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:181 pc=0x7fd8a45d57e4
makeTryWithNoUnwrap<const milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:254 pc=0x7fd8a45d57e4
makeTryWith<const milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7fd8a45d57e4
setWith<const milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/SharedPromise-inl.h:81 pc=0x7fd8a45d57e4
operator()<folly::Try<folly::Unit> >
	/go/src/github.com/milvus-io/milvus/internal/core/src/futures/Future.h:187 pc=0x7fd8a45d57e4
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:941 pc=0x7fd8a45d57e4
invoke<folly::Executor::KeepAlive<folly::Executor>, folly::Try<folly::Unit> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:139 pc=0x7fd8a45d57e4
invoke<folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::CoreCallbackState<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >, folly::Unit>
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:341 pc=0x7fd8a45d57e4
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:388 pc=0x7fd8a45d57e4
makeTryWithNoUnwrap<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:265 pc=0x7fd8a45d57e4
makeTryWith<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> mutable::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Try-inl.h:276 pc=0x7fd8a45d57e4
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/Future-inl.h:387 pc=0x7fd8a45d57e4
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/futures/detail/Core.h:629 pc=0x7fd8a45d57e4
callBig<folly::futures::detail::Core<folly::Unit>::setCallback<folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)> >(folly::futures::detail::FutureBase<folly::Unit>::thenImplementation<folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void> >(folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, folly::futures::detail::tryExecutorCallableResult<folly::Unit, folly::Future<folly::Unit>::thenTry<milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&>(milvus::futures::Future<milvus::SearchResult>::asyncProduce<AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)> >(folly::Executor::KeepAlive<>, int, AsyncSearch(CTraceContext, CSegmentInterface, CSearchPlan, CPlaceholderGroup, uint64_t)::<lambda(milvus::futures::CancellationToken)>&&)::<lambda(auto:85&&)>&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>, void>, folly::futures::detail::InlineContinuation)::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Unit>&&)>&&, std::shared_ptr<folly::RequestContext>&&, folly::futures::detail::InlineContinuation)::<lambda(folly::futures::detail::CoreBase&, folly::Executor::KeepAlive<>&&, folly::exception_wrapper*)> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/package/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/include/folly/Function.h:359 pc=0x7fd8a45d57e4
_ZN5folly6detail8function14FunctionTraitsIFvRNS_7futures6detail8CoreBaseEONS_8Executor9KeepAliveIS7_EEPNS_17exception_wrapperEEEclES6_SA_SC_
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7fd89c0d2d3f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/futures/detail/Core.cpp:616 pc=0x7fd89c0d2d3f
operator()
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:187 pc=0x7fd89c0d2d3f
callSmall<folly::Executor::KeepAlive<>::add<folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)> >(folly::futures::detail::CoreBase::doCallback(folly::Executor::KeepAlive<>&&, folly::futures::detail::State)::<lambda(folly::Executor::KeepAlive<>&&)>&&) &&::<lambda()> >
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7fd89c0d2d3f
_ZN5folly6detail8function14FunctionTraitsIFvvEEclEv
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:374 pc=0x7fd89bff2e75
_ZN5folly15catch_exceptionIRNS_8FunctionIFvvEEERDoFvPKcEJRS6_EvEET2_OT_OT0_DpOT1_
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/lang/Exception.h:286 pc=0x7fd89bff2e75
_ZN5folly8Executor18invokeCatchingExnsINS_8FunctionIFvvEEEEEvPKcT_
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Executor.h:234 pc=0x7fd89bff2e75
_ZN5folly18ThreadPoolExecutor7runTaskERKSt10shared_ptrINS0_6ThreadEEONS0_4TaskE
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/ThreadPoolExecutor.cpp:102 pc=0x7fd89bff2e75
_ZN5folly21CPUThreadPoolExecutor9threadRunESt10shared_ptrINS_18ThreadPoolExecutor6ThreadEE
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/executors/CPUThreadPoolExecutor.cpp:333 pc=0x7fd89bfb5e74
_ZSt13__invoke_implIvRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEERPS1_JRS4_EET_St21__invoke_memfun_derefOT0_OT1_DpOT2_
	/usr/include/c++/12/bits/invoke.h:74 pc=0x7fd89bff544b
_ZSt8__invokeIRMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEJRPS1_RS4_EENSt15__invoke_resultIT_JDpT0_EE4typeEOSC_DpOSD_
	/usr/include/c++/12/bits/invoke.h:96 pc=0x7fd89bff544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
	/usr/include/c++/12/functional:495 pc=0x7fd89bff544b
_ZNSt5_BindIFMN5folly18ThreadPoolExecutorEFvSt10shared_ptrINS1_6ThreadEEEPS1_S4_EEclIJEvEET0_DpOT_
	/usr/include/c++/12/functional:580 pc=0x7fd89bff544b
_ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallISt5_BindIFMNS_18ThreadPoolExecutorEFvSt10shared_ptrINS7_6ThreadEEEPS7_SA_EEEEvRNS1_4DataE
	/home/milvus/.conan/data/folly/2023.10.30.08/milvus/dev/build/71e52ec7e6bdcb39e8f12e598f0e25527e54965c/folly/Function.h:345 pc=0x7fd89bff544b
(null)
	(null):0 pc=0x7fd8a276b252
(null)
	(null):0 pc=0x7fd8a294fac2
(null)
	(null):0 pc=0x7fd8a29e184f
(null)
	(null):0 pc=0xffffffffffffffff

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_cron/detail/deploy_test_cron/2572/pipeline

log:
artifacts-pulsar-cluster-reinstall-2572-server-logs (1).tar.gz

cluster: 4am
namespace: chaos-testing
pod info

 + kubectl get pods -o wide
 + grep pulsar-cluster-reinstall-2572
 pulsar-cluster-reinstall-2572-bookie-0                            1/1     Running            0                 22m     10.104.33.75    4am-node36   <none>           <none>
 pulsar-cluster-reinstall-2572-bookie-1                            1/1     Running            0                 22m     10.104.19.236   4am-node28   <none>           <none>
 pulsar-cluster-reinstall-2572-broker-0                            1/1     Running            0                 22m     10.104.32.207   4am-node39   <none>           <none>
 pulsar-cluster-reinstall-2572-etcd-0                              1/1     Running            0                 22m     10.104.25.180   4am-node30   <none>           <none>
 pulsar-cluster-reinstall-2572-etcd-1                              1/1     Running            0                 22m     10.104.17.200   4am-node23   <none>           <none>
 pulsar-cluster-reinstall-2572-etcd-2                              1/1     Running            0                 22m     10.104.33.74    4am-node36   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-datacoord-db9dd794-xbqqv     1/1     Running            4 (20m ago)       22m     10.104.18.93    4am-node25   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-datanode-8648787665-kjtdt    1/1     Running            3 (21m ago)       22m     10.104.18.98    4am-node25   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-datanode-8648787665-qfswq    1/1     Running            3 (21m ago)       22m     10.104.30.159   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-indexcoord-698c5c7d77wmfgq   1/1     Running            0                 22m     10.104.23.161   4am-node27   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-indexnode-6949c674f6-mhpsn   1/1     Running            3 (21m ago)       22m     10.104.18.99    4am-node25   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-indexnode-6949c674f6-prs78   1/1     Running            3 (21m ago)       22m     10.104.30.161   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-indexnode-6949c674f6-zjgpb   1/1     Running            3 (21m ago)       22m     10.104.23.162   4am-node27   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-proxy-597bd87665-cltht       1/1     Running            3 (21m ago)       22m     10.104.23.163   4am-node27   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-querycoord-664f64865dwdlq9   1/1     Running            3 (21m ago)       22m     10.104.30.160   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-querynode-7646f4f445-hg76j   1/1     Running            3 (21m ago)       22m     10.104.4.50     4am-node11   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-querynode-7646f4f445-l6jl7   1/1     Running            4 (2m38s ago)     22m     10.104.30.165   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-querynode-7646f4f445-vbjdt   1/1     Running            3 (21m ago)       22m     10.104.23.164   4am-node27   <none>           <none>
 pulsar-cluster-reinstall-2572-milvus-rootcoord-7db55bc64b-4swt8   1/1     Running            3 (21m ago)       22m     10.104.30.163   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-minio-0                             1/1     Running            0                 22m     10.104.30.166   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-2572-minio-1                             1/1     Running            0                 22m     10.104.17.202   4am-node23   <none>           <none>
 pulsar-cluster-reinstall-2572-minio-2                             1/1     Running            0                 22m     10.104.19.238   4am-node28   <none>           <none>
 pulsar-cluster-reinstall-2572-minio-3                             1/1     Running            0                 22m     10.104.33.76    4am-node36   <none>           <none>
 pulsar-cluster-reinstall-2572-proxy-0                             1/1     Running            0                 22m     10.104.27.194   4am-node31   <none>           <none>
 pulsar-cluster-reinstall-2572-recovery-0                          1/1     Running            0                 22m     10.104.27.196   4am-node31   <none>           <none>
 pulsar-cluster-reinstall-2572-zookeeper-0                         1/1     Running            0                 22m     10.104.25.181   4am-node30   <none>           <none>

Anything else?

No response
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 25, 2024
@zhuwenxing zhuwenxing added this to the 2.5.0 milestone Jul 25, 2024
@zhuwenxing zhuwenxing added the test/chaos chaos test label Jul 25, 2024
@yanliang567
Copy link
Contributor

/assign @weiliu1031
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 26, 2024
@stale Stale
Copy link

stale bot commented Aug 25, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Aug 25, 2024
@stale stale bot closed this as completed Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days test/chaos chaos test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
3 participants
@zhuwenxing @yanliang567 @weiliu1031