Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: hybrid search failed on full text index with error code=2200, message=incomplete query result #38528

Open
1 task done
Squire-tomsk opened this issue Dec 17, 2024 · 3 comments
Open
1 task done

[Bug]: hybrid search failed on full text index with error code=2200, message=incomplete query result #38528

Squire-tomsk opened this issue Dec 17, 2024 · 3 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.2

Comments

@Squire-tomsk
Copy link

Squire-tomsk commented Dec 17, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.5.0-beta
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.5.0
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When I perform hybrid search call:

sparse_req = AnnSearchRequest(
    data=[keywords_separate_with_whitespace],
    anns_field="sparse_embedding",
    param={"metric_type": "BM25", "drop_ratio_search": 0.2},
    limit=10
)

dense_req = AnnSearchRequest(
    data=[dense_emb_1024_dim],
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {}},
    limit=10
)

ranker = RRFRanker(60)

collection.hybrid_search(
    [dense_req, sparse_req],
    rerank=ranker,
    limit=10,
    output_fields=["text"],
)

I see

RPC error: [hybrid_search], <MilvusException: (code=2200, message=incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result)>, <Time:{'RPC start': '2024-12-17 09:00:43.574024', 'RPC error': '2024-12-17 09:01:05.340891'}>

Explicit search by dense and sparse embeddings separatly works correct on same collection.

Steps To Reproduce

1. I have used default Milvus standalone 2.5.0-beta docker.
2. Collection schema


schema = MilvusClient.create_schema(auto_id=False, enable_dynamic_field=True)
schema.add_field(
    field_name="id",
    datatype=DataType.VARCHAR,
    max_length=65535,
    is_primary=True,
)
schema.add_field(
    field_name="embedding",
    datatype=DataType.FLOAT_VECTOR,
    dim=1024,
)
schema.add_field(
    field_name="text",
    datatype=DataType.VARCHAR, 
    max_length=65_535, 
    enable_analyzer=False
)
schema.add_field(
    field_name="keywords",
    datatype=DataType.VARCHAR, 
    max_length=65_535, 
    enable_analyzer=True
)
schema.add_field(
    field_name="sparse_embedding",
    datatype=DataType.SPARSE_FLOAT_VECTOR,
)
bm25_function = Function(
    name="text_bm25_emb",
    input_field_names=["keywords"],
    output_field_names=["sparse_embedding"],
    function_type=FunctionType.BM25,
)
schema.add_function(bm25_function)
milvusclient.create_collection(
    collection_name=collection_name, schema=schema
)
  1. Indexes
collection = Collection(collection_name, using=milvusclient._using)

sparse_index = {"index_type": "AUTOINDEX", "metric_type": "BM25"}
collection.create_index("sparse_embedding", sparse_index)

dense_index = {"params": {}, "metric_type": "COSINE", "index_type": "FLAT"}
collection.create_index("embedding", dense_index)
  1. Insert few thousands records and delete few hundreds.
  2. Bug occurs. The bug does not occur immediately on a small amount of data, but if it occurs, it is reproduced stably.


### Milvus Log

2024-12-17 12:20:19 [2024/12/17 09:20:19.977 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=fa34366b1ee90bcb3a6309d95265e658] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:21 [2024/12/17 09:20:21.389 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [proxy/impl.go:6169] ["connect received"] [traceID=6a83fc36b562fd77d6315fa7055af419] [sdk_type=Python] [sdk_version=2.5.0] [local_time="2024-12-17 09:20:22.020270"] [user=] [host=docker-desktop] [db=default]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/root_coord.go:1007] ["received request to list databases"] [traceID=6a83fc36b562fd77d6315fa7055af419] [msgID=0]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=1]
2024-12-17 12:20:22 [2024/12/17 09:20:22.022 +00:00] [INFO] [rootcoord/root_coord.go:1033] ["done to list databases"] [traceID=6a83fc36b562fd77d6315fa7055af419] [msgID=0] ["num of databases"=1]
2024-12-17 12:20:22 [2024/12/17 09:20:22.023 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=6a83fc36b562fd77d6315fa7055af419] [sdk_type=Python] [sdk_version=2.5.0] [local_time="2024-12-17 09:20:22.020270"] [user=] [host=docker-desktop] [identifier=454669689687638018] [last_active_time=2024/12/17 09:20:22.023 +00:00]
2024-12-17 12:20:22 [2024/12/17 09:20:22.026 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=2]
2024-12-17 12:20:22 [2024/12/17 09:20:22.029 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:22 [2024/12/17 09:20:22.031 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:24 [2024/12/17 09:20:24.393 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.123 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=0] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.378 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.845 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.845 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:25 [2024/12/17 09:20:25.846 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:26 [2024/12/17 09:20:26.702 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:27 [2024/12/17 09:20:27.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:28 [2024/12/17 09:20:28.359 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=4] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.975 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:29 [2024/12/17 09:20:29.977 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=ea79fce7f6e42eab14e68dfac07ca640] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [currentNodeID=2] [Action=UpdateVersion] [TargetVersion=1734427219977736171]
2024-12-17 12:20:29 [2024/12/17 09:20:29.977 +00:00] [INFO] [delegator/distribution.go:305] ["Update readable segment version"] [oldVersion=1734427209977139042] [newVersion=1734427219977736171] [growingSegmentNum=0] [sealedSegmentNum=1]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [observers/target_observer.go:531] ["observer trigger update current target"] [collectionID=454651567748415768]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/services.go:857] ["get recovery info request received"] [traceID=a5d9baf714036d0dfdcceb4bf1e9a1cc] [collectionID=454651567748415768] [partitionIDs="[]"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [numOfSegments=11] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454651567748415769\":0}"]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [posTs=454669679176187907] [posTime=2024/12/17 09:19:41.916 +00:00]
2024-12-17 12:20:29 [2024/12/17 09:20:29.978 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=a5d9baf714036d0dfdcceb4bf1e9a1cc] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:30 [2024/12/17 09:20:30.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:31 [2024/12/17 09:20:31.414 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:33 [2024/12/17 09:20:33.386 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:34 [2024/12/17 09:20:34.477 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:36 [2024/12/17 09:20:36.388 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:37 [2024/12/17 09:20:37.552 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.387 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.976 +00:00] [INFO] [rootcoord/task.go:176] [NewLockerKeyChain] [lockerKeys=3]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [querynodev2/services.go:1295] ["sync action"] [traceID=b5f8a653173085e961b873d451801ace] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [currentNodeID=2] [Action=UpdateVersion] [TargetVersion=1734427229979150968]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [delegator/distribution.go:305] ["Update readable segment version"] [oldVersion=1734427219977736171] [newVersion=1734427229979150968] [growingSegmentNum=0] [sealedSegmentNum=1]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [observers/target_observer.go:531] ["observer trigger update current target"] [collectionID=454651567748415768]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/services.go:857] ["get recovery info request received"] [traceID=f33b577d1a63ec1a98ab8f3bd3c008ae] [collectionID=454651567748415768] [partitionIDs="[]"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/handler.go:244] [GetQueryVChanPositions] [collectionID=454651567748415768] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [numOfSegments=11] ["result flushed"=1] ["result growing"=0] ["result L0"=0] ["partition stats"="{\"454651567748415769\":0}"]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/handler.go:348] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] [posTs=454669679176187907] [posTime=2024/12/17 09:19:41.916 +00:00]
2024-12-17 12:20:39 [2024/12/17 09:20:39.977 +00:00] [INFO] [datacoord/services.go:872] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=f33b577d1a63ec1a98ab8f3bd3c008ae] [collectionID=454651567748415768] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_454651567748415768v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=10] ["# of indexed segments"=0] ["# of l0 segments"=0]
2024-12-17 12:20:40 [2024/12/17 09:20:40.604 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.604 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.605 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:40 [2024/12/17 09:20:40.605 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [retried=8] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"]
2024-12-17 12:20:42 [2024/12/17 09:20:42.387 +00:00] [INFO] [balance/score_based_balancer.go:353] ["node workload status"] [collectionID=454651567748415768] [replicaID=454651567907799041] [nodes="[\"{NodeID: 2, AssignedScore: 26150.000000, CurrentScore: 28765.000000, Priority: 2615}\"]"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.676 +00:00] [WARN] [proxy/task_search.go:740] ["failed to requery"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [nq=1] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.677 +00:00] [WARN] [proxy/task_scheduler.go:488] ["Failed to post-execute task: "] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:43 [2024/12/17 09:20:43.677 +00:00] [WARN] [proxy/impl.go:3303] ["HybridSearch failed to WaitToFinish"] [traceID=08d003de00edaa47f0f0e9a02f45ae89] [role=proxy] [db=default] [collection=KEweb_DEV_6] [partitions="[]"] [OutputFields="[title,url,source,created,ref_doc_id,text]"] [ConsistencyLevel=Strong] [useDefaultConsistency=true] [error="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result"] [errorVerbose="incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768: inconsistent requery result\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrInconsistentRequery\n  | \t/workspace/source/pkg/util/merr/utils.go:1104\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Requery\n  | \t/workspace/source/internal/proxy/task_search.go:906\n  | github.com/milvus-io/milvus/internal/proxy.(*searchTask).PostExecute\n  | \t/workspace/source/internal/proxy/task_search.go:738\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/workspace/source/internal/proxy/task_scheduler.go:485\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/workspace/source/internal/proxy/task_scheduler.go:566\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/workspace/source/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1222\nWraps: (2) incomplete query result, missing id 5098ee92-20aa-4e6a-87df-7109bdf67bbc, len(searchIDs) = 10, len(queryIDs) = 1, collection=454651567748415768\nWraps: (3) inconsistent requery result\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106802]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750123608]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750157216]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567749026040]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106800]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750140412]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567748921126]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750089996]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750088021]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750106804]
2024-12-17 12:20:44 [2024/12/17 09:20:44.164 +00:00] [INFO] [delegator/exclude_info.go:79] ["remove segment from exclude info"] [segmentID=454651567750455507]
@Squire-tomsk Squire-tomsk added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 17, 2024
@chenyiminch
Copy link

The same problem. The bug does not occur on a small amount of data (e.g. thousand samples in database). But when the number of samples in the database is more than 100 thousand, the bug occurs.

@yanliang567
Copy link
Contributor

/assign @congqixia
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 24, 2024
@yanliang567 yanliang567 added this to the 2.5.1 milestone Dec 24, 2024
@DotaArtist
Copy link

The same problem.

@yanliang567 yanliang567 modified the milestones: 2.5.1, 2.5.2 Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.2
Development

No branches or pull requests
5 participants
@chenyiminch @Squire-tomsk @DotaArtist @yanliang567 @congqixia