[Bug]: Empty TopK query result with expr while non-empty result without expr

[liuxiang-github]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.3.1
- Deployment mode(standalone or cluster): GKE cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.3.1 
- OS(Ubuntu or CentOS): CentOs
- CPU/Memory: GKE deployment with enough resource
- GPU: none
- Others: none

Current Behavior

Hi team,

When we execute the topk query with expr, the result is empty.

search_params = {
    "metric_type": "L2", 
    "params": {"ef": 100}
}
results = collection.search(
    data=[embed], 
    anns_field="doc_vector",
    param=search_params,
    limit=5,
    expr="(product_status == 'Legacy') and (target == 'PPCP_Direct')",
    output_fields=["*"]
)

But if we remove expr, the result is non-empty.

search_params = {
    "metric_type": "L2", 
    "params": {"ef": 100}
}
results = collection.search(
    data=[embed], 
    anns_field="doc_vector",
    param=search_params,
    limit=5,
    output_fields=["*"]
)

Record number of entire collection: 5166
Record number that matches the expr: 833

Could you help check the root cause?

table schema:

{'collection_name': 'devdocs_code_assist_nov1_titktoken_1000_20241101081159', 'auto_id': True, 'num_shards': 1, 'description': 'Document Embeddings', 'fields': [
    {'field_id': 100, 'name': 'doc_id', 'description': '', 'type': 5, 'params': {}, 'auto_id': True, 'is_primary': True
    },
    {'field_id': 101, 'name': 'doc_text', 'description': '', 'type': 21, 'params': {'max_length': 65535
      }
    },
    {'field_id': 102, 'name': 'doc_metadata', 'description': '', 'type': 21, 'params': {'max_length': 8192
      }
    },
    {'field_id': 103, 'name': 'doc_vector', 'description': '', 'type': 101, 'params': {'dim': 1536
      }
    }
  ], 'aliases': ['devdocs_code_assist_nov1_titktoken_1000'
  ], 'collection_id': 1234567890, 'consistency_level': 2, 'properties': [], 'num_partitions': 1, 'enable_dynamic_field': True
}

Expected Behavior

For the query filter which attu returned non-empty result, the topk query should also return non-empty result.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @xiaocai2333
please help to take a look
/unassign

[liuxiang-github]

When I changed from "query" to "filter" in expression, the result is normal and non-empty. Quite strange. is there any difference between the two words?

[yanliang567]

When I changed from "query" to "filter" in expression, the result is normal and non-empty. Quite strange. is there any difference between the two words?

you mean changing the key word from "expr" to "filter" in the search() request?

[xiaocai2333]

@liuxiang-github
Could you provide the query code you used with the filter?

[xiaofan-luan]

@liuxiang-github

also we would recommend to use 2.4.14 on the test. We don't find similar issues so if you still hit that you will need to check you test script. We would definitely love to help.

[xiaocai2333]

It might be related to the index you are using. When scalar filtering is applied, it may disrupt the connectivity of the HNSW index graph, leading to degraded search performance. In version 2.4, we optimized this behavior to ensure that data can be retrieved.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.