[Bug]: the performance of partition keys is far inferior to scalar retrieval #38574
Comments
douglarek
added
kind/bug
|
@douglarek That's an interesting test. Thank you for you updates. Quick questions:
/assign @douglarek |
yanliang567
added
triage/needs-information
|
|
@yanliang567 All information has been provided. |
|
@douglarek thank you for your quick update. could you also provide the birdwatcher backup file for investigation. Please refer to this doc: https://github.com/milvus-io/birdwatcher to backup etcd backup with birdwatcher |
|
what is scalar search? are you search by partition? or you search with filter? |
|
and how does the table defined? |
|
the QPS seems to be too low under whatever test case you are testing |
|
@douglarek thank you for the update. I checked the logs, but the logs only have INFO logs, so we cannot find any search info in the logs. If convenient, could you please set the log level to debug, reproduce the issue and recollect the logs for investigation? |
Okay, I'll adjust the log to debug this afternoon and collect it again. Additionally, I will also provide information about birdwatcher. Thank you for paying attention to this issue. |
Perhaps it is like this, I will first provide some necessary information. I am not sure if it is my posture that is incorrect or an issue with Milvus, so I reserve judgment on this matter. |
@yanliang567 The following are the provided relevant logs and backup. milvus log(debug enabled): birdwatcher backup: |
|
@douglarek thank you for the update. unfortunately, it seems that the dirdwatcher backup file is broken or not backup successfully. After I loaded the backup file it show 0 collections and 0 segments. Could you please retry to backup a new file for us? |
|
@yanliang567 Before I started debugging, I made a backup (bw_etcd_ALL.241224-073220.bak.gz, perhaps this data is also acceptable?) . during debugging, I was too eager and forcibly used kubectl to delete some pods, causing certain nodes in etcd to be in a kill-triggered state. Birdwatcher exported with errors but still generated files; unexpectedly, the files are unusable.
yeah, my mistake. my test program previously tested the HNSW index, and this parameter might not have been deleted. Will this parameter affect IVF_FLAT(yes, the index is it!)? Perhaps I should correct this parameter and retest?
yes, the first one I queried when loading the collection in attu. |
Is there an existing issue for this?
Environment
Current Behavior
First, align a conclusion: whether partition key retrieval improves performance under the premise of scalar fields. If so, is the theoretical performance of partition keys higher than that of purely scalar field retrieval?
According to the test, I found that the retrieval performance of these two is vastly different.
Steps:
Insert data into the following set as shown in the figure
collection schema information:
json
{ "data": [ { "collection_name": "benchmark_v4", "schema": { "fields": [ { "type_params": [], "index_params": [], "fieldID": "100", "name": "id", "is_primary_key": true, "description": "", "data_type": "Int64", "autoID": true, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 5, "dimension": -1, "maxCapacity": -1, "maxLength": -1 }, { "type_params": [ { "key": "dim", "value": "768" } ], "index_params": [], "fieldID": "101", "name": "vec", "is_primary_key": false, "description": "", "data_type": "FloatVector", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 101, "index": { "params": [ { "key": "index_type", "value": "IVF_FLAT" }, { "key": "metric_type", "value": "COSINE" }, { "key": "params", "value": "{\"nlist\":1024}" } ], "index_name": "vec", "indexID": "452967585148662141", "field_name": "vec", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "IVF_FLAT", "metricType": "COSINE", "indexParameterPairs": [ { "key": "metric_type", "value": "COSINE" }, { "key": "nlist", "value": 1024 } ] }, "dimension": 768, "maxCapacity": -1, "maxLength": -1 }, { "type_params": [ { "key": "max_length", "value": "32" } ], "index_params": [], "fieldID": "102", "name": "scalar_colors", "is_primary_key": false, "description": "", "data_type": "VarChar", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 21, "index": { "params": [ { "key": "index_type", "value": "Trie" }, { "key": "metric_type", "value": "" }, { "key": "params", "value": "{}" } ], "index_name": "scalar_colors", "indexID": "452967585148662190", "field_name": "scalar_colors", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "Trie", "metricType": "", "indexParameterPairs": [ { "key": "metric_type", "value": "" } ] }, "dimension": -1, "maxCapacity": -1, "maxLength": 32 }, { "type_params": [ { "key": "max_length", "value": "32" } ], "index_params": [], "fieldID": "103", "name": "scalar_colors_k", "is_primary_key": false, "description": "", "data_type": "VarChar", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": true, "is_clustering_key": false, "dataType": 21, "index": { "params": [ { "key": "index_type", "value": "Trie" }, { "key": "metric_type", "value": "" }, { "key": "params", "value": "{}" } ], "index_name": "scalar_colors_k", "indexID": "452967585148662220", "field_name": "scalar_colors_k", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "Trie", "metricType": "", "indexParameterPairs": [ { "key": "metric_type", "value": "" } ] }, "dimension": -1, "maxCapacity": -1, "maxLength": 32 } ], "properties": [], "name": "benchmark_v4", "description": "", "autoID": false, "enable_dynamic_field": true, "primaryField": { "type_params": [], "index_params": [], "fieldID": "100", "name": "id", "is_primary_key": true, "description": "", "data_type": "Int64", "autoID": true, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 5, "dimension": -1, "maxCapacity": -1, "maxLength": -1 }, "hasVectorIndex": true, "enablePartitionKey": true, "scalarFields": [ { "type_params": [], "index_params": [], "fieldID": "100", "name": "id", "is_primary_key": true, "description": "", "data_type": "Int64", "autoID": true, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 5, "dimension": -1, "maxCapacity": -1, "maxLength": -1 }, { "type_params": [ { "key": "max_length", "value": "32" } ], "index_params": [], "fieldID": "102", "name": "scalar_colors", "is_primary_key": false, "description": "", "data_type": "VarChar", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 21, "index": { "params": [ { "key": "index_type", "value": "Trie" }, { "key": "metric_type", "value": "" }, { "key": "params", "value": "{}" } ], "index_name": "scalar_colors", "indexID": "452967585148662190", "field_name": "scalar_colors", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "Trie", "metricType": "", "indexParameterPairs": [ { "key": "metric_type", "value": "" } ] }, "dimension": -1, "maxCapacity": -1, "maxLength": 32 }, { "type_params": [ { "key": "max_length", "value": "32" } ], "index_params": [], "fieldID": "103", "name": "scalar_colors_k", "is_primary_key": false, "description": "", "data_type": "VarChar", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": true, "is_clustering_key": false, "dataType": 21, "index": { "params": [ { "key": "index_type", "value": "Trie" }, { "key": "metric_type", "value": "" }, { "key": "params", "value": "{}" } ], "index_name": "scalar_colors_k", "indexID": "452967585148662220", "field_name": "scalar_colors_k", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "Trie", "metricType": "", "indexParameterPairs": [ { "key": "metric_type", "value": "" } ] }, "dimension": -1, "maxCapacity": -1, "maxLength": 32 } ], "vectorFields": [ { "type_params": [ { "key": "dim", "value": "768" } ], "index_params": [], "fieldID": "101", "name": "vec", "is_primary_key": false, "description": "", "data_type": "FloatVector", "autoID": false, "state": "FieldCreated", "element_type": "None", "default_value": null, "is_dynamic": false, "is_partition_key": false, "is_clustering_key": false, "dataType": 101, "index": { "params": [ { "key": "index_type", "value": "IVF_FLAT" }, { "key": "metric_type", "value": "COSINE" }, { "key": "params", "value": "{\"nlist\":1024}" } ], "index_name": "vec", "indexID": "452967585148662141", "field_name": "vec", "indexed_rows": "11760180", "total_rows": "11760180", "state": "Finished", "index_state_fail_reason": "", "pending_index_rows": "0", "indexType": "IVF_FLAT", "metricType": "COSINE", "indexParameterPairs": [ { "key": "metric_type", "value": "COSINE" }, { "key": "nlist", "value": 1024 } ] }, "dimension": 768, "maxCapacity": -1, "maxLength": -1 } ], "dynamicFields": [ { "name": "$meta", "data_type": "JSON", "type_params": [], "description": "", "index_params": [], "dimension": -1, "maxCapacity": -1, "maxLength": -1, "autoID": false, "fieldID": "", "state": "", "dataType": 23 } ] }, "rowCount": 11760180, "createdTime": 1734073852866, "aliases": [], "description": "", "autoID": true, "id": "452967585148661839", "loadedPercentage": 100, "consistency_level": "Bounded", "replicas": [ { "partition_ids": [], "shard_replicas": [ { "node_ids": [ "55" ], "leaderID": "55", "leader_addr": "10.203.27.51:21123", "dm_channel_name": "milvus-zxy-sync-5-rootcoord-dml_14_452967585148661839v0" } ], "node_ids": [ "11", "4", "6", "5", "21", "56", "54", "1", "9", "46", "22", "55", "12", "7", "10", "20", "3", "47" ], "num_outbound_node": {}, "replicaID": "452967593885040663", "collectionID": "452967585148661839", "resource_group_name": "__default_resource_group" } ], "loaded": true, "status": "loaded", "properties": [] } ], "statusCode": 200 }Use a benchmark to call the golang-sdk, the following are key methods for retrieval
The
*exprhere is used to distinguish between scalar retrieval and partition key retrieval.scalar_colors == "red"for scalar search,scalar_colors_key == "red"for partition-key search.Search QPS benchmark
milvus 2.3.12
milvus 2.5.0-beta
Expected Behavior
One surprising aspect of milvus-2.5.0 is that the scalar search performance has indeed improved significantly, by about 40%. The gap between partition keys and scalars is too large, which doesn't quite align with the theory in the milvus documentation. Of course, it could also be an issue with my stress testing, which is why I'm raising this issue.
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: