Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] Drop partition raises error segment was removed from the loading map early in dql & multi-partition scene #38649

Open
1 task done
wangting0128 opened this issue Dec 23, 2024 · 1 comment
Open
1 task done

[Bug]: [benchmark][cluster] Drop partition raises error segment was removed from the loading map early in dql & multi-partition scene #38649

wangting0128 opened this issue Dec 23, 2024 · 1 comment
Assignees
@xiaocai2333
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.1

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.5-20241220-8cbe8da9-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-corn-1-1734789600
test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1734789600-1-etcd-0                           1/1     Running            0               3h7m    10.104.30.34    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-etcd-1                           1/1     Running            0               3h7m    10.104.26.159   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-etcd-2                           1/1     Running            0               3h7m    10.104.33.235   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-datanode-bbf7967962zh94   1/1     Running            2 (3h7m ago)    3h7m    10.104.21.190   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8fdh99j   1/1     Running            2 (3h7m ago)    3h7m    10.104.21.188   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8ffh8wd   1/1     Running            2 (3h7m ago)    3h7m    10.104.17.5     4am-node23   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8frxxvx   1/1     Running            2 (3h7m ago)    3h7m    10.104.25.134   4am-node30   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8fs2jrg   1/1     Running            2 (3h7m ago)    3h7m    10.104.27.116   4am-node31   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-mixcoord-99656c8f86bzxr   1/1     Running            2 (3h7m ago)    3h7m    10.104.17.4     4am-node23   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-proxy-757456df88-zstwf    1/1     Running            2 (3h7m ago)    3h7m    10.104.21.189   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-querynode-64c79698g5jxm   1/1     Running            2 (3h7m ago)    3h7m    10.104.15.143   4am-node20   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-querynode-64c79698kqx69   1/1     Running            2 (3h7m ago)    3h7m    10.104.20.58    4am-node22   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-0                          1/1     Running            0               3h7m    10.104.30.35    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-1                          1/1     Running            0               3h7m    10.104.26.158   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-2                          1/1     Running            0               3h7m    10.104.33.232   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-3                          1/1     Running            0               3h7m    10.104.32.134   4am-node39   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-0                1/1     Running            0               3h7m    10.104.30.37    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-1                1/1     Running            0               3h7m    10.104.33.237   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-2                1/1     Running            0               3h7m    10.104.26.164   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-init-b64fh       0/1     Completed          0               3h7m    10.104.6.74     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-broker-0                1/1     Running            0               3h7m    10.104.6.76     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-broker-1                1/1     Running            0               3h7m    10.104.14.4     4am-node18   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-proxy-0                 1/1     Running            0               3h7m    10.104.6.71     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-proxy-1                 1/1     Running            0               3h7m    10.104.9.51     4am-node14   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-pulsar-init-l62fs       0/1     Completed          0               3h7m    10.104.6.72     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-recovery-0              1/1     Running            0               3h7m    10.104.6.77     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-0             1/1     Running            0               3h7m    10.104.30.33    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-1             1/1     Running            0               3h7m    10.104.26.160   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-2             1/1     Running            0               3h7m    10.104.33.236   4am-node36   <none>           <none>

{pod=~"multi-vector-corn-1-1734789600-1-milvus-.*"} |~ "492bd40ad8b73896695b8ee7d7cd1e89"
截屏2024-12-23 11 15 58

client log:

, 'scene_test_partition_hybrid_search_WMV5LDzd', ''], kwargs: {}, [requestId: c0749536-bfa8-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:35:25,111 - DEBUG - fouram]: (api_response) : [Partition] {"name":"scene_test_partition_hybrid_search_WMV5LDzd","collection_name":"fouram_hsF7bpBM","description":""}, [requestId: c0749536-bfa8-11ef-b10b-7693a8272652] (api_request.py:44)
[2024-12-21 14:35:25,112 - DEBUG - fouram]: [Base] Create partition scene_test_partition_hybrid_search_WMV5LDzd of collection(fouram_hsF7bpBM) (base.py:821)
[2024-12-21 14:35:25,247 - DEBUG - fouram]: (api_request)  : [Collection.insert] args: <Collection.insert fields: 7, length: 3000, content: [ [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'list'>, dtype<>` [0.19579262613213944, 0. ... ], [ `type<class 'list'>, dtype<>` [0.85696574318921, 0.722 ... ], [ `type<class 'list'>, dtype<>` [0.6981304524714663, 0.5 ... ], [ `type<class 'list'>, dtype<>` [0.4047851630760757, 0.0 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'str'>, dtype<>` 0 ... ] ]>, ['scene_test_partition_hybrid_search_WMV5LDzd'], kwargs: {'timeout': 600}, [requestId: d0ee73be-bfa8-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:36:11,341 - DEBUG - fouram]: [Base] Start flush partition scene_test_partition_hybrid_search_WMV5LDzd, kwargs: {'timeout': 600} (base.py:836)
[2024-12-21 14:36:31,580 - DEBUG - fouram]: [Base] Partition scene_test_partition_hybrid_search_WMV5LDzd num entities: (3000) (base.py:832)
[2024-12-21 14:36:31,745 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:int64_1, index_params:{'index_type': 'INVERTED'} (base.py:1126)
[2024-12-21 14:36:32,259 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:varchar_1, index_params:{'index_type': 'INVERTED'} (base.py:1126)
[2024-12-21 14:36:33,259 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector, index_params:{'params': {'nlist': 1024}, 'index_type': 'IVF_FLAT', 'metric_type': 'L2'} (base.py:1126)
[2024-12-21 14:36:33,776 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_1, index_params:{'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}, 'index_type': 'HNSW'} (base.py:1126)
[2024-12-21 14:36:34,770 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_2, index_params:{'index_type': 'DISKANN', 'metric_type': 'IP', 'params': {}} (base.py:1126)
[2024-12-21 14:36:35,286 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_3, index_params:{'metric_type': 'L2', 'params': {'nlist': 2048}, 'index_type': 'IVF_SQ8'} (base.py:1126)
[2024-12-21 14:37:04,042 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:id, index_params:{} (base.py:1126)
[2024-12-21 14:37:58,791 - DEBUG - fouram]: [Base] Start load partition scene_test_partition_hybrid_search_WMV5LDzd, replica_number:1, kwargs:{'timeout': 600} (base.py:842)
[2024-12-21 14:39:45,129 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_WMV5LDzd hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_2', 'param': {'metric_type': 'IP', 'params': {'search_list': 32}}, 'limit': 30, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_3', 'param': {'metric_type': 'L2', 'params': {'nprobe': 16}}, 'limit': 400, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_response'} (base.py:871)
[2024-12-21 14:39:51,338 - DEBUG - fouram]: [Base] Start release partition scene_test_partition_hybrid_search_WMV5LDzd (base.py:848)
[2024-12-21 14:41:06,795 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_WMV5LDzd hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_2', 'param': {'metric_type': 'IP', 'params': {'search_list': 32}}, 'limit': 30, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_3', 'param': {'metric_type': 'L2', 'params': {'nprobe': 16}}, 'limit': 400, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_error_response', 'check_items': {'code': 65535, 'message': 'not loaded'}} (base.py:871)
[2024-12-21 14:41:10,411 - DEBUG - fouram]: (api_request)  : [Partition.drop] args: [], kwargs: {'timeout': 600}, [requestId: 9eaa535e-bfa9-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:41:42,516 - ERROR - fouram]: (api_response) : [Partition.drop] <MilvusException: (code=65535, message=segment was removed from the loading map early)>, [requestId: 9eaa535e-bfa9-11ef-b10b-7693a8272652] (api_request.py:57)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DQL & DML(partition)`
            verify concurrent DQL & DML(partition) scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data into 10 partitions
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - scene_test_partition_hybrid_search
                    (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop)
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-12-21 17:21:18,861 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     hybrid_search                                                                   3255     0(0.00%) |   3914      17   37035   2400 |    0.30        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     query                                                                            400     0(0.00%) |   4785      62   65199    490 |    0.04        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     scene_test_partition_hybrid_search                                               393     1(0.25%) | 317768   65544  669267 310000 |    0.04        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     search                                                                          3145     0(0.00%) |  23646    2548   52879  23000 |    0.29        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]:          Aggregated                                                                      7193     1(0.01%) |  29738      17  669267  14000 |    0.67        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]:  (stats.py:790)
[2024-12-21 17:21:18,866 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0', 'memory': '32Gi'}, 'requests': {'cpu': '17.0', 'memory': '17Gi'}}, 'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsarv3': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': '2.5-20241220-8cbe8da9-amd64'}}},
            'host': 'multi-vector-corn-1-1734789600-1-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}},
                                                    'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                        'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                        'partition_9'],
                                                                         'data_repeated': False},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_test_partition_hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10},
                                                                           {'search_param': {'search_list': 32}, 'anns_field': 'float_vector_2', 'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'hybrid_search_counts': 1,
                                                                  'data_size': 3000,
                                                                  'ni': 3000}},
                                                      {'type': 'search',
                                                       'weight': 8,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 >= 0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 8,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10},
                                                                           {'search_param': {'search_list': 32}, 'anns_field': 'float_vector_2', 'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95, 0.51, 0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1 && ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0, 100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'custom_expr': None,
                                                                  'custom_range': [0, 1],
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024122104547868,
            'datetime': '2024-12-21 14:14:14.362982',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 22.1728,
                                      'float_vector_1': {'RT': 0.5159},
                                      'float_vector_2': {'RT': 1.5223},
                                      'float_vector_3': {'RT': 0.5149},
                                      'id': {'RT': 0.5167},
                                      'int64_1': {'RT': 0.5163},
                                      'varchar_1': {'RT': 0.515}},
                            'insert': {'total_time': 121.5605, 'VPS': 8227.1488, 'batch_time': 1.2156, 'batch': 10000.0},
                            'flush': {'RT': 2.5249},
                            'load': {'RT': 2.2435},
                            'Locust': {'Aggregated': {'Requests': 7193,
                                                      'Fails': 1,
                                                      'RPS': 0.67,
                                                      'fail_s': 0.0,
                                                      'RT_max': 669267.21,
                                                      'RT_avg': 29738.27,
                                                      'TP50': 14000.0,
                                                      'TP99': 399000.0},
                                       'hybrid_search': {'Requests': 3255,
                                                         'Fails': 0,
                                                         'RPS': 0.3,
                                                         'fail_s': 0.0,
                                                         'RT_max': 37035.08,
                                                         'RT_avg': 3914.48,
                                                         'TP50': 2400.0,
                                                         'TP99': 21000.0},
                                       'query': {'Requests': 400,
                                                 'Fails': 0,
                                                 'RPS': 0.04,
                                                 'fail_s': 0.0,
                                                 'RT_max': 65199.15,
                                                 'RT_avg': 4785.13,
                                                 'TP50': 500.0,
                                                 'TP99': 48000.0},
                                       'scene_test_partition_hybrid_search': {'Requests': 393,
                                                                              'Fails': 1,
                                                                              'RPS': 0.04,
                                                                              'fail_s': 0.0,
                                                                              'RT_max': 669267.21,
                                                                              'RT_avg': 317768.67,
                                                                              'TP50': 310000.0,
                                                                              'TP99': 615000.0},
                                       'search': {'Requests': 3145,
                                                  'Fails': 0,
                                                  'RPS': 0.29,
                                                  'fail_s': 0.0,
                                                  'RT_max': 52879.95,
                                                  'RT_avg': 23646.62,
                                                  'TP50': 23000.0,
                                                  'TP99': 42000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Dec 23, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Dec 23, 2024
@yanliang567
Copy link
Contributor

/assign @xiaocai2333
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.5.0, 2.5.1 Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xiaocai2333 xiaocai2333

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.1
Development

No branches or pull requests
3 participants
@wangting0128 @xiaocai2333 @yanliang567