Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] hybrid_search raises error fail to search on QueryNode 2: channel not found in concurrent dql & ddl scene #37830

Closed
1 task done
wangting0128 opened this issue Nov 20, 2024 · 4 comments
Closed
1 task done

[Bug]: [benchmark][cluster] hybrid_search raises error fail to search on QueryNode 2: channel not found in concurrent dql & ddl scene #37830

wangting0128 opened this issue Nov 20, 2024 · 4 comments
Assignees
@wangting0128
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Nov 20, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20241119-b3fc5301-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc106
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: memory-opt-scenes-m2jwl
test case name: test_hybrid_search_locust_multi_ddl_dql_hybrid_search_cluster

server:

NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
memory-opt-scenes-m2jwl-4-etcd-0                                  1/1     Running     0                14h     10.104.21.71    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-etcd-1                                  1/1     Running     0                14h     10.104.23.39    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-etcd-2                                  1/1     Running     0                14h     10.104.18.213   4am-node25   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-datanode-566c74cb8b-j44f4        1/1     Running     5 (14h ago)      14h     10.104.14.116   4am-node18   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-lflmx       1/1     Running     0                14h     10.104.20.51    4am-node22   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-q87kz       1/1     Running     2 (14h ago)      14h     10.104.33.244   4am-node36   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-vpxfp       1/1     Running     2 (14h ago)      14h     10.104.19.11    4am-node28   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-indexnode-869b8fd5cc-vxs6m       1/1     Running     2 (14h ago)      14h     10.104.5.217    4am-node12   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-mixcoord-85f66f55d6-9444s        1/1     Running     0                14h     10.104.6.186    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-proxy-86ddbc987d-xtml2           1/1     Running     0                14h     10.104.6.187    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-querynode-6fd58c9669-74754       1/1     Running     0                14h     10.104.6.188    4am-node13   <none>           <none>
memory-opt-scenes-m2jwl-4-milvus-querynode-6fd58c9669-dmff7       1/1     Running     2 (14h ago)      14h     10.104.16.57    4am-node21   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-0                                 1/1     Running     0                14h     10.104.17.84    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-1                                 1/1     Running     0                14h     10.104.15.175   4am-node20   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-2                                 1/1     Running     0                14h     10.104.21.72    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-minio-3                                 1/1     Running     0                14h     10.104.23.42    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-0                         1/1     Running     0                14h     10.104.15.176   4am-node20   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-1                         1/1     Running     0                14h     10.104.24.24    4am-node29   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-2                         1/1     Running     0                14h     10.104.23.43    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-bookie-init-s7x8m                0/1     Completed   0                14h     10.104.21.61    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-broker-0                         1/1     Running     0                14h     10.104.24.16    4am-node29   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-proxy-0                          1/1     Running     0                14h     10.104.17.81    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-pulsar-init-c2sgn                0/1     Completed   0                14h     10.104.20.50    4am-node22   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-recovery-0                       1/1     Running     0                14h     10.104.21.62    4am-node24   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-0                      1/1     Running     0                14h     10.104.23.36    4am-node27   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-1                      1/1     Running     0                14h     10.104.17.88    4am-node23   <none>           <none>
memory-opt-scenes-m2jwl-4-pulsar-zookeeper-2                      1/1     Running     0                14h     10.104.21.84    4am-node24   <none>           <none>

image

client log:

hybrid search on collection scene_hybrid_search_test_FUdbgF8A failed

[2024-11-19 16:58:44,500 - ERROR - fouram]: RPC error: [hybrid_search], <MilvusException: (code=500, message=fail to search on QueryNode 2: channel not found[channel=by-dev-rootcoord-dml_0_454038175298478930v0])>, <Time:{'RPC start': '2024-11-19 16:58:40.889907', 'RPC error': '2024-11-19 16:58:44.500434'}> (decorators.py:140)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DDL & DQL`
            verify DDL & DQL scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - scene_hybrid_search_test: 4 vector fields, 3 scalar fields
                    (collection: create->insert->flush->index->load->hybrid_search->drop) <- hybrid_search raises error
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-11-20 00:19:16,684 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-11-20 00:19:16,685 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     hybrid_search                                                                   9092     0(0.00%) |   3013     114   17500   2400 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     query                                                                           9060     0(0.00%) |    855      75   29817    590 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     scene_hybrid_search_test                                                        9007     1(0.01%) |  83431   11993  479722  52000 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: grpc     search                                                                          8990     0(0.00%) |   8373    2773   49407   7200 |    0.21        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]:          Aggregated                                                                     36149     1(0.00%) |  23842      75  479722   4700 |    0.84        0.00 (stats.py:789)
[2024-11-20 00:19:16,685 -  INFO - fouram]:  (stats.py:790)
[2024-11-20 00:19:16,689 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0', 'memory': '32Gi'}, 'requests': {'cpu': '17.0', 'memory': '17Gi'}}, 'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n  segcore:\n    multipleChunkedEnable: true\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20241119-b3fc5301-amd64'}}},
            'host': 'memory-opt-scenes-m2jwl-4-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_multi_ddl_dql_hybrid_search_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '12h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_hybrid_search_test',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'nprobe': 32},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_1',
                                                                            'top_k': 10},
                                                                           {'search_param': {'ef': 32},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_2',
                                                                            'top_k': 5},
                                                                           {'search_param': {'search_list': 20},
                                                                            'anns_field': 'float_vector_scene_hybrid_search_test_3',
                                                                            'top_k': 10}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'dataset': 'local',
                                                                  'dim': 128,
                                                                  'shards_num': 2,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2',
                                                                  'other_fields': ['float_vector_scene_hybrid_search_test_1',
                                                                                   'float_vector_scene_hybrid_search_test_2',
                                                                                   'float_vector_scene_hybrid_search_test_3', 'int64_1', 'bool_1',
                                                                                   'varchar_1'],
                                                                  'replica_number': 1,
                                                                  'scalars_params': {'float_vector_scene_hybrid_search_test_1': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_scene_hybrid_search_test_2': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}},
                                                                                     'float_vector_scene_hybrid_search_test_3': {'params': {'dim': 128},
                                                                                                                                 'other_params': {'dataset': 'sift'}}},
                                                                  'scalars_index': {'int64_1': {},
                                                                                    'bool_1': {'index_type': 'INVERTED'},
                                                                                    'varchar_1': {'index_type': 'INVERTED'}},
                                                                  'vectors_index': {'float_vector_scene_hybrid_search_test_1': {'index_type': 'IVF_FLAT',
                                                                                                                                'index_param': {'nlist': 1024},
                                                                                                                                'metric_type': 'L2'},
                                                                                    'float_vector_scene_hybrid_search_test_2': {'index_type': 'HNSW',
                                                                                                                                'index_param': {'M': 8,
                                                                                                                                                'efConstruction': 200},
                                                                                                                                'metric_type': 'L2'},
                                                                                    'float_vector_scene_hybrid_search_test_3': {'index_type': 'DISKANN',
                                                                                                                                'index_param': {},
                                                                                                                                'metric_type': 'IP'}},
                                                                  'prepare_before_insert': False,
                                                                  'hybrid_search_counts': 10,
                                                                  'new_connect': False,
                                                                  'new_user': False}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 >= 0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 > 100000',
                                                                            'top_k': 100},
                                                                           {'search_param': {'ef': 64},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'id < 900000',
                                                                            'top_k': 10},
                                                                           {'search_param': {'search_list': 32},
                                                                            'anns_field': 'float_vector_2',
                                                                            'expr': 'varchar_1 > "1"',
                                                                            'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95, 0.51, 0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1 && ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 600,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0, 100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'custom_expr': None,
                                                                  'custom_range': [0, 1],
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024111972065950,
            'datetime': '2024-11-19 11:53:26.410333',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 62.9087,
                                      'float_vector_1': {'RT': 9.0731},
                                      'float_vector_2': {'RT': 142.9301},
                                      'float_vector_3': {'RT': 0.5176},
                                      'id': {'RT': 0.5732},
                                      'int64_1': {'RT': 0.5161},
                                      'varchar_1': {'RT': 0.5152}},
                            'insert': {'total_time': 153.3464, 'VPS': 6521.1834, 'batch_time': 1.5335, 'batch': 10000},
                            'flush': {'RT': 3.0494},
                            'load': {'RT': 4.6697},
                            'Locust': {'Aggregated': {'Requests': 36149,
                                                      'Fails': 1,
                                                      'RPS': 0.84,
                                                      'fail_s': 0.0,
                                                      'RT_max': 479722.31,
                                                      'RT_avg': 23842.91,
                                                      'TP50': 4700.0,
                                                      'TP99': 252000.0},
                                       'hybrid_search': {'Requests': 9092,
                                                         'Fails': 0,
                                                         'RPS': 0.21,
                                                         'fail_s': 0.0,
                                                         'RT_max': 17500.05,
                                                         'RT_avg': 3013.76,
                                                         'TP50': 2400.0,
                                                         'TP99': 10000.0},
                                       'query': {'Requests': 9060,
                                                 'Fails': 0,
                                                 'RPS': 0.21,
                                                 'fail_s': 0.0,
                                                 'RT_max': 29817.09,
                                                 'RT_avg': 855.73,
                                                 'TP50': 590.0,
                                                 'TP99': 5900.0},
                                       'scene_hybrid_search_test': {'Requests': 9007,
                                                                    'Fails': 1,
                                                                    'RPS': 0.21,
                                                                    'fail_s': 0.0,
                                                                    'RT_max': 479722.31,
                                                                    'RT_avg': 83431.53,
                                                                    'TP50': 52000.0,
                                                                    'TP99': 314000.0},
                                       'search': {'Requests': 8990,
                                                  'Fails': 0,
                                                  'RPS': 0.21,
                                                  'fail_s': 0.0,
                                                  'RT_max': 49407.92,
                                                  'RT_avg': 8373.27,
                                                  'TP50': 7200.0,
                                                  'TP99': 30000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Nov 20, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Nov 20, 2024
@yanliang567
Copy link
Contributor

/assign @liliu-z
/unassign

@sre-ci-robot sre-ci-robot assigned liliu-z and unassigned yanliang567 Nov 20, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 20, 2024
@liliu-z
Copy link
Member

liliu-z commented Nov 20, 2024

/assign @weiliu1031

@weiliu1031 weiliu1031 mentioned this issue Nov 20, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 21, 2024
@weiliu1031
fix: Channel may be released after balance (#37862)
b983ef9 
issue: #37830
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.

this PR fix the way to set proper version for channel.

Signed-off-by: Wei Liu <[email protected]>
@liliu-z
Copy link
Member

liliu-z commented Nov 21, 2024

/assign @wangting0128

weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Nov 22, 2024
@weiliu1031
fix: Channel may be released after balance (milvus-io#37862)
d421c4f 
issue: milvus-io#37830
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.

this PR fix the way to set proper version for channel.

Signed-off-by: Wei Liu <[email protected]>
@weiliu1031 weiliu1031 mentioned this issue Nov 22, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 25, 2024
@weiliu1031
fix: Channel may be released after balance (#37862) (#37940)
bb66636 
issue: #37830
pr: #37862
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.

this PR fix the way to set proper version for channel.

Signed-off-by: Wei Liu <[email protected]>
JsDove pushed a commit to JsDove/milvus that referenced this issue Nov 26, 2024
@weiliu1031 @JsDove
fix: Channel may be released after balance (milvus-io#37862)
a4cb82d 
issue: milvus-io#37830
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.

this PR fix the way to set proper version for channel.

Signed-off-by: Wei Liu <[email protected]>
@wangting0128
Copy link
Contributor Author

verification passed

argo task: memory-opt-scenes-wp9hm
image: master-20241127-302650ae-amd64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangting0128 wangting0128

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
4 participants
@wangting0128 @yanliang567 @liliu-z @weiliu1031