[Bug]: [benchmark][cluster] DQL request raises error `client is closed` in pod failure injection multi-replica testing #37718

wangting0128 · 2024-11-15T08:55:49Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: master-20241114-993051bb-amd64 
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc97
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: bitmap-chaos-c7sxm
test case name: test_bitmap_locust_resource_groups_multi_scalar_cluster

server:

NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
bitmap-chaos-c7sxm-etcd-0                                         1/1     Running     0               21h     10.104.32.164   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-etcd-1                                         1/1     Running     0               21h     10.104.18.39    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-etcd-2                                         1/1     Running     0               21h     10.104.17.218   4am-node23   <none>           <none>
bitmap-chaos-c7sxm-milvus-datanode-5968fff48c-tcw7c               1/1     Running     2 (21h ago)     21h     10.104.17.216   4am-node23   <none>           <none>
bitmap-chaos-c7sxm-milvus-indexnode-664679f498-2qlfg              1/1     Running     2 (21h ago)     21h     10.104.18.28    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-milvus-indexnode-664679f498-kqhwl              1/1     Running     1 (21h ago)     21h     10.104.32.150   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-milvus-mixcoord-7bddbb9f9f-ksnsb               1/1     Running     1 (21h ago)     21h     10.104.32.148   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-milvus-proxy-66f6d9d7cf-tfcpm                  1/1     Running     1 (21h ago)     21h     10.104.32.151   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-milvus-querynode-dc8f8dbb4-c6ptn               1/1     Running     1 (21h ago)     21h     10.104.15.157   4am-node20   <none>           <none>
bitmap-chaos-c7sxm-milvus-querynode-dc8f8dbb4-c7qbm               1/1     Running     28 (8h ago)     21h     10.104.5.218    4am-node12   <none>           <none>
bitmap-chaos-c7sxm-milvus-querynode-dc8f8dbb4-mhc9k               1/1     Running     54 (150m ago)   21h     10.104.9.100    4am-node14   <none>           <none>
bitmap-chaos-c7sxm-minio-0                                        1/1     Running     0               21h     10.104.18.36    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-minio-1                                        1/1     Running     0               21h     10.104.32.162   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-minio-2                                        1/1     Running     0               21h     10.104.16.50    4am-node21   <none>           <none>
bitmap-chaos-c7sxm-minio-3                                        1/1     Running     0               21h     10.104.17.223   4am-node23   <none>           <none>
bitmap-chaos-c7sxm-pulsar-bookie-0                                1/1     Running     0               21h     10.104.32.163   4am-node39   <none>           <none>
bitmap-chaos-c7sxm-pulsar-bookie-1                                1/1     Running     0               21h     10.104.16.52    4am-node21   <none>           <none>
bitmap-chaos-c7sxm-pulsar-bookie-2                                1/1     Running     0               21h     10.104.18.42    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-pulsar-bookie-init-vwcdl                       0/1     Completed   0               21h     10.104.15.156   4am-node20   <none>           <none>
bitmap-chaos-c7sxm-pulsar-broker-0                                1/1     Running     0               21h     10.104.18.31    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-pulsar-proxy-0                                 1/1     Running     0               21h     10.104.1.202    4am-node10   <none>           <none>
bitmap-chaos-c7sxm-pulsar-pulsar-init-pgbcz                       0/1     Completed   0               21h     10.104.18.29    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-pulsar-recovery-0                              1/1     Running     0               21h     10.104.1.201    4am-node10   <none>           <none>
bitmap-chaos-c7sxm-pulsar-zookeeper-0                             1/1     Running     0               21h     10.104.18.38    4am-node25   <none>           <none>
bitmap-chaos-c7sxm-pulsar-zookeeper-1                             1/1     Running     0               21h     10.104.17.227   4am-node23   <none>           <none>
bitmap-chaos-c7sxm-pulsar-zookeeper-2                             1/1     Running     0               21h     10.104.15.160   4am-node20   <none>           <none>

replica=2 and inject 1h pod failure into one of the queryNodes every 2 hours

client error log:
fouram_log.log

client error rate

client error time：
2024-11-14 14:16 ～ 2024-11-14 14:19
2024-11-14 17:15 ～ 2024-11-14 18:28

queryNode pod failure recover：[2024-11-14 12:16:25 - INFO - root]: all pods are ready (test_chaos_apply.py:134)
queryNode pod failure during：2024-11-14 17:16 ～ 2024-11-14 18:16

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key int64_1 field`
            verify DQL scenario on RG & replicas scene
            which has 2 vector fields(IVF_FLAT & SPARSE_WAND), building `BITMAP` index on all supported 12 scalar fields

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector_1': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector

                all scalar fields: varchar max_length=100, array max_capacity=13
                    'int64_1': partition_key, num_partitions=1024， data range: 0 ~ 9
            2. build indexes:
                IVF_FLAT: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector_1'

                BITMAP: all scalar fields
            3. insert 10 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 2
                resource groups: 2
            7. concurrent request: <- concurrent_number=100, Inject 1h pod failure into one of the queryNodes every 2 hours
                - search
                - hybrid_search

Milvus Log

No response

Anything else?

test result:

[2024-11-15 05:42:33,953 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]: grpc     hybrid_search                                                                  59923 48876(81.56%) |  61003       0  806806      0 |    0.83        0.68 (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]: grpc     search                                                                         59941 49041(81.82%) |  51243       0  793397      0 |    0.83        0.68 (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]:          Aggregated                                                                    119864 97917(81.69%) |  56122       0  806806      0 |    1.67        1.36 (stats.py:789)
[2024-11-15 05:42:33,953 -  INFO - fouram]:  (stats.py:790)
[2024-11-15 05:42:33,956 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_8c16m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '48Gi'}, 'requests': {'cpu': '5.0', 'memory': '25Gi'}}, 'replicas': 3},
                       'indexNode': {'resources': {'limits': {'cpu': '4.0', 'memory': '8Gi'}, 'requests': {'cpu': '3.0', 'memory': '5Gi'}}, 'replicas': 2},
                       'dataNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '16Gi'}, 'requests': {'cpu': '5.0', 'memory': '9Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': 'master-20241114-993051bb-amd64'}}},
            'host': 'bitmap-chaos-c7sxm-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_bitmap_locust_resource_groups_multi_scalar_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'int8_1': {'index_type': 'BITMAP'},
                                                                      'int16_1': {'index_type': 'BITMAP'},
                                                                      'int32_1': {'index_type': 'BITMAP'},
                                                                      'int64_1': {'index_type': 'BITMAP'},
                                                                      'varchar_1': {'index_type': 'BITMAP'},
                                                                      'bool_1': {'index_type': 'BITMAP'},
                                                                      'array_int8_1': {'index_type': 'BITMAP'},
                                                                      'array_int16_1': {'index_type': 'BITMAP'},
                                                                      'array_int32_1': {'index_type': 'BITMAP'},
                                                                      'array_int64_1': {'index_type': 'BITMAP'},
                                                                      'array_varchar_1': {'index_type': 'BITMAP'},
                                                                      'array_bool_1': {'index_type': 'BITMAP'}},
                                                    'vectors_index': {'sparse_float_vector_1': {'index_type': 'SPARSE_WAND',
                                                                                                'index_param': {'drop_ratio_build': 0.2},
                                                                                                'metric_type': 'IP'}},
                                                    'scalars_params': {'array_int8_1': {'params': {'max_capacity': 9},
                                                                                        'other_params': {'dataset': 'random_algorithm',
                                                                                                         'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                              'specify_range': [-128, 128],
                                                                                                                              'max_capacity': 9}}},
                                                                       'array_int16_1': {'params': {'max_capacity': 9},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 9}}},
                                                                       'array_int32_1': {'params': {'max_capacity': 9},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                               'specify_range': [-300, 300],
                                                                                                                               'max_capacity': 9}}},
                                                                       'array_int64_1': {'params': {'max_capacity': 9},
                                                                                         'other_params': {'dataset': 'random_algorithm',
                                                                                                          'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                               'specify_range': [-400, 432],
                                                                                                                               'batch': 50,
                                                                                                                               'max_capacity': 9}}},
                                                                       'array_varchar_1': {'params': {'max_capacity': 9},
                                                                                           'other_params': {'dataset': 'random_algorithm',
                                                                                                            'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                                 'specify_range': [-1500, 1500],
                                                                                                                                 'max_capacity': 9}}},
                                                                       'array_bool_1': {'params': {'max_capacity': 9}},
                                                                       'int64_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                         'specify_range': [0, 10],
                                                                                                                         'max_capacity': 1}}},
                                                                       'int8_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                   'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                        'specify_range': [-128, 128],
                                                                                                                        'max_capacity': 9}}},
                                                                       'int32_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'specify_scope',
                                                                                                                         'specify_range': [-300, 300],
                                                                                                                         'max_capacity': 9}}},
                                                                       'int16_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                    'algorithm_params': {'algorithm_name': 'fixed_value_range',
                                                                                                                         'specify_range': [-400, 432],
                                                                                                                         'batch': 50,
                                                                                                                         'max_capacity': 9}}},
                                                                       'varchar_1': {'other_params': {'dataset': 'random_algorithm',
                                                                                                      'algorithm_params': {'algorithm_name': 'random_range',
                                                                                                                           'specify_range': [-1500, 1500],
                                                                                                                           'max_capacity': 9}}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 10000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['sparse_float_vector_1', 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'varchar_1',
                                                                        'bool_1', 'array_int8_1', 'array_int16_1', 'array_int32_1', 'array_int64_1',
                                                                        'array_varchar_1', 'array_bool_1'],
                                                       'shards_num': 2},
                                 'load_params': {'replica_number': 2, '_resource_groups': 2},
                                 'resource_groups_params': {'groups': [1, 2], 'reset': True},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 100, 'during_time': '20h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 64},
                                                                  'expr': None,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 1200,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'nq': 1000}}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 10,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'varchar_1 > "0" && int8_1 >= 50 && id > -1',
                                                                            'top_k': 703},
                                                                           {'search_param': {'drop_ratio_search': 0.3},
                                                                            'anns_field': 'sparse_float_vector_1',
                                                                            'expr': 'id < 9000000 && int8_1 >= 0',
                                                                            'top_k': 57}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 1200,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'output_fields': ['sparse_float_vector_1', 'int8_1', 'int16_1', 'int32_1',
                                                                                                    'int64_1', 'varchar_1', 'bool_1', 'array_int8_1',
                                                                                                    'array_int16_1', 'array_int32_1', 'array_int64_1',
                                                                                                    'array_varchar_1', 'array_bool_1', 'id', 'float_vector'],
                                                                                  'nq': 10}}}]},
            'run_id': 2024111421478983,
            'datetime': '2024-11-14 08:15:47.776132',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 2810.6158,
                                      'sparse_float_vector_1': {'RT': 635.0244},
                                      'int8_1': {'RT': 227.9973},
                                      'int16_1': {'RT': 96.8698},
                                      'int32_1': {'RT': 99.9327},
                                      'int64_1': {'RT': 0.5241},
                                      'varchar_1': {'RT': 7.6035},
                                      'bool_1': {'RT': 1.0243},
                                      'array_int8_1': {'RT': 0.5177},
                                      'array_int16_1': {'RT': 0.5184},
                                      'array_int32_1': {'RT': 0.5202},
                                      'array_int64_1': {'RT': 0.5174},
                                      'array_varchar_1': {'RT': 0.5211},
                                      'array_bool_1': {'RT': 0.5307}},
                            'insert': {'total_time': 794.2879, 'VPS': 12589.8934, 'batch_time': 0.7943, 'batch': 10000},
                            'flush': {'RT': 3.0244},
                            'load': {'RT': 10.9402},
                            'Locust': {'Aggregated': {'Requests': 119864,
                                                      'Fails': 97917,
                                                      'RPS': 1.67,
                                                      'fail_s': 0.82,
                                                      'RT_max': 806806.2,
                                                      'RT_avg': 56122.84,
                                                      'TP50': 0,
                                                      'TP99': 519000.0},
                                       'hybrid_search': {'Requests': 59923,
                                                         'Fails': 48876,
                                                         'RPS': 0.83,
                                                         'fail_s': 0.82,
                                                         'RT_max': 806806.2,
                                                         'RT_avg': 61003.97,
                                                         'TP50': 0,
                                                         'TP99': 543000.0},
                                       'search': {'Requests': 59941,
                                                  'Fails': 49041,
                                                  'RPS': 0.83,
                                                  'fail_s': 0.82,
                                                  'RT_max': 793397.1,
                                                  'RT_avg': 51243.17,
                                                  'TP50': 0,
                                                  'TP99': 498000.0}}}}}

The text was updated successfully, but these errors were encountered:

issue: #37718 This PR refine the shard client ref counter, dec ref counter won't release client anymore, and only permit shard client manager to remove client. --------- Signed-off-by: Wei Liu <[email protected]>

weiliu1031 · 2024-11-18T10:59:54Z

please verify this with latest image

weiliu1031 · 2024-11-18T11:00:02Z

/assign @wangting0128

wangting0128 · 2024-11-19T05:01:55Z

working on it

wangting0128 · 2024-11-19T09:20:56Z

verification passed

argo task: master-20241118-16e20616-amd64
image: master-20241118-16e20616-amd64

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Nov 15, 2024

wangting0128 added this to the 2.5.0 milestone Nov 15, 2024

wangting0128 assigned yanliang567 and weiliu1031 Nov 15, 2024

weiliu1031 mentioned this issue Nov 15, 2024

fix: Get shard client failed by client is closed #37729

Merged

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 15, 2024

yanliang567 removed their assignment Nov 15, 2024

sre-ci-robot assigned wangting0128 Nov 18, 2024

yanliang567 unassigned weiliu1031 Nov 18, 2024

wangting0128 closed this as completed Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark][cluster] DQL request raises error `client is closed` in pod failure injection multi-replica testing #37718

[Bug]: [benchmark][cluster] DQL request raises error `client is closed` in pod failure injection multi-replica testing #37718

wangting0128 commented Nov 15, 2024

weiliu1031 commented Nov 18, 2024

weiliu1031 commented Nov 18, 2024

wangting0128 commented Nov 19, 2024

wangting0128 commented Nov 19, 2024

[Bug]: [benchmark][cluster] DQL request raises error client is closed in pod failure injection multi-replica testing #37718

[Bug]: [benchmark][cluster] DQL request raises error client is closed in pod failure injection multi-replica testing #37718

Comments

wangting0128 commented Nov 15, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

weiliu1031 commented Nov 18, 2024

weiliu1031 commented Nov 18, 2024

wangting0128 commented Nov 19, 2024

wangting0128 commented Nov 19, 2024

verification passed

[Bug]: [benchmark][cluster] DQL request raises error `client is closed` in pod failure injection multi-replica testing #37718

[Bug]: [benchmark][cluster] DQL request raises error `client is closed` in pod failure injection multi-replica testing #37718