[Bug]: test_load_balance_with_one_sealed_segment_id_not_exist test case fails occasionally

[sunby]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

test_load_balance_with_one_sealed_segment_id_not_exist test case fails occasionally

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[zhuwenxing]

It seems like a server bug

failed job: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20HA%20CI/detail/PR-37392/11/pipeline/

The log shows that get_query_segment_info got empty result after insert , flush, insert, and load

[pytest : test] __ TestUtilityAdvanced.test_load_balance_with_one_sealed_segment_id_not_exist __

[pytest : test] [gw3] linux -- Python 3.8.17 /usr/local/bin/python3

[pytest : test] [gw3] linux -- Python 3.8.17 /usr/local/bin/python3[gw3] linux -- Python 3.8.17 /usr/local/bin/python3

[pytest : test] 

[pytest : test] self = <test_utility.TestUtilityAdvanced object at 0x7f29ac192ac0>

[pytest : test] 

[pytest : test]     @pytest.mark.tags(CaseLabel.L1)

[pytest : test]     def test_load_balance_with_one_sealed_segment_id_not_exist(self):

[pytest : test]         """

[pytest : test]         target: test load balance of collection

[pytest : test]         method: init a collection and load balance with one of sealed segment ids not exist

[pytest : test]         expected: raise exception

[pytest : test]         """

[pytest : test]         # init a collection

[pytest : test]         c_name = cf.gen_unique_str(prefix)

[pytest : test]         collection_w = self.init_collection_wrap(name=c_name)

[pytest : test]         collection_w.create_index(default_field_name, default_index_params)

[pytest : test]         ms = MilvusSys()

[pytest : test]         nb = 3000

[pytest : test]         df = cf.gen_default_dataframe_data(nb)

[pytest : test]         collection_w.insert(df)

[pytest : test]         # get sealed segments

[pytest : test]         collection_w.num_entities

[pytest : test]         # get growing segments

[pytest : test]         collection_w.insert(df)

[pytest : test]         collection_w.load()

[pytest : test]         # prepare load balance params

[pytest : test]         res, _ = self.utility_wrap.get_query_segment_info(c_name)

[pytest : test]         segment_distribution = cf.get_segment_distribution(res)

[pytest : test]         all_querynodes = [node["identifier"] for node in ms.query_nodes]

[pytest : test]         all_querynodes = sorted(all_querynodes,

[pytest : test]                                 key=lambda x: len(segment_distribution[x]["sealed"])

[pytest : test]                                 if x in segment_distribution else 0, reverse=True)

[pytest : test]         src_node_id = all_querynodes[0]

[pytest : test]         dst_node_ids = all_querynodes[1:]

[pytest : test]         sealed_segment_ids = segment_distribution[src_node_id]["sealed"]

[pytest : test]         # add a segment id which is not exist

[pytest : test] >       sealed_segment_ids.append(max(segment_distribution[src_node_id]["sealed"]) + 1)

[pytest : test] E       ValueError: max() arg is an empty sequence

[pytest : test] 

[pytest : test] testcases/test_utility.py:1639: ValueError

[pytest : test] ------------------------------ Captured log setup ------------------------------

[pytest : test] [2024-11-07 12:17:04 - INFO - ci_test]: *********************************** setup *********************************** (client_base.py:47)

[pytest : test] [2024-11-07 12:17:04 - INFO - ci_test]: pymilvus version: 2.5.0rc106 (client_base.py:48)

[pytest : test] [2024-11-07 12:17:04 - INFO - ci_test]: [setup_method] Start setup test case test_load_balance_with_one_sealed_segment_id_not_exist. (client_base.py:49)

[pytest : test] ------------------------------ Captured log call -------------------------------

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int64', <DataType.INT64: 5>, ''], kwargs: {'is_primary': False, 'is_partition_key': False, 'nullable': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['varchar', <DataType.VARCHAR: 21>, ''], kwargs: {'max_length': 65535, 'is_primary': False, 'is_partition_key': False, 'nullable': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float_vector', <DataType.FLOAT_VECTOR: 101>, ''], kwargs: {'dim': 128, 'is_primary': False, 'nullable': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float', <DataType.FLOAT: 10>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['json_field', <DataType.JSON: 23>, ''], kwargs: {'is_primary': False, 'nullable': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'name': 'json_field', 'description': '', 'type': <DataType.JSON: 23>}  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [CollectionSchema] args: [[{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}, {'name': 'json_field', 'description': '', 'type': <DataTyp......, kwargs: {'primary_field': 'int64', 'auto_id': False, 'enable_dynamic_field': False} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params......  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : False  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default', '', '', 'default', ''], kwargs: {'host': 'ms-37392-11-py-pr-milvus.milvus-ci', 'port': '19530'} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - INFO - ci_test]: server version: 35e10a3 (client_base.py:166)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['utility_i4nEosSg', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType......., kwargs: {'consistency_level': 'Strong'} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_response) : <Collection>:

[pytest : test] -------------

[pytest : test] <name>: utility_i4nEosSg

[pytest : test] <description>: 

[pytest : test] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>},......  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:04 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'IVF_SQ8', 'metric_type': 'L2', 'params': {'nlist': 64}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:05 - DEBUG - ci_test]: (api_response) : Status(code=0, message=)  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:05 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                         json_field                                       float_vector

[pytest : test] 0         0     0.0       0        {'number': 0, 'float': 0.0}  [0.010380139152805888, 0.10424943405169092, 0....

[pytest : test] 1         1     1.0       1        {'number': 1, 'float': 1.0}  [0.0661......, kwargs: {'timeout': 180} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:05 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 453766499732815876, success count: 3000, err count: 0  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:05 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:08 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:08 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                         json_field                                       float_vector

[pytest : test] 0         0     0.0       0        {'number': 0, 'float': 0.0}  [0.010380139152805888, 0.10424943405169092, 0....

[pytest : test] 1         1     1.0       1        {'number': 1, 'float': 1.0}  [0.0661......, kwargs: {'timeout': 180} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:09 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 453766500612046849, success count: 3000, err count: 0  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:09 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:14 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[pytest : test] [2024-11-07 12:17:14 - DEBUG - ci_test]: (api_request)  : [get_query_segment_info] args: ['utility_i4nEosSg', 20, 'default'], kwargs: {} (api_request.py:62)

[pytest : test] [2024-11-07 12:17:14 - DEBUG - ci_test]: (api_response) : []  (api_request.py:37)

[zhuwenxing]

/assign @sunby

[sunby]

@zhuwenxing it's because the flushed segment is set to invisible

milvus/internal/datacoord/server.go

Line 1015 in 0f51470

operators = append(operators, SetSegmentIsInvisible(segmentID, true))

and it will be treated as growing segment when loading

milvus/internal/datacoord/handler.go

Lines 157 to 158 in 0f51470

	case !isFlushState(s.GetState()) || s.GetIsInvisible():
	growingIDs.Insert(s.GetID())

.

[zhuwenxing]

fix pr already merged