[Bug]: QueryNode oom when delete 23% of 100 million varchar pks

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20241125-e928e15b-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server config

    queryNode:
      replicas: 5
      resources:
        limits:
          cpu: "8" 
          memory: 32Gi
        requests:
          cpu: "8" 
          memory: 32Gi
  config:
    dataCoord:
      segment:
        expansionRate: 1.15
        maxSize: 2048
        sealProportion: 0.12
    quotaAndLimits:
      dml:
        deleteRate:
          max: 2
        enabled: true
        insertRate:
          max: 16
      limitWriting:
        deleteBufferRowCountProtection:
          enabled: true
          highWaterLevel: 25000000
          lowWaterLevel: 12000000
        deleteBufferSizeProtection:
          enabled: true
          highWaterLevel: 536870912
          lowWaterLevel: 134217728
        growingSegmentsSizeProtection:
          enabled: true
          highWaterLevel: 0.2
          lowWaterLevel: 0.1
          minRateRatio: 0.5
        l0SegmentsRowCountProtection:
          enabled: true
          highWaterLevel: 50000000
          lowWaterLevel: 12000000
        memProtection:
          dataNodeMemoryHighWaterLevel: 0.85
          dataNodeMemoryLowWaterLevel: 0.75
          queryNodeMemoryHighWaterLevel: 0.85
          queryNodeMemoryLowWaterLevel: 0.75
      limits:
        complexDeleteLimitEnable: true

client

• collection schema

{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 64}, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'enable_dynamic_field': False}

• 100m entities with pk avg length 63

metrics

• memory usage before/after delete
  
  metrics of compact-opt-100m-3

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/fa63b3a6-ac30-45ab-bb99-0787304fd919?nodeId=zong-concurrent-delete-varchar-3-a

Milvus Log

compact-opt-100m-3-milvus-datanode-c65548448-zzcdp                1/1     Running       0               67m     10.104.13.170   4am-node16   <none>           <none>
compact-opt-100m-3-milvus-indexnode-696586cf69-ljnhr              1/1     Running       0               70m     10.104.32.52    4am-node39   <none>           <none>
compact-opt-100m-3-milvus-indexnode-696586cf69-ps5n5              1/1     Running       0               62m     10.104.18.130   4am-node25   <none>           <none>
compact-opt-100m-3-milvus-indexnode-696586cf69-wmth6              1/1     Running       0               62m     10.104.13.176   4am-node16   <none>           <none>
compact-opt-100m-3-milvus-mixcoord-586b477846-9n5m7               1/1     Running       0               69m     10.104.20.124   4am-node22   <none>           <none>
compact-opt-100m-3-milvus-proxy-d8484956d-mlf8b                   1/1     Running       0               66m     10.104.23.26    4am-node27   <none>           <none>
compact-opt-100m-3-milvus-querynode-1-596f4f9444-4tzgq            1/1     Running       9 (12m ago)     62m     10.104.20.128   4am-node22   <none>           <none>
compact-opt-100m-3-milvus-querynode-1-596f4f9444-p64jl            1/1     Running       8 (13m ago)     62m     10.104.5.84     4am-node12   <none>           <none>
compact-opt-100m-3-milvus-querynode-1-596f4f9444-qxpw9            1/1     Running       8 (11m ago)     62m     10.104.6.207    4am-node13   <none>           <none>
compact-opt-100m-3-milvus-querynode-1-596f4f9444-r4qnv            1/1     Running       9 (9m57s ago)   68m     10.104.9.212    4am-node14   <none>           <none>
compact-opt-100m-3-milvus-querynode-1-596f4f9444-z2427            1/1     Running       9               62m     10.104.32.74    4am-node39   <none>           <none>

Anything else?

No response

[yanliang567]

/assign @XuanYang-cn
/unassign

[xiaofan-luan]

One thing @zhagnlu is working on is to store offset in each segment instead of store PK

[XuanYang-cn]

Milvus delete ability diffs a lot when dealing with int64 and varchar.

We lack of a valid protection for Varchar pk continous deletions. Adding it now.