[Bug]: Dynamic configuration refresh does not take effect

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20241029-3d1e81fb-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server config

    quotaAndLimits:
      dml:
        deleteRate:
          max: 0.5
        enabled: true
        insertRate:
          max: 8
      limitWriting:
        deleteBufferRowCountProtection:
          enabled: true
          highWaterLevel: 10000000
          lowWaterLevel: 1000000
        deleteBufferSizeProtection:
          enabled: true
          highWaterLevel: 536870912
          lowWaterLevel: 134217728
        growingSegmentsSizeProtection:
          enabled: true
          highWaterLevel: 0.2
          lowWaterLevel: 0.1
          minRateRatio: 0.5
        l0SegmentsRowCountProtection:
          enabled: true
          highWaterLevel: 10000000
          lowWaterLevel: 1000000
        memProtection:
          dataNodeMemoryHighWaterLevel: 0.9
          dataNodeMemoryLowWaterLevel: 0.85
          queryNodeMemoryHighWaterLevel: 0.9
          queryNodeMemoryLowWaterLevel: 0.85
      limits:
        complexDeleteLimitEnable: true

refresh config by etcd

compact-opt-rate-100m-1/config/dataCoord.compaction.timeout
3600
compact-opt-rate-100m-1/config/quotaAndLimits.limits.maxOutputSize
209715200
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferRowCountProtection.enabled
false
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferRowCountProtection.highWaterLevel
100000
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferRowCountProtection.lowWaterLevel
50000
**compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferSizeProtection.enabled
true
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferSizeProtection.highWaterLevel
524288000
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.deleteBufferSizeProtection.lowWaterLevel
104857600**
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.l0SegmentsRowCountProtection.enabled
false
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.l0SegmentsRowCountProtection.highWaterLevel
50000000
compact-opt-rate-100m-1/config/quotaandlimits.limitwriting.l0SegmentsRowCountProtection.lowWaterLevel
5000000

results

Loki logs
However, from the milvus log, the refreshed value has been obtained from etcd, but the actual rate limit uses the previous value.

• 2024/10/29 11:25:47 refresh value succ, lowWaterLevel] [value=104857600]
  
• 2024/10/29 11:30:35 lowWatermark=52428800
  

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

pods:

compact-opt-rate-100m-1-milvus-datanode-6847897fc4-zwgjp        1/1     Running     0                3h46m   10.104.6.238    4am-node13   <none>           <none>
compact-opt-rate-100m-1-milvus-indexnode-84b4f9c6df-59snb       1/1     Running     0                4h35m   10.104.13.51    4am-node16   <none>           <none>
compact-opt-rate-100m-1-milvus-indexnode-84b4f9c6df-l9g45       1/1     Running     0                3h54m   10.104.9.188    4am-node14   <none>           <none>
compact-opt-rate-100m-1-milvus-indexnode-84b4f9c6df-ml5v9       1/1     Running     0                3h53m   10.104.20.189   4am-node22   <none>           <none>
compact-opt-rate-100m-1-milvus-mixcoord-85dcd4664d-pstb5        1/1     Running     0                3h52m   10.104.1.240    4am-node10   <none>           <none>
compact-opt-rate-100m-1-milvus-proxy-84cc974d46-2hlzh           1/1     Running     0                3h45m   10.104.5.148    4am-node12   <none>           <none>
compact-opt-rate-100m-1-milvus-querynode-1-5f8455c74c-22jq7     1/1     Running     0                3h47m   10.104.18.25    4am-node25   <none>           <none>
compact-opt-rate-100m-1-milvus-querynode-1-5f8455c74c-66xn2     1/1     Running     0                3h51m   10.104.17.190   4am-node23   <none>           <none>
compact-opt-rate-100m-1-milvus-querynode-1-5f8455c74c-84szv     1/1     Running     0                3h48m   10.104.30.174   4am-node38   <none>           <none>
compact-opt-rate-100m-1-milvus-querynode-1-5f8455c74c-wcnhg     1/1     Running     0                3h49m   10.104.16.153   4am-node21   <none>           <none>

Anything else?

No response

[XuanYang-cn]

/assign

[ThreadDao]

can't reproduce