[Bug]: milvus rootcoord always crashes and restarts, the error message is oomkill, the memory has been allocated to 48G

[TonyAnn]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.2.16
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar 
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

milvus rootcoord always crashes and restarts, the error message is oomkill, the memory has been allocated to 48G

Expected Behavior

The detailed rootcoord log is shown in the attachment
rootcoord_280160cbb3b-json.log
rootcoord_json.log
rootcoord_280160cbb3b-json.log

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

are you sure it's coordinator reach memory limit?

from the log querynode-11298 is full of memory and this numbers shows that some node reboot a lot of times.

If you do say coordinators have so much memory usage, please offer a pprof file of rootcoord so we can do futher analysis

[xiaofan-luan]

I don't think it is possible for rootcoord to have a lot of memory in a short period of time.

[xiaofan-luan]

from the log, rootcoord works find without seeing any problems

[yanliang567]

@TonyAnn it looks like you have created thousands of collections in the cluster, which could be already improved in latest milvus relase. could you please retry on milvus 2.4.7 or 2.3.20

[xiaofan-luan]

@TonyAnn it looks like you have created thousands of collections in the cluster, which could be already improved in latest milvus relase. could you please retry on milvus 2.4.7 or 2.3.20

even with 10k collections, the memory usage is not expected?

[xiaofan-luan]

we need pprof to understand why. maybe it's due to some kind of retry

[yanliang567]

pprof.milvus.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
here received the pprof from user
/assign @congqixia
please help to take a look

[congqixia]

inuse space is very low, but the alloc space is huge
how many collection do you have in you instance?
@TonyAnn

[congqixia]

@TonyAnn also 2.2.16 is a very old version. There might be some optimization related to this issue in 2.3.x and 2.4.x. It highly recommended to upgrade you cluster to a more stable version

[TonyAnn]

inuse space is very low, but the alloc space is huge how many collection do you have in you instance? @TonyAnn

@congqixia There are a large number of collections in the cluster. There is a liveness detection script that continuously creates and deletes collections. This script may be the cause.

[TonyAnn]

why

OK, it will be upgraded in a while.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.