When the search method traffic surges to a certain level, a response timeout will occur. But when it slowly increases to the above level, the response is normal.

[wutachiang]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.2.11
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2): milvus-sdk-java v2.2.11
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 64C128G * 6
- GPU: None
- Others:

Current Behavior

When there is an instantaneous sudden increase in traffic, such as from 1000qps to 5000qps instantaneously, the response time of the search method of milvus will slow down, from the previous tens of milliseconds to tens of seconds, and will return to normal after a few seconds. But when the traffic increases steadily from 1000qps to 5000qps, the response of the search method of milvus is relatively normal. The current qps supported by the cluster is about 10,000 qps, so the above situation is very strange.
The monitoring of prometheus is as follows:

Expected Behavior

The search method under instantaneous traffic surge should be consistent with the performance of steady increase.

Steps To Reproduce

1. You can observe the performance of the service in this situation by temporarily increasing the traffic and then immediately returning it to before.

Milvus Log

Relevant log information is attached to the network disk link below.
https://pan.baidu.com/s/1RocYduOAqkOCR6TNo84YjQ

Anything else?

No response

[yanliang567]

I this was caused by the policy of nq combination. @czs007 could you please help to double check?
/assign @czs007
/unassign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[bigsheeper]

Hi, @wutachiang
Could you please provide us with the differences in NQ monitorings between the normal scenario and the abnormal scenario? The monitorings in question are as follows:

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.