[Bug]: Search failed with error: Collection not fully loaded

[aashishsingh2803]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.19
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

We have a cllection that contains 1000 vectors only, and during load collection its waiting for 60 secs, still the collection is not loaded.
During search we are getting error collection not fully loaded.

/workspace/source/internal/proxy/task_scheduler.go:532 github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1: collection not fully loaded[collection=454573880481312361]] [traceID: ec79832ef618d254a3e4a0343d765c79] [timeCost: 3.039977796s] [database: milvusdefaultv2] [collection: a360_prod_36c281b1adbd47c0940854b275e85009d1] [partitions: []] [expr: userAge > 50]

This is happening intermittently and frequency of this is large like our health check runs every in 15 mins and it s happening every 1 hour or so.
This issue occurs intermittently but with a high frequency. Our health check runs every 15 minutes, and the problem manifests approximately once every hour.

Expected Behavior

As the collection is very small i.e. 1000 vectors, it should be loaded instantly.

Steps To Reproduce

No response

Milvus Log

Added the compressed logs in the google drive as github had 25MB limit.
https://drive.google.com/file/d/1tQqyyxCqYeZSdjwC4i8dfXJDOngM4Y42/view?usp=drive_link

Anything else?

Database Name: milvusdefaultv2
Collection Name: a360_prod_36c281b1adbd47c0940854b275e85009d1
This Database contains 11.5K partitions.

I also checked the issue - #32498 but it doesn't mention any fix.

Please investigate this issue and provide guidance on resolving it. If additional details are required, let us know.

[yanliang567]

/assign @weiliu1031
/unassign