[Bug]: Deadlock on compaction when stopping datanode

[bigsheeper]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3 only
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Phenomenon:
Before stopping the old datanode, it received two compaction tasks, A and B, both targeting the same segment, seg0.

Process:

1. Task A starts compaction and successfully injects seg0 into the flush manager.
2. Task B starts compaction and attempts to inject seg0 into the flush manager but gets stuck since the previous seg0 has not yet executed injectDone.
3. Stopping datanode...
4. Task B is stopped first, stop process getting stuck at step 2.
5. Task B's stop being stuck prevents Task A from stopping, further blocking Task A injectDone for seg0, causing a deadlock between A and B.

Q: Why is this a probabilistic issue?
A: The deadlock does not occur if Task A is stopped before Task B.

Q: Why were compaction tasks received for the same segment?
A: The datanode received compaction tasks from both the new and old datacoords simultaneously.

Solution:
Stop all compaction tasks in parallel to avoid interdependencies and prevent deadlocks.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.