enhance: Use partitionID when delete by partitionKey #38232
Conversation
sre-ci-robot
added
size/XXL
mergify
bot
added
dco-passed
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.4 #38232 +/- ##
==========================================
- Coverage 80.44% 79.46% -0.99%
==========================================
Files 1077 1077
Lines 169442 169440 -2
==========================================
- Hits 136316 134639 -1677
- Misses 28677 30326 +1649
- Partials 4449 4475 +26
|
XuanYang-cn
force-pushed
the
enhance/delete-partitionkey2id
branch
from
c7b09e5 to
fb7e0a1
Compare
|
@XuanYang-cn E2e jenkins job failed, comment |
2 similar comments
|
@XuanYang-cn E2e jenkins job failed, comment |
|
@XuanYang-cn E2e jenkins job failed, comment |
XuanYang-cn
force-pushed
the
enhance/delete-partitionkey2id
branch
from
95afbe7 to
6fa91fc
Compare
|
@XuanYang-cn Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco. |
mergify
bot
added
needs-dco
When delete by partition_key, Milvus will generates L0 segments globally. During L0 Compaction, those L0 segments will touch all partitions collection wise. Due to the false-positive rate of segment bloomfilters, L0 compactions will append false deltalogs to completed irrelevant partitions, which causes *partition deletion amplification. This PR uses partition_key to set targeted partitionID when producing deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to partition level, and remove the false-positive influence collection-wise. However, due to DeleteMsg structure, we can only label one partition to one deleteMsg, so this enhancement fails if user wants to delete over 2 partition_keys in one deletion. Signed-off-by: yangxuan <[email protected]>
XuanYang-cn
force-pushed
the
enhance/delete-partitionkey2id
branch
from
6fa91fc to
c4bbbae
Compare
mergify
bot
added
dco-passed
|
@XuanYang-cn E2e jenkins job failed, comment |
|
/run-cpu-e2e |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tedxu, XuanYang-cn The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
When delete by partition_key, Milvus will generates L0 segments globally. During L0 Compaction, those L0 segments will touch all partitions collection wise. Due to the false-positive rate of segment bloomfilters, L0 compactions will append false deltalogs to completed irrelevant partitions, which causes *partition deletion amplification.
This PR uses partition_key to set targeted partitionID when producing deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to partition level, and remove the false-positive influence collection-wise.
However, due to DeleteMsg structure, we can only label one partition to one deleteMsg, so this enhancement fails if user wants to delete over 2 partition_keys in one deletion.
pr: #38231
See also: #34665