[Enhancement]: Refine scalar filtering with better understanding of sort and index

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

1. Data in Milvus should be resort in PK
2. For many operations like range and like a%, we should respect sort order if the operation is happened on PK
3. For term operation, think of using BF to avoid extra filtering
4. For range operation, think of using ordered index
5. Delete/MVCC should be handled with less cost

Why is this needed?

No response

Anything else?

No response

[yah01]

The data sort by pk will be done after L0 segment work as it contains the refactor of compaction, we will sort the data while flushing/compacting, not loading (this will break the offset to vector index, it's possible to work with this but would introduce overhead).

[yah01]

Query related impl plan:

• Split the default pk index with insert record, and move it into the index records, to make the behavior consistent while processing filters
• Leverage the ordered index to process range filter and prefix matching
• Split the data into multiple chunks, and filter the chunks by min/max stats (or BF)

[yah01]

Delete related impl plan:

• Store the offset but not pk for delete records to reduce the overhead Optimize delete records memory usage #27855
• Maintain the cached bitset while deleting but not querying, also change the cache policy

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.