support skip-index based on chunk-metrics to accelerate expr filter(#27925) #28297

MrPresent-Han · 2023-11-09T04:01:36Z

related: #27925

MrPresent-Han

need review for interface design and basic functions has been testified correctly on my local macOS

MrPresent-Han · 2023-11-09T04:02:36Z

internal/core/src/index/ScalarIndexSort-inl.h

@@ -307,4 +313,44 @@ ScalarIndexSort<T>::Reverse_Lookup(size_t idx) const {
    auto offset = idx_to_offsets_[idx];
    return data_[offset].a_;
 }
+


aggregate unary and binary judgment into one function

MrPresent-Han · 2023-11-09T04:03:10Z

internal/core/src/query/visitors/ExecExprVisitor.cpp

@@ -293,11 +302,18 @@ ExecExprVisitor::ExecRangeVisitorImpl(FieldId field_id,
                   "[ExecExprVisitor]Data size not equal to size_per_chunk");
        results.emplace_back(std::move(data));
    }
+
    for (auto chunk_id = indexing_barrier; chunk_id < num_chunk; ++chunk_id) {
        auto this_size = chunk_id == num_chunk - 1
                             ? row_count_ - chunk_id * size_per_chunk
                             : size_per_chunk;


use skip Index here when executing expr

MrPresent-Han · 2023-11-09T04:04:31Z

internal/core/src/segcore/SegmentInterface.cpp

@@ -293,4 +293,19 @@ SegmentInternalInterface::timestamp_filter(BitsetType& bitset,
    }
 }


at present, SkipIndex is a segment-level component, and relating load is also triggered by segment

MrPresent-Han · 2023-11-09T04:06:45Z

internal/core/src/segcore/SegmentInterface.cpp

+    skipIndex_.Load(field_id, chunk_id, data_type, chunk_data, count);
+}
+
+


maybe we need a returned value to indicate whether successful or not for loading skipIndex

MrPresent-Han · 2023-11-09T04:07:51Z

internal/core/src/segcore/SkipIndex.h

+
+    FieldChunkMetrics() : hasValue_(false){};
+};
+


SkipIndex is a separate class and can be extended for using fieldChunkMetrics in various ways.

MrPresent-Han · 2023-11-09T04:09:36Z

internal/core/unittest/test_sealed.cpp

@@ -1381,3 +1381,98 @@ TEST(Sealed, LoadArrayFieldDataWithMMap) {
    SealedLoadFieldData(dataset, *segment, {}, true);
    segment->Search(plan.get(), ph_group.get());
 }
+


old ut, I ' m changing now

mergify · 2023-11-09T04:15:26Z

@MrPresent-Han E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-11-09T04:24:49Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-11-09T08:25:46Z

@MrPresent-Han E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-11-09T08:36:24Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

codecov · 2023-11-09T11:39:02Z

Codecov Report

Merging #28297 (f0e3aff) into master (f8aa464) will increase coverage by 0.09%.
Report is 12 commits behind head on master.
The diff coverage is 92.45%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #28297      +/-   ##
==========================================
+ Coverage   81.79%   81.89%   +0.09%     
==========================================
  Files         847      839       -8     
  Lines      119750   120076     +326     
==========================================
+ Hits        97950    98336     +386     
+ Misses      18542    18478      -64     
- Partials     3258     3262       +4

Files	Coverage Δ
internal/core/src/index/ScalarIndexSort.h	`50.00% <ø> (ø)`
...nternal/core/src/query/generated/ExecExprVisitor.h	`100.00% <ø> (ø)`
internal/core/src/segcore/AckResponder.h	`100.00% <ø> (ø)`
internal/core/src/segcore/SegmentInterface.cpp	`92.63% <100.00%> (-1.41%)`	⬇️
internal/core/src/segcore/SegmentInterface.h	`100.00% <ø> (ø)`
internal/core/src/segcore/SegmentSealedImpl.cpp	`76.97% <100.00%> (-0.48%)`	⬇️
internal/core/src/segcore/SkipIndex.cpp	`100.00% <100.00%> (ø)`
internal/querynodev2/segments/segment_loader.go	`74.18% <100.00%> (-0.61%)`	⬇️
internal/core/src/index/ScalarIndexSort-inl.h	`67.50% <83.33%> (+3.25%)`	⬆️
internal/core/src/segcore/SkipIndex.h	`90.21% <90.21%> (ø)`
... and 1 more

... and 177 files with indirect coverage changes

mergify · 2023-11-09T12:45:08Z

@MrPresent-Han E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-11-09T13:44:06Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-11-10T03:18:37Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-11-10T04:13:48Z

@MrPresent-Han E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-11-10T07:31:22Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2023-11-10T09:25:03Z

@MrPresent-Han ut workflow job failed, comment rerun ut can trigger the job again.

MrPresent-Han · 2023-11-10T10:06:43Z

rerun ut

jiaoew1991 · 2023-11-13T02:27:49Z

internal/core/src/query/visitors/ExecExprVisitor.cpp

@@ -1641,8 +1749,11 @@ ExecExprVisitor::visit(UnaryRangeExpr& expr) {
        }
        case DataType::VARCHAR: {
            if (segment_.type() == SegmentType::Growing) {
+                std::cout << "hc===execute unary range: string" << std::endl;


mergify · 2023-11-13T03:15:34Z

@MrPresent-Han E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

MrPresent-Han · 2023-11-13T06:02:26Z

rerun ut

yah01

skip for term expr would come later?

MrPresent-Han · 2023-11-14T03:18:58Z

skip for term expr would come later?

emm， I forget this, good catch

…ilvus-io#27925) Signed-off-by: MrPresent-Han <[email protected]>

MrPresent-Han · 2023-11-15T02:02:32Z

skip for term expr would come later?

changed

MrPresent-Han

support skip term, need review

MrPresent-Han · 2023-11-15T02:04:08Z

internal/core/src/query/visitors/ExecExprVisitor.cpp

@@ -2522,9 +2633,19 @@ ExecExprVisitor::ExecTermVisitorImplTemplate<bool>(TermExpr& expr_raw)
        // return std::binary_search(terms.begin(), terms.end(), x);
        return term_set.find(x) != term_set.end();
    };
+    auto skip_index_func =


changed, support term @yah01

yah01

/lgtm

jiaoew1991 · 2023-11-15T03:19:34Z

/approve

sre-ci-robot · 2023-11-15T03:19:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiaoew1991, MrPresent-Han

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/core/OWNERS~~ [jiaoew1991]
~~internal/querynodev2/segments/OWNERS~~ [jiaoew1991]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot requested review from aoiasd and godchen0212 November 9, 2023 04:01

sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. area/compilation labels Nov 9, 2023

mergify bot added the dco-passed DCO check passed. label Nov 9, 2023

MrPresent-Han commented Nov 9, 2023

View reviewed changes

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch 2 times, most recently from 91f064b to 4884e45 Compare November 9, 2023 08:14

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch 3 times, most recently from 569dca0 to 47fb9fa Compare November 9, 2023 10:39

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 47fb9fa to 5aa0086 Compare November 9, 2023 12:30

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 5aa0086 to aa61c0f Compare November 10, 2023 02:36

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from aa61c0f to cc8924b Compare November 10, 2023 03:57

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from cc8924b to 6481542 Compare November 10, 2023 07:14

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 6481542 to 92a0270 Compare November 10, 2023 08:33

mergify bot added the ci-passed label Nov 10, 2023

jiaoew1991 reviewed Nov 13, 2023

View reviewed changes

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 92a0270 to 842396d Compare November 13, 2023 03:00

mergify bot removed the ci-passed label Nov 13, 2023

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 842396d to 81e8312 Compare November 13, 2023 03:25

mergify bot added the ci-passed label Nov 13, 2023

yah01 reviewed Nov 14, 2023

View reviewed changes

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 81e8312 to bffca94 Compare November 14, 2023 09:19

mergify bot removed the ci-passed label Nov 14, 2023

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from bffca94 to 1a11a06 Compare November 14, 2023 09:23

support skip-index based on chunk-metrics to accelerate expr filter(m…

f0e3aff

…ilvus-io#27925) Signed-off-by: MrPresent-Han <[email protected]>

MrPresent-Han force-pushed the support-field-chunk-metrics-new branch from 1a11a06 to f0e3aff Compare November 14, 2023 09:24

mergify bot added the ci-passed label Nov 14, 2023

MrPresent-Han commented Nov 15, 2023

View reviewed changes

yah01 reviewed Nov 15, 2023

View reviewed changes

sre-ci-robot assigned yah01 Nov 15, 2023

sre-ci-robot added the lgtm label Nov 15, 2023

sre-ci-robot added the approved label Nov 15, 2023

sre-ci-robot merged commit 836f300 into milvus-io:master Nov 15, 2023
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support skip-index based on chunk-metrics to accelerate expr filter(#27925) #28297

support skip-index based on chunk-metrics to accelerate expr filter(#27925) #28297

MrPresent-Han commented Nov 9, 2023

MrPresent-Han left a comment

MrPresent-Han Nov 9, 2023

MrPresent-Han Nov 9, 2023

MrPresent-Han Nov 9, 2023

MrPresent-Han Nov 9, 2023

MrPresent-Han Nov 9, 2023

MrPresent-Han Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

codecov bot commented Nov 9, 2023 •

edited

Loading

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

MrPresent-Han commented Nov 10, 2023

jiaoew1991 Nov 13, 2023

MrPresent-Han Nov 13, 2023

mergify bot commented Nov 13, 2023

MrPresent-Han commented Nov 13, 2023

yah01 left a comment

MrPresent-Han commented Nov 14, 2023

MrPresent-Han commented Nov 15, 2023

MrPresent-Han left a comment

MrPresent-Han Nov 15, 2023

yah01 left a comment

jiaoew1991 commented Nov 15, 2023

sre-ci-robot commented Nov 15, 2023

		@@ -293,4 +293,19 @@ SegmentInternalInterface::timestamp_filter(BitsetType& bitset,
		}
		}

		skipIndex_.Load(field_id, chunk_id, data_type, chunk_data, count);
		}

support skip-index based on chunk-metrics to accelerate expr filter(#27925) #28297

support skip-index based on chunk-metrics to accelerate expr filter(#27925) #28297

Conversation

MrPresent-Han commented Nov 9, 2023

MrPresent-Han left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

codecov bot commented Nov 9, 2023 • edited Loading

Codecov Report

mergify bot commented Nov 9, 2023

mergify bot commented Nov 9, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

mergify bot commented Nov 10, 2023

MrPresent-Han commented Nov 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Nov 13, 2023

MrPresent-Han commented Nov 13, 2023

yah01 left a comment

Choose a reason for hiding this comment

MrPresent-Han commented Nov 14, 2023

MrPresent-Han commented Nov 15, 2023

MrPresent-Han left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yah01 left a comment

Choose a reason for hiding this comment

jiaoew1991 commented Nov 15, 2023

sre-ci-robot commented Nov 15, 2023

codecov bot commented Nov 9, 2023 •

edited

Loading