Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support inverted index for array #33452

Merged
merged 10 commits into from
May 31, 2024
Merged

feat: support inverted index for array #33452

merged 10 commits into from
May 31, 2024

Conversation

longjiquan
Copy link
Contributor

issue: #27704

longjiquan added 4 commits May 29, 2024 15:38
@longjiquan
feat: support inverted index for array
88a9619 
Signed-off-by: longjiquan <[email protected]>
@longjiquan
InvertedIndex: parse array data
9171e7e 
Signed-off-by: longjiquan <[email protected]>
@longjiquan
query
fb7794b 
Signed-off-by: longjiquan <[email protected]>
@longjiquan
fix code checker
ff45c50 
Signed-off-by: longjiquan <[email protected]>
@sre-ci-robot sre-ci-robot requested review from aoiasd and bigsheeper May 29, 2024 07:53
@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label May 29, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels May 29, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented May 29, 2024

@longjiquan ut workflow job failed, comment rerun ut can trigger the job again.

@longjiquan
fix code checker
ce12669 
Signed-off-by: longjiquan <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented May 29, 2024

@longjiquan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@longjiquan
Copy link
Contributor Author

comment fixed at 073e5aa

@zhagnlu
Copy link
Contributor

zhagnlu commented May 30, 2024

comment fixed at 073e5aa

/lgtm

@mergify Mergify
Copy link
Contributor

mergify bot commented May 30, 2024

@longjiquan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@longjiquan
fix ut & ci
5169804 
Signed-off-by: longjiquan <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented May 30, 2024

@longjiquan E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented May 30, 2024

@longjiquan ut workflow job failed, comment rerun ut can trigger the job again.

@longjiquan
fix ut
8c5e7b7 
Signed-off-by: longjiquan <[email protected]>
@longjiquan
Copy link
Contributor Author

rerun ut

@codecov Codecov
Copy link

codecov bot commented May 30, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 40.87302% with 149 lines in your changes are missing coverage. Please review.

Project coverage is 82.06%. Comparing base (7763718) to head (8c5e7b7).
Report is 2 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #33452      +/-   ##
==========================================
- Coverage   82.12%   82.06%   -0.07%     
==========================================
  Files        1015     1005      -10     
  Lines      129767   130040     +273     
==========================================
+ Hits       106574   106719     +145     
- Misses      19231    19353     +122     
- Partials     3962     3968       +6
Files Coverage Δ
...ternal/core/src/exec/expression/JsonContainsExpr.h 100.00% <ø> (ø)
internal/core/src/index/IndexFactory.h 100.00% <ø> (ø)
internal/core/src/index/InvertedIndexTantivy.h 47.36% <100.00%> (ø)
internal/core/src/indexbuilder/IndexFactory.h 43.47% <ø> (-2.53%) ⬇️
internal/core/src/storage/Types.h 22.64% <ø> (-1.00%) ⬇️
pkg/util/indexparamcheck/inverted_checker.go 100.00% <100.00%> (ø)
internal/core/src/indexbuilder/index_c.cpp 24.15% <0.00%> (+0.27%) ⬆️
internal/core/src/expr/ITypeExpr.h 51.11% <20.00%> (+6.06%) ⬆️
internal/core/src/index/IndexFactory.cpp 72.04% <78.12%> (-0.24%) ⬇️
internal/querynodev2/segments/load_index_info.go 21.93% <65.00%> (-51.40%) ⬇️
... and 4 more

... and 226 files with indirect coverage changes

czs007
czs007 reviewed May 30, 2024
View reviewed changes
internal/core/thirdparty/tantivy/tantivy-wrapper.h
@@ -49,6 +51,15 @@ struct RustArrayWrapper {
std::cout << ss.str() << std::endl;
}

std::set<uint32_t>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use following code to accelerate insert of std::set.

std::set<uint32_t>
to_set() {
    std::set<uint32_t> s;
    for (int i = 0; i < array_.len; i++) {
        s.insert(s.end(), array_.array[i]);
    }
    return s;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This api is just used in test. to_set should never be used in real filtering sceniar.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll delete this api in the next pr.

@czs007
Copy link
Collaborator

czs007 commented May 30, 2024

/approve
/lgtm

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, longjiquan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@czs007 czs007 added ci-passed manual-pass manually set pass before ci-passed labeled labels May 31, 2024
@sre-ci-robot sre-ci-robot merged commit 0c5d866 into milvus-io:master May 31, 2024
14 of 15 checks passed
@longjiquan longjiquan deleted the array-inverted-index branch May 31, 2024 01:58
longjiquan added a commit to longjiquan/milvus that referenced this pull request Jun 21, 2024
@longjiquan
feat: support inverted index for array (milvus-io#33452)
f1cab41 
Signed-off-by: longjiquan <[email protected]>
@longjiquan longjiquan mentioned this pull request Jun 21, 2024
Closed
longjiquan added a commit to longjiquan/milvus that referenced this pull request Jun 21, 2024
@longjiquan
feat: support inverted index for array (milvus-io#33452)
afd767a 
Signed-off-by: longjiquan <[email protected]>
@longjiquan longjiquan mentioned this pull request Jun 21, 2024
Merged
longjiquan added a commit to longjiquan/milvus that referenced this pull request Jun 21, 2024
@longjiquan
feat: support inverted index for array (milvus-io#33452)
aed2991 
Signed-off-by: longjiquan <[email protected]>
sre-ci-robot pushed a commit that referenced this pull request Jun 24, 2024
@longjiquan @xiaocai2333
feat: support inverted index for array (#33452) (#34053)
22e6807 
pr: #33184
pr: #33452
pr: #33633
issue: #27704
Co-authored-by: xiaocai2333 <[email protected]>

---------

Signed-off-by: Cai Zhang <[email protected]>
Signed-off-by: longjiquan <[email protected]>
Co-authored-by: cai.zhang <[email protected]>
yellow-shine pushed a commit to yellow-shine/milvus that referenced this pull request Jul 2, 2024
@longjiquan @yellow-shine
feat: support inverted index for array (milvus-io#33452)
d1b35f5 
issue: milvus-io#27704

---------

Signed-off-by: longjiquan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@czs007 czs007 czs007 left review comments

@zhagnlu zhagnlu zhagnlu left review comments

@aoiasd aoiasd Awaiting requested review from aoiasd

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

Assignees

@zhagnlu zhagnlu

@czs007 czs007

Labels
approved area/compilation area/test ci-passed dco-passed DCO check passed. kind/feature Issues related to feature request from users lgtm manual-pass manually set pass before ci-passed labeled sig/testing size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@longjiquan @zhagnlu @czs007 @sre-ci-robot