Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Implement custom function module in milvus expr #36560

Merged
merged 1 commit into from
Oct 25, 2024
Merged

feat: Implement custom function module in milvus expr #36560

merged 1 commit into from
Oct 25, 2024

Conversation

jiangyinzuo
Copy link
Contributor

@jiangyinzuo jiangyinzuo commented Sep 27, 2024 •
edited
Loading

OSPP 2024 project: https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org

Solutions:

  • parser (planparserv2)
    • add CallExpr in planparserv2/Plan.g4
    • update parser_visitor and show_visitor
  • grpc protobuf
    • add CallExpr in plan.proto
  • execution (core/src/exec)
    • add CallExpr ValueExpr and ColumnExpr (both logical and physical) for function call and function parameters
  • function factory (core/src/exec/expression/function)
    • create a global hashmap when starting milvus (see server.go)
    • the global hashmap stores function signatures and their function pointers, the CallExpr in execution engine can get the function pointer by function signature.
  • custom functions
    • empty(string)
    • starts_with(string, string)
  • add cpp/go unittests and E2E tests

closes: #36559

@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Sep 27, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Sep 27, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Sep 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Sep 27, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

jiangyinzuo
jiangyinzuo commented Sep 27, 2024
View reviewed changes
Copy link
Contributor Author

@jiangyinzuo jiangyinzuo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some todos

internal/core/src/exec/expression/CallExpr.cpp Outdated
throw std::runtime_error("FilterScalarFunction " + func_sig.toString() +
" not found. " + this->expr_->ToString());
}
// eval inputs in function
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evaluate the input arguments here, not in function

internal/core/src/exec/expression/CallExpr.cpp Outdated
expression::FilterFunctionRegisterKey func_sig{
this->expr_->fun_name(), std::move(func_param_type_list)};
auto& factory = expression::FunctionFactory::Instance();
auto function = factory.GetFilterFunction(func_sig);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Call factory.GetFilterFunction at logical expr or parser

@jiangyinzuo
Copy link
Contributor Author

@zhagnlu

@mergify Mergify
Copy link
Contributor

mergify bot commented Sep 27, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Sep 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/filter-function branch 4 times, most recently from a9369b8 to 3e564b6 Compare October 13, 2024 13:39
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 13, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 13, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 13, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/filter-function branch 2 times, most recently from 0ab6321 to b651d73 Compare October 14, 2024 03:22
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 14, 2024

@jiangyinzuo Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

@mergify mergify bot added needs-dco DCO is missing in this pull request. dco-passed DCO check passed. and removed dco-passed DCO check passed. needs-dco DCO is missing in this pull request. labels Oct 14, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

jiangyinzuo

This comment was marked as duplicate.

Sign in to view

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

rerun cpp-unit-test

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@jiangyinzuo
Copy link
Contributor Author

rerun cpp-unit-test

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@jiangyinzuo cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

rerun cpp-unit-test

1 similar comment
@jiangyinzuo
Copy link
Contributor Author

rerun cpp-unit-test

@jiangyinzuo jiangyinzuo force-pushed the feat/filter-function branch 2 times, most recently from 7f1ced8 to 8382c28 Compare October 24, 2024 15:15
@jiangyinzuo
feat: Implement custom function module in milvus expr
8382c28 
OSPP 2024 project: https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org

Solutions:

- parser (planparserv2)
    - add CallExpr in planparserv2/Plan.g4
    - update parser_visitor and show_visitor
- grpc protobuf
    - add CallExpr in plan.proto
- execution (`core/src/exec`)
    - add `CallExpr`, `ValueExpr` and `ColumnExpr` (both logical and
      physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
    - create a global hashmap when starting milvus (see server.go)
    - the global hashmap stores function signatures and their function
      pointers, the CallExpr in execution engine can get the function pointer by
      function signature.
- `common/Vector.h`
    - add ConstantVector, update ColumnVector
- `segcore/SegmentChunkReader.h`
    - extracted some methods from `CompareExpr.h`
- custom functions
    - empty(string)
    - starts_with(string, string)
- add cpp/go unittests and E2E tests

closes: milvus-io#36559

Signed-off-by: Yinzuo Jiang <[email protected]>
@mergify mergify bot added the ci-passed label Oct 24, 2024
@zhagnlu
Copy link
Contributor

zhagnlu commented Oct 25, 2024

/lgtm

@czs007
Copy link
Collaborator

czs007 commented Oct 25, 2024

/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, jiangyinzuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit 3628593 into milvus-io:master Oct 25, 2024
15 of 17 checks passed
jiangyinzuo added a commit to jiangyinzuo/milvus that referenced this pull request Nov 14, 2024
@jiangyinzuo
doc: add contributing guide for filtering functions
adc205a 
Document about how to add a new filtering function.
PR milvus-io#36560 has implemented the custom filtering function module.

Signed-off-by: Yinzuo Jiang <[email protected]>
Signed-off-by: Yinzuo Jiang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smellthemoon smellthemoon smellthemoon left review comments

@czs007 czs007 czs007 left review comments

@zhagnlu zhagnlu zhagnlu left review comments

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

Assignees

@zhagnlu zhagnlu

Labels
approved area/compilation area/test ci-passed dco-passed DCO check passed. kind/feature Issues related to feature request from users lgtm sig/testing size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature]: Implement custom function module in milvus expr
5 participants
@jiangyinzuo @zhagnlu @czs007 @sre-ci-robot @smellthemoon