Support more than 1024 IP/masks with indexed field #16391

mkhludnev · 2024-10-19T10:20:44Z

Now

terms_query over IP with mask / values parsed as BooleanQuery.
BooleanQuery is prone to max clauses limit
as a result 1025 masks fails with too many clauses limit

Proposal

for indexed field combine masks with MultiRangePointQuery
for doc_values only field still fall back to Boolean Query

Related Issues

Resolves #16200

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-10-19T10:30:40Z

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-19T17:36:20Z

❌ Gradle check result for c9e2bd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-19T19:45:30Z

❌ Gradle check result for b6c3410: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev · 2024-10-20T07:03:26Z

it's an alt of #16202

server/src/main/java/org/opensearch/index/mapper/IpFieldMapper.java

github-actions · 2024-11-08T15:32:08Z

❌ Gradle check result for 6a11b54: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-11T23:29:16Z

❌ Gradle check result for 26ff736: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

* Updating Ip fields to use doc_values to search Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix IP tests Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix skip to allow yaml test to pass on main Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Update tests to use existing test file Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Changing skip version to match bwc Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Using exact match instead of range Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Spotless Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix IP field tests Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix spotless + precommit failure Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Get point out of query and into value Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Fix term tests Signed-off-by: Harsha Vamsi Kalluri <[email protected]> * Add skip test logic to only doc_values test Signed-off-by: Harsha Vamsi Kalluri <[email protected]> --------- Signed-off-by: Harsha Vamsi Kalluri <[email protected]>

github-actions · 2024-11-13T08:50:44Z

❌ Gradle check result for 01db875: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-19T21:20:31Z

❌ Gradle check result for 5849b96: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T09:53:54Z

❌ Gradle check result for d03b618: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T10:36:14Z

❌ Gradle check result for 7e1f8b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T15:24:05Z

❌ Gradle check result for c429cf0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev · 2024-11-20T15:51:02Z

Let's discuss edge cases, which are not obvious:

doc_values only field with mask/ values

In this case we can just create BooleanQuery { ssDvRangeQuery, ...}. It may lately fail with too many clauses, and it's reasonable. Note: it will be a way better with ssDvMultiRange

index & doc_values field with mask/ values

a strawman approach: forget about dv and IndexOrDvQuery, and just create MultiRangePointQuery
~~attempt to create IndexOrDvQuery(MultiRangePointQuery, BooleanQuery { ssDvRangeQuery, ...}) when number of BQ is less than MaxClauses limit.~~

The problem is that this boundary limit can't be decided on query parsing because sibling filter clauses may exceed MaxClauses limit.

github-actions · 2024-11-20T17:13:40Z

❌ Gradle check result for cfa3904: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-11-20T21:40:00Z

✅ Gradle check result for a4d65db: SUCCESS

codecov · 2024-11-20T21:40:43Z

Codecov Report

Attention: Patch coverage is 86.15385% with 9 lines in your changes missing coverage. Please review.

Project coverage is 72.14%. Comparing base (3b4fa0e) to head (341e256).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
...ava/org/opensearch/index/mapper/IpFieldMapper.java	86.15%	4 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16391      +/-   ##
============================================
+ Coverage     72.11%   72.14%   +0.02%     
- Complexity    65192    65251      +59     
============================================
  Files          5318     5318              
  Lines        303903   303949      +46     
  Branches      43970    43985      +15     
============================================
+ Hits         219166   219288     +122     
+ Misses        66786    66719      -67     
+ Partials      17951    17942       -9

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Mikhail Khludnev <[email protected]>

github-actions · 2024-11-21T08:05:44Z

❌ Gradle check result for 75b2719: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mkhludnev · 2024-11-21T19:38:01Z

Hi there!
Kindly asking for review. And tagging as backport.
Thanks!

reta · 2024-11-22T01:38:02Z

In this case we can just create BooleanQuery { ssDvRangeQuery, ...}. It may lately fail with too many clauses, and it's reasonable. Note: it will be a way better with ssDvMultiRange

Just a question, since with this change we still don't solve the problem completely (by and large, the limit might be hit anyway), is it worth the effort?

mkhludnev · 2024-11-22T06:26:49Z

is it worth the effort?

I think it is: it unblocks large terms_queries for indexed fields right now. Regarding doc_values solution: Lucene#13974, I'm not sure if I'm able to complete it and how long it take.

PS. Now, if terms_query has 2K IPs it works fine even for doc_values-only, but adding single mask flip it to too many clauses. This fix make it (2K IPs + less than 1K masks) work (even for doc_values-only).

Signed-off-by: Mikhail Khludnev <[email protected]>

github-actions · 2024-11-22T07:30:15Z

✅ Gradle check result for e8e769f: SUCCESS

reta · 2024-11-22T13:56:05Z

LGTM, thanks @mkhludnev ! @msfroh anything from your side?

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java

msfroh

Thanks, @mkhludnev! Looks good.

I agree that your new test should probably be marked as an IT, but otherwise I'm happy to merge.

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java

Signed-off-by: Mikhail Khludnev <[email protected]>

github-actions · 2024-11-26T14:20:10Z

✅ Gradle check result for 341e256: SUCCESS

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java

Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Mikhail Khludnev <[email protected]>

Signed-off-by: Mikhail Khludnev <[email protected]>

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Capabilities labels Oct 19, 2024

opensearch-ci-bot mentioned this pull request Oct 20, 2024

[AUTOCUT] Gradle Check Flaky Test Report for VerifyVersionConstantsIT #14585

Open

mkhludnev commented Oct 20, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/mapper/IpFieldMapper.java Outdated Show resolved Hide resolved

mkhludnev mentioned this pull request Oct 20, 2024

[Feature Request] relax max Clauses Count limitation of termS query over IP field #16200

Open

msfroh mentioned this pull request Nov 12, 2024

Updating Ip fields to use doc_values to search #11508

Merged

8 tasks

mkhludnev mentioned this pull request Nov 16, 2024

use terms in set for concrete IPs keep disjunctions over ranges. fix #16200 #16202

Closed

3 tasks

mkhludnev force-pushed the 16200-multirange branch from 01db875 to 5849b96 Compare November 19, 2024 21:10

opensearch-ci-bot mentioned this pull request Nov 15, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RestoreShallowSnapshotV2IT #16658

Open

mkhludnev changed the title ~~IP field via MultrangeQuery fix #16200~~ Support more than 1024 IP/masks with indexed field Nov 20, 2024

mkhludnev marked this pull request as ready for review November 20, 2024 20:53

mkhludnev requested review from anasalkouz, andrross, ashking94, Bukhtawar and CEHENKLE as code owners November 20, 2024 20:53

MultiRangeQuery for searching IP masks more 1025 masks in indexed field.

75b2719

Signed-off-by: Mikhail Khludnev <[email protected]>

mkhludnev force-pushed the 16200-multirange branch from a4d65db to 75b2719 Compare November 21, 2024 07:14

mkhludnev requested review from jainankitk and linuxpi as code owners November 21, 2024 07:14

opensearch-ci-bot mentioned this pull request Nov 21, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

reta added backport 2.x Backport to 2.x branch v3.0.0 Issues and PRs related to version 3.0.0 v2.19.0 Issues and PRs related to version 2.19.0 labels Nov 22, 2024

Rename

e8e769f

Signed-off-by: Mikhail Khludnev <[email protected]>

reta approved these changes Nov 22, 2024

View reviewed changes

mkhludnev commented Nov 25, 2024

View reviewed changes

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java Outdated Show resolved Hide resolved

msfroh approved these changes Nov 25, 2024

View reviewed changes

server/src/test/java/org/opensearch/search/SearchIpFieldTermsTests.java Outdated Show resolved Hide resolved

mkhludnev added 2 commits November 26, 2024 15:52

test comment

5ff6d11

Signed-off-by: Mikhail Khludnev <[email protected]>

move heavy test to IT

341e256

Signed-off-by: Mikhail Khludnev <[email protected]>

mkhludnev requested review from reta and msfroh November 26, 2024 15:13

reta reviewed Nov 26, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java Outdated Show resolved Hide resolved

reta reviewed Nov 26, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/search/fields/SearchIpFieldTermsIT.java Outdated Show resolved Hide resolved

reta approved these changes Nov 26, 2024

View reviewed changes

mkhludnev and others added 2 commits November 26, 2024 22:47

Apply suggestions from code review

c7382e3

Co-authored-by: Andriy Redko <[email protected]> Signed-off-by: Mikhail Khludnev <[email protected]>

move heavy test to IT

7aa15da

Signed-off-by: Mikhail Khludnev <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support more than 1024 IP/masks with indexed field #16391

Support more than 1024 IP/masks with indexed field #16391

mkhludnev commented Oct 19, 2024 •

edited

Loading

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

mkhludnev commented Oct 20, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 13, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

mkhludnev commented Nov 20, 2024 •

edited

Loading

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

codecov bot commented Nov 20, 2024 •

edited

Loading

github-actions bot commented Nov 21, 2024

mkhludnev commented Nov 21, 2024

reta commented Nov 22, 2024

mkhludnev commented Nov 22, 2024 •

edited

Loading

github-actions bot commented Nov 22, 2024

reta commented Nov 22, 2024

msfroh left a comment

github-actions bot commented Nov 26, 2024

Support more than 1024 IP/masks with indexed field #16391

Are you sure you want to change the base?

Support more than 1024 IP/masks with indexed field #16391

Conversation

mkhludnev commented Oct 19, 2024 • edited Loading

Now

Proposal

Related Issues

Check List

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

github-actions bot commented Oct 19, 2024

mkhludnev commented Oct 20, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 11, 2024

github-actions bot commented Nov 13, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

mkhludnev commented Nov 20, 2024 • edited Loading

doc_values only field with mask/ values

index & doc_values field with mask/ values

github-actions bot commented Nov 20, 2024

github-actions bot commented Nov 20, 2024

codecov bot commented Nov 20, 2024 • edited Loading

Codecov Report

github-actions bot commented Nov 21, 2024

mkhludnev commented Nov 21, 2024

reta commented Nov 22, 2024

mkhludnev commented Nov 22, 2024 • edited Loading

github-actions bot commented Nov 22, 2024

reta commented Nov 22, 2024

msfroh left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 26, 2024

mkhludnev commented Oct 19, 2024 •

edited

Loading

mkhludnev commented Nov 20, 2024 •

edited

Loading

codecov bot commented Nov 20, 2024 •

edited

Loading

mkhludnev commented Nov 22, 2024 •

edited

Loading