[Enhancement] improve sql digest for massive compound predicates #53207
Conversation
Signed-off-by: Murphy <[email protected]>
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
| exprs.add((SlotRef) x); | ||
| } | ||
| }); | ||
| return "$massive_compounds[" + exprs.stream().map(SlotRef::toSqlImpl) |
There was a problem hiding this comment.
SlotRefs are query-specific so unstable, suggest to use original SlotRefs(slotref references to table column) and convert original slotrefs into table columns then sort these columns lexicographically.
| if (pair.first + pair.second >= MASSIVE_COMPOUND_LIMIT) { | ||
| // Only record de-duplicated slots if there are too many compounds | ||
| Set<SlotRef> exprs = Sets.newHashSet(); | ||
| traverse(node, x -> { |
There was a problem hiding this comment.
why not use node.collectAllSlotRefs() directly?
There was a problem hiding this comment.
well, I should use it. but it seems I should improve the performance of that method first.
There was a problem hiding this comment.
it constructs a lot of intermediate list,, i'm afraid the performance can be terrible
| @Override | ||
| public String visitCompoundPredicate(CompoundPredicate node, Void context) { | ||
| Pair<Integer, Integer> pair = countCompound(node); | ||
| if (pair.first + pair.second >= MASSIVE_COMPOUND_LIMIT) { |
There was a problem hiding this comment.
How about Expr#flattenPredicate;
[FE Incremental Coverage Report]✅ pass : 33 / 33 (100.00%) file detail
|
Signed-off-by: Murphy <[email protected]>
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
Why I'm doing:
What I'm doing:
In cases where an OR predicate is dynamically constructed with a fixed column, the SQL digest varies due to a differing number of predicates. To address this, we consolidate extensive compound predicates into a compact format, ensuring consistent SQL digests.
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: