fix: Try to convert a static list into a set in Rust #184
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #184 +/- ##
============================================
+ Coverage 33.29% 33.30% +0.01%
- Complexity 766 767 +1
============================================
Files 107 107
Lines 35385 35372 -13
Branches 7658 7657 -1
============================================
+ Hits 11781 11782 +1
+ Misses 21157 21144 -13
+ Partials 2447 2446 -1 ☔ View full report in Codecov by Sentry. |
|
I believe the failure is due to a BUG in Datafusion, which I have just filed one: apache/datafusion#9530 I will add a temporary fix here. |
| Ok(Arc::new(InListExpr::new(value, list, expr.negated, None))) | ||
| // in_list doesn't handle value being dictionary type correctly, so we need to fall | ||
| // back to InListExpr if in_list fails. | ||
| // TODO: remove the fallback when https://github.com/apache/arrow-datafusion/issues/9530 is fixed |
There was a problem hiding this comment.
nit: maybe we can check if input_schema contains any dictionary type and use InListExpr if so?
Unfortunately dictionary type is treated as a logical type in DF, which caused many issues like this before. We created apache/datafusion#7421 but haven't got resources to work on it yet.
There was a problem hiding this comment.
nit: maybe we can check if input_schema contains any dictionary type and use InListExpr if so?
Sound good to me. I will address it in a following commit in probably ~8 hours as I'm going to bed now.
Unfortunately dictionary type is treated as a logical type in DF, which caused many issues like this before. We created apache/datafusion#7421 but haven't got resources to work on it yet.
yea.. I can imagine that a lot of issues/special handling of dictionary array would be introduced.
sunchao
changed the title
|
Merged, thanks |
Which issue does this PR close?
Closes #183
Rationale for this change
Leverage
InSetoptimization and improve performance.What changes are included in this PR?
in_listfunction to create anInListExprHow are these changes tested?
Existing tests.