Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Only delegate to DataFusion cast when we know that it is compatible with Spark #461

Merged
merged 16 commits into from
May 25, 2024
+115 −67
Merged

fix: Only delegate to DataFusion cast when we know that it is compatible with Spark #461

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3a1c387
only delegate to DataFusion cast when we know that it is compatible w…
andygrove May 23, 2024
60354bd
add more supported casts
andygrove May 23, 2024
35585aa
improve support for dictionary-encoded string arrays
andygrove May 23, 2024
dc4f99c
clippy
andygrove May 23, 2024
f8a6f94
Merge remote-tracking branch 'apache/main' into cast-df-compat
andygrove May 23, 2024
2b21917
fix merge conflict
andygrove May 23, 2024
c2c2546
fix a regression
andygrove May 23, 2024
7f1951a
fix a regression
andygrove May 23, 2024
c0f913b
fix a regression
andygrove May 23, 2024
34edf00
fix regression
andygrove May 23, 2024
54043ae
fix regression
andygrove May 23, 2024
2f22e06
fix regression
andygrove May 23, 2024
bc18fae
remove TODO comment now that issue has been filed
andygrove May 24, 2024
340e000
remove cast int32/int64 -> decimal from datafusion compatible list
andygrove May 25, 2024
7d804f5
Revert "remove cast int32/int64 -> decimal from datafusion compatible…
andygrove May 25, 2024
3fb2258
add comment
andygrove May 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion core/src/execution/datafusion/expressions/cast.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,14 +644,16 @@ impl Cast {
| DataType::Float32
| DataType::Float64
),
DataType::Decimal128(_, _) => matches!(
DataType::Decimal128(_, _) | DataType::Decimal256(_, _) => matches!(
viirya marked this conversation as resolved.
Show resolved Hide resolved
to_type,
DataType::Int8
| DataType::Int16
| DataType::Int32
| DataType::Int64
| DataType::Float32
| DataType::Float64
| DataType::Decimal128(_, _)
| DataType::Decimal256(_, _)
),
DataType::Utf8 => matches!(to_type, DataType::Binary),
DataType::Date32 => matches!(to_type, DataType::Utf8),
Expand Down
Loading