You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found the issue when CometCollectExec is added during #100.
The error msg is: data type Binary not supported in shuffle write
Reproduced in the local environment, the whole stack trace is :
org.apache.comet.CometNativeException: not implemented: data type Binary not supported in shuffle write
at std::backtrace_rs::backtrace::libunwind::trace(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93)
at std::backtrace_rs::backtrace::trace_unsynchronized(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/../../backtrace/src/backtrace/mod.rs:66)
at std::backtrace::Backtrace::create(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/backtrace.rs:331)
at comet::errors::init::{{closure}}(./core/src/errors.rs:134)
at as core::ops::function::Fn>::call(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/alloc/src/boxed.rs:2021)
at std::panicking::rust_panic_with_hook(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/panicking.rs:757)
at std::panicking::begin_panic_handler::{{closure}}(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/panicking.rs:631)
at std::sys_common::backtrace::__rust_end_short_backtrace(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/sys_common/backtrace.rs:170)
at rust_begin_unwind(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/std/src/panicking.rs:619)
at core::panicking::panic_fmt(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/panicking.rs:72)
at comet::execution::datafusion::shuffle_writer::slot_size(./core/src/execution/datafusion/shuffle_writer.rs:319)
at comet::execution::datafusion::shuffle_writer::PartitionBuffer::init_active_if_necessary::{{closure}}(./core/src/execution/datafusion/shuffle_writer.rs:225)
at core::iter::adapters::map::map_fold::{{closure}}(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/iter/adapters/map.rs:84)
at core::iter::traits::iterator::Iterator::fold(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/iter/traits/iterator.rs:2639)
at as core::iter::traits::iterator::Iterator>::fold(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/iter/adapters/map.rs:124)
at ::sum(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/iter/traits/accum.rs:50)
at core::iter::traits::iterator::Iterator::sum(/rustc/ec08a0337f3556212525dbf1d3b41e19bdf27621/library/core/src/iter/traits/iterator.rs:3634)
at comet::execution::datafusion::shuffle_writer::PartitionBuffer::init_active_if_necessary(./core/src/execution/datafusion/shuffle_writer.rs:221)
at comet::execution::datafusion::shuffle_writer::PartitionBuffer::append_rows(./core/src/execution/datafusion/shuffle_writer.rs:246)
at comet::execution::datafusion::shuffle_writer::PartitionBuffer::append_batch(./core/src/execution/datafusion/shuffle_writer.rs:237)
at comet::execution::datafusion::shuffle_writer::ShuffleRepartitioner::insert_batch::{{closure}}(./core/src/execution/datafusion/shuffle_writer.rs:730)
at comet::execution::datafusion::shuffle_writer::external_shuffle::{{closure}}(./core/src/execution/datafusion/shuffle_writer.rs:930)
It's clear that slot_size and append_columns doesn't include binary support.
Steps to reproduce
No response
Expected behavior
Binary data could be wrote by shuffle writer.
Additional context
I can submit a PR for this.
The text was updated successfully, but these errors were encountered:
Describe the bug
Found the issue when
CometCollectExec
is added during #100.The error msg is:
data type Binary not supported in shuffle write
Reproduced in the local environment, the whole stack trace is :
It's clear that
slot_size
andappend_columns
doesn't include binary support.Steps to reproduce
No response
Expected behavior
Binary data could be wrote by shuffle writer.
Additional context
I can submit a PR for this.
The text was updated successfully, but these errors were encountered: