You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, CometExec has default outputPartitioning implementation that reuses Spark original plan's outputPartitioning. In most cases, it is correct. But Spark AQE has special node AQEShuffleRead which can possibly change output partitioning for coalescing purpose, e.g.:
It causes many test failures in org.apache.spark.sql.connector.WriteDistributionAndOrderingSuite on #250. For example, CometSort's output partitioning is the original plan's hashpartitioning(data#3312, 5) of original exchange. But after AQE, AQEShuffleRead is added and it modifies output partitioning to coalescedhashpartitioning(hashpartitioning('data, 5), CoalescedBoundary(0,5)). Because we replace Spark SortExec with CometSort before AQE interferes Spark query plan, CometSort uses SortExec's output partitioning hashpartitioning(data#3312, 5).
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Describe the bug
Currently,
CometExec
has defaultoutputPartitioning
implementation that reuses Spark original plan'soutputPartitioning
. In most cases, it is correct. But Spark AQE has special nodeAQEShuffleRead
which can possibly change output partitioning for coalescing purpose, e.g.:It causes many test failures in
org.apache.spark.sql.connector.WriteDistributionAndOrderingSuite
on #250. For example,CometSort
's output partitioning is the original plan'shashpartitioning(data#3312, 5)
of original exchange. But after AQE,AQEShuffleRead
is added and it modifies output partitioning tocoalescedhashpartitioning(hashpartitioning('data, 5), CoalescedBoundary(0,5))
. Because we replace SparkSortExec
withCometSort
before AQE interferes Spark query plan,CometSort
usesSortExec
's output partitioninghashpartitioning(data#3312, 5)
.Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: