You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #651, Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls lexsort_to_indices and the function fallbacks to sort_to_indices for single column case. However, sort_to_indices doesn't support all data types, e.g., struct. For unsupported data types, lexsort_to_indices should not fallback to sort_to_indices.
[info] - vectorized reader: struct *** FAILED *** (515 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 644.0 failed 1 times, most recent failure: Lost task 0.0 in stage 644.0 (TID 862) (192.168.86.44 executor driver): org.apache.comet.CometNa
tiveException: Compute error: Sort not supported for data type Struct([Field { name: "_1", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "_2", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])
I'm going to fix it at arrow-rs side: apache/arrow-rs#6226. But before we have new releases, we need to fallback to Spark for unsupported cases like struct.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Describe the bug
In #651, Comet encountered some errors on Spark test like following. It is because DataFusion SortExec calls
lexsort_to_indices
and the function fallbacks tosort_to_indices
for single column case. However,sort_to_indices
doesn't support all data types, e.g., struct. For unsupported data types,lexsort_to_indices
should not fallback tosort_to_indices
.I'm going to fix it at arrow-rs side: apache/arrow-rs#6226. But before we have new releases, we need to fallback to Spark for unsupported cases like struct.
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: