chore: Enable Comet shuffle with AQE coalesce partitions

[viirya]

Which issue does this PR close?

Closes #387.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[viirya]

This is a copy of #651 but removed some changes. I'd like to see if "os cannot spawn new native thread" error could be removed.

And I also want to check if core-1 can pass without these changes on ubuntu-24.04.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 33.92%. Comparing base (380f03d) to head (0850404).

Files	Patch %	Lines
...org/apache/comet/CometSparkSessionExtensions.scala	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #834      +/-   ##
============================================
+ Coverage     33.83%   33.92%   +0.09%     
  Complexity      870      870              
============================================
  Files           112      112              
  Lines         42970    42909      -61     
  Branches       9466     9473       +7     
============================================
+ Hits          14538    14557      +19     
+ Misses        25446    25352      -94     
- Partials       2986     3000      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[viirya]

After changing to ubuntu 24.04, the negative ref count error like following disappears on Spark 3.4.3, although it still happens on Spark 4.0.0 and Spark 3.5.1.

Note that it only happens on spark-sql-sql/core-1, other Spark SQL tests are passed.

[info]   		Suppressed: java.lang.IllegalStateException: RefCnt has gone negative
[info]   			at org.apache.comet.shaded.arrow.util.Preconditions.checkState(Preconditions.java:465)
[info]   			at org.apache.comet.shaded.arrow.memory.BufferLedger.release(BufferLedger.java:131)
[info]   			at org.apache.comet.shaded.arrow.memory.BufferLedger.release(BufferLedger.java:105)
[info]   			at org.apache.comet.shaded.arrow.vector.BaseValueVector.releaseBuffer(BaseValueVector.java:114)
[info]   			at org.apache.comet.shaded.arrow.vector.BaseFixedWidthVector.clear(BaseFixedWidthVector.java:248)
[info]   			at org.apache.comet.shaded.arrow.vector.BaseFixedWidthVector.close(BaseFixedWidthVector.java:238)
[info]   			at org.apache.comet.vector.CometVector.close(CometVector.java:210)```

[viirya]

As it cannot be reproduced locally and on internal CI, I'm going to disable columnar shuffle in spark-sql-sql/core-1 for Spark 4.0.0 and Spark 3.5.1 temporarily. We can revisit this issue once we find a certain way to reproduce it.

[viirya]

For Spark 3.4.3, it is also probably negative ref count error happening on core-1 (e.g., in last run). So I disabled Comet shuffle for it too.

[andygrove]

LGTM. Thank you @viirya!

[viirya]

Thanks @andygrove