fix: Reduce RowPartition memory allocation

[viirya]

Which issue does this PR close?

Closes #.

Rationale for this change

RowPartition allocates memory for storing row addresses and sizes. We call ArrayBuffer.clear in RowPartition.reset method to clean up the memory allocation. But ArrayBuffer.clear doesn't actually deallocate the internal array but just initiates its content.

When we insert elements into the array, it will ensure the array size by doubling the array size every time if the space is not enough. So the array grows. That's said if the arrow grows, clear cannot deallocate the array after spilling but still uses unnecessary size of array.

This patch fixes it.

Besides, as we limit columnar batch size in native writer, it doesn't make sense to have JVM row buffer larger than the size. In this patch, it uses COMET_COLUMNAR_SHUFFLE_BATCH_SIZE as the initial size of RowPartition.

What changes are included in this PR?

How are these changes tested?

[viirya]

cc @sunchao @parthchandra

[sunchao]

LGTM

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 88.88889% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 33.58%. Comparing base (d76c113) to head (63f5554).

Files	Patch %	Lines
.../comet/execution/shuffle/CometDiskBlockWriter.java	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #244      +/-   ##
============================================
- Coverage     33.58%   33.58%   -0.01%     
  Complexity      780      780              
============================================
  Files           107      107              
  Lines         37211    37212       +1     
  Branches       8160     8161       +1     
============================================
  Hits          12496    12496              
  Misses        22076    22076              
- Partials       2639     2640       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[viirya]

Merged. Thanks.