Speedup CI and reduce test run time

[advancedxy]

What is the problem the feature request solves?

It looks like that CIs are quite slow, the most time consuming parts varies from 60m to 100+m.

It would be great that we could speed them up and hopefully keep them under ~30m.

Describe the potential solution

No response

Additional context

No response

[sunchao]

We can probably split the shuffle related tests into separate CI jobs

[sunchao]

... or reduce the amount of data in shuffle, right now it is 10000 in many tests.

[advancedxy]

right now it is 10000 in many tests.

10000 means number of records? That doesn't seem too much. Probably we can print java/scala test durations in the CI and then we can decided which parts are most time consuming. We can optimize these tests or split them then.

[advancedxy]

Probably we can print java/scala test durations in the CI and then we can decided which parts are most time consuming

After enable test duration, the most time consuming tests(exceeds one minute) on X86 mac runners with JDK 1.8 are:

- Comet columnar shuffle shuffle: different data type (8 minutes, 5 seconds)
- all types, with nulls (23 minutes, 55 seconds)
- Comet columnar shuffle shuffle: different data type (8 minutes, 18 seconds)
- Comet columnar shuffle shuffle: different data type (5 minutes, 59 seconds)
- Comet columnar shuffle shuffle: different data type (6 minutes, 50 seconds)
- fix: Too many task completion listener of ArrowReaderIterator causes OOM (34 minutes, 33 seconds)
- Comet columnar shuffle shuffle: different data type (7 minutes, 22 seconds)
- Comet columnar shuffle shuffle: different data type (6 minutes, 59 seconds)

The source is https://github.com/apache/arrow-datafusion-comet/actions/runs/8047518357/job/21976917198?pr=116

The fix: Too many task completion listener of ArrowReaderIterator causes OOM (34 minutes, 33 seconds) case looks extreme. Maybe we should fix it first, cc @viirya

[sunchao]

Thanks! I didn't realize "fix: Too many task completion listener of ArrowReaderIterator causes OOM" is taking so much time. I'm trying to improve the test time in #109.

The "all types, with nulls" are from the aggregation suite. We should reduce it too.

[sunchao]

@advancedxy @viirya what do you think if we move all the MacOS CI pipelines to post-commit, and only keep the ubuntu-latest pipelines as pre-commit? It seems the former are pretty slow especially with Java 8, while the latter are usually faster (~40min).

I think having ubuntu-latest should be sufficient to catch any potential issue in the PRs.

[viirya]

Okay for me. If we find any thing wrong with Mac OS, we can take action after commit to revert it or fix it.

[advancedxy]

what do you think if we move all the MacOS CI pipelines to post-commit,

The Silicon ones are pretty fast, faster than ubuntu-latest. So I think we should enable that on CI as per-commit. I believe a large percent of developers are on Mac, it would be nice to find problems earlier.

For X86 with Java8, it's pretty slow. I think we should at least move it to a post-commit manner, or even remove it totally.
I'm neutral for other JDKs on X86 MacOSs.

[sunchao]

OK, let's keep ubuntu-latest and MacOS Apple Silicon in the pre-commit then, and move MacOS X86 to post-commit. I'm also making changes in #109 to improve the test time in shuffle & aggregation. Will ping you once it is ready.