enhance: Optimize the performance of stats task

[bigsheeper]

1. Increase the writer's batchSize to avoid multiple serialization operations.
2. Perform asynchronous upload of binlog files to prevent blocking the data processing flow.
3. Reduce multiple calls to writer.Flush().

issue: #37373

[bigsheeper]

These optimizations reduced the stats task time from 15s to 7s (1 million 128-dim end-to-end test).
Before:

[2024/11/01 19:48:11.424 +08:00] [INFO] [indexnode/task_stats.go:301] ["sort segment end"] [clusterID=by-dev] [taskID=453630131593571107] [collectionID=453630131591970875] [partitionID=453630131591970876] [segmentID=453630131592170892] [subTaskType=Sort] ["target segmentID"=453630131593571106] ["old rows"=1000000] ["valid rows"=1000000] ["binlog batch count"=8] ["download elapse"=3.001232668s] ["upload binlogs elapse"=4.072131613s] ["sort elapse"=417.540613ms] ["serWrite elapse"=671.678849ms] ["write elapse"=7.321798417s] ["total1 elapse"=15.525105081s] ["total elapse"=15.524792759s]

After:

[2024/11/01 20:06:34.003 +08:00] [INFO] [indexnode/task_stats.go:296] ["sort segment end"] [clusterID=by-dev] [taskID=453630403454556934] [collectionID=453630403453156710] [partitionID=453630403453156711] [segmentID=453630403453156728] [subTaskType=Sort] ["target segmentID"=453630403454556933] ["old rows"=1000000] ["valid rows"=1000000] ["binlog batch count"=8] ["download elapse"=2.640881298s] ["upload binlogs elapse"=0s] ["sort elapse"=355.944256ms] ["serWrite elapse"=546.764809ms] ["write elapse"=3.359702497s] ["total elapse"=7.058947854s]

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 86.04651% with 6 lines in your changes missing coverage. Please review.

Project coverage is 81.02%. Comparing base (aed3b94) to head (73db30f).
Report is 20 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/flushcommon/io/binlog_io.go	78.57%	2 Missing and 1 partial ⚠️
internal/indexnode/task_stats.go	90.00%	1 Missing and 1 partial ⚠️
...ternal/datanode/compaction/clustering_compactor.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #37374      +/-   ##
==========================================
- Coverage   81.03%   81.02%   -0.02%     
==========================================
  Files        1305     1305              
  Lines      183190   183212      +22     
==========================================
- Hits       148455   148453       -2     
- Misses      29533    29548      +15     
- Partials     5202     5211       +9     

Components	Coverage Δ
Client	∅ <ø> (∅)
Core	67.30% <ø> (-0.01%)	⬇️
Go	83.26% <86.04%> (-0.01%)	⬇️

Files with missing lines	Coverage Δ
internal/datanode/compaction/compactor_common.go	84.40% <ø> (ø)
internal/datanode/compaction/segment_writer.go	76.41% <100.00%> (ø)
...ternal/datanode/compaction/clustering_compactor.go	67.14% <50.00%> (ø)
internal/indexnode/task_stats.go	68.94% <90.00%> (-0.58%)	⬇️
internal/flushcommon/io/binlog_io.go	85.00% <78.57%> (+1.66%)	⬆️

... and 30 files with indirect coverage changes

[xiaocai2333]

/lgtm

need to fix the ut

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify]

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[bigsheeper]

/run-cpu-e2e

[bigsheeper]

rerun cpp-unit-test

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[bigsheeper]

/run-cpu-e2e

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment