enhance: Support tracing for async watch channels

[tinswzy]

issue: #35917
Support tracing for async watch channels

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tinswzy
To complete the pull request process, please assign congqixia after the PR has been reviewed.
You can assign the PR to them by writing /assign @congqixia in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 93.17406% with 20 lines in your changes missing coverage. Please review.

Project coverage is 81.13%. Comparing base (4df444e) to head (7066afe).
Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/datacoord/channel_manager.go	87.50%	5 Missing and 2 partials ⚠️
internal/datacoord/channel_store.go	83.33%	2 Missing and 1 partial ⚠️
internal/flushcommon/pipeline/data_sync_service.go	76.92%	2 Missing and 1 partial ⚠️
internal/datacoord/channel.go	83.33%	1 Missing ⚠️
internal/datacoord/services.go	91.66%	0 Missing and 1 partial ⚠️
internal/flushcommon/broker/datacoord.go	66.66%	1 Missing ⚠️
internal/flushcommon/syncmgr/meta_writer.go	0.00%	1 Missing ⚠️
internal/querycoordv2/services.go	75.00%	1 Missing ⚠️
pkg/config/etcd_source.go	0.00%	1 Missing ⚠️
pkg/mq/msgstream/mq_msgstream.go	96.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #38444      +/-   ##
==========================================
+ Coverage   81.09%   81.13%   +0.04%     
==========================================
  Files        1383     1385       +2     
  Lines      195636   195959     +323     
==========================================
+ Hits       158642   158996     +354     
+ Misses      31404    31376      -28     
+ Partials     5590     5587       -3     

Components	Coverage Δ
Client	78.73% <ø> (+0.47%)	⬆️
Core	69.45% <ø> (+0.01%)	⬆️
Go	83.12% <93.17%> (+0.04%)	⬆️

Files with missing lines	Coverage Δ
internal/datacoord/cluster.go	97.22% <100.00%> (ø)
internal/datacoord/meta.go	89.68% <100.00%> (+0.01%)	⬆️
internal/datacoord/metrics_info.go	84.93% <100.00%> (ø)
internal/datacoord/policy.go	85.71% <100.00%> (ø)
internal/datacoord/sync_segments_scheduler.go	100.00% <100.00%> (ø)
internal/datacoord/util.go	98.68% <100.00%> (ø)
internal/datanode/channel/channel_manager.go	89.54% <100.00%> (+0.36%)	⬆️
internal/datanode/services.go	85.01% <100.00%> (ø)
...hcommon/pipeline/flow_graph_dmstream_input_node.go	94.11% <100.00%> (+5.88%)	⬆️
internal/kv/etcd/etcd_kv.go	80.68% <100.00%> (+0.38%)	⬆️
... and 36 more

... and 35 files with indirect coverage changes

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[tinswzy]

rerun go-sdk

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[tinswzy]

rerun go-sdk