Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: Reduce GetRecoveryInfo calls #37863

Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open

enhance: Reduce GetRecoveryInfo calls #37863

wants to merge 23 commits into from
+999 −33

Conversation

bigsheeper
Copy link
Contributor

  1. Introduce a data view mechanism for DataCoord, attempting to update each collection's data view periodically.
  2. QueryCoord maintains a cache of data view versions. Before batch-fetching recovery info, it retrieves all versions and only fetches recovery info for collections with updated versions.
  3. Return DataCoord's current data view when fetching RecoverInfo.

issue: #37743, #37630

@sre-ci-robot sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Nov 20, 2024
@bigsheeper bigsheeper changed the title enhance: Reduce GetRecoveryInfo calls from querycoord enhance: Reduce GetRecoveryInfo calls Nov 20, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Nov 20, 2024
@bigsheeper
Copy link
Contributor Author

bigsheeper commented Nov 20, 2024 •
edited
Loading

/hold
ut need to be added

@bigsheeper bigsheeper added the 10kcp PR related to 10k collection partitions Project label Nov 20, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 20, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 20, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 20, 2024

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Nov 21, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 86.36364% with 39 lines in your changes missing coverage. Please review.

Project coverage is 80.58%. Comparing base (accfacf) to head (fb6db4b).

Files with missing lines Patch % Lines
internal/datacoord/services.go 81.70% 11 Missing and 4 partials ⚠️
internal/querycoordv2/observers/target_observer.go 75.00% 8 Missing and 2 partials ⚠️
internal/datacoord/dataview/view_manager.go 93.13% 6 Missing and 1 partial ⚠️
internal/querycoordv2/meta/coordinator_broker.go 76.47% 3 Missing and 1 partial ⚠️
internal/datacoord/handler.go 0.00% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #37863      +/-   ##
==========================================
- Coverage   81.13%   80.58%   -0.55%     
==========================================
  Files        1385     1085     -300     
  Lines      196070   168659   -27411     
==========================================
- Hits       159083   135918   -23165     
+ Misses      31397    27168    -4229     
+ Partials     5590     5573      -17
Components Coverage Δ
Client 79.12% <ø> (ø)
Core ∅ <ø> (∅)
Go 80.63% <86.36%> (-2.47%) ⬇️
Files with missing lines Coverage Δ
internal/datacoord/dataview/update_chan.go 100.00% <100.00%> (ø)
internal/datacoord/server.go 68.53% <100.00%> (-4.70%) ⬇️
internal/distributed/datacoord/client/client.go 92.57% <100.00%> (-3.64%) ⬇️
internal/distributed/datacoord/service.go 86.23% <100.00%> (-3.19%) ⬇️
pkg/util/paramtable/component_param.go 98.39% <100.00%> (+<0.01%) ⬆️
pkg/util/typeutil/map.go 100.00% <100.00%> (+8.75%) ⬆️
internal/datacoord/handler.go 77.86% <0.00%> (-1.89%) ⬇️
internal/querycoordv2/meta/coordinator_broker.go 86.73% <76.47%> (-4.71%) ⬇️
internal/datacoord/dataview/view_manager.go 93.13% <93.13%> (ø)
internal/querycoordv2/observers/target_observer.go 82.38% <75.00%> (-7.71%) ⬇️
... and 1 more

... and 542 files with indirect coverage changes

@bigsheeper bigsheeper mentioned this pull request Nov 21, 2024
Merged
czs007 pushed a commit that referenced this pull request Nov 21, 2024
@bigsheeper
enhance: [10kcp] Reduce GetRecoveryInfo calls (#37891)
bf90e55 
1. Introduce a data view mechanism for DataCoord, attempting to update
each collection's data view periodically.
2. QueryCoord maintains a cache of data view versions. Before
batch-fetching recovery info, it retrieves all versions and only fetches
recovery info for collections with updated versions.
3. Return DataCoord's current data view when fetching RecoverInfo.

issue: #37743,
#37630

pr: #37863

Signed-off-by: bigsheeper <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 21, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@sre-ci-robot sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Nov 21, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 21, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 9, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@bigsheeper
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 9, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@bigsheeper
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 9, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 12, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 12, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

@bigsheeper
Copy link
Contributor Author

image
image

@bigsheeper
Copy link
Contributor Author

/run-cpu-e2e

@bigsheeper
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 13, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 13, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 13, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 17, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 17, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

@bigsheeper
Copy link
Contributor Author

/hold

@bigsheeper bigsheeper mentioned this pull request Dec 27, 2024
Open
39 tasks
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bigsheeper
To complete the pull request process, please assign yanliang567 after the PR has been reviewed.
You can assign the PR to them by writing /assign @yanliang567 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 30, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Dec 30, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 1, 2025

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@bigsheeper bigsheeper mentioned this pull request Jan 1, 2025
Merged
@mergify Mergify
Copy link
Contributor

mergify bot commented Jan 1, 2025

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@congqixia congqixia Awaiting requested review from congqixia

@cydrain cydrain Awaiting requested review from cydrain

Assignees
No one assigned
Labels
10kcp PR related to 10k collection partitions Project dco-passed DCO check passed. do-not-merge/hold kind/enhancement Issues or changes related to enhancement size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bigsheeper @sre-ci-robot