enhance: Reduce GetRecoveryInfo calls #37863

bigsheeper · 2024-11-20T14:20:45Z

Introduce a data view mechanism for DataCoord, attempting to update each collection's data view periodically.
QueryCoord maintains a cache of data view versions. Before batch-fetching recovery info, it retrieves all versions and only fetches recovery info for collections with updated versions.
Return DataCoord's current data view when fetching RecoverInfo.

sre-ci-robot · 2024-11-20T14:20:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bigsheeper
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bigsheeper · 2024-11-20T14:22:41Z

/hold
ut need to be added

mergify · 2024-11-20T14:45:44Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-20T14:45:45Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-20T14:45:45Z

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Signed-off-by: bigsheeper <[email protected]>

codecov · 2024-11-21T05:49:46Z

Codecov Report

Attention: Patch coverage is 90.55944% with 27 lines in your changes missing coverage. Please review.

Project coverage is 82.78%. Comparing base (2afe2ea) to head (0147eb6).
Report is 81 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/datacoord/services.go	81.70%	11 Missing and 4 partials ⚠️
internal/datacoord/dataview/view_manager.go	93.13%	6 Missing and 1 partial ⚠️
internal/datacoord/handler.go	0.00%	3 Missing ⚠️
internal/querycoordv2/observers/target_observer.go	95.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #37863      +/-   ##
==========================================
+ Coverage   80.91%   82.78%   +1.87%     
==========================================
  Files        1375     1085     -290     
  Lines      193645   167770   -25875     
==========================================
- Hits       156692   138892   -17800     
+ Misses      31418    23349    -8069     
+ Partials     5535     5529       -6

Components	Coverage Δ
Client	`74.58% <ø> (ø)`
Core	`∅ <ø> (∅)`
Go	`83.02% <90.55%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
internal/datacoord/dataview/update_chan.go	`100.00% <100.00%> (ø)`
internal/datacoord/server.go	`73.34% <100.00%> (+0.11%)`	⬆️
internal/distributed/datacoord/client/client.go	`96.28% <100.00%> (+0.07%)`	⬆️
internal/distributed/datacoord/service.go	`89.49% <100.00%> (-1.02%)`	⬇️
internal/querycoordv2/meta/coordinator_broker.go	`93.85% <100.00%> (+2.41%)`	⬆️
pkg/util/paramtable/component_param.go	`98.38% <100.00%> (+<0.01%)`	⬆️
pkg/util/typeutil/map.go	`100.00% <100.00%> (+8.75%)`	⬆️
internal/querycoordv2/observers/target_observer.go	`87.30% <95.00%> (+1.18%)`	⬆️
internal/datacoord/handler.go	`74.62% <0.00%> (-0.69%)`	⬇️
internal/datacoord/dataview/view_manager.go	`93.13% <93.13%> (ø)`
... and 1 more

... and 321 files with indirect coverage changes

1. Introduce a data view mechanism for DataCoord, attempting to update each collection's data view periodically. 2. QueryCoord maintains a cache of data view versions. Before batch-fetching recovery info, it retrieves all versions and only fetches recovery info for collections with updated versions. 3. Return DataCoord's current data view when fetching RecoverInfo. issue: #37743, #37630 pr: #37863 Signed-off-by: bigsheeper <[email protected]>

…1-dc-version

mergify · 2024-11-21T08:45:42Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: bigsheeper <[email protected]>

…1-dc-version

Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-11-21T13:25:28Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: bigsheeper <[email protected]>

…sion

Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-11-29T10:32:37Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…1-dc-version

mergify · 2024-12-03T10:09:59Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…1-dc-version

Signed-off-by: bigsheeper <[email protected]>

bigsheeper · 2024-12-09T02:53:58Z

/unhold

…1-dc-version

mergify · 2024-12-09T03:38:10Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

bigsheeper · 2024-12-09T03:40:00Z

/run-cpu-e2e

mergify · 2024-12-09T04:00:20Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

bigsheeper · 2024-12-09T06:07:52Z

/run-cpu-e2e

mergify · 2024-12-09T06:25:50Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…1-dc-version

mergify · 2024-12-12T08:50:14Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-12T09:03:58Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2024-12-12T12:35:22Z

bigsheeper · 2024-12-13T12:59:58Z

/run-cpu-e2e

bigsheeper · 2024-12-13T13:00:04Z

rerun go-sdk

mergify · 2024-12-13T13:19:54Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…1-dc-version

mergify · 2024-12-13T13:47:24Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-13T14:01:30Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

…1-dc-version

mergify · 2024-12-17T03:36:05Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-17T03:39:36Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2024-12-27T03:40:03Z

/hold

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Nov 20, 2024

sre-ci-robot requested review from congqixia and cydrain November 20, 2024 14:21

bigsheeper changed the title ~~enhance: Reduce GetRecoveryInfo calls from querycoord~~ enhance: Reduce GetRecoveryInfo calls Nov 20, 2024

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Nov 20, 2024

sre-ci-robot added the do-not-merge/hold label Nov 20, 2024

bigsheeper added the 10kcp PR related to 10k collection partitions Project label Nov 20, 2024

bigsheeper force-pushed the 2411-dc-version branch from 27557d2 to bba3dae Compare November 21, 2024 03:57

enhance: Reduce GetRecoveryInfo calls from querycoord

1960dec

Signed-off-by: bigsheeper <[email protected]>

bigsheeper force-pushed the 2411-dc-version branch from bba3dae to 1960dec Compare November 21, 2024 04:00

bigsheeper mentioned this pull request Nov 21, 2024

enhance: [10kcp] Reduce GetRecoveryInfo calls #37891

Merged

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

e18a8fc

…1-dc-version

bigsheeper added 3 commits November 21, 2024 20:26

add more test

da09929

Signed-off-by: bigsheeper <[email protected]>

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

ec50719

…1-dc-version

add more test

0d63554

Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Nov 21, 2024

bigsheeper added 3 commits November 26, 2024 15:07

fix conflicts

5689fc7

Signed-off-by: bigsheeper <[email protected]>

Merge branch 'master' of github.com:milvus-io/milvus into 2411-dc-ver…

185ccf5

…sion

add info

d0d096b

Signed-off-by: bigsheeper <[email protected]>

fix conflicts

ce6466c

Signed-off-by: bigsheeper <[email protected]>

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

7ba3dda

…1-dc-version

bigsheeper added 2 commits December 4, 2024 19:41

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

c194485

…1-dc-version

fix ut

c010808

Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot removed the do-not-merge/hold label Dec 9, 2024

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

e8dc8ee

…1-dc-version

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

f226d77

…1-dc-version

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

4770d1b

…1-dc-version

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

0147eb6

…1-dc-version

sre-ci-robot added the do-not-merge/hold label Dec 27, 2024

bigsheeper mentioned this pull request Dec 27, 2024

[Bug]: Test 10000 collections and 1000 partitions #37630

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: Reduce GetRecoveryInfo calls #37863

enhance: Reduce GetRecoveryInfo calls #37863

bigsheeper commented Nov 20, 2024

sre-ci-robot commented Nov 20, 2024

bigsheeper commented Nov 20, 2024 •

edited

Loading

mergify bot commented Nov 20, 2024

mergify bot commented Nov 20, 2024

mergify bot commented Nov 20, 2024

codecov bot commented Nov 21, 2024 •

edited

Loading

mergify bot commented Nov 21, 2024

mergify bot commented Nov 21, 2024

mergify bot commented Nov 29, 2024

mergify bot commented Dec 3, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

mergify bot commented Dec 12, 2024

mergify bot commented Dec 12, 2024

bigsheeper commented Dec 12, 2024

bigsheeper commented Dec 13, 2024

bigsheeper commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 17, 2024

mergify bot commented Dec 17, 2024

bigsheeper commented Dec 27, 2024

enhance: Reduce GetRecoveryInfo calls #37863

Are you sure you want to change the base?

enhance: Reduce GetRecoveryInfo calls #37863

Conversation

bigsheeper commented Nov 20, 2024

sre-ci-robot commented Nov 20, 2024

bigsheeper commented Nov 20, 2024 • edited Loading

mergify bot commented Nov 20, 2024

mergify bot commented Nov 20, 2024

mergify bot commented Nov 20, 2024

codecov bot commented Nov 21, 2024 • edited Loading

Codecov Report

mergify bot commented Nov 21, 2024

mergify bot commented Nov 21, 2024

mergify bot commented Nov 29, 2024

mergify bot commented Dec 3, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

bigsheeper commented Dec 9, 2024

mergify bot commented Dec 9, 2024

mergify bot commented Dec 12, 2024

mergify bot commented Dec 12, 2024

bigsheeper commented Dec 12, 2024

bigsheeper commented Dec 13, 2024

bigsheeper commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 17, 2024

mergify bot commented Dec 17, 2024

bigsheeper commented Dec 27, 2024

bigsheeper commented Nov 20, 2024 •

edited

Loading

codecov bot commented Nov 21, 2024 •

edited

Loading