enhance: Make dynamic load/release partition follow targets #38059

congqixia · 2024-11-27T16:41:22Z

Related to #37849

mergify · 2024-11-27T17:11:28Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-27T17:43:51Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

codecov · 2024-11-27T18:01:00Z

Codecov Report

Attention: Patch coverage is 89.21569% with 11 lines in your changes missing coverage. Please review.

Project coverage is 81.08%. Comparing base (af288a9) to head (019cdfa).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/querynodev2/segments/validate.go	28.57%	3 Missing and 2 partials ⚠️
internal/querycoordv2/job/utils.go	72.72%	3 Missing ⚠️
internal/querycoordv2/meta/target_manager.go	72.72%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #38059       +/-   ##
===========================================
+ Coverage   68.85%   81.08%   +12.23%     
===========================================
  Files         289     1372     +1083     
  Lines       25508   191467   +165959     
===========================================
+ Hits        17563   155252   +137689     
- Misses       7945    30727    +22782     
- Partials        0     5488     +5488

Components	Coverage Δ
Client	`74.43% <ø> (∅)`
Core	`68.85% <ø> (ø)`
Go	`83.21% <89.21%> (∅)`

Files with missing lines	Coverage Δ
internal/querycoordv2/job/job_load.go	`87.17% <100.00%> (ø)`
internal/querycoordv2/job/job_release.go	`81.35% <100.00%> (ø)`
internal/querycoordv2/job/job_sync.go	`80.95% <100.00%> (ø)`
internal/querycoordv2/job/undo.go	`73.52% <ø> (ø)`
internal/querycoordv2/observers/target_observer.go	`90.08% <100.00%> (ø)`
internal/querycoordv2/services.go	`86.59% <100.00%> (ø)`
internal/querycoordv2/task/executor.go	`80.57% <100.00%> (ø)`
internal/querycoordv2/utils/meta.go	`71.22% <100.00%> (ø)`
internal/querynodev2/delegator/delegator.go	`87.16% <100.00%> (ø)`
internal/querynodev2/delegator/delegator_data.go	`84.82% <100.00%> (ø)`
... and 6 more

... and 1067 files with indirect coverage changes

mergify · 2024-11-28T00:39:51Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-28T01:10:10Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-28T05:01:40Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-28T05:07:36Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-28T12:27:04Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

congqixia · 2024-11-28T12:45:39Z

rerun go-sdk

mergify · 2024-11-28T12:57:29Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-28T12:57:40Z

/run-cpu-e2e

mergify · 2024-11-28T13:37:15Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-28T13:42:54Z

/run-cpu-e2e

mergify · 2024-11-28T13:57:38Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-28T14:15:11Z

/run-cpu-e2e

mergify · 2024-11-28T14:58:33Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-28T15:50:18Z

/run-cpu-e2e

mergify · 2024-11-28T17:00:16Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-28T23:44:27Z

/run-cpu-e2e

mergify · 2024-11-29T00:55:28Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-11-29T01:09:32Z

/run-cpu-e2e

mergify · 2024-11-29T02:19:06Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-12-03T07:09:12Z

internal/querycoordv2/job/job_sync.go

 	if err != nil {
 		msg := "failed to store partitions"
 		log.Warn(msg, zap.Error(err))
 		return errors.Wrap(err, msg)
 	}

-	return nil
+	// manual trigger update next target
+	ready, err := job.targetObserver.UpdateNextTarget(req.GetCollectionID())


Compared to before, awaiting the next target here, if the next target includes an accompanying segment change, the execution time here will be prolonged, potentially affecting the time taken by RootCoord to execute create_partition

xiaofan-luan · 2024-12-03T13:50:33Z

good job, this looks to be much cleaner to me.

The major draw back czs point out is ok to me

mergify · 2024-12-05T00:32:45Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-05T00:34:21Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

congqixia · 2024-12-05T00:44:07Z

/run-cpu-e2e

congqixia · 2024-12-05T00:44:17Z

rerun go-sdk

mergify · 2024-12-05T00:55:13Z

@congqixia go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-05T00:58:14Z

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Related to milvus-io#37849 Signed-off-by: Congqi Xia <[email protected]>

Signed-off-by: Congqi Xia <[email protected]>

xiaofan-luan · 2024-12-05T08:23:51Z

/lgtm
/approve

sre-ci-robot · 2024-12-05T08:24:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/querycoordv2/OWNERS~~ [congqixia,xiaofan-luan]
~~internal/querynodev2/OWNERS~~ [congqixia,xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Related pr: #38233 #38059 Signed-off-by: Congqi Xia <[email protected]>

sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. approved labels Nov 27, 2024

sre-ci-robot requested review from aoiasd and bigsheeper November 27, 2024 16:41

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Nov 27, 2024

czs007 reviewed Dec 3, 2024

View reviewed changes

congqixia force-pushed the enhance/dyn_partitions_targets branch from 829bc7f to 021a796 Compare December 5, 2024 00:22

congqixia force-pushed the enhance/dyn_partitions_targets branch from 021a796 to 7f0191e Compare December 5, 2024 01:12

congqixia added 6 commits December 5, 2024 11:23

enhance: Make dynamic load/release partition follow targets

ac16926

Related to milvus-io#37849 Signed-off-by: Congqi Xia <[email protected]>

Fix ut ctx nil and remove wait partition loaded

6b586fb

Signed-off-by: Congqi Xia <[email protected]>

Fix unit tests and move sync partition check into job

10296e9

Signed-off-by: Congqi Xia <[email protected]>

Remove describe collection before sync in case of deadlock

70e3ae8

Signed-off-by: Congqi Xia <[email protected]>

Remove commented code

87635d7

Signed-off-by: Congqi Xia <[email protected]>

Fix sync part behavior and fix unit tests

019cdfa

Signed-off-by: Congqi Xia <[email protected]>

congqixia force-pushed the enhance/dyn_partitions_targets branch from 7f0191e to 019cdfa Compare December 5, 2024 03:24

mergify bot added the ci-passed label Dec 5, 2024

sre-ci-robot assigned xiaofan-luan Dec 5, 2024

sre-ci-robot added the lgtm label Dec 5, 2024

sre-ci-robot merged commit 051bc28 into milvus-io:master Dec 5, 2024
20 checks passed

congqixia mentioned this pull request Dec 5, 2024

enhance: Resolve compilation error due to PR conflict #38252

Merged

sre-ci-robot pushed a commit that referenced this pull request Dec 5, 2024

enhance: Resolve compilation error due to PR conflict (#38252)

6ff1948

Related pr: #38233 #38059 Signed-off-by: Congqi Xia <[email protected]>

congqixia mentioned this pull request Dec 6, 2024

[Bug]: Query with count(*) reports error "failed to query: target not ready: collection not fully loaded" after release collection and partition load #36538

Closed

1 task

bigsheeper mentioned this pull request Dec 13, 2024

enhance: Accelerate the loading of collection #37741

Closed

bigsheeper mentioned this pull request Dec 27, 2024

[Bug]: Test 10000 collections and 1000 partitions #37630

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: Make dynamic load/release partition follow targets #38059

enhance: Make dynamic load/release partition follow targets #38059

congqixia commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 29, 2024

congqixia commented Nov 29, 2024

mergify bot commented Nov 29, 2024

czs007 Dec 3, 2024

xiaofan-luan commented Dec 3, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

congqixia commented Dec 5, 2024

congqixia commented Dec 5, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

xiaofan-luan commented Dec 5, 2024

sre-ci-robot commented Dec 5, 2024

enhance: Make dynamic load/release partition follow targets #38059

enhance: Make dynamic load/release partition follow targets #38059

Conversation

congqixia commented Nov 27, 2024

mergify bot commented Nov 27, 2024

mergify bot commented Nov 27, 2024

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 28, 2024

congqixia commented Nov 28, 2024

mergify bot commented Nov 29, 2024

congqixia commented Nov 29, 2024

mergify bot commented Nov 29, 2024

czs007 Dec 3, 2024

Choose a reason for hiding this comment

xiaofan-luan commented Dec 3, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

congqixia commented Dec 5, 2024

congqixia commented Dec 5, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

xiaofan-luan commented Dec 5, 2024

sre-ci-robot commented Dec 5, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading