Wait for user data propagation in deployment workflow #6978
+2,162
−1,363
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ShahabT
approved these changes
Dec 13, 2024
| case <-ctx.Done(): | ||
| return ctx.Err() | ||
| case err := <-complete: | ||
| return err |
There was a problem hiding this comment.
Should we log the error? Also, I'm going to make task for adding metrics and alerts we need them for across the board for effective monitoring of user data processing and deployment WFs.
| @@ -302,11 +311,19 @@ func (d *DeploymentWorkflowRunner) handleSyncState(ctx workflow.Context, args *d | |||
| } | |||
| } | |||
| activityCtx := workflow.WithActivityOptions(ctx, defaultActivityOptions) | |||
| err = workflow.ExecuteActivity(activityCtx, d.a.SyncUserData, syncReq).Get(ctx, nil) | |||
| var syncRes deploymentspb.SyncUserDataResponse | |||
| err = workflow.ExecuteActivity(activityCtx, d.a.SyncUserData, syncReq).Get(ctx, &syncRes) | |||
There was a problem hiding this comment.
Do you think we'd need to limit the number of TQs we pass to each Activity call here? It seems 1000 TQs at one time can become very fragile.
| if err != nil { | ||
| // TODO: if this fails, should we roll back anything? | ||
| return nil, err | ||
| } | ||
| // wait for propagation | ||
| err = workflow.ExecuteActivity(activityCtx, d.a.CheckUserDataPropagation, &deploymentspb.CheckUserDataPropagationRequest{ |
There was a problem hiding this comment.
Same for this call. In here the problem would be more sever because there is another level of fanout to all partitions.
ShahabT
reviewed
Dec 13, 2024
| if err != nil { | ||
| // TODO: if this fails, should we roll back anything? | ||
| return nil, err | ||
| } | ||
| // wait for propagation | ||
| err = workflow.ExecuteActivity(activityCtx, d.a.CheckUserDataPropagation, &deploymentspb.CheckUserDataPropagationRequest{ |
There was a problem hiding this comment.
Need to path for this change, right?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
The deployment workflow waits for user data to propagate to all task queue partitions before updating its state.
Why?
We should ensure that the desired dispatch semantics will be in effect on all task queue partitions.
How did you test it?
existing tests, will add new one if this approach is good