Batch operation rate limit #366
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stephanos
force-pushed
the
batch-operation-rate-limit
branch
4 times, most recently
from
09e85ca to
20ef19e
Compare
stephanos
force-pushed
the
batch-operation-rate-limit
branch
from
20ef19e to
f1c1d96
Compare
stephanos
commented
Oct 30, 2023
| FlagQueryTerminate = "Terminate Workflow Executions with given List Filter." | ||
| FlagEventIDDefinition = "The Event Id for any Event after WorkflowTaskStarted you want to reset to (exclusive). It can be WorkflowTaskCompleted, WorkflowTaskFailed or others." | ||
| FlagQueryResetBatch = "Visibility Query of Search Attributes describing the Workflow Executions to reset. See https://docs.temporal.io/docs/tctl/workflow/list#--query." |
There was a problem hiding this comment.
To align it with the other flags, I changed this into FlagQueryReset and standardized the text.
stephanos
commented
Oct 30, 2023
| @@ -83,8 +83,10 @@ func ListBatchJobs(c *cli.Context) error { | |||
| // BatchTerminate terminate a list of workflows | |||
| func BatchTerminate(c *cli.Context) error { | |||
| operator := common.GetCurrentUserFromEnv() | |||
| rps := float32(c.Float64(common.FlagRPS)) | |||
There was a problem hiding this comment.
AFAICT, the CLI library doesn't have a built-in Float32 flag, so we need to downcast here. Which is fine; the required precision we expect here will most likely never be more than 2 decimals.
stephanos
commented
Oct 30, 2023
| @@ -16,24 +16,24 @@ require ( | |||
| github.com/temporalio/tctl-kit v0.0.0-20230328153839-577f95d16fa0 | |||
| github.com/temporalio/ui-server/v2 v2.18.2 | |||
| github.com/urfave/cli/v2 v2.25.7 | |||
| go.temporal.io/api v1.24.0 | |||
| go.temporal.io/api v1.25.0 | |||
There was a problem hiding this comment.
Upgraded to new API version to pull in the max_operations_per_second field.
3 tasks
yiminc
approved these changes
Oct 30, 2023
stephanos
added a commit
to temporalio/temporal
that referenced
this pull request
Oct 30, 2023
<!-- Describe what has changed in this PR --> **What changed?** Allow rate limiting a batch operation. Fixes #4926. <!-- Tell your future self why have you made these changes --> **Why?** Batch operations are run server side and may effect millions of executions, this in turn may overload workers and disrupt normal operations. <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> **How did you test it?** I started the Server locally and initiated a batch operation from the CLI ([CLI changes be found here](temporalio/cli#366)): - [x] uses provided limit - [x] uses server limit when not provided - [x] caps it at server limit <!-- Assuming the worst case, what can be broken when deploying this change to production? --> **Potential risks** Rate limiting to be incorrect and slow down processing or disrupt cluster. <!-- Is this PR a hotfix candidate or require that a notification be sent to the broader community? (Yes/No) --> **Is hotfix candidate?** No
stephanos
added a commit
that referenced
this pull request
Nov 6, 2023
stephanos
added a commit
that referenced
this pull request
Nov 6, 2023
Reverts #366 - but leaving in the dependency updates. It was merged too early; I'll bring it back when the new Server release is available.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was changed
Added a new flag
--rpsto the 4 batch operations.Note that although
batch-resetis a batch operation, it performs its work locally instead of on the server - so the flag does not apply here.Why?
To fix temporalio/temporal#4926.
Checklist
Closes OSS-1681
How was this tested: