Add MetricsContainer::getPerWorker{Counter|Histogram} (#28903) #28923
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
johnjcasey
reviewed
Oct 10, 2023
sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/NoOpCounter.java
Show resolved
Hide resolved
johnjcasey
reviewed
Oct 10, 2023
| // When enabled, the Pipeline will record Per-Worker metrics that will be piped to WMW. | ||
| StreamingStepMetricsContainer.setEnablePerWorkerMetrics( | ||
| options.isEnableStreamingEngine() | ||
| && DataflowRunner.hasExperiment(options, "enable_per_worker_metrics")); |
There was a problem hiding this comment.
Should this be an experiment, or just on by default?
There was a problem hiding this comment.
Will leave this as an experiment until I've done some load tests. Then I will enable it as default for Streaming Engine Jobs.
|
Assigning reviewers. If you would like to opt out of this review, comment R: @damccorm added as fallback since no labels match configuration Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
retest this please |
github-actions
bot
added
java
runners
dataflow
and removed
java
runners
dataflow
labels
johnjcasey
approved these changes
Oct 11, 2023
|
retest this please |
JayajP
force-pushed
the
perworker-metrics
branch
from
2e82f87 to
53d2262
Compare
Merge master to forked repo. 10/13/2023-1
JayajP
force-pushed
the
perworker-metrics
branch
from
53d2262 to
01ec123
Compare
|
I think the precommit Java GCP IO Direct failure might be related to the GHA migration, and so it may not actually be an issue. cc: @Abacn to confirm |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add getPerWorker{Counter|Histogram} methods to the MetricsContainer interface. By default, these methods will return no-op metrics that do not record any values.
For the Legacy Java runner, perWorkerMetrics are stored when the experiment
enable_per_worker_metricsis enabled. Otherwise, the Legacy runner defaults to the no-op metrics.These methods will be used to to store BigQuery related metrics that will be piped back to WMW.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.