Upload PerWorkerMetrics every 30 second instead of every 10 seconds #30795
Conversation
JayajP
force-pushed
the
uploadfrequency-v3
branch
from
bcc3af3 to
0dc2c4b
Compare
|
R: @m-trieu |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
| // perWorkerMetricsUpdateFrequency RPC call. If 0, PerWorkerMetrics are not reported. | ||
| long perWorkerMetricsUpdateFrequency = 0L; | ||
| // Used to track the number of WorkerMessages that have been sent without PerWorkerMetrics. | ||
| long workerMessagesIndex = 0L; |
There was a problem hiding this comment.
this will not be threadsafe, how about using an AtomicLong
| @@ -81,6 +83,12 @@ public final class StreamingWorkerStatusReporter { | |||
| private final ScheduledExecutorService globalWorkerUpdateReporter; | |||
| private final ScheduledExecutorService workerMessageReporter; | |||
|
|
|||
| // PerWorkerMetrics are sent on the WorkerMessages channel, and are sent one in every | |||
| // perWorkerMetricsUpdateFrequency RPC call. If 0, PerWorkerMetrics are not reported. | |||
| long perWorkerMetricsUpdateFrequency = 0L; | |||
There was a problem hiding this comment.
if this never changes lets make it private and final and initialize in the constructor
you can modify the constructor to take in this instead of passing it into start() and have start() take no params.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #30795 +/- ##
============================================
+ Coverage 70.95% 71.47% +0.51%
============================================
Files 1257 710 -547
Lines 140917 104815 -36102
Branches 4305 0 -4305
============================================
- Hits 99989 74915 -25074
+ Misses 37451 28268 -9183
+ Partials 3477 1632 -1845
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
Run Java PreCommit |
Currently PerWorkerMetrics are reported on the WorkerMessages RPC call. By default this RPC is sent every 10 seconds. After this PR we will report PerWorkerMetrics every 30 seconds by extracting/sending these metrics on every 3rd WorkerMessagesRPC.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.