Report Outstanding Bundles and Bytes

[edman124]

These metrics are needed as a part of an internal project for thread scaling.
The outstanding bundles and bytes will provide signals for how we want to autoscale the workers. The outstanding bundles tracks how many bundles of work are currently being processed (currently very similar to number of active threads). The outstanding bytes tracks how many bytes are currently being processed. These two can be compared to the respective limits to see if the pipeline needs to be scaled up or down.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[edman124]

ptal @twang126
ptal @m-trieu

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[twang126]

General nit: can you add a bit more color in the description about:

1. What is a bundle. Why is it important
2. What is this bytes referring to. Is it backlog, queued, etc?

[edman124]

Run Java PreCommit

[edman124]

ptal: @bvolpato

[edman124]

assign set of reviewers

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva added as fallback since no labels match configuration

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[Abacn]

Hi, what is the status of this PR? @edman124 @m-trieu @twang126

It was noted in #29065 (comment) that this PR will fix a flaky test of Java PreCommit that seen quite frequently. We may need this before release -- current Java PreCommit is too flaky

[kennknowles]

This flakiness is recent, right? Was there a change that caused it to become flaky? (could be fine but we might want to check)

[kennknowles]

It is OK to rollback a change that went in that does not have reliable tests, too.

[Abacn]

This flakiness is recent

Yes. I suggest if this was not made into release cut we should revert #28513

[edman124]

I think this PR is ready, @m-trieu can you double check and give a LGTM if everything looks good. I think the PR should be good to submit after approval

[m-trieu]

LGTM
reminder to rebase and squash commits

[Abacn]

thanks, as long as there is no merging conflict rebase is not needed, or is there any recent commit to master branch would affect this PR? Otherwise I will just merge this PR

[edman124]

I don't think there are any conflicts

[AnandInguva]

@Abacn or I will merge it once tests are green.