Implement a handful of perworkermetrics the bigquery sink (#28903).

[JayajP]

Implement the following set of metrics in BigQuery's StorageWriteAPI transforms. These metrics will be stored in the PerWorkerMetrics container defined in (#28923)

1. AppendRowsRowStatus Counter

Metric that tracks the status of BigQuery rows after making an AppendRows RPC call.

• If the AppendRows RPC succeeds, it tracks the number of rows that been successfully scheduled for a flush.
• If the AppendRows RPC fails, it tracks the number of rows that will be retried and failed.

Metric has labels the following labels:

• RowStatus: Status of the BigQuery rows after , one of SUCCESSFUL, RETRIED, FAILED
• Status: gRPC status of the AppendRows RPC
• TableId: 'datasets/{ }/tables/{ }' that the rows are sent to.

2. ThrottledTime Counter

Tracks the total time spent waiting between RPC retries.

Metric has labels the following labels:

• Method: BigQuery method that's causing the throttle. One of AppendRows, FlushRows, FinalizeStream.

3. RpcRequests Counter

Tracks the Rpc Status of various BigQuery write methods.

Metric has labels the following labels:

• Method: BigQuery sink method. One of AppendRows, FlushRows, FinalizeStream.
• Status: gRPC status of the RPC
• TableId: 'datasets/{ }/tables/{ }' that the rows are sent to.

4. RpcLatency Histogram:

Tracks the Rpc Latency of various BigQuery write methods.

Metric has labels the following labels:

• Method: BigQuery sink method. One of AppendRows, FlushRows, FinalizeStream.

Notes

• We pass a Counter to FluentBackoff.java which tracks the total time spent backing off aka ThrottleTime.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[johnjcasey]

We have this pattern all over the place, of grabbing the start time, updating a metric, grabbing error code, etc.

Is it possible to repackage this into a reusable utility method for clarity?

[JayajP]

Repacked updating the RpcRequests and RpcLatency metrics into utility methods since that are reused across all write methods. These updates are now handled by reportSuccessfulRpcMetrics and reportFailedRPCMetrics.

It's not straightforward to refactor updates AppendRowsRowStatus into a utility. When we update this metric we need to know the number of rows that were failed/retried/succeeded so it's more embedded in the code.

[johnjcasey]

This LGTM. I'm rerunning the tests before merge however.

[johnjcasey]

Retest this please

[johnjcasey]

That test failure is a flake, this is good to go!

[JayajP]

There's a single jenkins test failure but the corresponding Github actions for that test passes.