[Java BQ] Storage API streaming load test

[ahmedabu98]

Adding a streaming load test for writing via Storage API sink. Includes exactly-one and at-least-once semantics.

This test is set up to first write rows using batch FILE_LOADS mode to a "source of truth" table. Afterwards, it will write the same rows in streaming mode with Storage API to a second table. Then it will query between these two tables to check that they are identical. There is also the option of providing an existing table with the expected data, in which case the test will skip the first step.

The throughput, length of test (in minutes), and data shape can be changed by adding a new configuration line.

Also including a small addition: we can set an interval for the sink to intentionally crash every now and then. This is intended to test retry resilience. The sink will sometimes throw an exception to simulate a work item failure, and other times will exit the system to simulate a worker failure. Either way, we expect the pipeline to pick up where it left off and deliver data appropriately.

[ahmedabu98]

R: @johnjcasey

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[johnjcasey]

We should have two different test configurations. One should publish performance metrics to the table, and should be the "healthy" scenario with no deliberate crashes. The other should not do this publication, and should include the intermittent failures.

[ahmedabu98]

Got it, I'll remove the crashSink option from TestProperties so that it's not exposed to the performance testing framework.
I'll still include the crashing logic. We can later create a test that makes use of this same class by just passing `crashStorageApiSinkEverySeconds=" to pipeline options. If this option is set, the test will run the pipelines normally without publishing any metrics.

[ahmedabu98]

R: @johnjcasey
R: @reuvenlax

PTAL

[ahmedabu98]

Run PostCommit_Java_Dataflow

[ahmedabu98]

Run PostCommit_Java_DataflowV2