[Failing Test]: beam_PostRelease_NightlySnapshot red due to various flakes in Dataflow example tests

[kennknowles]

What happened?

Signal is red for days, e.g. https://github.com/apache/beam/actions/runs/8052190987 (and the few tests before it)

Failures are different each day, but span these tests:

• :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom
• :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow
• :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Some error messages

2024-02-28T16:32:54.4498139Z > Task :runners:direct-java:runMobileGamingJavaDirect
2024-02-28T16:32:54.4500524Z .....................................................................................................................................................................................................................................DELAY(522365, 1)
2024-02-28T16:32:54.4502043Z {timestamp_ms=1709137451000}
2024-02-28T16:32:54.4504023Z late data for: user2_AliceBlueEchidna,AliceBlueEchidna,9,1709137451000,2024-02-28 08:32:54.298
2024-02-28T16:33:02.1489911Z bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__
2024-02-28T16:33:04.0488488Z 
2024-02-28T16:33:05.1491216Z Waiting on bqjob_r6fa404a3d8bc3b12_0000018df08ff71e_1 ... (0s) Current status: RUNNING
2024-02-28T16:33:05.1494635Z                                                                                       
2024-02-28T16:33:05.3491840Z Waiting on bqjob_r6fa404a3d8bc3b12_0000018df08ff71e_1 ... (0s) Current status: DONE   
2024-02-28T16:33:05.3494694Z +-----------------------------------+
2024-02-28T16:33:05.3495662Z |             table_id              |
2024-02-28T16:33:05.3496733Z +-----------------------------------+
2024-02-28T16:33:05.3497829Z | hourly_team_score_python_dataflow |
2024-02-28T16:33:05.3499013Z | hourly_team_score_python_direct   |
2024-02-28T16:33:05.3500079Z | leaderboard_DirectRunner_team     |
2024-02-28T16:33:05.3500886Z +-----------------------------------+
2024-02-28T16:33:05.4491349Z Waiting for pipeline to produce more results...
2024-02-28T16:33:05.6494425Z [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
2024-02-28T16:33:05.6500764Z [ERROR] POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DirectRunner_team/insertAll?prettyPrint=false
2024-02-28T16:33:05.6502502Z [ERROR] {
2024-02-28T16:33:05.6502898Z [ERROR]   "code" : 404,
2024-02-28T16:33:05.6503452Z [ERROR]   "errors" : [ {
2024-02-28T16:33:05.6504204Z [ERROR]     "domain" : "global",
2024-02-28T16:33:05.6505425Z [ERROR]     "message" : "Table 844138762903:beam_postrelease_mobile_gaming.leaderboard_DirectRunner_team not found.",
2024-02-28T16:33:05.6506679Z [ERROR]     "reason" : "notFound"
2024-02-28T16:33:05.6507342Z [ERROR]   } ],
2024-02-28T16:33:05.6511961Z [ERROR]   "message" : "Table 844138762903:beam_postrelease_mobile_gaming.leaderboard_DirectRunner_team not found.",
2024-02-28T16:33:05.6512832Z [ERROR]   "status" : "NOT_FOUND"
2024-02-28T16:33:05.6513286Z [ERROR] }
2024-02-28T16:33:05.6514147Z [ERROR] -> [Help 1]
2024-02-28T16:33:05.6514629Z [ERROR] 
2024-02-28T16:33:05.6515470Z [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
2024-02-28T16:33:05.6516540Z [ERROR] Re-run Maven using the -X switch to enable full debug logging.

[damondouglas]

I will go ahead and close this issue as beam_PostRelease_NightlySnapshot seems to be green now from the recent fixes.