[Stuck Test]: GitHub Action issue_comment trigger not scalable

[Abacn]

What happened?

See https://github.com/apache/beam/actions?page=100

We are experiencing a huge queue of GHA tests. Current phrase trigger is not scalable - every issue comments triggers hundreds of jobs. This become especially problematic when a mass commit script is running

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 2

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

An issue is that when this happens, it's queued in GitHub side, even before the queue for self-hosted runners, so no check show in the PR, which may be confusing and merge code before the check run (which happened before)

[Abacn]

We have reduced 64 issue_comment triggers, downgrade this to P2 for a long term solution

[Abacn]

This still persists, now the action queue is more than 1 hr old.