Skip runner Dataflow v1 steps when a job will run on v2

[kennknowles]

Fixes #30469

Currently there is inconsistent rewriting and validation.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[damccorm]

LGTM if it goes green

[kennknowles]

R: @damccorm

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[damccorm]

Can we keep the files to run the postcommits?

[kennknowles]

Sure. I dropped them because the only risk of this change is catastrophically making all Dataflow v1 jobs fail, and we already have them on precommit. I can add them back.

[kennknowles]

Looks like there are some replacements and also things that happen in traversal that are explicitly expected by tests. TBD what the deal is. I expect we have an anti-pattern of doing some replacements and then producing a "portable" pipeline that is specialized to Dataflow on the client side.

[kennknowles]

Nope, it looks like there are just tests that set use_runner_v2 but then check that things happen during the v1 translation. Specifically the runner v1 translation was used to do client-side validation of what features were supported on v2. That definitely needs to be service-side so that service can evolve support without an SDK release. So actually this PR is a major upgrade in that sense, but we need to check that the service-side translation has adequate validation. Might commit a smaller version like your PR first, considering all this.

[kennknowles]

R: @scwhittle for the removed tests. They were testing things that honestly wouldn't work right, but curious where are your thoughts on this, as it pertains to the status of the service's level of care in v2 codepaths:

• level of validation for SetState and MapState support which should happen in the backend
• don't really understand what is the deal with the GroupIntoBatches stuff
• whether there is something in the service that somehow was trying to depend on the pubsub sink v1 translation even in v2?

To be clear, all these tests are testing whether the v1 translation has some behavior when the actual pipeline is running on v2.

[kennknowles]

R: @reuvenlax

[kennknowles]

Laughing to myself that the response to failing tests here was to delete the tests, but I think it is the right thing to do.

[kennknowles]

Test failures were GCE quota.

[kennknowles]

Green, but conflicts in the trigger files. I will drop that commit then merge.

[kennknowles]

Internal tests green as well.