Feature/add retry to gcp auth #28475

johnjcasey · 2023-09-15T14:50:00Z

Updating our GCP auth flow to add a retry if we don't receive credentails

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

# Conflicts: # CHANGES.md

Create gradle task and github actions config for GCS using this.

…tests Feature/automate performance tests

…tests fix call to gradle

…tests run on hosted runner for testing

…tests add additional checkout

…tests add destination for triggered tests

…tests move env variables to correct location

…tests try uploading against separate dataset

…tests try without a user

…tests update branch checkout, try to view the failure log

…tests run on failure

…tests update to use correct BigQuery instance

…tests convert to matrix

codecov · 2023-09-15T17:15:01Z

Codecov Report

Merging #28475 (e4a9746) into master (f09955e) will decrease coverage by 0.27%.
Report is 153 commits behind head on master.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #28475      +/-   ##
==========================================
- Coverage   72.33%   72.07%   -0.27%     
==========================================
  Files         683      679       -4     
  Lines      100709   100013     -696     
==========================================
- Hits        72848    72082     -766     
- Misses      26284    26354      +70     
  Partials     1577     1577

Flag	Coverage Δ
python	`82.54% <100.00%> (-0.30%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
sdks/python/apache_beam/internal/gcp/auth.py	`85.00% <100.00%> (+5.83%)`	⬆️

... and 39 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2023-09-15T18:06:21Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

github-actions · 2023-09-15T18:34:43Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @liferoad for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

liferoad · 2023-09-15T18:53:06Z

sdks/python/apache_beam/internal/gcp/auth.py

@@ -159,12 +159,19 @@ def _get_service_credentials(pipeline_options):
          'Credentials.')
      return credentials
    except Exception as e:
-      _LOGGER.warning(
+      _LOGGER.error(


I am not sure we need to change this to error since connecting anonymously is allowed. And can we also make the message much clearer? e.g, Unable to find default credentials to use: %s\n, Connecting anonymously. This is expected if no credentials are required to access GCP resources.

sdks/python/apache_beam/internal/gcp/auth_test.py

liferoad · 2023-09-21T20:29:48Z

LGTM. make sure the tests run fine. Thanks.

johnjcasey · 2023-09-22T19:50:23Z

Run Python_Coverage PreCommit

johnjcasey · 2023-09-22T19:50:29Z

Run Python PreCommit

kennknowles · 2023-09-25T16:59:41Z

This is marked as blocking release. It seems like a nice feature but not a release blocker. Is there more context?

johnjcasey · 2023-09-25T17:14:42Z

We had a customer issue where this not retrying caused a pipeline to fail. Ultimately, the issue was downstream, so we can defer this for 2.51

kennknowles · 2023-09-25T17:15:33Z

I suggest given the nature of this, no need to specify a milestone.

Change retry timeout to facilitate shorter retrys for anonymous access cases

johnjcasey · 2023-09-26T17:07:09Z

Run Python PreCommit

…s cases

johnjcasey · 2023-09-26T20:39:29Z

Run Python PreCommit

johnjcasey · 2023-09-27T14:44:53Z

Run Python PreCommit

johnjcasey and others added 30 commits July 5, 2023 16:05

Update 2.50 release notes to include new Kafka topicPattern feature

c22b8c1

Merge remote-tracking branch 'origin/master'

f7cf5de

# Conflicts: # CHANGES.md

Merge remote-tracking branch 'origin/master'

ab68ecb

Merge remote-tracking branch 'origin/master'

40ad1d5

Create groovy class for io performance tests

6c9c28d

Create gradle task and github actions config for GCS using this.

delete unnecessary class

520c9d1

fix env call

062de23

Merge pull request #181 from johnjcasey/feature/automate-performance-…

01fc25b

…tests Feature/automate performance tests

fix call to gradle

9c9f86b

Merge pull request #182 from johnjcasey/feature/automate-performance-…

92306fa

…tests fix call to gradle

run on hosted runner for testing

925ce55

Merge pull request #183 from johnjcasey/feature/automate-performance-…

2dcfb70

…tests run on hosted runner for testing

add additional checkout

117ef8b

Merge pull request #184 from johnjcasey/feature/automate-performance-…

1f73cda

…tests add additional checkout

add destination for triggered tests

cb6e01b

Merge pull request #185 from johnjcasey/feature/automate-performance-…

a9e86aa

…tests add destination for triggered tests

move env variables to correct location

8ea6c51

Merge pull request #186 from johnjcasey/feature/automate-performance-…

d8822d7

…tests move env variables to correct location

try uploading against separate dataset

320a4cc

Merge pull request #187 from johnjcasey/feature/automate-performance-…

e89b59e

…tests try uploading against separate dataset

try without a user

1cd4e55

Merge pull request #188 from johnjcasey/feature/automate-performance-…

4473f17

…tests try without a user

update branch checkout, try to view the failure log

4fc5b8e

Merge pull request #189 from johnjcasey/feature/automate-performance-…

706650d

…tests update branch checkout, try to view the failure log

run on failure

59069f2

Merge pull request #190 from johnjcasey/feature/automate-performance-…

7f79b62

…tests run on failure

update to use correct BigQuery instance

6f51976

Merge pull request #191 from johnjcasey/feature/automate-performance-…

e95d920

…tests update to use correct BigQuery instance

convert to matrix

df716cb

Merge pull request #192 from johnjcasey/feature/automate-performance-…

4d8eded

…tests convert to matrix

Add test case

1bacada

liferoad added this to the 2.51.0 Release milestone Sep 15, 2023

github-actions bot added the Next Action: Reviewers label Sep 15, 2023

liferoad reviewed Sep 15, 2023

View reviewed changes

sdks/python/apache_beam/internal/gcp/auth_test.py Show resolved Hide resolved

johnjcasey added 4 commits September 19, 2023 10:56

Update exception log

8eca7d0

Add failure test

59310a2

Update removal of retrying method

0da1faa

rework via mock

e7e66f2

liferoad approved these changes Sep 21, 2023

View reviewed changes

johnjcasey modified the milestones: 2.51.0 Release, 2.52.0 Release Sep 25, 2023

kennknowles removed this from the 2.52.0 Release milestone Sep 25, 2023

johnjcasey added 2 commits September 25, 2023 13:57

Clear credentials cache for idempotent tests

a54cd14

Remove handler after test

aceae65

Change retry timeout to facilitate shorter retrys for anonymous access cases

Change retry timeout to facilitate shorter retrys for anonymous acces…

9bd54a7

…s cases

reset credentials before and after test

e4a9746

johnjcasey merged commit 725a2d6 into apache:master Sep 27, 2023
77 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/add retry to gcp auth #28475

Feature/add retry to gcp auth #28475

johnjcasey commented Sep 15, 2023 •

edited

Loading

codecov bot commented Sep 15, 2023 •

edited

Loading

github-actions bot commented Sep 15, 2023

github-actions bot commented Sep 15, 2023

liferoad Sep 15, 2023

liferoad commented Sep 21, 2023

johnjcasey commented Sep 22, 2023

johnjcasey commented Sep 22, 2023

kennknowles commented Sep 25, 2023

johnjcasey commented Sep 25, 2023

kennknowles commented Sep 25, 2023

johnjcasey commented Sep 26, 2023

johnjcasey commented Sep 26, 2023

johnjcasey commented Sep 27, 2023

Feature/add retry to gcp auth #28475

Feature/add retry to gcp auth #28475

Conversation

johnjcasey commented Sep 15, 2023 • edited Loading

Updating our GCP auth flow to add a retry if we don't receive credentails

GitHub Actions Tests Status (on master branch)

codecov bot commented Sep 15, 2023 • edited Loading

Codecov Report

github-actions bot commented Sep 15, 2023

github-actions bot commented Sep 15, 2023

liferoad Sep 15, 2023

Choose a reason for hiding this comment

liferoad commented Sep 21, 2023

johnjcasey commented Sep 22, 2023

johnjcasey commented Sep 22, 2023

kennknowles commented Sep 25, 2023

johnjcasey commented Sep 25, 2023

kennknowles commented Sep 25, 2023

johnjcasey commented Sep 26, 2023

johnjcasey commented Sep 26, 2023

johnjcasey commented Sep 27, 2023

johnjcasey commented Sep 15, 2023 •

edited

Loading

codecov bot commented Sep 15, 2023 •

edited

Loading