Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement "stage_file_with_retry" for robust file staging #33436

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

shunping
Copy link
Contributor

@shunping shunping commented Dec 21, 2024

Job creation involves staging multiple files (artifacts, pipeline proto, job file, template). While artifact staging has retry logic, these other types do not, leading to potential job failures due to transient service issues.

This commit introduces stage_file_with_retry, a new function with built-in retry logic. Several stage_file calls in apiclient.py have been replaced with this new function.

Internal bug id: 384579652.

Copy link

codecov bot commented Dec 21, 2024

Codecov Report

Attention: Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 57.14%. Comparing base (4a91890) to head (e891af8).
Report is 25 commits behind head on master.

Files with missing lines Patch % Lines
...apache_beam/runners/dataflow/internal/apiclient.py 87.50% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #33436      +/-   ##
============================================
- Coverage     57.44%   57.14%   -0.30%     
  Complexity     1475     1475              
============================================
  Files           979      979              
  Lines        155237   155218      -19     
  Branches       1076     1076              
============================================
- Hits          89181    88705     -476     
- Misses        63840    64297     +457     
  Partials       2216     2216              
Flag Coverage Δ
python 80.64% <87.50%> (-0.66%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label python.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant