Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[yaml] support gcs location for main.py yaml input file #29014

Merged
merged 2 commits into from
Oct 20, 2023
Merged

[yaml] support gcs location for main.py yaml input file #29014

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fe9179c
[yaml] support gcs location for main.py yaml input file
Polber Oct 16, 2023
b69b834
add with to file opening
Polber Oct 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions sdks/python/apache_beam/yaml/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import yaml

import apache_beam as beam
from apache_beam.io.filesystems import FileSystems
from apache_beam.typehints.schemas import LogicalType
from apache_beam.typehints.schemas import MillisInstant
from apache_beam.yaml import yaml_transform
Expand All @@ -43,8 +44,8 @@ def _pipeline_spec_from_args(known_args):
raise ValueError(
"Exactly one of pipeline_spec or pipeline_spec_file must be set.")
elif known_args.pipeline_spec_file:
with open(known_args.pipeline_spec_file) as fin:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still want to use the with syntax to make sure it gets closed properly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

pipeline_yaml = fin.read()
with FileSystems.open(known_args.pipeline_spec_file) as fin:
pipeline_yaml = fin.read().decode()
elif known_args.pipeline_spec:
pipeline_yaml = known_args.pipeline_spec
else:
Expand Down
Loading