Skip to content

Commit

Permalink
Merge pull request apache#33287 [YAML] Document jinja templatization …
Browse files Browse the repository at this point in the history
…features.
  • Loading branch information
robertwb authored Dec 5, 2024
2 parents e509e70 + 7df49a5 commit 1de636b
Showing 1 changed file with 43 additions and 0 deletions.
43 changes: 43 additions & 0 deletions website/www/site/content/en/documentation/sdks/yaml.md
Original file line number Diff line number Diff line change
Expand Up @@ -708,6 +708,49 @@ options:
streaming: true
```
## Jinja Templatization
It is a common to want to run a single Beam pipeline in different contexts
and/or with different configurations.
When running a YAML pipeline using `apache_beam.yaml.main` or via gcloud,
the yaml file can be parameterized with externally provided variables using
the [jinja variable syntax](https://jinja.palletsprojects.com/en/stable/templates/#variables).
The values are then passed via a `--jinja_variables` command line flag.
For example, one could start a pipeline with
```
pipeline:
transforms:
- type: ReadFromCsv
config:
path: {{input_pattern}}
```
and then run it with
```sh
python -m apache_beam.yaml.main \
--yaml_pipeline_file=pipeline.yaml \
--jinja_variables='{"input_pattern": "gs://path/to/this/runs/files*.csv"}'
```

Arbitrary [jinja control structures](https://jinja.palletsprojects.com/en/stable/templates/#list-of-control-structures),
such as looping and conditionals, can be used as well if desired as long as the
output results in a valid Beam YAML pipeline.

We also expose the [`datetime`](https://docs.python.org/3/library/datetime.html)
module as a variable by default, which can be particularly useful in reading
or writing dated sources and sinks, e.g.

```
- type: WriteToJson
config:
path: "gs://path/to/{{ datetime.datetime.now().strftime('%Y/%m/%d') }}/dated-output.json"
```

would write to files like `gs://path/to/2016/08/04/dated-output*.json`.

## Other Resources

* [Example pipeline](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml/examples)
Expand Down

0 comments on commit 1de636b

Please sign in to comment.