forked from meltano/meltano
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Add a
--merge-state flag to meltano run to merge the curren…
…t pipeline state with that of the latest run (meltano#8258) * Removed --verbose from CLI * Removed Verbose From CLI * Resolved Build Error * feat: Added a flag that tells Meltano whether to merge or overwrite state at the end of a run * Remove global flag * Update tests/meltano/core/runner/test_runner.py * [pre-commit.ci] auto fixes from pre-commit.ci hooks for more information, see https://pre-commit.ci * Update docs * Add option to `run` * Add integration test * Add test case to matrix * Fix shebang * Use --environment * Link from docs to example * Add state output to md example * Improve wording a bit * docs: Add a note explaining that setting a value with `--from-file` for objects and arrays expects valid JSON contents Also fixed some typos. --------- Co-authored-by: Rohan Arora <[email protected]> Co-authored-by: Rohan Arora <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Loading branch information
4 people
authored
Dec 5, 2023
1 parent
f2358d3
commit 8d86e65
Showing
18 changed files
with
426 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 changes: 3 additions & 0 deletions
3
integration/example-library/meltano-run-merge-states/.gitignore
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| /venv | ||
| /.meltano | ||
| .env |
27 changes: 27 additions & 0 deletions
27
integration/example-library/meltano-run-merge-states/ending-meltano.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| version: 1 | ||
| default_environment: dev | ||
| project_id: a3e6d53c-8ccc-4cac-a89c-08b70120f243 | ||
| environments: | ||
| - name: dev | ||
| - name: staging | ||
| - name: prod | ||
| send_anonymous_usage_stats: false | ||
| plugins: | ||
| extractors: | ||
| - name: tap-with-state | ||
| namespace: tap_with_state | ||
| variant: custom | ||
| executable: ./tap.py | ||
| capabilities: | ||
| - discover | ||
| - catalog | ||
| - state | ||
| settings: | ||
| - name: ts | ||
| kind: date_iso8601 | ||
| description: Dummy timestamp | ||
| select: ["*.*"] | ||
| loaders: | ||
| - name: target-jsonl | ||
| variant: andyh1203 | ||
| pip_url: target-jsonl |
89 changes: 89 additions & 0 deletions
89
integration/example-library/meltano-run-merge-states/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Get setup | ||
|
|
||
| This example shows how state from sequential invocations of `meltano run` can be merged together to create a state object that combines bookmarks from streams synced in different runs. | ||
|
|
||
| This is useful for when you want to backfill/refresh a single stream, without losing the state of other streams. | ||
|
|
||
| ```shell | ||
| meltano install | ||
| ``` | ||
|
|
||
| ## Without state merging (default) | ||
|
|
||
| ### Extract all streams | ||
|
|
||
| ```shell | ||
| TAP_WITH_STATE_TS='2023-01-01T00:00:00Z' \ | ||
| meltano run tap-with-state target-jsonl --state-id-suffix=no-merge | ||
| ``` | ||
|
|
||
| ### Extract a single stream | ||
|
|
||
| Run a 'full refresh' pipeline of a single stream. | ||
|
|
||
| ```shell | ||
| TAP_WITH_STATE_TS='2023-01-01T01:00:00Z' \ | ||
| TAP_WITH_STATE__SELECT_FILTER='["stream_1"]' \ | ||
| meltano run tap-with-state target-jsonl --full-refresh --state-id-suffix=no-merge | ||
| ``` | ||
|
|
||
| Note that the state will only contain the bookmark for `stream_1`. | ||
|
|
||
| ```shell | ||
| meltano --environment=dev state get dev:tap-with-state-to-target-jsonl:no-merge | ||
| ``` | ||
|
|
||
| ```json | ||
| { | ||
| "singer_state": { | ||
| "bookmarks": { | ||
| "stream_1": { | ||
| "created_at": "2023-01-01T01:00:00Z" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## With state merging | ||
|
|
||
| ### Extract all streams | ||
|
|
||
| ```shell | ||
| TAP_WITH_STATE_TS='2023-01-01T00:00:00Z' \ | ||
| meltano run tap-with-state target-jsonl --state-id-suffix=merge | ||
| ``` | ||
|
|
||
| ### Filter a single stream, merging states | ||
|
|
||
| Run a 'full refresh' pipeline of a single stream, but merge the current pipelines state with the latest stored state. | ||
|
|
||
| ```shell | ||
| TAP_WITH_STATE_TS='2023-01-01T01:00:00Z' \ | ||
| TAP_WITH_STATE__SELECT_FILTER='["stream_1"]' \ | ||
| meltano run tap-with-state target-jsonl --full-refresh --state-id-suffix=merge --merge-state | ||
| ``` | ||
|
|
||
| Note that the state will now contain both the new bookmark for `stream_1` and the old bookmarks for the other streams. | ||
|
|
||
| ```shell | ||
| meltano --environment=dev state get dev:tap-with-state-to-target-jsonl:merge | ||
| ``` | ||
|
|
||
| ```json | ||
| { | ||
| "singer_state": { | ||
| "bookmarks": { | ||
| "stream_1": { | ||
| "created_at": "2023-01-01T01:00:00Z" | ||
| }, | ||
| "stream_2": { | ||
| "created_at": "2023-01-01T00:00:00Z" | ||
| }, | ||
| "stream_3": { | ||
| "created_at": "2023-01-01T00:00:00Z" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` |
27 changes: 27 additions & 0 deletions
27
integration/example-library/meltano-run-merge-states/meltano.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| version: 1 | ||
| default_environment: dev | ||
| project_id: a3e6d53c-8ccc-4cac-a89c-08b70120f243 | ||
| environments: | ||
| - name: dev | ||
| - name: staging | ||
| - name: prod | ||
| send_anonymous_usage_stats: false | ||
| plugins: | ||
| extractors: | ||
| - name: tap-with-state | ||
| namespace: tap_with_state | ||
| variant: custom | ||
| executable: ./tap.py | ||
| capabilities: | ||
| - discover | ||
| - catalog | ||
| - state | ||
| settings: | ||
| - name: ts | ||
| kind: date_iso8601 | ||
| description: Dummy timestamp | ||
| select: ["*.*"] | ||
| loaders: | ||
| - name: target-jsonl | ||
| variant: andyh1203 | ||
| pip_url: target-jsonl |
34 changes: 34 additions & 0 deletions
34
...ion/example-library/meltano-run-merge-states/plugins/loaders/target-jsonl--andyh1203.lock
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| { | ||
| "plugin_type": "loaders", | ||
| "name": "target-jsonl", | ||
| "namespace": "target_jsonl", | ||
| "variant": "andyh1203", | ||
| "label": "JSON Lines (JSONL)", | ||
| "docs": "https://hub.meltano.com/loaders/target-jsonl--andyh1203", | ||
| "repo": "https://github.com/andyh1203/target-jsonl", | ||
| "pip_url": "target-jsonl", | ||
| "description": "JSONL loader", | ||
| "logo_url": "https://hub.meltano.com/assets/logos/loaders/jsonl.png", | ||
| "settings": [ | ||
| { | ||
| "name": "destination_path", | ||
| "kind": "string", | ||
| "value": "output", | ||
| "label": "Destination Path", | ||
| "description": "Sets the destination path the JSONL files are written to, relative\nto the project root.\n\nThe directory needs to exist already, it will not be created\nautomatically.\n\nTo write JSONL files to the project root, set an empty string (`\"\"`).\n" | ||
| }, | ||
| { | ||
| "name": "do_timestamp_file", | ||
| "kind": "boolean", | ||
| "value": false, | ||
| "label": "Include Timestamp in File Names", | ||
| "description": "Specifies if the files should get timestamped.\n\nBy default, the resulting file will not have a timestamp in the file name (i.e. `exchange_rate.jsonl`).\n\nIf this option gets set to `true`, the resulting file will have a timestamp associated with it (i.e. `exchange_rate-{timestamp}.jsonl`).\n" | ||
| }, | ||
| { | ||
| "name": "custom_name", | ||
| "kind": "string", | ||
| "label": "Custom File Name Override", | ||
| "description": "Specifies a custom name for the filename, instead of the stream name.\n\nThe file name will be `{custom_name}-{timestamp}.jsonl`, if `do_timestamp_file` is `true`.\nOtherwise the file name will be `{custom_name}.jsonl`.\n\nIf custom name is not provided, the stream name will be used.\n" | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.