Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] include option to parse overwrite flag to spark submit #62

Open
jeppe742 opened this issue Apr 18, 2024 · 0 comments
Open

[FEATURE] include option to parse overwrite flag to spark submit #62

jeppe742 opened this issue Apr 18, 2024 · 0 comments
Assignees
Labels
code/new-feature New feature or request

Comments

@jeppe742
Copy link

jeppe742 commented Apr 18, 2024

Description

One of the default arguments to all our spark jobs is the --overwrite flag.
This is mainly used for the few places we have jobs that do incremental loading, to fully reload all historical data.
(e.g https://github.com/SneaksAndData/shrek/blob/93dc85842f55434bd82029360284148cc7980513/shrek/kits/generalized/copy_data_merge_by_key.py#L161)

However, currently it's not possible to specify this argument when submitting a beast job using the CLI, using the --overrides .
Instead you have to manually change the CRD, or create a new one

Possible solution

Allow one to specify the overwrite flag in the --overrides argument

Alternatives

No response

Context

No response

@jeppe742 jeppe742 added the code/new-feature New feature or request label Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/new-feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants