Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Warn users about misspelled pipeline options #28402

Open
1 of 15 tasks
tvalentyn opened this issue Sep 11, 2023 · 9 comments
Open
1 of 15 tasks

[Feature Request]: Warn users about misspelled pipeline options #28402

tvalentyn opened this issue Sep 11, 2023 · 9 comments

Comments

@tvalentyn
Copy link
Contributor

tvalentyn commented Sep 11, 2023

What would you like to happen?

I believe after 3dcdcf3 we are no longer getting warnings like "Discarding unparseable args".

It is very easy to mistype an option. For example, one can supply --extra-package instead of --extra_package, which I did. In 2.40.0 I would have gotten
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--extra-package=package.whl'] in console logs, but we are not getting such logs anymore.

We should warn users about incorrect options, or even better suggest corrections: did you mean X ?

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@tvalentyn
Copy link
Contributor Author

cc: @riteshghorse @liferoad

@github-actions github-actions bot added the P2 label Sep 11, 2023
@riteshghorse riteshghorse self-assigned this Nov 17, 2023
@riteshghorse
Copy link
Contributor

riteshghorse commented Nov 17, 2023

Validated that the warning is shown while running on Direct runner but not for Dataflow runner on master

@riteshghorse
Copy link
Contributor

I think this is the actual change which prevented the warnings https://github.com/apache/beam/pull/22132/files

@riteshghorse
Copy link
Contributor

riteshghorse commented Nov 17, 2023

On dataflow runner it is not shown because we are retaining the pipeline options. So any unknown flag is getting added as a valid argument and parsed again. This seems to be working as intended. But as a caution, we can add a log stating that unknown arg found.

WDYT @tvalentyn ?

@Abacn
Copy link
Contributor

Abacn commented Nov 17, 2023

I can still see "Discarding unparseable args" in recent dataflow pipeline, e.g. this load test: "2023-11-17_07_58_43-10220172113544842485", run in https://github.com/apache/beam/actions/runs/6906173679/job/18790592122

@tvalentyn
Copy link
Contributor Author

There is probably some off-the-shelf library or algorithm we can reuse to detect misspellings specifically.

@riteshghorse
Copy link
Contributor

Sounds good.. I'll take a look on this more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants