Add TF MNIST classification cost benchmark #33391
Conversation
|
Assigning reviewers. If you would like to opt out of this review, comment R: @shunping for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| -PloadTest.mainClass=apache_beam.testing.benchmarks.inference.tensorflow_mnist_classification_cost_benchmark \ | ||
| -Prunner=DataflowRunner \ | ||
| -PpythonVersion=3.10 \ | ||
| '-PloadTest.args=${{ env.beam_Inference_Python_Benchmarks_Dataflow_test_arguments_1 }} --job_name=benchmark-tests-tf-mnist-classification-python-${{env.NOW_UTC}} --input_file=gs://apache-beam-ml/testing/inputs/it_mnist_data.csv --output_file=gs://temp-storage-for-end-to-end-tests/wordcount/result_tf_mnist-${{env.NOW_UTC}}.txt --model=gs://apache-beam-ml/models/tensorflow/mnist/' \ |
There was a problem hiding this comment.
Should we consider running multiple benchmarks in the same workflow instead of a workflow per test? The advantage would be having fewer things to monitor/maintain
There was a problem hiding this comment.
Eventually we can bundle them either by framework or put them all together in one workflow, this is largely just me building on the wordcount example benchmark by having a RunInference-specific instance (the most important distinction is the need to include a requirements file for Dataflow workers, but the pattern will largely hold for custom containers with CUDA deps too.)
If we wanted to go ahead and choose one of those routes we could go ahead and do that now + set the workflow up for cron scheduling, I'm not opposed to that.
There was a problem hiding this comment.
I'd be in favor of just doing that now - I think a single workflow will end up being easier to manage, and we can always parallelize via jobs within the workflow if needed
There was a problem hiding this comment.
Can we move https://github.com/apache/beam/blob/master/.github/workflows/beam_Wordcount_Python_Cost_Benchmark_Dataflow.yml into this file as well?
There was a problem hiding this comment.
easy enough, done
|
Build wheel failure is unrelated, merging |
* Add TF MNIST classification cost benchmark * linting * Generalize to single workflow file for cost benchmarks * fix incorrect UTC time in comment * move wordcount to same workflow * update workflow job name
* Add TF MNIST classification cost benchmark * linting * Generalize to single workflow file for cost benchmarks * fix incorrect UTC time in comment * move wordcount to same workflow * update workflow job name
Adds the Tensorflow MNIST classification example as a dataflow cost benchmark workflow as a second example for benchmark implementation. Has the added wrinkle of additional dependencies that need to be installed on workers, unlike the wordcount example.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.