Add uses bulkdata argument to paasta spark run #3995
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes the change to paasta spark run so that https://github.yelpcorp.com/sysgit/yelpsoa-configs/pull/52010 will work as expected I'm not checking here if the /nail/bulkdata volume is specified in the spark config, e.g `spark.kubernetes.executor.volumes.hostPath.0.mount.path=/nail/bulkdata` - doing this and setting uses_bulkdata set to True would result in multiple docker volumes being set which would cause a failure. This follows on from [this conversation in slack](https://yelp.slack.com/archives/CA8BWU65D/p1729768030212919) and will allow us to complete [this project](https://yelpwiki.yelpcorp.com/display/PRODENG/Project+Incredible+Bulk)
nemacysts
reviewed
Dec 16, 2024
There was a problem hiding this comment.
lgtm (and i think it's perfectly fine to not check if there's a spark config for mounting /nail/bulkdata since it looks like that's not something anyone is currently doing - and i doubt any of our spark users would add such a mount un-prompted)
that said i'll let someone for ml compute ship since they own this file in its entirety :)
chi-yelp
approved these changes
Dec 16, 2024
SuperMatt
reviewed
Dec 17, 2024
| "--uses-bulkdata", | ||
| help="Mount /nail/bulkdata in the container", | ||
| action="store_true", | ||
| default=False, |
There was a problem hiding this comment.
Should we not set the default to true for now, then roll out my change to add the flag everywhere, and then set the default to false?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes the change to paasta spark run so that
https://github.yelpcorp.com/sysgit/yelpsoa-configs/pull/52010 will work as expected
I'm not checking here if the /nail/bulkdata volume is specified in the spark config, e.g
spark.kubernetes.executor.volumes.hostPath.0.mount.path=/nail/bulkdataThis follows on from this conversation in
slack and will allow us to complete this
project