Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Parallelize Validator Installs #1179

Open
zsimjee opened this issue Dec 3, 2024 · 1 comment
Open

[feat] Parallelize Validator Installs #1179

zsimjee opened this issue Dec 3, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@zsimjee
Copy link
Collaborator

zsimjee commented Dec 3, 2024

Description
Validator installations currently happen serially, end to end. A few steps could be parallelized for faster installs, while keeping post-install steps sequential.

Why is this needed
Faster, better.

Implementation details
Collect validators, validate that they're real validators upfront by mapping to manifests. Then, in the installation, we can do a pip install with all the validator packages at once.

End result
Same hub install syntax for multiple validators.

@zsimjee zsimjee added the enhancement New feature or request label Dec 3, 2024
@CalebCourier
Copy link
Collaborator

We perform file writes to build the init files under guardrails/hub. We could install all the modules at once with pip, but we'll need to process post-installs sequentially or have some sort of lock on the files to ensure queing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants