Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opinion: Use Asyncio instead of ThreadPoolExecutor #4

Open
maximusunc opened this issue May 20, 2024 · 3 comments
Open

Opinion: Use Asyncio instead of ThreadPoolExecutor #4

maximusunc opened this issue May 20, 2024 · 3 comments

Comments

@maximusunc
Copy link

I'm not sure a ThreadPoolExecutor is necessarily the right tool for what we want to do here. Each query doesn't really need its own thread as we aren't doing anything super CPU intensive, and we will arguably want to send more than 30 or so queries at a time. I find this answer to be very informative about the differences: https://stackoverflow.com/a/61360215

I would like to advocate for using asyncio as I think it's more suited for this task, as well as being cleaner in code (no literal parsing of the html response)

@ShervinAbd92
Copy link
Collaborator

switched to asyncio.

@maximusunc
Copy link
Author

It looks like the stress_utilities block still uses the ThreadPoolExecutor

@maximusunc
Copy link
Author

Also, I think aiohttp is not needed. httpx has plenty of exception types you can use: https://www.python-httpx.org/exceptions/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants