Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading some modules fails #15

Open
lucacasonato opened this issue Jul 16, 2020 · 2 comments
Open

Uploading some modules fails #15

lucacasonato opened this issue Jul 16, 2020 · 2 comments
Labels
bug Something isn't working

Comments

@lucacasonato
Copy link
Member

lucacasonato commented Jul 16, 2020

Currently some file uploads fail with the error with error trying to connect: Connection reset by peer (os error 104). I am not 100% sure what causes this. I assume it is because we are sending requests to S3 too quickly.

Previously we also had issues with DNS rate limiting. I was able to mostly resolve this by introducing a upload rate limit of 6 files a second.

I think both issues could be tackled by reusing the HTTP connections to S3. This is something that needs to be fixed upstream in Deno. It should also increase the overall speed of uploading, because many uploads can be multiplexed onto a single HTTP/2 connection and the client side rate limit can be removed.

@lucacasonato lucacasonato changed the title Speed up uploading Uploading some modules fails Jul 17, 2020
@lucacasonato lucacasonato added bug Something isn't working high priority labels Jul 17, 2020
@lucacasonato
Copy link
Member Author

Status update: Deno now uses a connection pool (single reqwest client), but this uncovered an issue with reqwests http/2 connection pooling implementation: seanmonstar/reqwest#976

@lucacasonato
Copy link
Member Author

I am going to remove the high priority flag, because this issue occurs very, very rarely now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant