Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One concurrent download per host for all hosts #3557

Merged
merged 1 commit into from
Apr 6, 2022

Conversation

HebaruSan
Copy link
Member

Background

In #3054 we changed the async downloader to sequentialize downloads from GitHub to avoid triggering GitHub's download throttling. Only one GitHub download will ever be active concurrently, and this seems to have solved the problems we were having with throttling.

Problems

Causes

  • It's hard to know for sure, but I suspect that the first few downloads from SpaceDock quickly fill the pipe, and then the remaining ones either fail to start due to congestion or are deliberately throttled by the server.
  • Too many UI updates at once can overwhelm Mono. A fix for this was submitted in Fix unhandled UnixIOException in XplatUIX11 mono/mono#21136, but it shows no signs of having been looked at by the Mono team in the last 9 months, if such a team still exists. CKAN can be affected by this Mono bug if there are many concurrent downloads, because each of them has its own progress bar that we update every time another chunk of the download arrives.

Changes

Now the single concurrent download behavior is applied to all download hosts, per host. When you start a large group of downloads, we will start one from each host in the group (GitHub, SpaceDock, etc.) in parallel, and as each download finishes the next one from that host will be started, until they're all done. This should fix problems arising from too many downloads from one host.

Fixes #3314 (probably, depending on which downloads were failing for that user).
Fixes #3555 (hopefully, by reducing the number of GUI updates during downloads).

@HebaruSan HebaruSan added Bug Something is not working as intended Easy This is easy to fix GUI Issues affecting the interactive GUI Core (ckan.dll) Issues affecting the core part of CKAN Pull request Network Issues affecting internet connections of CKAN labels Apr 6, 2022
Copy link
Member

@DasSkelett DasSkelett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works like a charm, let's hope they do resolve some of the download issues we've seen.

@HebaruSan HebaruSan merged commit 04484a6 into KSP-CKAN:master Apr 6, 2022
@HebaruSan HebaruSan deleted the fix/limit-downloads branch April 6, 2022 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something is not working as intended Core (ckan.dll) Issues affecting the core part of CKAN Easy This is easy to fix GUI Issues affecting the interactive GUI Network Issues affecting internet connections of CKAN
Projects
None yet
2 participants