nix-index does not support fetching cache.nixos.org paths over HTTPS

[delroth]

We tried yesterday to configure cache.nixos.org to serve only redirects to https:// on its http:// endpoint (NixOS/infra#415). This ended up breaking channel scripts due to nix-index being unable to:

1. Follow redirects (Follow redirects when generating index #100);
2. Handle TLS connections.

Both of these need to be fixed as they impede improvements on the cache. @K900 made an attempt at fixing this with #247

[polarathene]

I have run the basic example from the README nix run github:nix-community/nix-index#nix-index which says it should only take 5 mins but it's been over 20 and still going. While the memory intensive portion has since finished (unclear how much RAM you're meant to have as mine maxed out and went into swap), there is a constant stream of errors like this:

# ...
+ generating index: 00000 paths found :: 00197 paths not in binary cache :: 132905 paths in queue
Error: fetching the references of store path '/nix/store/l19jh4jka0caqhszyhfd124gizpbclwc-coc-pairs-1.4.2' failed
Caused by: error trying to connect: invalid URL, scheme is not http
Caused by: invalid URL, scheme is not http
+ generating index: 00000 paths found :: 00198 paths not in binary cache :: 132904 paths in queue
Error: fetching the references of store path '/nix/store/6g3g7yyrhwvj4qpghnb0zbik6j3c86mn-coc-git-2.7.0' failed
Caused by: error trying to connect: invalid URL, scheme is not http
# ...

Is this related to this issue regarding https://? or is it a separate one I should file?

Given the count of 00198 paths not in binary cache :: 132904 paths in queue, I assume that is progress of 198 with 132904 more to go?... Some of these take only 1-4 seconds, while a few stall close to a minute 😕 I'll give up on trying to use this as this doesn't seem like it'll complete that iteration in a timely manner, and the errors don't give much hope that it'll even work once it's done.

[leon-thomm]

same here. I'm behind a proxy and nix-index just gives me these errors

[K900]

This is fixed in master now.