Resource management with cgroups and friends

[zimbatm]

A tracking issue.

Now that NixOS/nix#11412 has been merged, it might be worth deploying Nix with groups enabled in the next release.

Arian also built this prom exporter that is nicely complementary and could be used to collect metrics and better understand resource utilisation: https://github.com/arianvp/cgroup-exporter

[Mic92]

Part of nix-experimental.

[zowoq]

Maybe I should just apply the cgroups patches to our current nix version so we can try to make some progress on this?

#1466

[zimbatm]

@Mic92 do you know if a new release will be cut soon?

[Mic92]

We are behind schedule actually for a release. I will ask if we can do it next week.

[zowoq]

Remote builds support passing max-jobs, how feasible would it be to add cores so we could set only for hydra jobs? Could be useful as a complement to cgroups and would also work for darwin jobs. If upstream wasn't very keen on it we could even carry the patch (if it isn't too complicated)?