userns for nixbld?

[terinjokes]

Describe the bug
In a multi-user installation of nix, the builders do not have uid and gid mappings to use user namespaces. Adding them manually does not work, as /etc/subuid and /etc/subgid are not in the sandbox.

To Reproduce

# default.nix
{ pkgs ? import <nixpkgs> { }, ... }:

let
  imageAmd64 = pkgs.dockerTools.buildImage {
    name = "hello";
    config.Cmd = [ "${pkgs.hello}/bin/hello" ];
  };
  imageArm64 = pkgs.pkgsCross.aarch64-multiplatform.dockerTools.buildImage {
    name = "hello";
    config.Cmd = [ "${pkgs.hello}/bin/hello" ];
  };
in pkgs.runCommand "oci-dir-hello" { nativeBuildInputs = [ pkgs.buildah ]; } ''
  buildah manifest create --all hello-list "docker-archive:${imageAmd64}" "docker-archive:${imageArm64}"
  buildah manifest push --all --format v2s2 --purge hello-list "oci:$out:latest"
''

$ nix-build default.nix
these derivations will be built:
  /nix/store/d764hn715ri08xxyvja6zc2bm528lbyi-oci-dir-hello.drv
building '/nix/store/d764hn715ri08xxyvja6zc2bm528lbyi-oci-dir-hello.drv'...
WARN error reading allowed ID mappings: error reading subuid mappings for user "nixbld" and subgid mappings for group "nixbld": open /etc/subuid: no such file or directory
WARN Found no UID ranges set aside for user "nixbld" in /etc/subuid.
WARN Found no GID ranges set aside for user "nixbld" in /etc/subgid.
Error: error writing "allow" to setgroups: write /proc/9/setgroups: operation not permitted
ERRO error opening "allow" to /proc/9/setgroups: write /proc/9/setgroups: operation not permitted
ERRO (unable to determine exit status)
builder for '/nix/store/d764hn715ri08xxyvja6zc2bm528lbyi-oci-dir-hello.drv' failed with exit code 1
error: build of '/nix/store/d764hn715ri08xxyvja6zc2bm528lbyi-oci-dir-hello.drv' failed

Expected behavior
If it's decided builders should have access to user namespaces, we should allow user namespaces and this should build a multi-architecture OCI list in $out.

Additional context
I'm honestly not sure this is something we want to do, but I didn't see an issue for this functionality.

I'll continue to investigate ways of building multi-architecture images in Nix. Ideally this would be supported by skopeo, but see containers/skopeo#1136. At worse, I'll create a builder that manually builds the structure and list manifest. /cc @roberth in case you have an idea for this aspect.

Notify maintainers

Metadata

• system: "x86_64-linux"
• host os: Linux 5.4.81, NixOS, 20.09.2205.65c9cc79f1d (Nightingale)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.9
• channels(root): "nixos-20.09.2205.65c9cc79f1d, nixos-hardware"
• channels(terin): "home-manager-20.09"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[roberth]

If I understand correctly, your goal is to create multi-arch docker images and one way to accomplish this is by having user namespace support in Nix. I think @Ericson2314 did some work during NixCon Hackday 2020 on a PR that might make this easier. I'm a little concerned that it reduces the portability of Nix to restrictive Linux environments like hardened installations or containers. We can probably address that with requiredSystemFeatures. If you like, you could create an issue on the NixOS/nix repo.

Regarding your goal of creating multi-arch images, did you try uploading the skopeo-generated image through other means? It seems like skopeo may be suitable for combining the images, but just not for uploading it, at this time. It's sort of the opposite of what you tried in your comment here.

Alternatively, it may be possible to improve buildah, so that it does only attempts to set up user namespaces when it needs them. That's of course assuming it doesn't need it in this case.

[terinjokes]

As mentioned in this ticket, even I'm not convinced this is something Nix should do. I agree, if it does happen, it should be a separate feature for now.

Regarding your goal of creating multi-arch images, did you try uploading the skopeo-generated image through other means? It seems like skopeo may be suitable for combining the images, but just not for uploading it, at this time.

The directory created by the skopeo copies isn't valid, or at least it doesn't look like any multi-image manifest list I've seen thus far. I also haven't found any tool that could push it.

I have found that skopeo will copy a multi-image OCI path created by buildah when given an otherwise ignored tag (the directory created by the push command here: containers/buildah#2858 (comment)).

This OCI directory is very similar to the output created by the double skopeo copy. I think it's close enough that a little shell scripting and turn one into the other. I'll give this a try after work, if successful I won't need buildah at all.

I'm going to go ahead and close this in the meantime.