"su: cannot set groups: Operation not permitted" when switching users inside a bare runc container

[codethief]

Hi there!

I'm dabbling with runc in order to understand rootless container internals a bit better and, in particular, how to nest them. However, even before nesting them I'm struggling to switch to a non-root user inside a container I started with runc.

For the record, rootless Podman works fine on my machine and even nesting rootless Podman containers works (podman run --user podman quay.io/podman/stable podman run --user podman quay.io/podman/stable whoami etc.), which leads to me to believe I either messed up my OCI bundle's config.json, its rootfs (its file permissions) or both. But let me start from the beginning:

I use Podman to create a rootfs from the following Dockerfile:

FROM debian:bookworm-slim

RUN useradd --user-group --create-home --shell /bin/bash user
SHELL ["/bin/bash", "-c"]

RUN apt-get update && apt-get install -y podman buildah

# The following was meant for allowing me to nest containers later:
RUN echo "root:100000:9999" > /etc/subuid
RUN echo "root:100000:9999" > /etc/subgid
RUN echo "user:200000:9999" >> /etc/subuid
RUN echo "user:200000:9999" >> /etc/subgid

USER user
WORKDIR /home/user

To create the rootfs I run:

cd $(mktemp -d)
podman build -t mytestimage /path/to/the/folder/containing/the/above/Dockerfile
podman export $(podman create mytestimage) | tar -C "ro-rootfs" -xpvf -

I also create the following auxiliary folders in the same working dir:

mkdir dummy-rootfs
mkdir upper
mkdir overlayfs-workdir

Then inside that working directory I create a config.json with the contents below. It is a modified default config (runc spec --rootless), where I modified the rootfs (made it an overlayfs), and adjusted the uidMappings/gidMappings, the working directory, default command/shell, and additionally mounted the ro-rootfs from above as "external volume" under /home/user/ro-rootfs (again, the idea was to nest containers later):

{
  "ociVersion": "1.0.2-dev",
  "process": {
    "terminal": true,
    "user": {
      "uid": 0,
      "gid": 0
    },
    "args": [
      "/bin/bash"
    ],
    "env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "TERM=xterm"
    ],
    "cwd": "/home/user",
    "capabilities": {
      "bounding": [
        "CAP_AUDIT_WRITE",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE"
      ],
      "effective": [
        "CAP_AUDIT_WRITE",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE"
      ],
      "permitted": [
        "CAP_AUDIT_WRITE",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE"
      ],
      "ambient": [
        "CAP_AUDIT_WRITE",
        "CAP_KILL",
        "CAP_NET_BIND_SERVICE"
      ]
    },
    "rlimits": [
      {
        "type": "RLIMIT_NOFILE",
        "hard": 1024,
        "soft": 1024
      }
    ],
    "noNewPrivileges": true
  },
  "root": {
    "path": "/tmp/tmp.YY6VLXFSYA/dummy-rootfs",
    "readonly": false
  },
  "hostname": "runc",
  "mounts": [
    {
      "destination": "/",
      "type": "overlay",
      "source": "overlay",
      "options": [
        "rw",
        "upperdir=/tmp/tmp.YY6VLXFSYA/upper",
        "lowerdir=/tmp/tmp.YY6VLXFSYA/ro-rootfs",
        "workdir=/tmp/tmp.YY6VLXFSYA/overlayfs-workdir"
      ]
    },
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc"
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "tmpfs",
      "source": "shm",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "mode=1777",
        "size=65536k"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys",
      "type": "none",
      "source": "/sys",
      "options": [
        "rbind",
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "source": "cgroup",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "ro"
      ]
    },
    {
      "destination": "/home/user/ro-rootfs",
      "type": "bind",
      "source": "/tmp/tmp.YY6VLXFSYA/ro-rootfs",
      "options": [
        "rbind",
        "ro"
      ]
    }
  ],
  "linux": {
    "uidMappings": [
      {
        "containerID": 0,
        "hostID": 1000,
        "size": 1
      },
      {
        "containerID": 1,
        "hostID": 100000,
        "size": 8192
      }
    ],
    "gidMappings": [
      {
        "containerID": 0,
        "hostID": 1000,
        "size": 1
      },
      {
        "containerID": 1,
        "hostID": 100000,
        "size": 8192
      }
    ],
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "cgroup"
      },
      {
        "type": "user"
      },
      {
        "type": "network"
      }
    ],
    "maskedPaths": [
      "/proc/acpi",
      "/proc/asound",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/sys/firmware",
      "/proc/scsi"
    ],
    "readonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  }
}

Then I start my container using runc run mytestcontainer. So far, so good. Unfortunately, when I run su user inside my container I get:

root@runc:/home/user# su user
su: cannot set groups: Operation not permitted

Similarly, when trying apt update:

root@runc:/home/user# apt update
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 42 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 42 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (1: Operation not permitted)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)

Meanwhile, setting process.user.(uid|gid) in config.json to UID/GID 1000 directly leads to runc run mytestcontainer failing right from the start.

What gives?

A couple things I double-checked:

• The su binary in my container has the setuid bit set:

root@runc:/home/user# ls -l /usr/bin/su
-rwsr-xr-x 1 root root 72000 Mar 23  2023 /usr/bin/su

• /etc/passwd, /etc/group, /etc/shadow inside the container look as expected:

root@runc:/home/user# cat /etc/passwd
[…]
user:x:1000:1000::/home/user:/bin/bash
[…]

root@runc:/home/user# cat /etc/group
[…]
user:x:1000:
[…]

root@runc:/home/user# cat /etc/shadow
[…]
user:!:19798:0:99999:7:::
[…]

• On my host machine, I set up subuids and subgids as follows:

codethief@host:/# cat /etc/subuids
codethief:100000:65536
codethief@host:/# cat /etc/subgid
codethief:100000:65536

• My host machine's /tmp (under which I create the rootfs/bundle) is not mounted as tmpfs (i.e. not mounted with noexec or nosuid, either). df /tmp outputs my root partition.

I'm really at a loss here and would very much appreciate any pointers! Thanks so much!

[codethief]

I just came across the (archived) https://github.com/rootless-containers/runrootless project and the README there mentions the exact same issues with apt update that I ran into above. So it seems I'm missing something fundamental here about why apt update doesn't work inside a rootless runc container. (This issue mentions that there is a difference between "rootless" and "fully rootless"? Whatever that means exactly.)

Finally, I'm wondering what rootless Podman & Docker do differently to get things to work?

My apologies for very likely asking dumb questions here! As you can tell I'm relatively new to the topic of how containers work under the hood and I've already spent hours trying to research my questions – without really getting anywhere.

[AkihiroSuda]

"noNewPrivileges": true

This has to be false for su

[codethief]

Thanks! Unfortunately, this doesn't seem to do the trick. I still get the exact same errors when doing su user or apt update.

[cyphar]

So, there are a couple of issues here:

• For a general purpose container to work properly you need to have at least 65535 UIDs in order for tools like apt to be happy. apt intentionally changes users to nobody:nogroup in order to check what kind of sandbox it is, among other things. Your first configuration wasn't working because you didn't have enough IDs mapped. Strictly speaking, you only need enough UIDs to map the ones the container actually uses, but in order to be safe you really want 64K IDs mapped.

• In order to be able to change users inside the container, you need to have the CAP_SETUID and CAP_SETGID capabilities in the capability sets. Even a setuid binary won't be able to gain capabilities if the bounding set doesn't allow it.

Making those two changes makes your examples work on my machine. However, there are a few other issues you might run into, so I'll list them here:

• Another issue is that setgroups on Linux has annoying behaviour when you are a member of a group that is not mapped in your namespace. This happens with rootless containers because an unprivileged user cannot drop their membership of a group, and so when runc creates a container, that container inherits all of the groups the host process was a member of.

  (Assuming the process doesn't have setgroups disabled -- unprivileged user namespaces require setgroups to be disabled but with newgidmap you can avoid that) You can drop all of your groups without issue, but if you want to leave a single group you have to get the results from getgroups and then modify the list and call setgroups. The problem is that if a group is not mapped, it will appear in the output of getgroups as 65534 -- if that group is mapped, then the subsequent setgroups will make you join the nobody group (even though you actually didn't want the host groups). Programs could deal with this more nicely, but they rarely do and so you end up with errors or weird behaviours as a result. I sent a patch many years ago to hide the groups from getgroups but it was rejected.

  IIRC aptitude might handle this nicely but only if you have a user namespace with 64K IDs mapped -- it has code to "verify" that the sandbox is "safe" and it will error out in a variety of cases where it detects that things it expects don't work.

• noNewPrivileges will cause issues with any setuid binary in the container, as this bit restricts processes from gaining privileges in various ways. This doesn't matter if you are running su as root, as you are in the example -- but it's something you need to be aware of.

• (You mentioned you wanted to nest the containers later -- you cannot map IDs in a user namespace that don't exist in the parent namespace, so you need at least 9999 IDs in the container if you want to map 9999 IDs in a nested container. Also, I suspect you will run into a few issues with nesting but you can cross that bridge when you get to it...)

(rootless-containers/runrootless#23 mentions that there is a difference between "rootless" and "fully rootless"? Whatever that means exactly.)

When I first implemented rootless containers for runc, the intention was to allow you to run containers on a system where you have no administrative permissions at all and cannot use any setuid helpers. On Linux, this means you can only map a single user in a user namespace. Since we later added the ability to use setuid helpers, I used "fully rootless" to refer to the original model where there is only a single user mapped.

I just came across the (archived) https://github.com/rootless-containers/runrootless project and the README there mentions the exact same issues with apt update that I ran into above. So it seems I'm missing something fundamental here about why apt update doesn't work inside a rootless runc container.

runrootless is a tool that wraps a bunch of libc functions in order to allow for a fully rootless container to look like a container that has more than one user, for compatibility reasons. The "nicer" solution for this was our fork of PRoot, and we had a prototype for how we could use SECCOMP_USER_NOTIF to make it have even less overhead, but these days almost everyone uses newuidmap to set up containers and thus it seems like a bit of a waste of time to work on "fully rootless" containers (even though I still think it's a better idea from a security perspective, though it makes using them quite painful).

Finally, I'm wondering what rootless Podman & Docker do differently to get things to work?

They map enough IDs and include the necessary capabilities, which should be sufficient to fix most of the issues you're running into.

[AkihiroSuda]

runrootless is a tool I wrote

I am you 😆

[cyphar]

Ah, I got runrootless confused with remainroot 😅. It's been a while, sorry about that...

[codethief]

Thanks so much for your comprehensive answer, @cyphar! I am currently on vacation and will report back as soon as I'm home again!

[codethief]

@cyphar Once again thank you for helping me! I just returned from my vacation and followed your suggestions:

• I changed linux.(uid|gid)Mappings and made 65535 UIDs/GIDs available to the container
• I added CAP_SETUID and CAP_SETGID to process.capabilities.(bounding|effective|permitted|ambient|inheritable).

Moreover, I added a few more capabilities to enable usage of chown/chmod/ping inside the container, and bind-mounted /etc/resolv.conf while removing the "network" namespace, so as to allow processes inside the container to access the internet & my local DNS resolver.) Here is the resulting config.json.

At this point, su user and apt update no longer complained about setgroups/seteuid/setgid failing, nor – in the case of apt – about connection issues, but both still failed with various "Permission denied" errors (e.g. su: failed to execute /bin/bash: Permission denied).

After a while,

• I realized I might need to make sure that extracting the rootfs tarball would preserve file ownership. I did so by writing a script which runs tar inside the same user namespace that I use for my container afterwards.
• I noticed that the container root / still lacked permissions go+rX. This was likely due to the restrictive umask 077 I use on my host machine. 🤦 Running chmod go+rX on the upper folder of the overlay rootfs fixed that.

With these two changes, su user now works! Only apt update is still giving me trouble:

root@runc:/home/user# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Reading package lists... Done
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_bookworm_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_bookworm-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian-security_dists_bookworm-security_InRelease - PrepareFiles (13: Permission denied)
E: Unable to read /var/lib/apt/lists/partial/ - open (13: Permission denied)

Do you happen to have an idea what this could be about?

[rata]

You need to chown the rootfs to match the user namespace mapping. You can try to create a container with userns with containerd or crio, and then you can see the permissions there.

For a quick hack, doing a chown hostUID:hostGID rootfs -R should help. With hostUID I mean the user on the host that the container root ID is mapped to (idem for hostGID). But you are changing all to be owned by those UIDs, that will be root inside the container, instead of the more granular permissions that it has.

Let's see if that fixes it first and, if it does, then you can create the right solution.

[codethief]

@rata Sorry for the late response and thank you for taking the time to look into this!

I just double-checked again and I think what you're asking is something I already did in my last post but maybe I didn't express this clearly enough:

I realized I might need to make sure that extracting the rootfs tarball would preserve file ownership. I did so by writing a script which runs tar inside the same user namespace that I use for my container afterwards.

By running tar inside the same user namespace used in the container later (and by running tar with the --same-owner option), the aforementioned script ensures that root-owned files in the tarball will be owned by UID 0 inside the container later (which corresponds to my personal user account codethief / hostUID on the host), and so on for any other UIDs 1, 2, 3…

After extracting the tarball using the above script and before entering the container, I therefore get the following permissions on the host:

$ ls -l /tmp/runc-tests/2024-09-03_22-38-02/ro-rootfs
insgesamt 60
lrwxrwxrwx  1 codethief codethief    7 Mai 13 02:00 bin -> usr/bin
drwxr-xr-x  2 codethief codethief 4096 Jan 28  2024 boot
drwxr-xr-x  2 codethief codethief 4096 Mai 13 02:00 dev
drwxr-xr-x 42 codethief codethief 4096 Sep  4 00:18 etc
drwxr-xr-x  3 codethief codethief 4096 Sep  4 00:17 home
lrwxrwxrwx  1 codethief codethief    7 Mai 13 02:00 lib -> usr/lib
lrwxrwxrwx  1 codethief codethief    9 Mai 13 02:00 lib64 -> usr/lib64
drwxr-xr-x  2 codethief codethief 4096 Mai 13 02:00 media
drwxr-xr-x  2 codethief codethief 4096 Mai 13 02:00 mnt
drwxr-xr-x  2 codethief codethief 4096 Mai 13 02:00 opt
drwxr-xr-x  2 codethief codethief 4096 Jan 28  2024 proc
drwx------  3 codethief codethief 4096 Sep  4 00:18 root
drwxr-xr-x  8 codethief codethief 4096 Sep  4 00:18 run
lrwxrwxrwx  1 codethief codethief    8 Mai 13 02:00 sbin -> usr/sbin
drwxr-xr-x  2 codethief codethief 4096 Mai 13 02:00 srv
drwxr-xr-x  2 codethief codethief 4096 Jan 28  2024 sys
drwxrwxrwt  2 codethief codethief 4096 Sep  4 00:18 tmp
drwxr-xr-x 12 codethief codethief 4096 Mai 13 02:00 usr
drwxr-xr-x 11 codethief codethief 4096 Mai 13 02:00 var

i.e. system files are owned by my personal account on the host (= hostUID/hostGID) and hence will be owned by UID 0 inside the container.

Just to be absolutely sure I ran sudo chown codethief:codethief -R $RO_ROOTFS again on the host, as you suggested, to make the entire rootfs owned by root inside the container, overriding the usual more fine-granular file ownership. The error message when running apt update inside the container has remained the same, though:

root@runc:/home/user# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Reading package lists... Done
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_bookworm_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian_dists_bookworm-updates_InRelease - PrepareFiles (13: Permission denied)
W: Problem unlinking the file /var/lib/apt/lists/partial/deb.debian.org_debian-security_dists_bookworm-security_InRelease - PrepareFiles (13: Permission denied)
E: Unable to read /var/lib/apt/lists/partial/ - open (13: Permission denied)

[cyphar]

You probably need to add CAP_DAC_OVERRIDE, since it seems apt relies on root being able to override DAC permissions (at least, that's how it looks when I see #4229 (reply in thread)).

[codethief]

@cyphar Thanks, I will give this a shot!

[codethief]

CAP_DAC_OVERRIDE did the trick! Thanks so much, @cyphar!

Now that basic command line operations inside the container seem to work, I'll continue my quest to run another runc container nested inside the one I already have. It doesn't seem to work out of the box yet, but I want to investigate this some more before I bother you guys again. :)