Rootless build permission errors

[savemuri]

Issue Description

Issue with running podman on a rootless container in k8s.

Steps to reproduce the issue

Steps to reproduce the issue

1. Include the following piece in ubuntu-22.04 LTS Containerfile to install podman

FROM docker.io/ubuntu:22.04

# Configurations
ARG HOST_ARCH=amd64
ARG OS_TYPE=x86_64
ARG PYTHON_VERSION=3.10

ENV JUPYTER_USER jovyan
ENV NB_UID 1000
ENV HOME=/home/$JUPYTER_USER
ENV JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre/"
ENV SSH_HOME=/etc/ssh/$JUPYTER_USER

USER root

RUN useradd -M -s /bin/bash -N -u ${NB_UID} ${JUPYTER_USER} \
    && mkdir -p ${HOME} && mkdir -p /etc/ssh/$JUPYTER_USER

RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get -yq update \
    && apt-get remove --auto-remove -yq openjdk* \
    && apt-get purge -yq openjdk* \
    && apt-get install -yq \
    lsb-release \
    openjdk-8-jdk \
    openjdk-8-jre \
    curl \
    git \
    gnupg \
    openssh-server \
    && apt-get autoremove -y \
    && apt-get remove --auto-remove -yq python3-blinker \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Install Podman
RUN key_url="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_22.04/Release.key" \
    sources_url="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_22.04" \
    && echo "deb $sources_url/ /" | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list \
    && curl -fsSL $key_url | gpg --dearmor | tee /etc/apt/trusted.gpg.d/devel_kubic_libcontainers_unstable.gpg > /dev/null \
    && apt -yq update \
    && apt install -yq podman fuse-overlayfs buildah

RUN echo -e "${JUPYTER_USER}:1:999\n${JUPYTER_USER}:1001:64535" > /etc/subuid \
    && echo -e "${JUPYTER_USER}:1:999\n${JUPYTER_USER}:1001:64535" > /etc/subgid

ARG _REPO_URL="https://raw.githubusercontent.com/containers/podman/main/contrib/podmanimage/stable"
ADD $_REPO_URL/containers.conf /etc/containers/containers.conf
ADD $_REPO_URL/podman-containers.conf /home/${JUPYTER_USER}/.config/containers/containers.conf

RUN mkdir -p /home/${JUPYTER_USER}/.local/share/containers \
    && chmod 644 /etc/containers/containers.conf

RUN sed -e 's|^#mount_program|mount_program|g' \
        -e '/additionalimage.*/a "/var/lib/shared",' \
        -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
        /usr/share/containers/storage.conf \
        > /etc/containers/storage.conf

# Setup internal Podman to pass subscriptions down from host to internal container
RUN printf '/run/secrets/etc-pki-entitlement:/run/secrets/etc-pki-entitlement\n/run/secrets/rhsm:/run/secrets/rhsm\n' > /etc/containers/mounts.conf

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN chown -R ${JUPYTER_USER}:users ${HOME} /usr/local/etc /usr/local/share 

USER $NB_UID

CMD ["bin/sh", "-c", "while :; do echo '.'; sleep 5 ; done"]

Describe the results you received

Case 1: Pulling image without buildah unshare

jovyan@sasank-test-0:~$ podman pull docker.io/nginx:latest
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
ERRO[0000] running `/usr/bin/newuidmap 456 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: write to uid_map failed: Operation not permitted 
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

Case 2: Pulling image after buildah unshare

jovyan@sasank-test-0:~$ buildah unshare
WARN[0000] running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted 
WARN[0000] /usr/bin/newgidmap should be setgid or have filecaps setgid 
WARN[0000] Falling back to single mapping               
WARN[0000] Error running newuidmap: exit status 1: newuidmap: write to uid_map failed: Operation not permitted 
WARN[0000] Falling back to single mapping               
root@sasank-test-0:~# podman --storage-opt mount_program=/usr/bin/fuse-overlayfs --storage-opt ignore_chown_errors=true run -it --rm docker.io/library/python:latest
Trying to pull docker.io/library/python:latest...
Getting image source signatures
Copying blob fa6efb383f81 done  
Copying blob 1c74526957fc done  
Copying blob ad5739181616 done  
Copying blob 1b13d4e1a46e done  
Copying blob 30d855997954 done  
Copying blob 75e2b45cbee5 done  
Copying blob 0c6213e9c29b done  
Copying blob 1b1e18053fdc done  
Copying config e7177b0afd done  
Writing manifest to image destination
WARN[0014] Path "/run/secrets/etc-pki-entitlement" from "/etc/containers/mounts.conf" doesn't exist, skipping 
WARN[0014] Path "/run/secrets/rhsm" from "/etc/containers/mounts.conf" doesn't exist, skipping 
Error: crun: mount `proc` to `proc`: Operation not permitted: OCI permission denied

Describe the results you expected

Expected to pull images without error.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon_2:2.1.10-0ubuntu22.04+obs18.10_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.34
    systemPercent: 0.21
    userPercent: 0.45
  cpus: 8
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  freeLocks: 2047
  hostname: sasank-test-0
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
  kernel: 5.10.205-195.804.amzn2.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2866364416
  memTotal: 66584010752
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.14-0ubuntu22.04+obs67.1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14
      commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
      rundir: /var/tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /var/tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.2-0ubuntu22.04+obs12.36_amd64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 262h 59m 10.00s (Approximately 10.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/jovyan/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jovyan/.local/share/containers/storage
  graphRootAllocated: 10464022528
  graphRootUsed: 1298386944
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/jovyan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

### Podman in a container

Yes

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

Running as pod in EKS.

### Additional information

I used this as reference since running that image in rootless pod worked for me - https://github.com/containers/podman/blob/d418d164a99cb9b3a95fbcaf72eaf379c8c100ab/contrib/podmanimage/stable/Containerfile

[giuseppe]

make sure the newuidmap and newgidmap binaries have the right capabilities set (i.e. the equivalent of rpm --setcaps shadow-utils in the Containerfile you've linked) as it is specified in the warning message:

/usr/bin/newgidmap should be setgid or have filecaps setgid 

[rhatdan]

Also make sure the file system that contains newuidmap and newgidmap is not mounted nosuid.

[savemuri]

Thanks for the quick response. This fixed it for me. For future reference, here is a working Containerfile with ubuntu

FROM docker.io/ubuntu:22.04

# Configurations
ARG HOST_ARCH=amd64
ARG CONDA_VER=latest
ARG OS_TYPE=x86_64
ARG CONDA_ENV_NAME=deploy
ARG PYTHON_VERSION=3.10

# Setup enviroment variables
ENV NB_USER jovyan
ENV NB_UID 1000
ENV HOME=/home/$NB_USER
ENV PIP_NO_CACHE_DIR=off
ENV DEBUG_IGNORE_VERSION_MISMATCH="1"
ENV JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre/"
ENV SSH_HOME=/etc/ssh/$NB_USER

USER root

# Setup the user jovyan
RUN useradd -M -s /bin/bash -N -u ${NB_UID} ${NB_USER} \
    && mkdir -p ${HOME} && mkdir -p /etc/ssh/$NB_USER

# Install Ubuntu Packages
RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get -yq update \
    && apt-get remove --auto-remove -yq openjdk* \
    && apt-get purge -yq openjdk* \
    && apt-get install -yq \
    lsb-release \
    openjdk-8-jdk \
    openjdk-8-jre \
    curl \
    git \
    gnupg \
    openssh-server \
    && apt-get autoremove -y \
    && apt-get remove --auto-remove -yq python3-blinker \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Install Podman
RUN key_url="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_22.04/Release.key" \
    sources_url="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_22.04" \
    && echo "deb $sources_url/ /" | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list \
    && curl -fsSL $key_url | gpg --dearmor | tee /etc/apt/trusted.gpg.d/devel_kubic_libcontainers_unstable.gpg > /dev/null \
    && apt -yq update \
    && apt install -yq podman fuse-overlayfs buildah slirp4netns uidmap

RUN echo "${NB_USER}:1001:64535" > /etc/subuid && \
    echo "${NB_USER}:1001:64535" > /etc/subgid

ARG _REPO_URL="https://raw.githubusercontent.com/containers/podman/main/contrib/podmanimage/stable"
ADD $_REPO_URL/containers.conf /etc/containers/containers.conf
ADD $_REPO_URL/podman-containers.conf /home/${NB_USER}/.config/containers/containers.conf

RUN apt-get install -y libcap2-bin && \
    chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap && \
    setcap cap_setuid=ep /usr/bin/newuidmap && \
    setcap cap_setgid=ep /usr/bin/newgidmap && \
    apt-get autoremove --purge -y libcap2-bin

RUN mkdir -p /home/${NB_USER}/.local/share/containers \
    && chmod 644 /etc/containers/containers.conf

RUN sed -e 's|^#mount_program|mount_program|g' \
        -e '/additionalimage.*/a "/var/lib/shared",' \
        -e 's|^mountopt[[:space:]]*=.*$|mountopt = "nodev,fsync=0"|g' \
        /usr/share/containers/storage.conf \
        > /etc/containers/storage.conf

# Setup internal Podman to pass subscriptions down from host to internal container
RUN printf '/run/secrets/etc-pki-entitlement:/run/secrets/etc-pki-entitlement\n/run/secrets/rhsm:/run/secrets/rhsm\n' > /etc/containers/mounts.conf

ENV _CONTAINERS_USERNS_CONFIGURED=""

# Setup Environment
RUN chown -R ${NB_USER}:users ${HOME} /usr/local/etc /usr/local/share \
    && if [ -d /opt/tritonserver ]; then chown -R ${NB_USER}:users /opt/tritonserver; fi
USER $NB_UID

CMD ["bin/sh", "-c", "while :; do echo '.'; sleep 5 ; done"]

And this post I found in another issue was helpful - https://samuel.forestier.app/blog/security/podman-rootless-in-podman-rootless-the-debian-way

[savemuri]

Apologies for closing issue prematurely. The above solution lets me run podman pull and podman push commands. However I can't seem to be able to build or run an image as it fails as shown below

jovyan@savemuri-test-0:~$ podman run -it --rm docker.io/nginx:latest --entrypoint sh
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
Trying to pull docker.io/library/nginx:latest...
Getting image source signatures
Copying blob dc9c4fdb83d6 done  
Copying blob 8b7dd3ed1dc3 done  
Copying blob 36664b6ce66b done  
Copying blob 2f44b7a888fa done  
Copying blob 2d455521f76c done  
Copying blob 35497dd96569 done  
Copying blob 8056d2bcf3b6 done  
Copying config a8758716bb done  
Writing manifest to image destination
Error: crun: mount `proc` to `proc`: Operation not permitted: OCI permission denied

[rhatdan]

This means podman is running within a mount namespace that the /proc has been modified. Podman wants to modify the /proc itself, which is not allowed by the kernel.

You could try a -v /proc:/proc

[savemuri]

Thanks, that did get me past the issue. However when trying to run a docker build with /proc mounted, it lands to a different issue

STEP 1/2: FROM docker.io/ubuntu:22.04
STEP 2/2: RUN apt-get -yq update
E: setgroups 65534 failed - setgroups (22: Invalid argument)
E: setegid 65534 failed - setegid (22: Invalid argument)
Reading package lists...
E: setgroups 65534 failed - setgroups (22: Invalid argument)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (22: Invalid argument)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (22: Invalid argument)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)
Error: building at STEP "RUN apt-get -yq update": while running runtime: exit status 100

Containerfile that I am trying to build

FROM docker.io/ubuntu:22.04

RUN apt-get -yq update

[giuseppe]

let's not reuse the same issue for a different question.

Using -v /proc:/proc means you cannot create a new PID namespace, otherwise the PIDs won't match, you need to add --pid=host.

The other error you've reported seems like you've not enough GIDs available. Please verify that in the idMappings field in the podman info output, or check the ones currently available with cat /proc/self/gid_map