[RHEL8]Get the Function not implemented error by using crun runtime when building a multi-arch image

[jianzhangbjz]

Issue Description

I got the permission error by using the default runtime runc, as follows,

[linux/arm64] STEP 5/6: RUN ["/bin/opm", "serve", "/configs", "--cache-dir=/tmp/cache", "--cache-only"]
error running container: from /usr/bin/runc creating container for [/bin/opm serve /configs --cache-dir=/tmp/cache --cache-only]: time="2024-07-16T14:12:55+08:00" level=warning msg="unable to get oom kill count" error="no directory specified for memory.oom_control"
time="2024-07-16T14:12:55+08:00" level=error msg="runc create failed: unable to start container process: error during container init: error mounting \"sysfs\" to rootfs at \"/sys\": mount sysfs:/sys (via /proc/self/fd/6), flags: 0xf: operation not permitted"
: exit status 1
ERRO[0003] did not get container create message from subprocess: EOF 

And then, I followed the suggestion from #15314 to use the crun runtime, but got the below error.

[linux/amd64] STEP 5/6: RUN ["/bin/opm", "serve", "/configs", "--cache-dir=/tmp/cache", "--cache-only"]
error running container: from /usr/bin/crun creating container for [/bin/opm serve /configs --cache-dir=/tmp/cache --cache-only]: mount_setattr `/sys`: Function not implemented
: exit status 1
ERRO[0002] did not get container create message from subprocess: EOF 

Steps to reproduce the issue

Steps to reproduce the issue

1. create a Dockerfile, like

[cloud-user@preserve-olm-env2 opm]$ cat catalog.Dockerfile 
# The base image is expected to contain
# /bin/opm (with a serve subcommand) and /bin/grpc_health_probe
FROM registry.redhat.io/openshift4/ose-operator-registry-rhel9:v4.16
#FROM quay.io/operator-framework/opm:latest

# Configure the entrypoint and command
ENTRYPOINT ["/bin/opm"]
CMD ["serve", "/configs", "--cache-dir=/tmp/cache"]

# Copy declarative config root into image at /configs and pre-populate serve cache
ADD catalog /configs
RUN ["/bin/opm", "serve", "/configs", "--cache-dir=/tmp/cache", "--cache-only"]

# Set DC-specific label for the location of the DC root directory
# in the image
LABEL operators.operatorframework.io.index.configs.v1=/configs

2, build the multi-arch image based on this Dockerfile.

Describe the results you received

Describe the results you received

[cloud-user@preserve-olm-env2 opm]$  podman --runtime crun manifest create quay.io/openshifttest/etcd-index:33450
b57994d9bfc0ada75b3ba98113580f6019fb0160d97abc92e0b6e3b836ae734a
[cloud-user@preserve-olm-env2 opm]$ podman build --runtime crun --platform linux/amd64,linux/arm64,linux/ppc64le,linux/s390x  --manifest quay.io/openshifttest/etcd-index:33450 -f catalog.Dockerfile  
[linux/s390x] STEP 1/6: FROM registry.redhat.io/openshift4/ose-operator-registry-rhel9:v4.16
Trying to pull registry.redhat.io/openshift4/ose-operator-registry-rhel9:v4.16...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 3bcb3add8bf2 skipped: already exists  
Copying blob 51b9b79d167f skipped: already exists  
Copying blob d6359be12a3e skipped: already exists  
Copying blob 91099ff70275 skipped: already exists  
Copying config dbd4f8129d done   | 
Writing manifest to image destination
Storing signatures
[linux/s390x] STEP 2/6: ENTRYPOINT ["/bin/opm"]
--> Using cache b17fae611b4ca898d708a4485c86279a93b48613ee8bfc5bc1386e1376ae9d42
--> b17fae611b4c
[linux/s390x] STEP 3/6: CMD ["serve", "/configs", "--cache-dir=/tmp/cache"]
--> Using cache 307dc57ad952aece20050406e78f3e11168dc6ea03158214f96fa949d9c383e4
--> 307dc57ad952
[linux/s390x] STEP 4/6: ADD catalog /configs
--> Using cache 2273653d85fad6f88ec23ac8c26d5287d4dcafccd25ace4dc8bd2b5e6db4df90
--> 2273653d85fa
[linux/s390x] STEP 5/6: RUN ["/bin/opm", "serve", "/configs", "--cache-dir=/tmp/cache", "--cache-only"]
error running container: from /usr/bin/crun creating container for [/bin/opm serve /configs --cache-dir=/tmp/cache --cache-only]: mount_setattr `/sys`: Function not implemented
: exit status 1
ERRO[0003] did not get container create message from subprocess: EOF 
[linux/amd64] STEP 1/6: FROM registry.redhat.io/openshift4/ose-operator-registry-rhel9:v4.16
Trying to pull registry.redhat.io/openshift4/ose-operator-registry-rhel9:v4.16...
Getting image source signatures
Checking if image destination supports signatures
Copying blob bcd7dc0c0b02 skipped: already exists  
Copying blob 74b582a585c1 skipped: already exists  
Copying blob ca1636478fe5 skipped: already exists  
Copying blob 2c1c281f5b6a skipped: already exists  
Copying config 74e4eaf472 done   | 
Writing manifest to image destination
Storing signatures
[linux/amd64] STEP 2/6: ENTRYPOINT ["/bin/opm"]
--> Using cache 86fba5797960a3036176ed5c368bbd971af4e0273df6e3fced9748ae265c5d62
--> 86fba5797960
[linux/amd64] STEP 3/6: CMD ["serve", "/configs", "--cache-dir=/tmp/cache"]
--> Using cache 66cfc5e659660d7d33b08d757b170051e184001c803a72434f04ffedb59de3ef
--> 66cfc5e65966
[linux/amd64] STEP 4/6: ADD catalog /configs
--> Using cache d0fb13686a1a88cb305676e4d908cf5ae1c879f7040222a4323ce4f2e68749e6
--> d0fb13686a1a
[linux/amd64] STEP 5/6: RUN ["/bin/opm", "serve", "/configs", "--cache-dir=/tmp/cache", "--cache-only"]
error running container: from /usr/bin/crun creating container for [/bin/opm serve /configs --cache-dir=/tmp/cache --cache-only]: mount_setattr `/sys`: Function not implemented
: exit status 1
ERRO[0006] did not get container create message from subprocess: EOF 
...
...

Describe the results you expected

Podman builds multi-arch images successfully.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

[cloud-user@preserve-olm-env2 opm]$ podman version
Client:       Podman Engine
Version:      4.9.4-rhel
API Version:  4.9.4-rhel
Go Version:   go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
Built:        Tue Jun 18 17:34:14 2024
OS/Arch:      linux/amd64

[cloud-user@preserve-olm-env2 opm]$ podman version
Client:       Podman Engine
Version:      4.9.4-rhel
API Version:  4.9.4-rhel
Go Version:   go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
Built:        Tue Jun 18 17:34:14 2024
OS/Arch:      linux/amd64
[cloud-user@preserve-olm-env2 opm]$ podman info 
host:
  arch: amd64
  buildahVersion: 1.33.8
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.10-1.module+el8.10.0+21962+8143777b.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 8f1552a71b76511fb9b4a96282513d17686c56fc'
  cpuUtilization:
    idlePercent: 99.29
    systemPercent: 0.27
    userPercent: 0.43
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: rhel
    version: "8.10"
  eventLogger: file
  freeLocks: 2035
  hostname: preserve-olm-env2
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-287.el8.dt4.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 453201920
  memTotal: 16600383488
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.4.0-dev
        commit: unknown
        CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
    package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21962+8143777b.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.12-1.module+el8.10.0+21974+acd2159c.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.module+el8.10.0+21962+8143777b.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 6213h 13m 12.00s (Approximately 258.88 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/cloud-user/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 0
    stopped: 11
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/cloud-user/.local/share/containers/storage
  graphRootAllocated: 128731557888
  graphRootUsed: 124240289792
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 115
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/cloud-user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4-rhel
  Built: 1718703254
  BuiltTime: Tue Jun 18 17:34:14 2024
  GitCommit: ""
  GoVersion: go1.21.7 (Red Hat 1.21.7-2.module+el8.10.0+21638+b01be198)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-rhel

[cloud-user@preserve-olm-env2 opm]$ uname -a 
Linux preserve-olm-env2 4.18.0-287.el8.dt4.x86_64 #1 SMP Thu Feb 18 13:31:55 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

[cloud-user@preserve-olm-env2 opm]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.10 (Ootpa)

[cloud-user@preserve-olm-env2 opm]$ rpm -q podman
podman-4.9.4-4.module+el8.10.0+21995+81e8507c.x86_64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

if I understand correctly you are trying to run a nested container. Please use the same configuration as the podman container (quay.io/podman/stable:latest), you can grab it from the /etc/containers/containers.conf file

[jianzhangbjz]

Thanks! But, no. I'm building a multi-arch image. I guess I need to install the qemu-system-x86_64 first.

[cloud-user@preserve-olm-env2 opm]$ podman machine init
Error: exec: "qemu-system-x86_64": executable file not found in $PATH

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Luap99]

Note podman machine is not supported in RHEL AFAIK.
If you have a problem with the proper builds in RHEL where crun is not working please file these bugs through the proper Red Hat channels not upstream.