.buildkit_qemu_emulator: operation not supported

[anthr76]

Hello! I'm trying to use the Docker CLI with the DOCKER_HOST set to the Podman socket on MacOS.

I have a project I'm trying to build (https://github.com/goauthentik/authentik) that uses buildx.

I'm using a arm64 Mac and the command I'm trying to invoke is

DOCKER_BUILDKIT=1 docker build . --tag authentik:test1 --platform=linux/amd64 -f Dockerfile --load

This fails with a error

/dev/.buildkit_qemu_emulator ... : did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qemu-emulator2534885165/dev/.buildkit_qemu_emulator: operation not supported

This does run okay on the native architecture.

Has anyone ever seen this before and know if it's a bug or a workaround?

I found this related issue dagger/dagger#8054

I know disabling SELinux on CoreOS is usually not a route you can really take so not sure how to approach this.

[nalind]

Can you crank up the logging level here using the -D option? I'm not sure the client actually ends up talking to podman, since that's not an error message that I think it produces.

[anthr76]

Sure!

I'm not seeing anything particularly interesting here

Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "HEAD /_ping HTTP/1.1" 200 0 "" "Docker-Client/unknown-version (darwin)"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 2m+0h/4t connection(s)" X-Reference-Id=0x4000534a90
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:new 2m+0h/4t connection(s)" X-Reference-Id=0x4000534a98
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 2m+0h/5t connection(s)" X-Reference-Id=0x4000534a98
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "HEAD /_ping HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 3m+0h/5t connection(s)" X-Reference-Id=0x4000534a98
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:new 3m+0h/5t connection(s)" X-Reference-Id=0x4000534aa0
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 3m+0h/6t connection(s)" X-Reference-Id=0x4000534aa0
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=info msg="Failed Request: (404:Not Found) for POST:'/grpc'"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 4m+0h/6t connection(s)" X-Reference-Id=0x4000534aa0
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:closed 4m+0h/6t connection(s)" X-Reference-Id=0x4000534aa0
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:new 3m+0h/6t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 3m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "HEAD /_ping HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 4m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 4m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Looking up image \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" in local containers storage"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Trying \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" ..."
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="parsed reference into \"[overlay@/var/home/core/.local/share/containers/storage+/run/user/502/containers]@32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\""
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Found image \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" as \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" in local containers storage"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Found image \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" as \"32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b\" in local containers storage ([overlay@/var/home/core/.local/share/containers/storage+/run/user/502/containers]@32aa1a49331703fbc2ad933b11debfe90406ddc1aa233462396781e8c610973b)"
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "GET /v1.41/containers/buildx_buildkit_default/json HTTP/1.1" 200 7614 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 4m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 4m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=info msg="Created exec session 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 in container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7"
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "POST /v1.41/containers/buildx_buildkit_default/exec HTTP/1.1" 201 74 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 4m+0h/7t connection(s)" X-Reference-Id=0x4000534aa8
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:new 4m+0h/7t connection(s)" X-Reference-Id=0x4000892000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 4m+0h/8t connection(s)" X-Reference-Id=0x4000892000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Starting exec session 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 of container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=info msg="Going to start container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 exec session 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 and attach to it"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Set user to "
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="/usr/bin/conmon messages will be logged to syslog"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -c 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 -u 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 -r /usr/bin/crun -b /var/home/core/.local/share/containers/storage/overlay-containers/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7/userdata/964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 -p /var/home/core/.local/share/containers/storage/overlay-containers/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7/userdata/964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626/exec_pid -n buildx_buildkit_default --exit-dir /var/home/core/.local/share/containers/storage/overlay-containers/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7/userdata/964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626/exit --persist-dir /var/home/core/.local/share/containers/storage/overlay-containers/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7/userdata/964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626/persist/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 --full-attach -s -l none --log-level debug --syslog -i -e --exec-attach --exec-process-spec /var/home/core/.local/share/containers/storage/overlay-containers/0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7/userdata/964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626/exec-process-2410713519 --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/core/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/502/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/502/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /var/home/core/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg --exec --exit-command-arg 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 --exit-command-arg 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 --exit-delay 300]"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=info msg="Running conmon under slice user.slice and unitName libpod-conmon-0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7.scope"
Sep 06 13:40:46 localhost.localdomain conmon[3291]: conmon 0ce927da97fb77aa6866 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Attaching to container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 exec session 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Received: 0"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:hijacked 5m+0h/8t connection(s)" X-Reference-Id=0x4000892000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Performing non-terminal HTTP attach for container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Beginning STDIN copy"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Received: 3296"
Sep 06 13:40:46 localhost.localdomain podman[3277]: 2024-09-06 13:40:46.057116859 -0400 EDT m=+2.325791784 container exec 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7 (image=docker.io/moby/buildkit:buildx-stable-1, name=buildx_buildkit_default)
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Successfully started exec session 964f11e52aa85010ae715118175acc675b79bc11d61b68bc95cc12ad45d33626 in container 0ce927da97fb77aa6866460ac681a538197beda6d48594be4259f3171814eec7"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:new 4m+1h/8t connection(s)" X-Reference-Id=0x400052e000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 4m+1h/9t connection(s)" X-Reference-Id=0x400052e000
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "HEAD /_ping HTTP/1.1" 200 0 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 5m+1h/9t connection(s)" X-Reference-Id=0x400052e000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:active 5m+1h/9t connection(s)" X-Reference-Id=0x400052e000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Cached value indicated that volatile is being used"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Cached value indicated that idmapped mounts for overlay are not supported"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Check for idmapped mounts support "
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/999-podman-machine.conf\""
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="Loading registries configuration \"/var/home/core/.config/containers/registries.conf.d/001-home-manager.conf\""
Sep 06 13:40:46 localhost.localdomain podman[3277]: @ - - [06/Sep/2024:13:40:46 -0400] "GET /v1.41/info HTTP/1.1" 200 2457 "" "Go-http-client/1.1"
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="IdleTracker:idle 5m+1h/9t connection(s)" X-Reference-Id=0x400052e000
Sep 06 13:40:46 localhost.localdomain podman[3277]: time="2024-09-06T13:40:46-04:00" level=debug msg="STDIN copy completed"

Here's some bonus logs from docker buildx

docker logs buildx_buildkit_default
time="2024-09-06T17:35:18Z" level=info msg="auto snapshotter: using overlayfs"
time="2024-09-06T17:35:18Z" level=info msg="found worker \"vtiedu9brfi75i9b4o7esln4b\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:0ce927da97fb org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm64 linux/riscv64 linux/ppc64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/loong64]"
time="2024-09-06T17:35:18Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2024-09-06T17:35:18Z" level=info msg="found 1 workers, default=\"vtiedu9brfi75i9b4o7esln4b\""
time="2024-09-06T17:35:18Z" level=warning msg="currently, only the default worker can be used."
time="2024-09-06T17:35:18Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2024-09-06T17:35:31Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"/dev/.buildkit_qemu_emulator /bin/sh -c apt-get update &&     apt-get install -y --no-install-recommends libpq5 libmaxminddb0 ca-certificates &&     apt-get install -y --no-install-recommends runit &&     apt-get clean &&     rm -rf /tmp/* /var/lib/apt/lists/* /var/tmp/ &&     adduser --system --no-create-home --uid 1000 --group --home /authentik authentik &&     mkdir -p /certs /media /blueprints &&     mkdir -p /authentik/.ssh &&     mkdir -p /ak-root &&     chown authentik:authentik /certs /media /authentik/.ssh /ak-root\" did not complete successfully: failed to copy xattrs: failed to set xattr \"security.selinux\" on /tmp/buildkit-qemu-emulator2954603484/dev/.buildkit_qemu_emulator: operation not supported"
time="2024-09-06T17:40:46Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"/dev/.buildkit_qemu_emulator /bin/sh -c rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages \\\"true\\\";' > /etc/apt/apt.conf.d/keep-cache\" did not complete successfully: failed to copy xattrs: failed to set xattr \"security.selinux\" on /tmp/buildkit-qemu-emulator2933629274/dev/.buildkit_qemu_emulator: operation not supported"
time="2024-09-06T17:42:47Z" level=warning msg="leaked execError detected and released"
time="2024-09-06T17:42:47Z" level=warning msg="leaked execError detected and released"
time="2024-09-06T17:44:09Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"/dev/.buildkit_qemu_emulator /bin/sh -c rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages \\\"true\\\";' > /etc/apt/apt.conf.d/keep-cache\" did not complete successfully: failed to copy xattrs: failed to set xattr \"security.selinux\" on /tmp/buildkit-qemu-emulator622860307/dev/.buildkit_qemu_emulator: operation not supported"

[nalind]

Huh, looks like I was barking up the wrong tree, and the attempt is not allowed, and I can't explain why it seems to be working in arm64-on-arm64 when it fails for me when I try chcon in an amd64-on-amd64 container that I run as a rootless user.
The rootfs for the container that podman runs has a label set for it when it's mounted, so I would expect attempts to change the SELinux label on, well, anything under it to be rejected as not supported.
I'd have to launch the container as root with --privileged and, from inside of it, mount a tmpfs on /tmp to avoid having a label set there at mount-time.

[anthr76]

@nalind do you suggest I should open a issue for this?

[nalind]

Yeah, I don't think we have one for that yet, but we should. I can't guarantee that it'll be fixed quickly, since it'll require patches in multiple packages to get the right things to happen, but I do think we should fix it.

[rhatdan]

Does it work if you do

docker build ... --security-opt label=disabled ...

[anthr76]

With --load this is what I get

DOCKER_BUILDKIT=1 docker build . --tag authentik:test1 --platform=linux/arm64,linux/amd64 -f Dockerfile --load --security-opt label=disabled
WARNING: security-opt flag is deprecated. "RUN --security=insecure" should be used with BuildKit.
[conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
[+] Building 0.0s (0/0)                                                                                                    docker-container:default
ERROR: docker exporter does not currently support exporting manifest lists

If I drop that arg I do end up seeing the same thing sadly

DOCKER_BUILDKIT=1 docker build . --tag authentik:test1 --platform=linux/arm64,linux/amd64 -f Dockerfile --security-opt label=disabled
WARNING: security-opt flag is deprecated. "RUN --security=insecure" should be used with BuildKit.
[conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
[+] Building 7.8s (23/93)
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt-get update &&     apt-get install -y --no-install-recommends libpq5 libmaxminddb0 ca-certificates &&     apt-get install -y --no-install-recommends runit &&     apt-get clean &&     rm -rf /tmp/* /var/lib/apt/lists/* /var/tmp/ &&     adduser --system --no-create-home --uid 1000 --group --home /authentik authentik &&     mkdir -p /certs /media /blueprints &&     mkdir -p /authentik/.ssh &&     mkdir -p /ak-root &&     chown authentik:authentik /certs /media /authentik/.ssh /ak-root" did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qemu-emulator1672885143/dev/.buildkit_qemu_emulator: operation not supported

[deej-io]

I am currently affected by this issue too. Is there any form of work around? A recommended different OS image instead of quay.io/podman/machine-os:5.2 perhaps?