Rootless Quadlet is running as a root

[sigulete]

Issue Description

I'm using Podman 5.2.3

My containers are defined as Quadlets in /etc/containers/systemd/users/..

My setup has been working as expected until this weekend when I updated my server. I'm experiencing a very strange issue.
When I rebooted the server, all quadlest located in the path above are being run as root.
podman ps produces an empty output, while sudo podman ps shows all of them.

I tried everything, and I rebooted my server various times, but for some reason the containers are always re-created as system containers instead of user containers.

Any clue? I need to sort this out urgently.

Steps to reproduce the issue

1. Create a container in /etc/containers/systemd/users/mycontainer.container

[Container]
ContainerName=mycontainer
SecurityLabelLevel=s0
Image=docker.io/freshrss/freshrss:latest
PublishPort=8082:80
AutoUpdate=registry
Volume=feeds-data:/var/www/FreshRSS/data

[Service]
Restart=always
TimeoutStartSec=90

[Install]
WantedBy=default.target

2. Reboot

Describe the results you received

The container is running as root.

Describe the results you expected

The container should run rootless, as user.

podman info output

host:
arch: amd64
buildahVersion: 1.37.3
cgroupControllers:

• cpu
• memory
• pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
  package: conmon-2.1.12-2.fc40.x86_64
  path: /usr/bin/conmon
  version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
  idlePercent: 98
  systemPercent: 0.58
  userPercent: 1.42
  cpus: 2
  databaseBackend: sqlite
  distribution:
  distribution: fedora
  version: "40"
  eventLogger: journald
  freeLocks: 2036
  hostname: venus
  idMappings:
  gidmap:
  • container_id: 0
    host_id: 1000
    size: 1
  • container_id: 1
    host_id: 100000
    size: 65536
    uidmap:
  • container_id: 0
    host_id: 1000
    size: 1
  • container_id: 1
    host_id: 100000
    size: 65536
    kernel: 6.11.4-201.fc40.x86_64
    linkmode: dynamic
    logDriver: journald
    memFree: 1910894592
    memTotal: 4077432832
    networkBackend: netavark
    networkBackendInfo:
    backend: netavark
    dns:
    package: aardvark-dns-1.12.2-2.fc40.x86_64
    path: /usr/libexec/podman/aardvark-dns
    version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
    ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
    crun version 1.17
    commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
    rundir: /run/user/1000/crun
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
    os: linux
    pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.x86_64
    version: |
    pasta 0^20240906.g6b38f07-1.fc40.x86_64
    Copyright Red Hat
    GNU General Public License, version 2 or later
    https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
    rootlessNetworkCmd: pasta
    security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
    serviceIsRemote: false
    slirp4netns:
    executable: ""
    package: ""
    version: ""
    swapFree: 5368705024
    swapTotal: 5368705024
    uptime: 0h 25m 31.00s
    variant: ""
    plugins:
    authorization: null
    log:
• k8s-file
• none
• passthrough
• journald
  network:
• bridge
• macvlan
• ipvlan
  volume:
• local
  registries:
  search:
• registry.fedoraproject.org
• registry.access.redhat.com
• docker.io
  store:
  configFile: /var/home/daniel/.config/containers/storage.conf
  containerStore:
  number: 1
  paused: 0
  running: 0
  stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/daniel/.local/share/containers/storage
  graphRootAllocated: 105772986368
  graphRootUsed: 10241253376
  graphStatus:
  Backing Filesystem: btrfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Supports shifting: "false"
  Supports volatile: "true"
  Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
  number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/daniel/.local/share/containers/storage/volumes
  version:
  APIVersion: 5.2.3
  Built: 1727136000
  BuiltTime: Tue Sep 24 10:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

The server is running as VM.
Hypervisor qemu - cockpit

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[sigulete]

Additional information:

• When I run /usr/libexec/podman/quadlet --user --dryrun it shows the systemd definition files as expected and it is empty when running /usr/libexec/podman/quadlet --dryrun. So it is scanning /etc/containers/systemd/users as expected.
• As temporary measure, I moved the Quadlet definition to .config/containers/systemd and it resolved the problem.

Finally, I have tried in another virtual machine using Fedora 41 (beta) with Podman 5.2.4, and it worked as expected.
Maybe this is a problem of version 5.2.3.

[rhatdan]

The quadlets should be in
/etc/containers/systemd/users/$(UID)?

So you need to pick a UID to run them in as I understand it.

[Luap99]

Systemd uses system and user sessions. The system session exists only as root as it what is used by default with the systemctl commands. But the user session exists once per user (including root) and is generally created if you log in and stopped if you log out (unless enable-linger is used).

Paths from /etc/containers/systemd/users/ are added to each user session so they can be triggered as root if root has a valid user session. You can easily verify if they run as user service with systemctl --user status <service name>.
And you should do the same as rootless user, I assume the units exists there too but likely just failed to start due conflicts such as the port number.
At least that is what I assume you are seeing. I only looked at the code in main, it could be that there is a legit bug in your release that mixes these things up.

I think our quadlet docs aren't aren't very good to highlight user vs system session. They talk about root vs rootless which is not generally how systemd works in that sense.

[sigulete]

In my case I don't login as root, only as a regular user, so root wouldn't have triggered the generation of containers.
It has been workinig well before v5.2.3 and it works again in v5.2.4. So it may be an issue with v5.2.3.

However, to make sure no other user triggers the creation of containers, I decided to keep the quadlet files under /etc/containers/systemd/users/1000. And yes, this user has linger enabled, so containers are always up and running.

Using /etc/containers/systemd/users/1000 also works in v5.2.3.

[Luap99]

There are no quadlet code changes in 5.2.3 compared to 5.2.2 or 5.2.4 so it theoretical impossible that the behaviour changed in any of the 5.2.z versions.

[Luap99]

Is /etc/containers/systemd a symlink then it should be fixed in 5.3 I think with #23498