User units and network-online.target

[ogarcia]

Issue Description

When you try to start a user container using quadlet it by default depends on the unit podman-user-wait-network-online.service which in turn runs until systemctl is-active network-online.target; do sleep 0.5; done. The problem is that (at least on Arch Linux) the network-online.target unit is not active and therefore the podman-user-wait-network-online.service unit always fails.

Steps to reproduce the issue

Steps to reproduce the issue

1. Configure a container unit in .config/containers/systemd/, for example:

   [Unit]
   Description=Ollama Open WebUI container
   
   [Container]
   Image=ghcr.io/open-webui/open-webui:ollama
   Pull=newer
   PublishPort=127.0.0.1:8090:8080
   PublishPort=127.0.0.1:11434:11434
   Environment=WEBUI_AUTH=false
   Environment=OLLAMA_HOST=0.0.0.0:11434
   PodmanArgs=--gpus=all
   
   [Service]
   Restart=always
   TimeoutStartSec=900
   

2. Start it up: systemctl --user start ollama
3. Check the journal journalctl --user -e

   dic 08 13:24:56 hell systemd[1601]: Starting Wait for system level network-online.target as user....
   dic 08 13:24:56 hell sh[3128]: inactive
   dic 08 13:24:56 hell sh[3130]: inactive
   dic 08 13:24:57 hell sh[3132]: inactive
   dic 08 13:24:57 hell sh[3134]: inactive
   dic 08 13:24:58 hell sh[3136]: inactive
   ...
   ...
   dic 08 13:26:25 hell sh[3604]: inactive
   dic 08 13:26:25 hell sh[3606]: inactive
   dic 08 13:26:26 hell systemd[1601]: podman-user-wait-network-online.service: start operation timed out. Terminating.
   dic 08 13:26:26 hell systemd[1601]: podman-user-wait-network-online.service: Main process exited, code=killed, status=15/TERM
   dic 08 13:26:26 hell systemd[1601]: podman-user-wait-network-online.service: Failed with result 'timeout'.
   dic 08 13:26:26 hell systemd[1601]: Failed to start Wait for system level network-online.target as user..
   

Describe the results you received

The system fails when trying to determine the network status. This happens because the network-online.target is not actually active:

% systemctl status network-online.target 
○ network-online.target - Network is Online
     Loaded: loaded (/usr/lib/systemd/system/network-online.target; static)
     Active: inactive (dead)
       Docs: man:systemd.special(7)
             https://systemd.io/NETWORK_ONLINE

This however does not affect other units that depend on this target, for example with this user unit in .config/systemd/user/test.service:

[Unit]
Description=test unit
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/echo hello

If we start it with systemctl --user start test.service it starts without problems.

Describe the results you expected

There should be another way to determine if the network online target has been reached for the user. As far as I see systemd does it, but I don't know how.

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 99.35
    systemPercent: 0.35
    userPercent: 0.3
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2045
  hostname: hell
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.12.3-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 54145740800
  memTotal: 66479091712
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19-1
    path: /usr/bin/crun
    version: |-
      crun version 1.19
      commit: db31c42ac46e20b5527f5339dcbf6f023fcd539c
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2024_11_27.c0fbc7e-1
    version: |
      pasta 2024_11_27.c0fbc7e
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 27m 22.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  harbor.hi.inet:
    Blocked: false
    Insecure: true
    Location: harbor.hi.inet
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: harbor.hi.inet
    PullFromMirror: ""
store:
  configFile: /home/ogarcia/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ogarcia/.local/share/containers/storage
  graphRootAllocated: 1000203091968
  graphRootUsed: 501712965632
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/ogarcia/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732225906
  BuiltTime: Thu Nov 21 22:51:46 2024
  GitCommit: 4cbdfde5d862dcdbe450c0f1d76ad75360f67a3c
  GoVersion: go1.23.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

It is a fully upgraded Arch Linux system.

Additional information

In the configuration of the container unit I have tried to put the following so that at least it would not try to execute the podman-user-wait-network-online.service.

[Quadlet]
DefaultDependencies=false

I don't know why but it still executes this dependency, maybe because my unit depends on some volumes?

[st0nie]

https://systemd.io/NETWORK_ONLINE
I think it not a bug
systemctl status network-online.target
If you're using systemd, then you'll definitely have this target, it's just that it makes a difference when this target is reached, depending on what network manager you're using

[st0nie]

The targets and units of systemd are categorized into system and user, and network-online.target does not exist in user by default, which means that if you fill network-online.target in the user-unit in this way, it won't actually do anything.

[st0nie]

If you're using NetworkManager or systemd-networkd, that shouldn't be a problem

[ogarcia]

The truth is that I thought it could be a bug in Arch. I'm using NetworkManager on my machine, but I've set up a virtual machine with clean Arch Linux using systemd-networkd and the exact same thing happens:

[vagrant@archlinux ~]$ systemctl status systemd-networkd
● systemd-networkd.service - Network Configuration
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-12-09 08:58:44 UTC; 1min 11s ago
 Invocation: a0b0da8a6f76411289d70f8303bf65dc
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
             man:org.freedesktop.network1(5)
   Main PID: 283 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 530)
   FD Store: 0 (limit: 512)
     Memory: 3.4M (peak: 3.7M)
        CPU: 22ms
     CGroup: /system.slice/systemd-networkd.service
             └─283 /usr/lib/systemd/systemd-networkd

Warning: some journal files were not opened due to insufficient permissions.

[vagrant@archlinux ~]$ systemctl status network-online.target
○ network-online.target - Network is Online
     Loaded: loaded (/usr/lib/systemd/system/network-online.target; static)
     Active: inactive (dead)
       Docs: man:systemd.special(7)
             https://systemd.io/NETWORK_ONLINE

More info

[vagrant@archlinux ~]$ uname -a
Linux archlinux 6.12.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 06 Dec 2024 11:15:43 +0000 x86_64 GNU/Linux

[vagrant@archlinux ~]$ systemctl --version
systemd 256 (256.9-1-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL
+ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY
+P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE

And the list of packages and their versions in case it clarifies something:

[vagrant@archlinux ~]$ pacman -Qn
acl 2.3.2-1
archlinux-keyring 20241203-1
argon2 20190702-6
attr 2.5.2-1
audit 4.0.2-2
base 3-2
bash 5.2.037-1
binutils 2.43+r4+g7999dae6961-1
brotli 1.1.0-2
btrfs-progs 6.12-1
bzip2 1.0.8-6
ca-certificates 20240618-1
ca-certificates-mozilla 3.107-1
ca-certificates-utils 20240618-1
coreutils 9.5-2
cryptsetup 2.7.5-1
curl 8.11.0-3
dbus 1.14.10-2
dbus-broker 36-4
dbus-broker-units 36-4
dbus-units 36-4
device-mapper 2.03.28-1
diffutils 3.10-1
dosfstools 4.2-5
e2fsprogs 1.47.1-4
efibootmgr 18-3
efivar 39-1
expat 2.6.4-1
file 5.46-2
filesystem 2024.11.21-1
findutils 4.10.0-2
gawk 5.3.1-1
gcc-libs 14.2.1+r134+gab884fffe3fc-1
gdbm 1.24-1
gettext 0.22.5-2
glib2 2.82.2-1
glibc 2.40+r16+gaa533d58ff-2
gmp 6.3.0-2
gnupg 2.4.7-1
gnutls 3.8.8-1
gpgme 1.24.1-1
grep 3.11-1
grub 2:2.12-3
gzip 1.13-4
hwdata 0.390-1
iana-etc 20241206-1
icu 75.1-1
iproute2 6.12.0-1
iptables 1:1.8.10-2
iputils 20240905-1
jansson 2.14-4
json-c 0.18-1
kbd 2.6.4-3
keyutils 1.6.3-3
kmod 33-3
krb5 1.21.3-1
libarchive 3.7.7-1
libassuan 3.0.0-1
libbpf 1.5.0-1
libcap 2.71-1
libcap-ng 0.8.5-2
libedit 20240517_3.1-1
libelf 0.192-2
libevent 2.1.12-4
libffi 3.4.6-1
libgcrypt 1.11.0-2
libgpg-error 1.51-1
libidn2 2.3.7-1
libksba 1.6.7-1
libldap 2.6.9-1
libmnl 1.0.5-2
libnetfilter_conntrack 1.0.9-2
libnfnetlink 1.0.2-2
libnftnl 1.2.8-1
libnghttp2 1.64.0-1
libnghttp3 1.6.0-1
libnl 3.11.0-1
libnsl 2.0.1-1
libp11-kit 0.25.5-1
libpcap 1.10.5-2
libpsl 0.21.5-2
libsasl 2.1.28-5
libseccomp 2.5.5-3
libsecret 0.21.4-1
libssh2 1.11.0-1
libsysprof-capture 47.2-1
libtasn1 4.19.0-2
libtirpc 1.3.6-1
libunistring 1.2-1
libusb 1.0.27-1
libverto 0.3.2-5
libxcrypt 4.4.36-2
libxml2 2.13.5-1
licenses 20240728-1
linux 6.12.3.arch1-1
linux-api-headers 6.10-1
lmdb 0.9.33-1
lz4 1:1.10.0-2
lzo 2.10-5
mkinitcpio 39.2-2
mkinitcpio-busybox 1.36.1-1
mpfr 4.2.1-4
ncurses 6.5-3
nettle 3.10-1
npth 1.8-1
openssh 9.9p1-2
openssl 3.4.0-1
p11-kit 0.25.5-1
pacman 7.0.0.r6.gc685ae6-1
pacman-mirrorlist 20240717-1
pam 1.7.0-1
pambase 20230918-2
pciutils 3.13.0-2
pcre2 10.44-1
pinentry 1.3.1-5
popt 1.19-2
procps-ng 4.0.4-3
psmisc 23.7-1
readline 8.2.013-1
rsync 3.3.0-2
sed 4.9-3
shadow 4.16.0-1
sqlite 3.47.2-1
sudo 1.9.16.p2-1
systemd 256.9-1
systemd-libs 256.9-1
systemd-sysvcompat 256.9-1
tar 1.35-2
tpm2-tss 4.1.3-1
tzdata 2024b-2
util-linux 2.40.2-1
util-linux-libs 2.40.2-1
xxhash 0.8.2-1
xz 5.6.3-1
zlib 1:1.3.1-2
zstd 1.5.6-1

[st0nie]

Did you enable NetworkManager-wait-online.service ?

[ogarcia]

Did you enable NetworkManager-wait-online.service ?

Yes:

% systemctl status NetworkManager-wait-online.service
○ NetworkManager-wait-online.service - Network Manager Wait Online
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager-wait-online.service; enabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:NetworkManager-wait-online.service(8)

[Luap99]

What is the output of systemctl list-dependencies network-online.target? The target unit itself is not so important and does not need to be enabled. It just acts as single unit where other units from the network management tool hook into, i.e. NetworkManager-wait-online.service in case of NetworkManager.

[ogarcia]

In my system where I use NetworkManager:

% systemctl list-dependencies network-online.target
network-online.target
○ └─NetworkManager-wait-online.service

In a virtual machine with a clean system using systemd-networkd:

$ systemctl list-dependencies network-online.target
network-online.target
○ └─systemd-networkd-wait-online.service

In both cases the dependent service is enabled and inactive.

[Luap99]

Ok that is odd but this all is far outside of the control of podman I believe.

Can you check systemctl list-dependencies --reverse network-online.target? I believe the enable command for NetworkManager-wait-online.service doesn't actually start the unit at boot by default, it just causes it to get triggered by network-online.target because of

...
[Install]
WantedBy=network-online.target

And network-online.target doesn't start on its own either, it only get run when other units declare:

After=network-online.target
Wants=network-online.target

So I think if you have no other root service starting it it might never activate.

I really wish we would not need to be in the business of that stuff but here we are until systemd/systemd#3312 is fixed. We need a work around it as rootless containers start without networking otherwise #22197

[ogarcia]

% systemctl list-dependencies --reverse network-online.target
network-online.target
○ └─archlinux-keyring-wkd-sync.service

archlinux-keyring-wkd-sync.service is a service that runs through a timer once a week.

Therefore, as you say, there is no other (system) service that activates this target and therefore it remains inactive.

[st0nie]

So we may need a dummy system level service requires this target
Can you try it？

[Luap99]

A dummy unit might not be required, enabling a service is just adding a symlink, adding a link like
/etc/systemd/system/multi-user.target.wants/network-online.target -> /usr/lib/systemd/system/network-online.target might be enough. That way the unit will get started when multi-user.target is reached which should always be there.

[ogarcia]

Creating a service like this and enabling it works:

# /etc/systemd/system/podman-network-online-dummy.service
[Unit]
Description=This service simply activates network-online.target
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/echo Activating network-online.target

[Install]
WantedBy=multi-user.target

$ systemctl status podman-network-online-dummy
● podman-network-online-dummy.service - This service simply activates network-online.target
     Loaded: loaded (/etc/systemd/system/podman-network-online-dummy.service; enabled; preset: disabled)
     Active: inactive (dead) since Mon 2024-12-09 12:37:14 UTC; 2min 14s ago
   Duration: 13ms
 Invocation: 2badd17dfc034d7098894fe20496afab
    Process: 573 ExecStart=/usr/bin/echo Activating network-online.target (code=exited, status=0/SUCCESS)
   Main PID: 573 (code=exited, status=0/SUCCESS)
   Mem peak: 1.6M
        CPU: 3ms


$ systemctl status network-online.target
● network-online.target - Network is Online
     Loaded: loaded (/usr/lib/systemd/system/network-online.target; static)
     Active: active since Mon 2024-12-09 12:37:14 UTC; 30s ago
 Invocation: cb64d12a239a4782a5fa3dc88e7cf750
       Docs: man:systemd.special(7)
             https://systemd.io/NETWORK_ONLINE