How to set up a systemd system service (with User=) that runs a caddy executable without conmon and the OCI runtime in the rootless podman network namespace?

[eriksjolund]

How to set up a systemd system service (with systemd directive User=) that runs a caddy executable in the rootless podman network namespace? (conmon and the OCI runtime should not be used)

I would like the caddy process to serve as an HTTP reverse proxy for podman containers running in a custom network. Rootless Podman with Pasta is used for the containers in the custom network.

First I tried with ExecStart=podman unshare --rootless-netns /usr/local/bin/caddy .... It worked fine but when I stopped the service with the command systemctl stop caddy.service the caddy process kept running.

I think systemd didn't kill the process because caddy is not a direct child process of systemd (or some similar explanation related to cgroups).

Anyway, I then tried using nsenter instead. Here are the relevant parts of the file caddy.service

User=test
ExecStart=bash -c "exec nsenter \
   --net=/proc/$(cat $XDG_RUNTIME_DIR/containers/networks/aardvark-dns/aardvark.pid)/ns/net \
   --user=/proc/$(cat $XDG_RUNTIME_DIR/libpod/tmp/pause.pid)/ns/user \
   --mount=/proc/$(cat $XDG_RUNTIME_DIR/libpod/tmp/pause.pid)/ns/mnt \
    /usr/local/bin/caddy run --environ --config /var/home/test/Caddyfile"
RestrictAddressFamilies=AF_INET AF_INET6
NoNewPrivileges=yes
PAMName=login

That worked better.  The command systemctl stop caddy.service stopped the caddy process.

These two lines were added to let the service run with extra restrictions

RestrictAddressFamilies=AF_INET AF_INET6
NoNewPrivileges=yes

This line

PAMName=login

was added to be able to use the environment variable $XDG_RUNTIME_DIR

Any suggestions of how to improve this hackish solution that makes use of nsenter?
(The solution currently has a race condition as it does not make use of file locking and the file
$XDG_RUNTIME_DIR/containers/networks/rootless-netns/ref-count)

Maybe Podman could add support for this use case by
introducing a new command-line option --exec to podman unshare
?
A sketch: (This syntax does not currently exist)

ExecStart=podman unshare --rootless-netns --exec command arg1 arg2 ...

A tricky problem is how to decrease $XDG_RUNTIME_DIR/containers/networks/rootless-netns/ref-count
when the service stops. Maybe an additional command would be needed

A sketch: (This syntax does not currently exist)

ExecStopPost=podman unshare --rootless-netns --decrease-refcounter

Note, this discussion thread has some similarities to

• support User= in systemd for running rootless services #20573

but here ExecStart= does not execute podman, conmon, or the OCI runtime (crun, runc).

[Luap99]

I think using podman unshare --rootless-netns should work, there should no need to add any new options.
I have no tried that but I guess it is possible that podman moves itself out of the cgroup because we do not own it? And the podman setup really wants a cgroup that it can modify?

I have not looked deeply at the cgroup stuff but if the move out of the unit cgroup is the issue then maybe registry.NoMoveProcess option in the code that can be set on a per podman command basis. I think we can set it for podman unshare.

[eriksjolund]

Thanks for the input and the tip about NoMoveProcess! I'll investigate more and try it out.

I started drafting a PR (it's work-in-progress):
eriksjolund/podman-caddy-socket-activation#22

Currently it makes use of

ExecStart=bash -c "exec nsenter \
    --preserve-credentials \
    --net=/proc/$(cat $XDG_RUNTIME_DIR/containers/networks/aardvark-dns/aardvark.pid)/ns/net \
    --user=/proc/$(cat $XDG_RUNTIME_DIR/libpod/tmp/pause.pid)/ns/user \
    --mount=/proc/$(cat $XDG_RUNTIME_DIR/libpod/tmp/pause.pid)/ns/mnt \
    /usr/local/bin/caddy run --environ --config /srv/caddy/Caddyfile"

but using ExecStart=podman unshare --rootless-netns ... would be a more robust solution.