Best practices for host-to-rootless connections, where sudo/suid is possible?

[adelton]

Hello,

for development, testing, and debugging, I prefer to use rootless containers but I also from time to time need a way to connect to those rootless containers from the host, as an unprivileged user. I know that I can use

$ podman unshare --rootless-netns curl http://10.88.42.42/

and the connection will work, seen as coming from 10.88.0.1. But the podman unshare is not often practical, for example when I need to check something with a browser or other more heavyweight mechanism.

I know that as an unprivileged user, my processes are not able to manipulate network interfaces on the host. I'm OK with having a sudo or suid helper script to create the ingress to the rootless network, on temporary basis.

I first tried to create a new veth link from the host netns to the rootless netns where the podman0 interface with 10.88.0.1 lives (for a typical bridge network and pasta setup) but I couldn't get pasta do the proper routing to the individual containers.

So I then decided to create a helper container (actually a pod) and move its end of the veth link that pasta creates to the host netns. That way pastra treats connections from that interface as if they were coming from one of the rootless containers, not knowing that they in fact originate from processes in the host netns.

My current WIP investigation and setup is below. I wonder if this is a reasonable approach to use or if I'm missing some simpler way to create the ingress interface / route on the host?

All commands below are run as unprivileged user; sudo commands indicate where we need root privileges.

Make sure NetworkManager does not touch our interface on the host

A one-off setup to prevent NetworkManager from attempting to handle IP address for the podman-ingress interface which we will create.

( echo '[keyfile]' ; echo unmanaged-devices=interface-name:podman-ingress ) | sudo tee /etc/NetworkManager/conf.d/podman-ingress.conf
sudo systemctl reload NetworkManager

Create a pod from which we will "steal" the link

podman pod create --name podman-ingress --infra-name podman-ingress-infra --network bridge
podman pod start podman-ingress

Get its pid and IP address

set -e
INGRESS_INFRA_PID=$( podman inspect podman-ingress-infra | jq -r '.[0].PidFile' )
test -n "$INGRESS_INFRA_PID"
INGRESS_INFRA_PID=$( cat $INGRESS_INFRA_PID )
test -n "$INGRESS_INFRA_PID"
INGRESS_INFRA_IP=$( podman inspect podman-ingress-infra | jq -r '.[0].NetworkSettings | .IPAddress + "/" + (.IPPrefixLen | tostring)' )
test -n "$INGRESS_INFRA_IP"
echo "We will use link from container (pid $INGRESS_INFRA_PID) with $INGRESS_INFRA_IP"

Move the link to the host netns

sudo ip netns attach podman-ingress-pod $INGRESS_INFRA_PID
sudo ip -n podman-ingress-pod link set eth0 name podman-ingress netns 1 up
sudo ip addr add $INGRESS_INFRA_IP dev podman-ingress

Check the setup

ip a show dev podman-ingress

In a different terminal, run a testing container:

podman run --rm --network podman --ip 10.88.42.42 docker.io/library/nginx

In the first terminal, connect to it:

curl http://10.88.42.42/

Tear down the setup

Move the link back to the pod, so that podman does not complain when removing it; avoiding
network: netavark: failed to delete container veth eth0: Netlink error: No such device (os error 19):

sudo ip link set podman-ingress netns podman-ingress-pod name eth0

Remove the podman-ingress-pod netns name:

sudo ip netns del podman-ingress-pod

Remove the ingress pod:

podman pod rm -f podman-ingress

[sbrivio-rh]

I also from time to time need a way to connect to those rootless containers from the host

One thing I'm completely missing here is: what prevents you from using plain port forwarding (Podman's -p or pasta's -t)? If you do that and connect to a loopback address on the host, you'll also reach the container.

[adelton]

So you differentiate between them using different addresses?

Yes. I basically treat those containers as VMs (and often they actually run systemd and multiple services).

pasta --freebind -t 1.2.3.4/443 --config-net

I'm getting

Failed to bind port 443 (Permission denied) for option '-t 1.2.3.4/443', exiting

I can use

sudo sysctl net.ipv4.ip_unprivileged_port_start=443

to change that limit but with that we are already getting to the sudo territory. :-)

# nc -l 443

I'm getting

Ncat: Failed to resolve default IPv4 address: Name or service not known. QUITTING.

Not sure what should or shouldn't have gotten setup by pasta at this point. Running ip a in that pasta process shows

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1a:bc:91:ee:40:62 brd ff:ff:ff:ff:ff:ff

It feels like playing with individual ports is more work than the wholesale ingress to the rootless network that I'm able to do by moving that veth link to the host netns.

[sbrivio-rh]

I can use

sudo sysctl net.ipv4.ip_unprivileged_port_start=443

to change that limit but with that we are already getting to the sudo territory. :-)

Yes, but that's the recommended way (for the reason explained in pasta(1), Binding to low numbered ports (well-known or system ports, up to 1023)) to bind to low ports, and it won't leave interfaces created as root around, so it's arguably the least privileged way to do things.

I'm getting

Ncat: Failed to resolve default IPv4 address: Name or service not known. QUITTING.

Not sure what should or shouldn't have gotten setup by pasta at this point. Running ip a in that pasta process shows

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1a:bc:91:ee:40:62 brd ff:ff:ff:ff:ff:ff

Did you forget --config-net perhaps?

[adelton]

Did you forget --config-net perhaps?

Ah, you are right, I did.

[adelton]

I confirm that the approach with

$ sudo ip route add local 10.88.0.0/16 dev lo

and then using either

$ podman run --rm --network pasta:--ipv4-only,-a,10.88.0.150,-n,16,-g,10.88.0.1,-t,10.88.0.150/80,-m,1500,--no-ndp,--no-dhcpv6,--no-dhcp docker.io/library/nginx

or

$ podman run --rm -p 10.88.0.150:80:80 --network pasta:--ipv4-only,-a,10.88.0.150,-n,16,-g,10.88.0.1,-m,1500,--no-ndp,--no-dhcpv6,--no-dhcp docker.io/library/nginx

works -- I'm then able to

$ curl http://10.88.0.150/

from the host, and from other containers that have network configured in the same way.

Now I wonder, how to combine that with the rootless bridge setup, to take advantage of the automatic assignment of the IP addresses for containers that are clients, where no well-known externally accessible IP address is needed. I tried adding

,--netns,/run/user/1000/containers/networks/rootless-netns/rootless-netns

to the --network pasta:... parameter list to force the interfaces to be created in the bridge network netns but that does not seem to be the way.

[sbrivio-rh]

$ podman run --rm -p 10.88.0.150:80:80 --network pasta:--ipv4-only,-a,10.88.0.150,-n,16,-g,10.88.0.1,-m,1500,--no-ndp,--no-dhcpv6,--no-dhcp docker.io/library/nginx

By the way, you don't strictly need all those --no-... options I guess.

Now I wonder, how to combine that with the rootless bridge setup, to take advantage of the automatic assignment of the IP addresses for containers that are clients, where no well-known externally accessible IP address is needed.

If you want to combine it with a rootless bridge setup, you can create a custom network instead, and the network diagram would look like #22943 (comment) (details about how to configure all the available rootless modes at https://github.com/eriksjolund/podman-networking-docs).

Or you can use IPv6, create an "AnyIP" route for a whole /64 prefix, and do something similar to the configuration outlined in https://bugs.passt.top/show_bug.cgi?id=48 (not "automatically" supported yet, but you can do something like that manually). Say:

# ip route add local 2001:db8:b1bb:1d1::/64 dev lo

...then let containers configure their own address via SLAAC (tell pasta to advertise your prefix via NDP) and use those automatically-generated addresses instead.