What does ServiceLB (Klipper LB) actually do ?

[nate-42]

Hi everyone !

I am fairly new to K8s and began recently playing with K3s.
My setup is the following :

• 3 masters/workers nodes v1.28.8+k3s1 on Debian 11 vanilla
• Cilium 1.15.3 as a CNI

I am trying to understand what does ServiceLB actually do.
The documentation states here :

For each LoadBalancer Service, a DaemonSet is created in the kube-system namespace. This DaemonSet in turn creates Pods with a svc- prefix, on each node. These Pods use iptables to forward traffic from the Pod's NodePort, to the Service's ClusterIP address and port.

So I've dived a bit in how the traefik Service is implemented.
I've found the /entry file in klipper-lb repo also the controller code in k3s repo (which I did not read fully for the latter).

But I cannot make sense of what I've found in my cluster.
I have the following nodes, services and pods :

> kubectl get nodes -o wide
NAME   STATUS   ROLES  AGE  VERSION INTERNAL-IP    EXTERNAL-IP   OS-IMAGE  KERNEL-VERSION  CONTAINER-RUNTIME
dev-master01   Ready    control-plane,etcd,master   6d20h   v1.28.8+k3s1   10.255.200.1   <none>  Debian 11   5.10.0-28-amd64                              containerd://1.7.11-k3s2
dev-master02   Ready    control-plane,etcd,master   6d20h   v1.28.8+k3s1   10.255.200.2   <none>  Debian 11   5.10.0-28-amd64                               containerd://1.7.11-k3s2
dev-master03   Ready    control-plane,etcd,master   6d18h   v1.28.8+k3s1   10.255.200.3   <none>  Debian 11   5.10.0-28-amd64                               containerd://1.7.11-k3s2

My external IPs are 10.13.37.1, 10.13.37.2, 10.13.37.3 (not declared in k3s systemd unit, so not printed in the above output).

> kubectl get svc -A -o wide
NAMESPACE     NAME              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kube-system   traefik           LoadBalancer   10.43.100.248   10.13.37.2    80:30174/TCP,443:32236/TCP   6d18h

> kubectl get pods -A -o wide
NAMESPACE  NAME   READY   STATUS      RESTARTS         AGE     IP             NODE        NOMINATED NODE   READINESS GATES
kube-system   svclb-traefik-e39c842e-89mb8     2/2     Running     0                15h     10.0.3.106     dev-master03   <none>           <none>
kube-system   svclb-traefik-e39c842e-d47zb      2/2     Running     0                15h     10.0.1.244     dev-master02   <none>           <none>
kube-system   svclb-traefik-e39c842e-qpz2s      2/2     Running     6                 15h     10.0.0.171     dev-master01   <none>           <none>
kube-system   traefik-f4564c4f4-962wh              1/1     Running     0                6d17h   10.0.3.42      dev-master03   <none>           <none>

But when I look at the iptables NAT rules on my k3s host, regarding this service, I get the following (shortened to ease reading):

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 370K   24M CILIUM_PRE_nat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cilium-feeder: CILIUM_PRE_nat */
 370K   24M KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ 

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SVC-CVG3OEGEH7H5P3HQ  tcp  --  *      *       0.0.0.0/0            10.43.100.248        /* kube-system/traefik:websecure cluster IP */ tcp dpt:443
    0     0 KUBE-EXT-CVG3OEGEH7H5P3HQ  tcp  --  *      *       0.0.0.0/0            10.13.37.2           /* kube-system/traefik:websecure loadbalancer IP */ tcp dpt:443

Chain KUBE-SVC-CVG3OEGEH7H5P3HQ (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.100.248        /* kube-system/traefik:websecure cluster IP */ tcp dpt:443
    0     0 KUBE-SEP-ZXP4PF2PKR2YGFAS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/traefik:websecure -> 10.0.3.42:8443 */

Chain KUBE-EXT-CVG3OEGEH7H5P3HQ (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* masquerade traffic for kube-system/traefik:websecure external destinations */
    0     0 KUBE-SVC-CVG3OEGEH7H5P3HQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-SEP-ZXP4PF2PKR2YGFAS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  all  --  *      *       10.0.3.42            0.0.0.0/0            /* kube-system/traefik:websecure */
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/traefik:websecure */ tcp to:10.0.3.42:8443

So this can be sum-up with : when there is a packet for 10.13.37.2:443, it is DNATed to 10.0.3.42:8443 the traefik pod IP.
To my understanding, those iptables rules are created by kube-proxy, whenever a service is fired up (rules may vary depending on Service configuration).

So I wondered what is in the svclb-traefik pods ?

> kubectl exec -n kube-system -it svclb-traefik-e39c842e-qpz2s -c lb-tcp-443 -- sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
23: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether ca:75:3c:e2:66:cb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.171/32 scope global eth0
       valid_lft forever preferred_lft forever

/ # iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
    0     0 ACCEPT     6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
    0     0 DROP       6    --  *      *       0.0.0.0/0            10.43.100.248        tcp dpt:80
    0     0 DROP       6    --  *      *       0.0.0.0/0            10.43.100.248        tcp dpt:443

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

/ # iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:10.43.100.248:443
    0     0 DNAT       6    --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:10.43.100.248:80

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  6    --  *      *       0.0.0.0/0            10.43.100.248       
    0     0 MASQUERADE  6    --  *      *       0.0.0.0/0            10.43.100.248 

So I can see that those rules are the ones created by the entry script in the container image of ServiceLB and are also the ones mentioned in the doc quoted at the beginning of this post ("These Pods use iptables to forward traffic from the Pod's NodePort, to the Service's ClusterIP address and port").
But, those rules never seems to be used (I generated traffic to try) as the packets and bytes counters remain to 0, generally, the svclb-traefik pods seems never used.

So this brings me to my final question :
What is the purpose of this /entry file?
How come traffic never reaches svclb-traefik ?
How does ServiceLB and kube-proxy interact ?
What does Service LB actually do ?
What is the usecase that ServiceLB is for ?

Thank you for reading my (way too long) post !
I'm looking forward to reading your answers !

[brandond]

ServiceLB exists to make LoadBalancer services work. Without it, or another LoadBalancer controller, services of this type would remain pending.

The pods occupy the service's ports on the host, and forward traffic to the service's ClusterIP address, or to the local NodePort, depending on configuration. The IPs of hosts running the svclb pods are advertised in the loadbalancer status.

[nate-42]

Thank you for your answer, your explanation match what I've been able to understand so far.
I understand that ServiceLB is just a simple placeholder to satisfy K3s needs for a LB implementation.
What I still don't understand is why traffic never reaches the svclb pods, as the traffic is caught earlier by the DNAT rules setup by kube-proxy ?

[IJMacD]

I think your observation is correct, and actually answers your initial question.

The ServiceLB pods never handle TCP/UDP traffic themselves, that's not their purpose. Their entire purpose as I understand it is to set up those DNAT iptables rules on the node. Once the iptables rules are set up, the pod's purpose is fulfilled but they stick around to watch for changes in the service and to remove the rules when the corresponding K8s service is deleted.

kube-proxy is only responsible for creating iptables rules to handle traffic within the cluster. ServiceLB is an add-on (mostly specific to K3s), only adding a couple of extra rules per LoadBalanced service to handle external traffic.

In the abstract, a K8s "LoadBalancer" is just some method to map an external IP address to a cluster IP and to report that external IP back to the control plane. On prem this could be implemented in hardware with a hardware load balancer. MetalLB is an alternative LoadBalancer to ServiceLB. It implements a LoadBalancer via Level 2 and Level 3 protocols to interact with various real network hardware. The approach ServiceLB uses is a software approach, leveraging iptables on the host node and reporting the node's own external IP.

[nate-42]

That makes sense, thank you !!

[IJMacD]

Just an update to my answer for myself and future travellers...

Doing more reading, the ServiceLB (a.k.a. klipper-lb) container is implemented as a single shell script, found here: https://github.com/k3s-io/klipper-lb/blob/master/entry

All it does is check which version of iptables the host is using, sets appropriate iptables rules, then do some pause trick.

The pod using this container image requests the corresponding hostPorts, but as noted traffic gets DNAT'd before it reaches any process. Does it request the hostPorts to "reserve" them? To make sure nothing else on the host is trying to use them? I have no idea.

Also another thing I'm unsure about is exactly by what mechanism the iptables rules are removed. Is it due to the kernel namespace features? Once the container is terminated does the kernel automatically clean up any rules created by it? That would explain the need for the pause trick.

[nate-42]

The iptables rules are indeed cleaned when the network namespace is terminated, happening upon pod termination.

[lindhe]

MetalLB is an alternative LoadBalancer to ServiceLB.

Interesting. The way I understand it, what MetalLB fundamentally does is to announce an IP to the network (e.g. using APR) the nodes are on, while ServiceLB just binds to a port and reports the IP of the node back to the service (but never announces the IP to the outside world).

Did I get that more or less right, would you say?