NetworkPolicy not applying

[JonathanWilbur]

Environmental Info:
K3s Version:

k3s version v1.26.4+k3s1 (8d0255af)
go version go1.19.8

Node(s) CPU architecture, OS, and Version:

Linux saturn 5.14.0-1059-oem #67-Ubuntu SMP Mon Mar 13 14:22:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Single node: my Xubuntu workstation. Modern, powerful hardware. Normal x86-64 processor.

Describe the bug:

If I create a NetworkPolicy that blocks the CNI IP (representing external traffic that is inbound into the cluster), nothing is blocked. Specifically, my testing works by reading the logs of an Nginx server and manually blocking the specific IP address recorded in the logs like so:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {} # Omitted
spec:
  egress:
  - {}
  ingress:
  - from:
    - ipBlock:
        cidr: 1.1.1.1/0 # Effectively "every IP address"
        except:
        - 203.0.113.1/32 # This is a Bogon IP address, here for technical reasons
        - 10.42.0.1/32 # This is the IP address of my CNI interface
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress
status: {}

Here is my CNI interface:

95: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ae:95:e6:81:e7:bd brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever

And for thoroughness, here is my service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx","namespace":"k2btest"},"spec":{"externalTrafficPolicy":"Local","ports":[{"port":8088,"targetPort":80}],"selector":{"app":"nginx"},"type":"LoadBalancer"}}
  creationTimestamp: "2023-05-28T18:59:47Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: nginx
  namespace: k2btest
  resourceVersion: "26496"
  uid: 96e01626-bae0-430c-93e3-1c9eb69ac4c0
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.182.236
  clusterIPs:
  - 10.43.182.236
  externalTrafficPolicy: Local
  healthCheckNodePort: 31913
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32247
    port: 8088
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.8.164

I have tested this technique on Azure Kubernetes Service (AKS) and it works there, but the big difference is that AKS just uses your real IP address, not a CNI bridge. I think I also tested this on Minikube, but I am not sure.

This might not be a bug. Maybe I'm not supposed to be able to block this IP, since it does seem to represent all traffic from the outside world. Is it even possible to do this in K3s? Again, it seems to work in Azure AKS.

Steps To Reproduce:

• Installed K3s: I used the script on the homepage with no further configuration whatsoever.
• Run an Nginx server on your cluster, and expose it using a LoadBalancer service.
• Make a request to this Nginx server using your local web browser.
• Observe that the logged IP address is your CNI interface's IP address.
• Apply the NetworkPolicy shown above, but replace the second except IP address with the IP of your CNI interface.
• Observe that you are still not blocked from the Nginx server.

Additional testing

• Change the externalTrafficPolicy: Local to externalTrafficPolicy: Cluster (the default). (I originally changed it to the former, because the former preserves external IP addresses, but oddly, that does not seem to matter in K3s. Everything still shows up as 10.42.0.1 for me.)

Expected behavior:

I kind of expect to be blocked from accessing the Nginx service. However, I could reasonably see this exclusion being governed by this bullet point:

The ability to prevent loopback or incoming host traffic (Pods cannot currently block localhost access, nor do they have the ability to block access from their resident node).

So, again, I'm not sure this is a bug or not. I'd just like clarification.

Actual behavior:

I can still make requests that reach the Nginx server.

Additional context / logs:

Nginx logs sample:

10.42.0.1 - - [29/May/2023:18:25:27 +0000] "GET /asdf.txt HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" "-"
2023/05/29 18:25:28 [error] 29#29: *27 open() "/usr/share/nginx/html/asdf.txt" failed (2: No such file or directory), client: 10.42.0.1, server: localhost, request: "GET /asdf.txt HTTP/1.1", host: "192.168.8.164:8088"
10.42.0.1 - - [29/May/2023:18:25:28 +0000] "GET /asdf.txt HTTP/1.1" 404 555 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" "-"

[mlazowik]

I'm 90% sure that the bullet point you quoted is the answer. In particular, limiting traffic from the resident node is problematic because the k8s ready/live/startup probes are coming from kubelet that's running outside of the cluster.

Service meshes that do mTLS need to work around that, for example see: linkerd/linkerd2#7050 It's easier to handle when you can do authz at the L7, because then you can allow the external traffic only to the probe endpoints. With NetworkPolicies you can only work at L3/L4, so it would only make sense for containers that expose probes on dedicated ports.