Unable to connect to pod's container IP; works in Docker

[budney]

Environmental Info:
K3s Version: v1.21.3+k3s1 (1d1f220)

Node(s) CPU architecture, OS, and Version: Linux k8s-01 5.11.0-1016-raspi #17-Ubuntu SMP PREEMPT aarch64 GNU/Linux

Cluster Configuration: 1 node, 1 server, out-of-the-box setup

My pod runs a DNS resolver on port 53. From a shell in the same container, I can connect to the resolver on 127.0.0.1, but NOT on the container's IP address (10.42.0.99). Running the same image in Docker, I can connect to either IP address. I've attached steps to reproduce the failure in k3s and the success in Docker. Just to add to the mystery, I installed nmap and it claims port 53 is open on the container IP -- but nslookup nevertheless times out trying to connect to it.

Steps To Reproduce:

1. Install k3s: curl -sfL https://get.k3s.io | sh -
2. Don't install any worker nodes.
3. sudo kubectl apply -f dnscache.yaml
4. sudo kubectl get pods
5. sudo kubectl exec --stdin --tty pod/dnscache-65b994b978-lwm59 -- /bin/sh
6. Inside the container: nslookup cnn.com 127.0.0.1 (This works as expected.)
7. nslookup cnn.com $(ifconfig eth0 | sed -nr '/.*inet addr:1/s/^.*inet addr:([^ ]+).*$/\1/p')
   The latter command times out with the message, "no servers could be reached."

• Installed K3s:

Expected behavior:

nslookup should succeed when pointed at either IP address. When these steps are followed in a Docker container based on the same image, it works exactly as expected.

Actual behavior:

nslookup times out when pointed at the eth0 IP address, but succeeds when pointed at 127.0.0.1.

Additional context / logs:

I've attached the deployment/service YAML file that I used, renamed to dnscache.txt because the .yaml was rejected.
dnscache.txt

Backporting

• Needs backporting to older releases

[ChristianCiach]

nslookup cnn.com $(ifconfig eth0

I don't understand what you're trying to do here. This will get you the IP address of the host, not the IP adress of the container. If this really works with your docker setup (which you haven't shown us), then you surely mapped the port 53/udp to your host.

There are multiple ways to do that with Kubernetes. For example, when using K3s, you can use the integrated load balancer (klipper-lb). To use that, change the type of your Service to LoadBalancer:

apiVersion: v1
kind: Service
metadata:
  name: dnscache
spec:
  selector:
    app: dnscache
  ports:
    - name: dnscache-53-udp
      port: 53
      protocol: UDP
      targetPort: 53
  type: LoadBalancer

Alternatively, you could ditch the Service altogether and just bind the Pod directly to the host by adding the hostPort attribute:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dnscache
  labels:
    app: dnscache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dnscache
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: dnscache
    spec:
      containers:
        - name: dnscache
          image: budney/djbdns
          env:
            - name: delegate
              value: lan.jeenyus.net:153.18.172.in-addr.arpa
            - name: service
              value: dnscache
          ports:
            - name: dnscache-53-udp
              containerPort: 53
              hostPort: 53
              protocol: UDP
          resources: {}
          volumeMounts:
            - mountPath: /docker-compose.yml
              name: dnscache-hostpath0
              readOnly: true
      restartPolicy: Always
      volumes:
        - hostPath:
            path: /etc/djbdns/docker-compose.yml
          name: dnscache-hostpath0

Anyway, why are you mounting a docker-compose-file into the container?

Edit: You seem to be new to Kubernetes and this looks more like a question than an issue. I think this issue should be moved to "Discussions".

[budney]

nslookup HOST SERVER contacts SERVER to ask for the IP address of HOST. In this case, I'm running a shell inside the container itself, which has a running DNS resolver, and I'm asking it to tell me the IP address of cnn.com. The service I'm running is a DNS server.

I can run this image in Docker, then exec a shell in the same container, and then contact the DNS server on either 127.0.0.1 or on the IP address of eth0.

But if I run the same image using k3s, I can contact the server on 127.0.0.1 but not on the IP address of eth0.

In other words the DNS server inside the container is binding all interfaces (0.0.0.0), but cannot receive any connections on eth0, even from within the container itself. Host ports aren't involved in any way at all; that's why I ran a shell inside the server's own container.

Before installing k3s, I had the service working just fine using docker-compose. After banging my head against the wall with k3s, I think I've been able to show that this is actually a bug of some sort. The first thing I tried was using LoadBalancer, but it's not going to do much if even processes inside the container itself can't connect to port 53.

...unless there's something I missed, which is entirely possible.

[budney]

I reread this:

I don't understand what you're trying to do here. This will get you the IP address of the host, not the IP adress of the container. If this really works with your docker setup (which you haven't shown us), then you surely mapped the port 53/udp to your host.

I think what you missed was that I'm running ifconfig eth0 inside the container itself. It's definitely giving me the address of the container, as it should. I even mentioned it (10.42.0.99) in my original report.

Step #5 of "steps to reproduce" was:

sudo kubectl exec --stdin --tty pod/dnscache-65b994b978-lwm59 -- /bin/sh

That runs a shell in the container. The steps that come after it are performed inside that shell.

[ChristianCiach]

Yes, you are right, I misunderstood that part.

Using your original example with the Service of type ClusterIP, what happens when you are using the IP that is shown when executing kubectl get services -o wide on the host?

[budney]

Same result: connection timed out.

[ChristianCiach]

I guess you are referring to my comment #3915 (reply in thread).

While I can say with confidence that I can connect to the IPs shown by kubectl get services -o wide from the host on all clusters that I've ever set up using k3s (which are a lot), this is not the way you usually should connect to your kubernetes services. You should probably better use a service of kind NodePort for this - or just use a Pod with hostPort as shown above..

Anyway, I strongly suspect some kind of iptables issue in your case, because I never had any issues connecting to service IPs.

[ChristianCiach]

Maybe it could help if you could show us exactly how you did it using docker.

[budney]

$ docker run -d --rm -e service=dnscache budney/djbdns
5db76c977a386a729ecc03e86c1cf10ca79768b94a327d4d87cf65ba0d1cac00

$ docker exec -it 5db76c977a38 /bin/sh
/ # echo nameserver 127.0.0.1 > /etc/resolv.conf
/ # dnsip cnn.com
151.101.1.67 151.101.65.67 151.101.129.67 151.101.193.67

/ # echo nameserver `ifconfig eth0 | sed -nr '/.*inet addr:1/s/^.*inet addr:([^ ]+).*$/\1/p'` > /etc/resolv.conf
/ # dnsip cnn.com
151.101.65.67 151.101.193.67 151.101.1.67 151.101.129.67

[ChristianCiach]

I am sorry, I totally skipped over this part:

In other words the DNS server inside the container is binding all interfaces (0.0.0.0), but cannot receive any connections on eth0, even from within the container itself.

This is very interesting and I never had this kind of issue. I think this is the core issue here, and everything I've said about host ports and stuff is irrelevant.

Edit: Please give me a few minutes to reproduce exactly your setup.

[budney]

This is all running on Ubuntu 21.04, with image ubuntu-21.04-preinstalled-server-arm64+raspi.img.xz downloaded from here.

[brandond]

dnscache doesn't respond to anything other than 127.0.0.1 by default. As per the docs:

dnscache listens for incoming UDP packets and TCP connections addressed to port 53 of $IP. Typically $IP is 127.0.0.1, but it can also be an externally accessible IP address. dnscache accepts a packet or connection from IP address 1.2.3.4 if it sees a file named ip/1.2.3.4 or ip/1.2.3 or ip/1.2 or ip/1.

[ChristianCiach]

10.42.1.1 is the IP of the cni0 device of my first k3s agent.

[budney]

Now that's interesting! Does k8s fiddle with the networking inside the container in order to make cluster magic work?Now that's interesting! Does k8s fiddle with the networking inside the container in order to make cluster magic work?

[ChristianCiach]

That being said, I don't understand why I cannot connect to this service using the Service (ClusterIP or LoadBalancer) or hostPort. I am not completely convinced that the dnscache really binds to all interfaces, even though it seems to be the case when using Docker.

Now I am really intrigued.

[brandond]

I think it binds to 0.0.0.0 no matter what, but it only responds to requests from ip/x?

[ChristianCiach]

I was just about to write just that. The documentation is quite clear about this, actually. I don't know why it works with Docker, but let me try this with K3s.

[ChristianCiach]

I got it!

Look at the files inside /srv/dnscache/root/ip:

Docker:

/srv/dnscache/root/ip # ls
127.0.0.1  192

K3s:

/srv/dnscache/root/ip # ls
10?10      127.0.0.1

As @brandond correctly stated in #3915 (comment), these file names describe the prefixes of the ip addresses that the dnscache process responds to.

So, where does the broken 10?10 come from? Actually from the entrypoint script at /start.sh, which does this:

        NET=$( netstat -rn | egrep UG | awk '{print $2}' | cut -f1 -d. )
        touch "/srv/$SERVICE/root/ip/$NET"`

The issue is that the first line actually returns 10 twice in K3s, with a line break in between.

So, in the end it comes down to a buggy entrypoint script of this dnscache image.

Edit: Running mv /srv/dnscache/root/ip/10?10 /srv/dnscache/root/ip/10 immediately fixes the issue.

[budney]

Oh my goodness. Thank you for spotting that! I've built new docker images and now the service works perfectly. Switching it to LoadBalancer also makes it accessible from the node's IP address.

Thanks!