Installing on EKS with K8S 1.27 and IPv6 fails due to kube-rs openssl bug

[passcod]

What is the issue?

This is the equivalent to this issue: vectordotdev/vector#17679 for linkerd.

It is caused by this bug in kube-rs: kube-rs/kube#991 in conjunction with using the openssl backend (reportedly fixed with rustls).

How can it be reproduced?

When installing the control plane (I used the helm chart version 1.16.2) on a new AWS EKS cluster with Kubernetes version 1.27 and IPv6 enabled, the policy container of linkerd-destination fails with a "Hostname mismatch" error.

Logs, error output, etc

2023-09-27T22:48:41.161444Z TRACE mio::poll: registering event source with poller: token=Token(0), interests=READABLE | WRITABLE    
2023-09-27T22:48:41.161526Z TRACE tower::buffer::worker: worker polling for next message
2023-09-27T22:48:41.165208Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE | WRITABLE    
2023-09-27T22:48:41.165274Z TRACE tower::buffer::service: sending request to buffer worker
2023-09-27T22:48:41.165307Z TRACE tower::buffer::worker: worker polling for next message
2023-09-27T22:48:41.165359Z TRACE tower::buffer::worker: processing new request
2023-09-27T22:48:41.165370Z TRACE tower::buffer::worker: resumed=false worker received request; waiting for service readiness
2023-09-27T22:48:41.165375Z DEBUG tower::buffer::worker: service.ready=true processing request
2023-09-27T22:48:41.165383Z TRACE tower::buffer::worker: returning response future
2023-09-27T22:48:41.165386Z TRACE tower::buffer::worker: worker polling for next message
2023-09-27T22:48:41.165495Z DEBUG HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: kube_client::client::builder: requesting
2023-09-27T22:48:41.165534Z TRACE HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: hyper::client::pool: checkout waiting for idle connection: ("https", [fdef:bf01:f7b::1])
2023-09-27T22:48:41.165589Z TRACE HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("[fdef:bf01:f7b::1]"), port=None
2023-09-27T22:48:41.165693Z DEBUG HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: hyper::client::connect::http: connecting to [fdef:bf01:f7b::1]:443
2023-09-27T22:48:41.165870Z TRACE HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: mio::poll: registering event source with poller: token=Token(2), interests=READABLE | WRITABLE    
2023-09-27T22:48:41.166237Z DEBUG HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: hyper::client::connect::http: connected to [fdef:bf01:f7b::1]:443
2023-09-27T22:48:41.169841Z TRACE HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: mio::poll: deregistering event source from poller    
2023-09-27T22:48:41.169945Z TRACE HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client"}: hyper::client::pool: checkout dropped for ("https", [fdef:bf01:f7b::1])
2023-09-27T22:48:41.169986Z ERROR HTTP{http.method=GET http.url=https://[fdef:bf01:f7b::1]/apis/apps/v1/namespaces/linkerd/deployments/linkerd-destination otel.name="get" otel.kind="client" otel.status_code="ERROR"}: kube_client::client::builder: failed with error error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921:: Hostname mismatch
2023-09-27T22:48:41.170012Z TRACE mio::poll: deregistering event source from poller    
2023-09-27T22:48:41.170038Z TRACE mio::poll: deregistering event source from poller    
2023-09-27T22:48:41.170094Z TRACE tower::buffer::worker: buffer already closed
Error: HyperError: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921:: Hostname mismatch

output of linkerd check -o short

linkerd-existence
-----------------
× control plane pods are ready
    pod/linkerd-destination-dc9d8c4d8-n2fcp container policy is not ready
    see https://linkerd.io/2.14/checks/#l5d-api-control-ready for hints

Status check results are ×

Environment

• Kubernetes version: 1.27
• AWS EKS with default AWS CNI etc and IPv6 enabled
• Linkerd version: stable-2.14.1

Possible solution

From this comment: vectordotdev/vector#17679 (comment) this can be worked around by adding this to the container environment:

env:
  - name: KUBERNETES_SERVICE_HOST
    value: "kubernetes.default.svc"

I wasn't able to find how to do that natively with linkerd's helm chart without editing the templates locally.

Alternatively you could switch to the rustls backend?

Or take over this upstream PR to fix it in hyper-openssl.

Additional context

I'm currently testing the above workaround by editing the resources directly.
Update: that did seem to fix the immediate issue of the hostname mismatch.

Would you like to work on fixing this bug?

maybe

[passcod]

Oh, linkerd just plain doesn't support IPv6. Right.