Tech debt: BackendTLSPolicy applied to a MinIO instance with certificates issued by Kubernetes CA

[ferdinandosimonetti]

Description:

Given this MinIO installation

[KS-Farmhub-admin|minio-dev] ➜  k8s git:(feature/envoygateway) ✗ k get po,sts,svc
NAME                      READY   STATUS    RESTARTS   AGE
pod/blob-store-pool-0-0   2/2     Running   0          50d
pod/blob-store-pool-0-1   2/2     Running   0          50d

NAME                                 READY   AGE
statefulset.apps/blob-store-pool-0   2/2     393d

NAME                          TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)             AGE
service/blob-store-console    ClusterIP      10.0.29.88     <none>         9443/TCP            393d
service/blob-store-hl         ClusterIP      None           <none>         9000/TCP,8022/TCP   393d
service/blob-store-sftp-svc   LoadBalancer   10.0.235.124   10.100.60.11   8022:32527/TCP      256d
service/minio                 ClusterIP      10.0.175.229   <none>         443/TCP             393d

and the Subject of the certificates served by MinIO Pods

[KS-Farmhub-admin|minio-dev] ➜  k8s git:(feature/envoygateway) ✗ k view-secret blob-store-tls public.crt|openssl x509 -subject
subject=O=system:nodes, CN=system:node:*.blob-store-hl.minio-dev.svc.cluster.local
-----BEGIN CERTIFICATE-----
MIIERzCCAi+gAwIBAgIRAPiNfnUTtLtIHn2PJ6w2o+owDQYJKoZIhvcNAQELBQAw
DTELMAkGA1UEAxMCY2EwHhcNMjQwMzExMTY1NTI2WhcNMjUwMzExMTY1NTI2WjBZ
MRUwEwYDVQQKEwxzeXN0ZW06bm9kZXMxQDA+BgNVBAMMN3N5c3RlbTpub2RlOiou
YmxvYi1zdG9yZS1obC5taW5pby1kZXYuc3ZjLmNsdXN0ZXIubG9jYWwwWTATBgcq
hkjOPQIBBggqhkjOPQMBBwNCAATy0c6nTxiRUEUDcnUQ3xm+K8zOFiHIGj788U/p
lDwoI4a5cfGcZrREvGr3vCQa4IkyxWkrsVltnDcnp9lANfc1o4IBHzCCARswDgYD
VR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAw
geUGA1UdEQSB3TCB2oJDYmxvYi1zdG9yZS1wb29sLTAtezAuLi4xfS5ibG9iLXN0
b3JlLWhsLm1pbmlvLWRldi5zdmMuY2x1c3Rlci5sb2NhbIIhbWluaW8ubWluaW8t
ZGV2LnN2Yy5jbHVzdGVyLmxvY2Fsgg9taW5pby5taW5pby1kZXaCE21pbmlvLm1p
bmlvLWRldi5zdmOCKyouYmxvYi1zdG9yZS1obC5taW5pby1kZXYuc3ZjLmNsdXN0
ZXIubG9jYWyCHSoubWluaW8tZGV2LnN2Yy5jbHVzdGVyLmxvY2FsMA0GCSqGSIb3
DQEBCwUAA4ICAQBIX3EiUNT5Sp6bmRzWODbj4yJY/cQzmgQmW+iq08r/NerwAf3n
RB4yvM97i+r9YBtN82QXtF2NvdWUYwRlhIqhYOlwhQDWT7984UqBQo307/bBwRNO
i/K1GeUkREhS1Z3AehVosdnlM1ZdJk8v9hYNNh+n/jpU6yi4m4Bz6ClHOvF+fedq
26R5rkJKK375fsA9RTh1anYuRj+eKzr5ZwLtbulvcKDWQ0oKEBK4biKvX3zfRLr9
Cejvmhvfu0J8rJfOMfgmpiW3rKWAZ7rN37pVe1duRBXargvSXjM3wqJ0eotyMCuq
nbip9FfYK+C03sKH5AC4B0hHXBrMxf8TLjvLc5GzdmAK8ZFEIWgS6ZfOBV+At/NO
keDjIeeCmgUBHfSj9TzShnakcN7ViQJG3jMMYGq9nEk0HJJGo8kyQlzmXN2WpaaT
U3k4zBLYJOhTWN7exVQDqVozD/wF52hHZ0mxrA/xSMUepWsISGyoz50/RtNCYYKj
vIcOaOew2C0KgmYWBuKo6ncbGH0AR9zp5jyYmVIFChPjIukNUPzdyDkr/mXM9ArP
dT943UwfvUhmwj9p30sKn8Wk1t0n653pAe4jzZ0pfXqypfQ9fP6axfqPPyF9QzjO
TuNbvkzWypo6bfjoymtmxN0o8T0XXfKAtCscQ7BbLOy/yM3x1S2pmBV4jw==
-----END CERTIFICATE-----

as well as the Subject Alternative Names

[KS-Farmhub-admin|minio-dev] ➜  k8s git:(feature/envoygateway) ✗ k view-secret blob-store-tls public.crt|openssl x509 -text|grep -A1 Alternative
            X509v3 Subject Alternative Name: 
                DNS:blob-store-pool-0-{0...1}.blob-store-hl.minio-dev.svc.cluster.local, DNS:minio.minio-dev.svc.cluster.local, DNS:minio.minio-dev, DNS:minio.minio-dev.svc, DNS:*.blob-store-hl.minio-dev.svc.cluster.local, DNS:*.minio-dev.svc.cluster.local

I am still unable to build an effective BackendTLSPolicy, my last tentative has been

---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: minio-console
  namespace: minio-dev
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: blob-store-console
  validation:
    wellKnownCACertificates: "System"
    hostname: minio.minio-dev.svc.cluster.local

Alternatively, it would be acceptable even disabling the certificate verification only for the HTTPRoutes regarding MinIO (not at a Gateway level).
But this, too, I don't know how to achieve.

Log excerpt:

[2024-11-22 17:57:10.483][13][debug][http] [source/common/http/conn_manager_impl.cc:393] [Tags: "ConnectionId":"34813"] new stream
[2024-11-22 17:57:10.483][13][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] request headers complete (end_stream=true):
':method', 'GET'
':authority', 'minio-dev.farmhub.nadara.com'
':scheme', 'https'
':path', '/'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'sec-ch-ua', '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"'
'sec-ch-ua-mobile', '?0'
'sec-ch-ua-platform', '"macOS"'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7'
'sec-fetch-site', 'none'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br, zstd'
'accept-language', 'en-US,en;q=0.9'
'priority', 'u=0, i'

[2024-11-22 17:57:10.483][13][debug][http] [source/common/http/conn_manager_impl.cc:1166] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] request end stream timestamp recorded
[2024-11-22 17:57:10.483][13][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"34813"] current connecting state: false
[2024-11-22 17:57:10.483][13][debug][router] [source/common/router/router.cc:527] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] cluster 'httproute/minio-dev/minio-console-route/rule/0' match for URL '/'
[2024-11-22 17:57:10.483][13][debug][router] [source/common/router/router.cc:756] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] router decoding headers:
':method', 'GET'
':authority', 'minio-dev.farmhub.nadara.com'
':scheme', 'https'
':path', '/'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'sec-ch-ua', '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"'
'sec-ch-ua-mobile', '?0'
'sec-ch-ua-platform', '"macOS"'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7'
'sec-fetch-site', 'none'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br, zstd'
'accept-language', 'en-US,en;q=0.9'
'priority', 'u=0, i'
'x-forwarded-for', '10.100.244.222'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '957eb903-4f91-4046-8885-0aa453422948'

[2024-11-22 17:57:10.483][13][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2024-11-22 17:57:10.483][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection
[2024-11-22 17:57:10.483][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2024-11-22 17:57:10.483][13][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"34949"] current connecting state: true
[2024-11-22 17:57:10.483][13][debug][client] [source/common/http/codec_client.cc:57] [Tags: "ConnectionId":"34949"] connecting
[2024-11-22 17:57:10.483][13][debug][connection] [source/common/network/connection_impl.cc:1017] [Tags: "ConnectionId":"34949"] connecting to 10.0.29.88:9443
[2024-11-22 17:57:10.484][13][debug][connection] [source/common/network/connection_impl.cc:1036] [Tags: "ConnectionId":"34949"] connection in progress
[2024-11-22 17:57:10.495][13][debug][connection] [source/common/network/connection_impl.cc:746] [Tags: "ConnectionId":"34949"] connected
[2024-11-22 17:57:10.496][13][debug][connection] [source/common/tls/cert_validator/default_validator.cc:321] verify cert failed: X509_verify_cert: certificate verification error at depth 0: unable to get local issuer certificate
[2024-11-22 17:57:10.496][13][debug][connection] [source/common/tls/ssl_socket.cc:246] [Tags: "ConnectionId":"34949"] remote address:10.0.29.88:9443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
[2024-11-22 17:57:10.496][13][debug][connection] [source/common/network/connection_impl.cc:276] [Tags: "ConnectionId":"34949"] closing socket: 0
[2024-11-22 17:57:10.496][13][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"34949"] disconnect. resetting 0 pending requests
[2024-11-22 17:57:10.496][13][debug][pool] [source/common/conn_pool/conn_pool_base.cc:495] [Tags: "ConnectionId":"34949"] client disconnected, failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
[2024-11-22 17:57:10.496][13][debug][router] [source/common/router/router.cc:1384] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] upstream reset: reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end
[2024-11-22 17:57:10.497][13][debug][http] [source/common/http/filter_manager.cc:1084] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] Sending local reply with details upstream_reset_before_response_started{remote_connection_failure|TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end}
[2024-11-22 17:57:10.497][13][debug][http] [source/common/http/conn_manager_impl.cc:1878] [Tags: "ConnectionId":"34813","StreamId":"4010610485661447579"] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '216'
'content-type', 'text/plain'
'date', 'Fri, 22 Nov 2024 17:57:10 GMT'

[ferdinandosimonetti]

Ok, guys.
Turned out I misunderstood completely the meaning of wellKnownCaCertificates validation option.

This

---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: minio
  namespace: minio
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: blob-store-console
  - group: ''
    kind: Service
    name: minio
  validation:
    caCertificateRefs:
    - name: kube-root-ca.crt
      group: ''
      kind: ConfigMap
    hostname: minio.minio.svc.cluster.local

was the answer to my problem, because the MinIO certificate was issued by the Kubernetes cluster's internal CA, whose certificate resides inside well-known kube-root-ca.crt ConfigMap.

And maybe someone else's, too: I hope that, by leaving this piece of YAML here, it could be of help.

I suggest, however, to put an example of usage of both wellKnowCaCertificates and the internal Kubernetes CA into the documentation.