kubernetes Fake Certificate

[saashqdev]

Hi Vito:

I followed this doc exactly: https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Setting%20up%20a%20cluster.md For some reason I always get the K8s Fake Cert and not my Letsencrypt one.

I'm with Cloudflare and here my clusterissuer.yaml:

metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-key
      selector:
        dnsZones:
        - 'saashq.org'
        - '*.saashq.org'
---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-key: <cloudflare-api-key>

and here's my helloworld:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: rancher/hello-world
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
spec:
  rules:
  - host: saashq.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80
  tls:
  - hosts:
    - saashq.org
    secretName: saashq-org-tls

It creates the cert and secret fine. But I keep getting the Kubernetes Fake Certificate. My services are as follows:

NAMESPACE       NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
cert-manager    cert-manager                         ClusterIP      10.43.78.234    <none>        9402/TCP                     28m
cert-manager    cert-manager-cainjector              ClusterIP      10.43.66.134    <none>        9402/TCP                     28m
cert-manager    cert-manager-webhook                 ClusterIP      10.43.150.212   <none>        443/TCP,9402/TCP             28m
default         hello-world                          ClusterIP      10.43.184.235   <none>        80/TCP                       14m
default         kubernetes                           ClusterIP      10.43.0.1       <none>        443/TCP                      41m
ingress-nginx   ingress-nginx-controller             LoadBalancer   10.43.212.233   saashq.org    80:30460/TCP,443:32358/TCP   38m
ingress-nginx   ingress-nginx-controller-admission   ClusterIP      10.43.209.17    <none>        443/TCP                      38m
kube-system     hcloud-csi-controller-metrics        ClusterIP      10.43.52.110    <none>        9189/TCP                     41m
kube-system     hcloud-csi-node-metrics              ClusterIP      10.43.91.108    <none>        9189/TCP                     41m
kube-system     kube-dns                             ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       41m

Everything looks fine but for some reason it won't go secure in the browser:

Is it that Letsencrypt going to port 80 on a Hetzner LB thing? My service is set do a domain name instead of an IP though like in your docs.

I created a test certificate and I got the following in the certificate logs (Issuing certificate as Secret does not exist):

...
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  saashq-org-tls-test-jj5nx
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    75s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  75s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "saashq-org-tls-test-jj5nx"
  Normal  Requested  75s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "saashq-org-tls-test-1"

Cheers,
Dave

[vitobotta]

Hi Dave,

I'm turning this into a discussion since it's not actually an issue with the tool itself.

If you're using Cloudflare in front of your web app, how did you set up the SSL profile for the site? If it's currently on "Flexible," try switching it to "Full" or "Full (Strict)".

[saashqdev]

Cool, I'll try that. btw I dug deeper into the cert order flow: kubectl describe challenge -n cert-manager

Status:
  Presented:   false
  Processing:  true
  Reason:      while attempting to find Zones for domain _acme-challenge.test.saashq.org.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.test.saashq.org" 
           Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header
  State:  pending
Events:
  Type     Reason        Age                From                     Message
  ----     ------        ----               ----                     -------
  Normal   Started       35m                cert-manager-challenges  Challenge scheduled for processing
  Warning  PresentError  13m (x9 over 35m)  cert-manager-challenges  Error presenting challenge: while attempting to find Zones for domain _acme-challenge.test.saashq.org.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.test.saashq.org" 
   Error: 6003: Invalid request headers<- 6111: Invalid format for Authorization header

[saashqdev]

oh my god --- I had the wrong auth-token/key for Cloudflare - aaaahhhh. I'll never ever get the naming subtleties with security stuff. key vs token