reuse of existing ingress load balancer fails in a weird way

[jampy]

I've noticed a problem of hetzner-k3s in combination with ingress-nginx. I'm not sure wether this is a problem of hetzner-k3s or the
hetznercloud/csi-driver, so I'm posting it here.

Basically the problem is as follows:

1. create cluster with hetzner-k3s create, without any resources pre-existing
2. configure ingress as described in the Wiki (steps 7 - 11); load balancer is created and becomes healthy
3. destroy cluster using hetzner-k3s delete (the ingress load balancer is not deleted)
4. repeat steps 1 and 2
5. new targets for the load balancer won't be set up, ingress dysfunctional, without any targets

This obviously means that recreating a cluster will leave it in a bad shape. I also wonder what will happen when nodes are added/replaced during the lifetime of a cluster since the targets configuration apparently is not very stable.

Detailed description to reproduce

I'm using hcloud-k3s Version 1.1.4 (even if --version reports 1.1.3, but that's expected).

My cluster_config.yaml:

---
cluster_name: lb-targets-test-cluster
kubeconfig_path: "../kubeconfig"
k3s_version: v1.28.2+k3s1

public_ssh_key_path: "~/.ssh/id_ed25519.pub"
private_ssh_key_path: "~/.ssh/id_ed25519"

use_ssh_agent: false
ssh_allowed_networks:
  - 0.0.0.0/0
api_allowed_networks:
  - 0.0.0.0/0
private_network_subnet: 10.0.0.0/16
disable_flannel: false
schedule_workloads_on_masters: false

cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.18.0/ccm-networks.yaml"

# NOTE: I've also tried version 2.5.1 instead of 2.4.0
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.4.0/deploy/kubernetes/hcloud-csi.yml"

system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"

masters_pool:
  instance_type: cpx21
  instance_count: 3
  location: nbg1
worker_node_pools:
  - name: small-static
    instance_type: cpx21
    instance_count: 4
    location: fsn1

My ingress-nginx-annotations.yaml:

controller:
  kind: DaemonSet
  service:
    annotations:
      load-balancer.hetzner.cloud/location: nbg1
      load-balancer.hetzner.cloud/name: lb-being-tested
      load-balancer.hetzner.cloud/use-private-ip: "true"
      load-balancer.hetzner.cloud/uses-proxyprotocol: 'true'
      load-balancer.hetzner.cloud/http-redirect-https: 'false'

      load-balancer.hetzner.cloud/hostname: doesnt.matter.example.com

I create the cluster using hetzner-k3s create --config cluster_config.yaml and that works fine, a total of 7 servers is started and the lb-targets-test-cluster-api load balancer is started/configured and is healthy.

Then I configure the Ingress and it's Load Balancer:

helm upgrade --install \
  ingress-nginx ingress-nginx/ingress-nginx \
  -f ./ingress-nginx-annotations.yaml \
  --namespace ingress-nginx \
  --create-namespace

This creates the lb-being-tested load balancer, which after a while becomes healthy.

Then I destroy the cluster using hetzner-k3s delete --config cluster_config.yaml. All servers are being destroyed, the API load balancer is deleted but the lb-being-tested load balancer remains (as expected).

Then I re-create the cluster using hetzner-k3s create --config cluster_config.yaml again, which brings up (different) Hetzner servers, of course. The API load balancer is recreated and becomes healthy again.

Then I install ingress-nginx using Helm, just like before.

I would expect that the existing Load Balancer (and, most importantly, it's public IP address) is reused. However, no targets are configured on the load balancer, thus it doesn't become healthy. So, no ingress.

Side note: weird LB behavior after deleting the cluster

When deleting the cluster using hetzner-k3s without manually removing targets from the load balancer, the lb-being-tested load balancer configuration is left as-is (as it is not managed by hetzner-k3s). Targets disappear very slowly (supposedly garbage collected by Hetzner itself) but the Hetzner UI shows inconsistent information about the targets, see here:

Here it says "4 servers" but also "no targets".

Workaround

The problem is avoided by manually removing all targets from the load balancer, before deleting the cluster.

for id in $(hcloud load-balancer describe lb-being-tested -o json | jq '.targets[].server.id') ; do hcloud load-balancer remove-target lb-being-tested --server "$id" ; done

Note that even if targets are not manually removed, the load balancer will eventually show "no targets" automatically after deleting the cluster. This takes a few minutes(!). However, the load balancer is still not reconfigured properly after re-creating the cluster.

Detailed commands, logs

See here: https://pastebin.com/raw/XLe85iNr

[vitobotta]

Hi, I just tested twice and I wasn't able to reproduce your issue. Each time I deleted the cluster, the targets were removed from the load balancer as soon as the servers were deleted as expected. When I recreated the cluster and reinstalled ingress-nginx, the existing load balancer was reused just fine and the new targets were added. Seems to work correctly for me. Perhaps you could raise this on the Hetzner Cloud Controller Manager project .

[derlinuxer]

It should work like a charm. I use it very often, because I lock my LBs to not loose the public IP (like you).
I've a bit different controller config (type: LoadBalance):

controller:
  config:
    use-proxy-protocol: "true"
  service:
    type: LoadBalancer
    annotations:
      load-balancer.hetzner.cloud/name: my-name
      load-balancer.hetzner.cloud/location: fsn1
      load-balancer.hetzner.cloud/use-private-ip: true
      load-balancer.hetzner.cloud/algorithm-type: least_connections
      load-balancer.hetzner.cloud/uses-proxyprotocol: true
      load-balancer.hetzner.cloud/hostname: my-hostname

But I don't think this will make any difference.
I'll try it again, my be an issue with current version of hetzner cloud controller.
See this Issue (and there are more):
hetznercloud/hcloud-cloud-controller-manager#521

Just checked my hcloud-controller-manager version. I use 1.16.0.

[vitobotta]

I'll convert this to a discussion since it doesn't seem to be a problem with hetzner-k3s. Thanks @derlinuxer for confirming it works for you too :)

[jampy]

Okay, many thanks for the replies.

I opened a support ticket (#2023100903011459) at Hetzner including a screen recording showing exactly what's happening, so they hopefully can find out what's wrong.

@derlinuxer Just to understand this better: Have you effectively tested the configuration shown here or are you just referring to your experience with your configurations?

[jampy]

Thanks for testing this. Can you tell me how/where I can see the logs showing errors you are talking about?

[derlinuxer]

Hi, yes. I mean the logs of the hcloud-cloud-controller-manager pod (pod-name: hcloud-cloud-controller-manager-xxxxxxx). There you can also find the communication against the hetzner api.

[jampy]

Hmm, strange, I don't see anything interesting in these logs: https://gist.github.com/jampy/766ae7a7e44ccbcdc7c2811ce69ffd3b

[derlinuxer]

Did you started helm install of ingress controller? There should be log entries with "load_balancer.go ..." where you see add targets and services.

[jampy]

Yes. First hetzner-k3s create ... then helm upgrade --install ....

Anyway I just did the same steps (recreating the cluster) and tadaa!... this time I see errors regarding a network ID not existing: https://gist.githubusercontent.com/jampy/448671c392595a79c96992673e4d02f7/raw/3ceba2f06612f36fc8c0c91e08df14123cd7cdd0/gistfile1.txt

[jampy]

FYI, Hetzner just responded that they assume a bug on their side and they are investigating it.

[vitobotta]

Nice, glad you got a confirmation from them :)

[jampy]

Hetzner informed me today that they deployed fixes that should resolve the problem.

Indeed, I can no longer reproduce the issue 🎉

Very nice :-)

[vitobotta]

I haven't tried the latest version yet

[jampy]

Now I'm getting confused. I meant hetzner-k3s release 1.1.4.

I'm not sure what you meant..?

[vitobotta]

Oh I meant the CCM 😄

[jampy]

Ah, ok. I did not upgrade any software - apparently Hetzner modified something within the infrastructure itself

[vitobotta]

Oh I see. Thanks

[vitobotta]

@jampy FYI I replied to your email yesterday but it seems like there is a problem with your mail server because of timeouts :)

[vitobotta]

can anyone help with testing rc1 of v2.0.0? see #385 for details