Customizing etcd parameters in rke2

[japereira1986]

Hello,

Is it possible to modify etcd setup parameters in rke2?

I am attempting to adjust both "ETCD_ADVERTISE_CLIENT_URLS" and "ETCD_INITIAL_ADVERTISE_PEER_URLS" in an airgapped environment (due to a slightly weird setup where our advertised ips need to be different from the node ips). Per the documentation, I am attempting to use etcd-extra-env to do this. The config below is our setup (using rke version 1.24.4+rke2r1)

write-kube-config-mode: "0644"
data-dir: /data/rancher/current
tls-san:
   -7.130.32.71
   -p-proj-rancher02.my.airgapped.dns
etcd-extra-env:
   - "ETCD_ADVERTISE_CLIENT_URLS=https://7.130.32.71:2379"
   - "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://7.130.32.71:2379"
advertise-address: 7.130.32.71
node-ip: 10.22.3.17
node-external-ip: 7.130.32.71

This configuration results in an etcd pod which does have both of those variables set in the environment section. Unfortunately, it doesn't seem to make a difference; in our etcd logs, our pod is still only advertising itself on 10.22.3.17 (leaving other pods unable to reach it).

When I consulted the etcd docs at https://etcd.io/docs/v3.5/op-guide/configuration/#configuration-file, I found the following warning:

 Caution: If you mix-and-match configuration options, then the following rules apply.

Command-line flags take precedence over environment variables.
If you provide a configuration file all command-line flags and environment variables are ignored.

That would seem to suggest to me that setting etcd-extra-env won't actually work- the default config file at (in our case) /data/rancher/current/server/db/config will overwrite anything we pass in.

Am I doing something wrong? Is there a different way to do this that I overlooked in the rke2 documentation? Is this simply not supported?

Appreciate any assistance! Apologies for the lack of logs- the nature of our environment makes it difficult to provide them, but hopefully the information I've provided is enough to describe what I'm encountering.

[brandond]

1. Etcd reads those settings from the config file, not the environment. You might try passing them through as extra args, not env variables.
2. We don't currently support directly overriding the address configuration; it is hard coded to use the internal IPs.

[japereira1986]

Thanks for the quick response! Do you anticipate this being something you'll support in the future, or is it not on the roadmap at this time?

[japereira1986]

Basically, the network we're operating off of has a few different availability zones (each of with its own set of private ips). We were hoping to set up a cluster with three server nodes and x agents with one server per availability zone. That way we'd be more tolerant to an availability zone going down.

The private ips aren't accessible across those zones, and we have minimum ability to do any sort of redirection from one zone to another. Each vm that we're installing a node on has its own attached public "routing" ip, so if we were able to change the advertising address, we'd be able to communicate between zones using those ips. But we can't actually "bind" to those routing ips because of the weird way the network is set up (basically the hypervisors and routers per availability zone know about them and redirect traffic, but the individual vms don't have them bound as a network interface).

Unfortunately this is an airgapped network that we're just tenants on, so we have minimal ability to make changes to the way it's structured. That's why we were hoping to make some configuration changes to see if we could work around the weird setup.

[brandond]

Each vm that we're installing a node on has its own attached public "routing" ip

Have you tried just using that "public" IP as the --node-ip?

[japereira1986]

We have, but since that "public" ip isn't a bound network interface, what seems to happen is that etcd tries and fails to bind itself on that ip plus port. As a result, the etcd containerd image fails to start.

This is the error in the etcd crictl logs (Bear with any typos since I had to copy this over from the airgapped network):

{"level":"warn", "ts":"2023-09-09T05:05:03.688Z","caller":"etcdomain/etcd.go:146","msg":"failed to start etcd","error":"listen tcp 7.130.32.71:2380: bind: cannot assign requested address"}

(we are working with some networking folks on our end to see if we can make that a bindable interface, but at the moment that's the barrier we're encountering with that approach)

[brandond]

Ah OK. So it's not actually attached - just NATed to the private IP? That would be what the --node-external-ip is meant to be used with, but we intentionally don't allow configuring etcd to use external IPs, to discourage folks from trying to run etcd over public or high-latency network segments.

[japereira1986]

Yeah, that makes sense...we did try setting the external ip, and everything worked for an agent, it's just when we tried making it a server instead it didn't work since etcd was still using the node-ip. Which makes sense given what you said above.

I think for now we're probably stuck with working with the owner of the airgapped cluster to see if we can make network adjustments, but I appreciate your time and responsiveness.