Is there a way to Helm uninstall RKE2/Cilium to install latest Cilium/Cilium snapshot to test new features on cluster

[Jamison1]

Cilium v1.14 snapshot 2 has wireguard support for node-to-node encryption and I would like to test that out on my cluster.

Since rke2/cilium is integrated at install - does rke2 support helm uninstall rke2-cilium and then helm install cilium cilium/cilium --version cilium-1.14.0-snapshot.2 or something similar?

Before I potentially frag the cluster or spend hours troubleshooting any issues, I wanted to check with others on this.

Features I want to test on rke2 from cilium PR notes:

This PR adds support for node-to-node encryption to WireGuard. To achieve this, we've completely changed the WireGuard integration in the datapath. Previously, WireGuard support was implemented by marking packets to be encrypted in "from-container" and redirecting it to the WireGuard tunnel via a hostns IP rule. This worked fine for traffic originating in pods - but for node-to-node traffic, we need to redirect the packets on the outgoing network interface. Thus, the new implementation attaches bpf_host to the outgoing device and redirects packets to the WireGuard tunnel from there. See commit descriptions for more details.

On the agent side, there are also changes to the implementation. Previously, the datapath assumed that any IPCache entry with an associated tunnel endpoint would need encryption. To determine if we need to encryption traffic to a remote endpoint, we now rely on the encrypt_key field instead. This allows us to more precisely track if traffic to a particular destination needs to be encrypted, and allows certain nodes to opt out of encryption (see below). The agent code has been updated to populate the CiliumEndpoint and CiliumNode CRDs with a static non-zero EncryptKey value if encryption for those resources is enabled.

Additional points worth noting:

This new implementation now supports L7 proxy with WireGuard (due to it not requiring skb marks pre-encryption)
The agent opts out of node-to-node encryption if it is running on a kube-apiserver control plane node. This is done to
ensure worker nodes are always able to communicate with the kube-apiserver node, in order to be able to manage their own encryption key. See docs for more details.
CI is handled in the new CI-Datapath suite via Cilium-CLI.

[brandond]

I would not recommend uninstalling the CNI on a running cluster if you want to avoid breaking it. You could try upgrading the existing chart if you have a modified version that includes the changes you're looking for, or providing a HelmChartConfig that changes the images to point at snapshot versions, assuming there are no other changes necessary to use the new version.

Alternately, you could just spin up a new cluster with --cni=none and deploy the upstream chart to a clean cluster.

[Jamison1]

When installing upstream cilium with helm and using Cilium gateway api you need to set a variable for the cilium IPAM clusterpool IP address like this (make sure it doesnt match the external CIDR):

ipam:
    operator:
      clusterPoolIPv4PodCIDRList: '10.50.2.0/16

Otherwise it will set an internal CIDR managed by Cilium that in my case matched the external CIDR managed externally.
When coredns went to ContainerCreating it caused the node to freeze.

Potential Issue with rke2-cilium and gateway api:

As I was looking to use the gateway api with rke2-cilium. This requires that 4 CRD's be installed to the server node before cilium starts up for the first time. I tried copying all 4 CRD's to the manifests folder to have them install with essentially kubectl -f CRD_file.yaml but its hit or miss which ones get installed before cilium spins up. sometimes none are installed before and sometimes one is.

Unfortunately, I havent found a way to force the CRD installation from the manifests folder before rke2-cilium is installed.

The 4 CRD's for cilium gateway api (latest version)

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.6.2/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.6.2/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.6.2/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.6.2/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml

Check if all 4 CRD's are installed before installing Cilium with gateway api:

kubectl get crd gatewayclasses.gateway.networking.k8s.io
kubectl get crd gateways.gateway.networking.k8s.io
kubectl get crd httproutes.gateway.networking.k8s.io
kubectl get crd referencegrants.gateway.networking.k8s.io
NAME                                       CREATED AT
gatewayclasses.gateway.networking.k8s.io   2023-05-06T19:38:22Z
NAME                                 CREATED AT
gateways.gateway.networking.k8s.io   2023-05-06T19:38:22Z
NAME                                   CREATED AT
httproutes.gateway.networking.k8s.io   2023-05-06T19:38:23Z
NAME                                        CREATED AT
referencegrants.gateway.networking.k8s.io   2023-05-06T19:38:23Z