Add ADR about failed multus integration #9434

manuelbuil · 2024-02-08T13:55:16Z

Proposed Changes

I worked for a while in the integration of k3s and Multus but it was pretty complicated. This ADR tries to explain what was tried and why I finally gave up

Types of Changes

New feature

Verification

Testing

Linked Issues

User-Facing Change

Further Comments

brandond

LGTM. I usually just write it up in the final format the ADR will be accepted in, so marked as Approved, and with the Design suggestion section as if it was the Decision... so that people can approve and you don't need to update it again and get another round of approvals just to do minor rewording.

brandond · 2024-02-10T00:21:44Z

docs/adrs/multus.md

+
+The suggestion is to include a new `--multus` boolean flag. When that flag gets activated, we deploy a Kubernetes job that will run in that node and will:
+* Run the [cni-plugins image](https://github.com/rancher/image-build-cni-plugins) which will deploy in the node all the widely used CNI binaries (e.g. macvlan), the multus binary and the whereabouts binary
+* Install the required multus CRD `network-attachment-definitions.k8s.cni.cncf.io` and the whereabouts CRDs `ippools.whereabouts.cni.cncf.io` & `overlappingrangeipreservations.whereabouts.cni.cncf.io`


I would probably do like we do with disabled components in RKE2, and ship these as a packaged manifest, and then just add them to the --disable list if --multus isn't set.

brandond · 2024-02-10T00:26:26Z

docs/adrs/multus.md

+* Create the appropriate Multus CNI config file and place it in the right path
+* Start the whereabouts controller
+
+These tasks are more or less what the upstream Multus and whereabouts daemonset do but, at least for Multus, it seems too overkill to have a daemonset running forever just for deploying the binaries and creating the correct multus cni config. A job that runs the `cni-plugins image` once the nodes comes up seems like a better solution.


I'm not sure a Job is the best way to do this, given that that the process for assigning job pods to specific nodes is a bit clumsy. We would need to reinvent much of the logic around creating and managing job-node assignment, and this logic already exists in the daemonset controller. I think we should either drop the required files directly from the k3s supervisor process, or use a daemonset to ensure that the install pod gets run on every node.

if it is too clumsy, then perhaps it is best for k3s to do that job but then, I don't want to increase the size of k3s by adding too many binaries that most people won't use... I'm starting to wonder if perhaps the idea of that user that suggested to add "macvlan" to the cni multi-exec might be the best one :D

macvlan + multus + whereabouts +.... I'll have fun in the next days

Adding multus+whereabouts to the multi-exec is not worth it.:

Multus binary increases the size of k3s significantly. Given that only a handful of users will benefit from this, I don't think it makes sense

whereabouts binary has very very old dependencies. That project actually looks sort of unmaintained. Anyway, it would creep in a lot of CVEs and we don't want that

Open question: do we want to add the multus images to the core k3s airgap image list, or should we have an optional image tarball, like we do for rke2?

Good point, probably the latter makes more sense. What would be the disadvantages of it?

Disadvantage to adding an optional tarball is that we have to update build scripts, QA processes, and documentation to handle the optional tarballs. We have never had more than 1 for k3s. How big are the additional images?

yeah. To minimize that pain, I'd leave the same name for the current tarball k3s-airgap-images-${ARCH}.tar and the images file k3s-images.txt

codecov · 2024-02-15T18:23:51Z

Codecov Report

Attention: Patch coverage is 33.33333% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 43.77%. Comparing base (06b6444) to head (59ef7c7).

❗ Current head 59ef7c7 differs from pull request most recent head 916b122. Consider uploading reports for the commit 916b122 to get more accurate results

Files	Patch %	Lines
pkg/deploy/zz_generated_bindata.go	0.00%	12 Missing ⚠️
pkg/server/server.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9434      +/-   ##
==========================================
- Coverage   48.66%   43.77%   -4.89%     
==========================================
  Files         158      151       -7     
  Lines       14017    13580     -437     
==========================================
- Hits         6821     5945     -876     
- Misses       5906     6416     +510     
+ Partials     1290     1219      -71

Flag	Coverage Δ
e2etests	`39.29% <33.33%> (-7.21%)`	⬇️
inttests	`39.38% <33.33%> (?)`
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

brandond

the k3s-multus chart has the same amd64 nodeSelector issue as the rke2 chart. We should fix that before we add the chart to k3s, since we do fully support arm64.

https://github.com/k3s-io/k3s-charts/blob/main/charts/multus/4.0.201%2Bupv4.0.2-build2024020801/values.yaml#L81

manuelbuil · 2024-03-11T07:51:25Z

This PR is postponed for the April release

brandond · 2024-03-12T18:21:46Z

pkg/server/server.go

@@ -279,6 +287,7 @@ func stageFiles(ctx context.Context, sc *Context, controlConfig *config.Control)
 		"%{SYSTEM_DEFAULT_REGISTRY}%":     registryTemplate(controlConfig.SystemDefaultRegistry),
 		"%{SYSTEM_DEFAULT_REGISTRY_RAW}%": controlConfig.SystemDefaultRegistry,
 		"%{PREFERRED_ADDRESS_TYPES}%":     addrTypesPrioTemplate(controlConfig.FlannelExternalIP),
+		"%{DATA_DIR}%":                    CNIBinDir,


Lets not call this DATA_DIR - this could be confusing as normally that would refer to the value of the --data-dir flag.

Suggested change

"%{DATA_DIR}%": CNIBinDir,

"%{CNI_BIN_DIR}%": CNIBinDir,

There are other problems with injecting the server's CNI bin dir path into the chart like this though:

CNI bin dir is version dependent. The value will change whenever the server is upgraded, and will only work if all nodes are on the exact same build of k3s:

brandond@seago:~$ grep bin_dir /var/lib/rancher/k3s/agent/etc/containerd/config.toml bin_dir = "/var/lib/rancher/k3s/data/0bd989dbe82165818ee98da4cc70b043adabccc23827c9bbd24b1460d4f3f2e1/bin"

CNI bin dir is prefixed by --data-dir, which may be set to different values on different hosts.

You're going to need to find some way to set this on per-node basis, instead of injecting it via the chart. You could MAYBE get away with hardcoding it as /var/lib/rancher/k3s/data/current but this only fixes problem 1, not problem 2.

Thanks Brad, those are great points I did not think about

brandond · 2024-03-12T18:23:14Z

pkg/daemons/config/types.go

@@ -157,6 +157,7 @@ type CriticalControlArgs struct {
 	FlannelIPv6Masq       bool         `cli:"flannel-ipv6-masq"`
 	FlannelExternalIP     bool         `cli:"flannel-external-ip"`
 	EgressSelectorMode    string       `cli:"egress-selector-mode"`
+	Multus				  bool         `cli:"multus"`	


looks like this needs tidying

brandond · 2024-03-12T18:24:32Z

pkg/cli/server/server.go

@@ -397,6 +398,11 @@ func run(app *cli.Context, cfg *cmds.Server, leaderControllers server.CustomCont
 		serverConfig.ControlConfig.Disables["ccm"] = true
 	}

+	if !serverConfig.ControlConfig.Multus {


Do we require that the bundled multus be used alongside the embedded flannel? Should we raise an error if multus is enabled, but flannel backend is set to none?

brandond · 2024-03-12T18:24:50Z

manifests/multus.yaml

+    config:
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: %{DATA_DIR}%


Suggested change

binDir: %{DATA_DIR}%

binDir: %{CNI_BIN_DIR}%

Signed-off-by: Manuel Buil <[email protected]>

thomasferrandiz · 2024-02-12T16:54:29Z

docs/adrs/multus.md

+
+## Alternatives
+
+* Deploy multus and whereabouts like traefik, reusing the multus chart in rke2-charts. However, that will deploy a daemonset for both multus and whereabouts which will consume more resources


considering Brad's comment above about using a daemonset, this looks it would be the simplest way to deploy multus+whereabouts?

I think you are reading an old version of the ADR 🤔 . In the latest version, it is explained the reasons why this option is complicated. In the end, it was the option I tried

manuelbuil requested a review from a team as a code owner February 8, 2024 13:55

brandond reviewed Feb 10, 2024

View reviewed changes

manuelbuil force-pushed the multusADR branch 5 times, most recently from 8455315 to 5a04cb4 Compare February 15, 2024 17:07

manuelbuil changed the title ~~Multus adr~~ Integrate k3s and multus Feb 15, 2024

manuelbuil mentioned this pull request Feb 15, 2024

Add multus chart k3s-io/k3s-charts#9

Merged

manuelbuil force-pushed the multusADR branch 4 times, most recently from 5073222 to a2f3a96 Compare February 23, 2024 07:02

manuelbuil force-pushed the multusADR branch 5 times, most recently from 47541a7 to 1efc154 Compare March 6, 2024 14:46

dereknola previously approved these changes Mar 8, 2024

View reviewed changes

brandond requested changes Mar 8, 2024

View reviewed changes

caroline-suse-rancher mentioned this pull request Mar 8, 2024

Integrate k3s and multus #9710

Open

4 tasks

manuelbuil dismissed dereknola’s stale review via db299d8 March 11, 2024 13:42

manuelbuil force-pushed the multusADR branch 2 times, most recently from db299d8 to 59ef7c7 Compare March 11, 2024 14:52

manuelbuil requested review from brandond and dereknola March 12, 2024 09:52

brandond requested changes Mar 12, 2024

View reviewed changes

manuelbuil force-pushed the multusADR branch from 59ef7c7 to 70189b8 Compare April 15, 2024 17:19

Write ADR about failed multus integration

916b122

Signed-off-by: Manuel Buil <[email protected]>

manuelbuil force-pushed the multusADR branch from 70189b8 to 916b122 Compare April 15, 2024 17:20

manuelbuil changed the title ~~Integrate k3s and multus~~ Add ADR about failed multus integration Apr 15, 2024

thomasferrandiz approved these changes Apr 16, 2024

View reviewed changes

brandond mentioned this pull request Sep 9, 2024

CNI bin dir changes with K3s version #10869

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ADR about failed multus integration #9434

Add ADR about failed multus integration #9434

manuelbuil commented Feb 8, 2024 •

edited

Loading

brandond left a comment

brandond Feb 10, 2024 •

edited

Loading

brandond Feb 10, 2024

manuelbuil Feb 12, 2024

manuelbuil Feb 12, 2024

manuelbuil Feb 13, 2024

brandond Feb 15, 2024

manuelbuil Feb 16, 2024 •

edited

Loading

brandond Feb 19, 2024 •

edited

Loading

manuelbuil Feb 23, 2024 •

edited

Loading

codecov bot commented Feb 15, 2024 •

edited

Loading

brandond left a comment •

edited

Loading

manuelbuil commented Mar 11, 2024

brandond Mar 12, 2024

brandond Mar 12, 2024

manuelbuil Mar 13, 2024

brandond Mar 12, 2024

brandond Mar 12, 2024

brandond Mar 12, 2024

thomasferrandiz Feb 12, 2024

manuelbuil Apr 16, 2024 •

edited

Loading


		## Alternatives

		* Deploy multus and whereabouts like traefik, reusing the multus chart in rke2-charts. However, that will deploy a daemonset for both multus and whereabouts which will consume more resources

Add ADR about failed multus integration #9434

Are you sure you want to change the base?

Add ADR about failed multus integration #9434

Conversation

manuelbuil commented Feb 8, 2024 • edited Loading

Proposed Changes

Types of Changes

Verification

Testing

Linked Issues

User-Facing Change

Further Comments

brandond left a comment

Choose a reason for hiding this comment

brandond Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manuelbuil Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

brandond Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

manuelbuil Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Feb 15, 2024 • edited Loading

Codecov Report

brandond left a comment • edited Loading

Choose a reason for hiding this comment

manuelbuil commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manuelbuil Apr 16, 2024 • edited Loading

Choose a reason for hiding this comment

manuelbuil commented Feb 8, 2024 •

edited

Loading

brandond Feb 10, 2024 •

edited

Loading

manuelbuil Feb 16, 2024 •

edited

Loading

brandond Feb 19, 2024 •

edited

Loading

manuelbuil Feb 23, 2024 •

edited

Loading

codecov bot commented Feb 15, 2024 •

edited

Loading

brandond left a comment •

edited

Loading

manuelbuil Apr 16, 2024 •

edited

Loading