add support for 'vmnet-*' networks in qemu with root privs #16339
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: prezha The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
k8s-ci-robot
added
size/L
@medyagh yes, you install qemu using brew with as for running qemu as root: you do not need to run anything upfront - ie, based on the platform, minikube will automatically pick the right qemu binary and then also run it as root (using sudo) for you i've amended pr to add more user-friendly handling of the password for sudo - there'll be two options:
|
|
@prezha thank you for this PR ! sorry for the long wait on the review I have two issues 1. qemu-system-aarch64 not in the path for sudo(I had to manually add /opt/homebrew/bin/ to "secure_path" in vi /etc/sudoers Issue 2. it goes stopped after I run minikube-guihttps://github.com/kubernetes-sigs/minikube-gui after a successful start (after adding the homebrew to secure_path) I can minikube ssh and I can minikube profile list and same for minikube status |
|
@medyagh thanks for the feedback! as for the issues you've faced, i, unfortunately, don't have a mac at hand to try to reproduce, but here are a couple of hints:
i don't remember i had to do anything in addition to
i suspect that it stopped because of the issue related to looking at the sudo password prompt you got in the previous step (under 1. qemu-system-aarch64 not in the path for sudo): |
|
@prezha can u find a solution for this problem ? |
@medyagh i used just the "official" one-liner to install qemu on m1 macos using homebrew unfortunately, i don't have access to that mac anymore to try to reproduce, but i also don't recall having the issue you saw on the other hand, i found a similar issue other users had with qemu installed with homebrew on macos that might help: so, based on that, in the case of the problem you've described, the steps could be:
|
and still same issue did u install it using brew command? |
| // vmnet network requires elevated privileges | ||
| if strings.HasPrefix(d.Network, "vmnet-") { | ||
| //TODO: handle windows | ||
| startProgram = "sudo" |
There was a problem hiding this comment.
@medyagh sorry for not coming back sooner to this, but i didn't have a mac at hand to test earlier - now i do :)
i was finally able to replicate the issue you described above (not sure what's the difference between this and prev mac i tried on and did not have that problem before)
i tried with -E (i guess you meant with the capital e) - ie, the preserve environment flag, but the issue remained
in the latest commit, i opted to retain the user's PATH explicitly, and it worked - can you please pull & try and let me know if that worked for you as well
before:
% minikube start --driver qemu --network vmnet-shared
😄 minikube v1.30.1 on Darwin 13.5 (arm64)
✨ Using the qemu2 driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...\
sudo password:-
sudo password:
sudo password: OUTPUT:
ERROR: Password:
sudo: qemu-system-aarch64: command not found
🔥 Deleting "minikube" in qemu2 ...
🤦 StartHost failed, but will try again: creating host: create: creating: Password:
sudo: qemu-system-aarch64: command not found: exit status 1
🔥 Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...|
sudo password:/
sudo password:
sudo password: OUTPUT:
ERROR: Password:
sudo: qemu-system-aarch64: command not found
😿 Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: Password:
sudo: qemu-system-aarch64: command not found: exit status 1
❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: Password:
sudo: qemu-system-aarch64: command not found: exit status 1
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
after:
% minikube start --driver qemu --network vmnet-shared
😄 minikube v1.30.1 on Darwin 13.5 (arm64)
✨ Using the qemu2 driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...\
sudo password:
❗ This VM is having trouble accessing https://registry.k8s.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳 Preparing Kubernetes v1.26.3 on Docker 20.10.23 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
|
Any news about this feature? I'm available to work on it if help is needed |
k8s-ci-robot
removed
the
needs-rebase
thanks @alessiodionisi ! the last time i checked this pr (a long time ago!) - it was working, i rebased it now can you please test yourself as well and see if there's anything additional we need to do |
|
I'm testing it and everything seems to work fine. I'll use it as a development environment in the next few days. |
k8s-ci-robot
added
the
do-not-merge/invalid-commit-message
k8s-ci-robot
removed
the
do-not-merge/invalid-commit-message
|
@alessiodionisi fyi: i've just synched with the current master |
|
/ok-to-test |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
These are the flake rates of all failed tests.
To see the flake rates of all tests by environment, click here. |
This comment has been minimized.
This comment has been minimized.
k8s-ci-robot
added
the
needs-rebase
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
do we still need this pr/functionality? |
|
Sorry, missed this PR, once the binary building fix is merged (#19862) we should rebase and re-run the tests. |
spowelljr
removed
the
lifecycle/stale
make check and exit message consistent with socket_vmnet Co-authored-by: Alessio Dionisi <[email protected]>
k8s-ci-robot
removed
the
needs-rebase
Co-authored-by: Steven Powell <[email protected]>
Co-authored-by: Steven Powell <[email protected]>
| @@ -473,6 +473,11 @@ func getNetwork(driverName string) string { | |||
| return n | |||
| } | |||
| switch n { | |||
| case "vmnet-host", "vmnet-shared", "vmnet-bridged": | |||
| //TODO: check if QEMU v7.1+ version is installed | |||
There was a problem hiding this comment.
Is there a required macOS version as well? Just wondering how far back the vmnet protocol is supported
There was a problem hiding this comment.
according to the apple docs: vmnet framework requires macOS 10.10+
There was a problem hiding this comment.
just tried it on sonoma 14.7
$ minikube -p vmnet start --driver=qemu --network=vmnet-shared
😄 [vmnet] minikube v1.34.0 on Darwin 14.7 (arm64)
✨ Using the qemu2 driver based on user configuration
👍 Starting "vmnet" primary control-plane node in "vmnet" cluster
🔥 Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) .../
sudo password:
🐳 Preparing Kubernetes v1.31.1 on Docker 27.3.1 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring bridge CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "vmnet" cluster and "default" namespace by default
There was a problem hiding this comment.
That's quite old, not sure if thats worth adding a version check for
There was a problem hiding this comment.
yeah, ten years now, so probably not
|
kvm2 driver with docker runtime Times for minikube start: 51.4s 53.8s 50.5s 50.9s 51.5s Times for minikube ingress: 18.5s 19.0s 15.6s 15.0s 19.0s docker driver with docker runtime Times for minikube start: 24.3s 21.1s 24.1s 21.1s 24.1s Times for minikube ingress: 12.3s 12.3s 12.3s 12.8s 13.3s docker driver with containerd runtime Times for minikube (PR 16339) start: 23.0s 23.7s 19.9s 22.1s 22.9s Times for minikube ingress: 39.3s 38.8s 22.8s 23.3s 38.8s |
|
Here are the number of top 10 failed tests in each environments with lowest flake rate.
Besides the following environments also have failed tests:
To see the flake rates of all tests by environment, click here. |
this pr adds support for "vmnet-host", "vmnet-shared" and "vmnet-bridged" networks in qemu v7.1+ run with
elevated-root privileges(for *nix) via existing--networkflagnote:
--network vmnet-sharedis probably what most users would want to useexample run on M1 Pro (Ventura 13.3.1) with QEMU 7.2.1:
Accessing apps example run with
minikube service with tunnel:takes about 30sec to start (depending on how fast you type root pass):