diff --git a/compute/resources_requests_and_limits/index.html b/compute/resources_requests_and_limits/index.html
index 1fdae466..488b56ca 100644
--- a/compute/resources_requests_and_limits/index.html
+++ b/compute/resources_requests_and_limits/index.html
@@ -2927,7 +2927,7 @@ <h2 id="cpu">CPU<a class="headerlink" href="#cpu" title="Permanent link">&para;<
 <p>Note: dedicated CPUs (and isolated emulator thread) are ignored here as they have a <a href="../dedicated_cpu_resources/">dedicated page</a>.</p>
 <h3 id="cpu-requests-on-the-container">CPU requests on the container<a class="headerlink" href="#cpu-requests-on-the-container" title="Permanent link">&para;</a></h3>
 <ul>
-<li>By default, the container requests (1/cpuAllocationRatio) CPU per vCPU. The number of vCPUs is sockets<em>cores</em>threads, defaults to 1.</li>
+<li>By default, the container requests (1/cpuAllocationRatio) CPU per vCPU. The number of vCPUs is sockets * cores * threads, defaults to 1.</li>
 <li>cpuAllocationRatio defaults to 10 but can be changed in the CR.</li>
 <li>If a CPU limit is manually set on the VM(I) and no CPU request is, the CPU requests on the container will match the CPU limits</li>
 <li>Manually setting CPU requests on the VM(I) will override all of the above and be the CPU requests for the container</li>
diff --git a/search/search_index.json b/search/search_index.json
index ae407fb7..0b93251a 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-,:!=\\[\\]\\(\\)\"/]+|\\.(?!\\d)","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome","text":"<p>The KubeVirt User Guide is divided into the following sections:</p> <ul> <li>Architecture: Technical and conceptual overview of KubeVirt components</li> <li>Quickstarts: A list of resources to help you learn KubeVirt basics</li> <li>Cluster Administration: Cluster-level administration concepts and tasks</li> <li>User Workloads: Creating, customizing, using, and monitoring virtual machines </li> <li>Compute: Resource allocation and optimization for the virtualization layer</li> <li>Network: Concepts and tasks for the networking and service layers </li> <li>Storage: Concepts and tasks for the storage layer, including importing and exporting.</li> <li>Release Notes: The release notes for all KubeVirt releases</li> <li>Contributing: How you can contribute to this guide or the KubeVirt project </li> <li>Virtualization Debugging: How to debug your KubeVirt cluster and virtual resources</li> </ul>"},{"location":"#try-it-out","title":"Try it out","text":"<ul> <li> <p>Kubevirt on Killercoda: https://killercoda.com/kubevirt</p> </li> <li> <p>Kubevirt on Minikube: https://kubevirt.io/quickstart_minikube/</p> </li> <li> <p>Kubevirt on Kind: https://kubevirt.io/quickstart_kind/</p> </li> <li> <p>Kubevirt on cloud providers: https://kubevirt.io/quickstart_cloud/</p> </li> </ul>"},{"location":"#kubevirt-labs","title":"KubeVirt Labs","text":"<ul> <li> <p>Use KubeVirt</p> </li> <li> <p>Experiment with Containerized Data Importer (CDI)</p> </li> <li> <p>Experiment with KubeVirt Upgrades</p> </li> <li> <p>Live Migration</p> </li> </ul>"},{"location":"#getting-help","title":"Getting help","text":"<ul> <li> <p>File a bug: https://github.com/kubevirt/kubevirt/issues</p> </li> <li> <p>Mailing list: https://groups.google.com/forum/#!forum/kubevirt-dev</p> </li> <li> <p>Slack: https://kubernetes.slack.com/messages/virtualization</p> </li> </ul>"},{"location":"#developer","title":"Developer","text":"<ul> <li> <p>Start contributing: Contributing</p> </li> <li> <p>API Reference: http://kubevirt.io/api-reference/</p> </li> </ul>"},{"location":"#privacy","title":"Privacy","text":"<ul> <li> <p>Check our privacy policy at: https://kubevirt.io/privacy/</p> </li> <li> <p>We do use https://netlify.com Open Source Plan for Rendering Pull   Requests to the documentation repository</p> </li> </ul>"},{"location":"architecture/","title":"Architecture","text":"<p>KubeVirt is built using a service oriented architecture and a choreography pattern.</p>"},{"location":"architecture/#stack","title":"Stack","text":"<pre><code>  +---------------------+\n  | KubeVirt            |\n~~+---------------------+~~\n  | Orchestration (K8s) |\n  +---------------------+\n  | Scheduling (K8s)    |\n  +---------------------+\n  | Container Runtime   |\n~~+---------------------+~~\n  | Operating System    |\n  +---------------------+\n  | Virtual(kvm)        |\n~~+---------------------+~~\n  | Physical            |\n  +---------------------+\n</code></pre> <p>Users requiring virtualization services are speaking to the Virtualization API (see below) which in turn is speaking to the Kubernetes cluster to schedule requested Virtual Machine Instances (VMIs). Scheduling, networking, and storage  are all delegated to Kubernetes, while KubeVirt provides the virtualization functionality.</p>"},{"location":"architecture/#additional-services","title":"Additional Services","text":"<p>KubeVirt provides additional functionality to your Kubernetes cluster, to perform virtual machine management</p> <p>If we recall how Kubernetes is handling Pods, then we remember that Pods are created by posting a Pod specification to the Kubernetes API Server. This specification is then transformed into an object inside the API Server, this object is of a specific type or kind - that is how it's called in the specification. A Pod is of the type <code>Pod</code>. Controllers within Kubernetes know how to handle these Pod objects. Thus once a new Pod object is seen, those controllers perform the necessary actions to bring the Pod alive, and to match the required state.</p> <p>This same mechanism is used by KubeVirt. Thus KubeVirt delivers three things to provide the new functionality:</p> <ol> <li>Additional types - so called Custom Resource Definition (CRD) - are added to the Kubernetes API</li> <li>Additional controllers for cluster wide logic associated with these new types</li> <li>Additional daemons for node specific logic associated with new types</li> </ol> <p>Once all three steps have been completed, you are able to</p> <ul> <li>create new objects of these new types in Kubernetes (VMIs in our   case)</li> <li>and the new controllers take care to get the VMIs scheduled on some host,</li> <li>and a daemon - the <code>virt-handler</code> - is taking care of a host - alongside the   <code>kubelet</code> - to launch the VMI and configure it until it matches the required   state.</li> </ul> <p>One final note; both controllers and daemons are running as Pods (or similar) on top of the Kubernetes cluster, and are not installed alongside it. The type is - as said before - even defined inside the Kubernetes API server. This allows users to speak to Kubernetes, but modify VMIs.</p> <p>The following diagram illustrates how the additional controllers and daemons communicate with Kubernetes and where the additional types are stored:</p> <p></p> <p>And a simplified version:</p> <p></p>"},{"location":"architecture/#application-layout","title":"Application Layout","text":"<ul> <li>Cluster</li> <li>KubeVirt Components<ul> <li>virt-controller</li> <li>virt-handler</li> <li>libvirtd</li> <li>\u2026</li> </ul> </li> <li>KubeVirt Managed Pods<ul> <li>VMI Foo</li> <li>VMI Bar</li> <li>\u2026</li> </ul> </li> <li>KubeVirt Custom Resources<ul> <li>VirtualMachine (VM) Foo     -&gt; VirtualMachineInstance (VMI) Foo</li> <li>VirtualMachineInstanceReplicaSet (VMIRS) Bar     -&gt; VirtualMachineInstance (VMI) Bar</li> </ul> </li> </ul> <p>VirtualMachineInstance (VMI) is the custom resource that represents the basic ephemeral building block of an instance. In a lot of cases this object won't be created directly by the user but by a high level resource. High level resources for VMI can be:</p> <ul> <li>VirtualMachine (VM) - StateFul VM that can be stopped and started while keeping the VM data and state.</li> <li>VirtualMachineInstanceReplicaSet (VMIRS) - Similar to pods ReplicaSet, a group of ephemeral VMIs with similar configuration defined in a template.</li> </ul>"},{"location":"architecture/#native-workloads","title":"Native Workloads","text":"<p>KubeVirt is deployed on top of a Kubernetes cluster. This means that you can continue to run your Kubernetes-native workloads next to the VMIs managed through KubeVirt.</p> <p>Furthermore: if you can run native workloads, and you have KubeVirt installed, you should be able to run VM-based workloads, too. For example, Application Operators should not require additional permissions to use cluster features for VMs, compared to using that feature with a plain Pod.</p> <p>Security-wise, installing and using KubeVirt must not grant users any permission they do not already have regarding native workloads. For example, a non-privileged Application Operator must never gain access to a privileged Pod by using a KubeVirt feature.</p>"},{"location":"architecture/#the-razor","title":"The Razor","text":"<p>We love virtual machines, think that they are very important and work hard to make them easy to use in Kubernetes. But even more than VMs, we love good design and modular, reusable components. Quite frequently, we face a dilemma: should we solve a problem in KubeVirt in a way that is best optimized for VMs, or should we take a longer path and introduce the solution to Pod-based workloads too?</p> <p>To decide these dilemmas we came up with the KubeVirt Razor: \"If something is useful for Pods, we should not implement it only for VMs\".</p> <p>For example, we debated how we should connect VMs to external network resources. The quickest way seems to introduce KubeVirt-specific code, attaching a VM to a host bridge. However, we chose the longer path of integrating with Multus and CNI and improving them.</p>"},{"location":"architecture/#virtualmachine","title":"VirtualMachine","text":"<p>A <code>VirtualMachine</code> provides additional management capabilities to a VirtualMachineInstance inside the cluster. That includes:</p> <ul> <li> <p>API stability</p> </li> <li> <p>Start/stop/restart capabilities on the controller level</p> </li> <li> <p>Offline configuration change with propagation on     VirtualMachineInstance recreation</p> </li> <li> <p>Ensure that the VirtualMachineInstance is running if it should be     running</p> </li> </ul> <p>It focuses on a 1:1 relationship between the controller instance and a virtual machine instance. In many ways it is very similar to a StatefulSet with <code>spec.replica</code> set to <code>1</code>.</p>"},{"location":"architecture/#how-to-use-a-virtualmachine","title":"How to use a VirtualMachine","text":"<p>A VirtualMachine will make sure that a VirtualMachineInstance object with an identical name will be present in the cluster, if <code>spec.running</code> is set to <code>true</code>. Further it will make sure that a VirtualMachineInstance will be removed from the cluster if <code>spec.running</code> is set to <code>false</code>.</p> <p>There exists a field <code>spec.runStrategy</code> which can also be used to control the state of the associated VirtualMachineInstance object. To avoid confusing and contradictory states, these fields are mutually exclusive.</p> <p>An extended explanation of <code>spec.runStrategy</code> vs <code>spec.running</code> can be found in Run Strategies</p>"},{"location":"architecture/#starting-and-stopping","title":"Starting and stopping","text":"<p>After creating a VirtualMachine it can be switched on or off like this:</p> <pre><code># Start the virtual machine:\nvirtctl start vm\n\n# Stop the virtual machine:\nvirtctl stop vm\n</code></pre> <p><code>kubectl</code> can be used too:</p> <pre><code># Start the virtual machine:\nkubectl patch virtualmachine vm --type merge -p \\\n    '{\"spec\":{\"running\":true}}'\n\n# Stop the virtual machine:\nkubectl patch virtualmachine vm --type merge -p \\\n    '{\"spec\":{\"running\":false}}'\n</code></pre> <p>Find more details about a VM's life-cycle in the relevant section</p>"},{"location":"architecture/#controller-status","title":"Controller status","text":"<p>Once a VirtualMachineInstance is created, its state will be tracked via <code>status.created</code> and <code>status.ready</code> fields of the VirtualMachine. If a VirtualMachineInstance exists in the cluster, <code>status.created</code> will equal <code>true</code>. If the VirtualMachineInstance is also ready, <code>status.ready</code> will equal <code>true</code> too.</p> <p>If a VirtualMachineInstance reaches a final state but the <code>spec.running</code> equals <code>true</code>, the VirtualMachine controller will set <code>status.ready</code> to <code>false</code> and re-create the VirtualMachineInstance.</p> <p>Additionally, the <code>status.printableStatus</code> field provides high-level summary information about the state of the VirtualMachine. This information is also displayed when listing VirtualMachines using the CLI:</p> <pre><code>$ kubectl get virtualmachines\nNAME     AGE   STATUS    VOLUME\nvm1      4m    Running\nvm2      11s   Stopped\n</code></pre> <p>Here's the list of states currently supported and their meanings. Note that states may be added/removed in future releases, so caution should be used if consumed by automated programs.</p> <ul> <li>Stopped: The virtual machine is currently stopped and isn't expected to start.</li> <li>Provisioning: Cluster resources associated with the virtual machine (e.g., DataVolumes) are being provisioned and prepared.</li> <li>Starting: The virtual machine is being prepared for running.</li> <li>Running: The virtual machine is running.</li> <li>Paused: The virtual machine is paused.</li> <li>Migrating: The virtual machine is in the process of being migrated to another host.</li> <li>Stopping: The virtual machine is in the process of being stopped.</li> <li>Terminating: The virtual machine is in the process of deletion, as well as its associated resources (VirtualMachineInstance, DataVolumes, \u2026).</li> <li>Unknown: The state of the virtual machine could not be obtained, typically due to an error in communicating with the host on which it's running.</li> </ul>"},{"location":"architecture/#restarting","title":"Restarting","text":"<p>A VirtualMachineInstance restart can be triggered by deleting the VirtualMachineInstance. This will also propagate configuration changes from the template in the VirtualMachine:</p> <pre><code># Restart the virtual machine (you delete the instance!):\nkubectl delete virtualmachineinstance vm\n</code></pre> <p>To restart a VirtualMachine named vm using virtctl:</p> <pre><code>$ virtctl restart vm\n</code></pre> <p>This would perform a normal restart for the VirtualMachineInstance and would reschedule the VirtualMachineInstance on a new virt-launcher Pod</p> <p>To force restart a VirtualMachine named vm using virtctl:</p> <pre><code>$ virtctl restart vm --force --grace-period=0\n</code></pre> <p>This would try to perform a normal restart, and would also delete the virt-launcher Pod of the VirtualMachineInstance with setting GracePeriodSeconds to the seconds passed in the command.</p> <p>Currently, only setting grace-period=0 is supported.</p> <p>Note</p> <p>Force restart can cause data corruption, and should be used in cases of kernel panic or VirtualMachine being unresponsive to normal restarts.</p>"},{"location":"architecture/#fencing-considerations","title":"Fencing considerations","text":"<p>A VirtualMachine will never restart or re-create a VirtualMachineInstance until the current instance of the VirtualMachineInstance is deleted from the cluster.</p>"},{"location":"architecture/#exposing-as-a-service","title":"Exposing as a Service","text":"<p>A VirtualMachine can be exposed as a service. The actual service will be available once the VirtualMachineInstance starts without additional interaction.</p> <p>For example, exposing SSH port (22) as a <code>ClusterIP</code> service using <code>virtctl</code> after the VirtualMachine was created, but before it started:</p> <pre><code>$ virtctl expose virtualmachine vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>All service exposure options that apply to a VirtualMachineInstance apply to a VirtualMachine.</p> <p>See Service Objects for more details.</p>"},{"location":"architecture/#when-to-use-a-virtualmachine","title":"When to use a VirtualMachine","text":""},{"location":"architecture/#when-api-stability-is-required-between-restarts","title":"When API stability is required between restarts","text":"<p>A <code>VirtualMachine</code> makes sure that VirtualMachineInstance API configurations are consistent between restarts. A classical example are licenses which are bound to the firmware UUID of a virtual machine. The <code>VirtualMachine</code> makes sure that the UUID will always stay the same without the user having to take care of it.</p> <p>One of the main benefits is that a user can still make use of defaulting logic, although a stable API is needed.</p>"},{"location":"architecture/#when-config-updates-should-be-picked-up-on-the-next-restart","title":"When config updates should be picked up on the next restart","text":"<p>If the VirtualMachineInstance configuration should be modifiable inside the cluster and these changes should be picked up on the next VirtualMachineInstance restart. This means that no hotplug is involved.</p>"},{"location":"architecture/#when-you-want-to-let-the-cluster-manage-your-individual-virtualmachineinstance","title":"When you want to let the cluster manage your individual VirtualMachineInstance","text":"<p>Kubernetes as a declarative system can help you to manage the VirtualMachineInstance. You tell it that you want this VirtualMachineInstance with your application running, the VirtualMachine will try to make sure it stays running.</p> <p>Note</p> <p>The current belief is that if it is defined that the VirtualMachineInstance should be running, it should be running. This is different from many classical virtualization platforms, where VMs stay down if they were switched off. Restart policies may be added if needed. Please provide your use-case if you need this!</p>"},{"location":"architecture/#example","title":"Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-cirros\n  name: vm-cirros\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-cirros\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 64M\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - name: containerdisk\n        containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n      - cloudInitNoCloud:\n          userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK\n        name: cloudinitdisk\n</code></pre> <p>Saving this manifest into <code>vm.yaml</code> and submitting it to Kubernetes will create the controller instance:</p> <pre><code>$ kubectl create -f vm.yaml\nvirtualmachine \"vm-cirros\" created\n</code></pre> <p>Since <code>spec.running</code> is set to <code>false</code>, no vmi will be created:</p> <pre><code>$ kubectl get vmis\nNo resources found.\n</code></pre> <p>Let's start the VirtualMachine:</p> <pre><code>$ virtctl start vm vm-cirros\n</code></pre> <p>As expected, a VirtualMachineInstance called <code>vm-cirros</code> got created:</p> <pre><code>$ kubectl describe vm vm-cirros\nName:         vm-cirros\nNamespace:    default\nLabels:       kubevirt.io/vm=vm-cirros\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachine\nMetadata:\n  Cluster Name:\n  Creation Timestamp:  2018-04-30T09:25:08Z\n  Generation:          0\n  Resource Version:    6418\n  Self Link:           /apis/kubevirt.io/v1/namespaces/default/virtualmachines/vm-cirros\n  UID:                 60043358-4c58-11e8-8653-525500d15501\nSpec:\n  Running:  true\n  Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        Kubevirt . Io / Ovmi:  vm-cirros\n    Spec:\n      Domain:\n        Devices:\n          Disks:\n            Disk:\n              Bus:        virtio\n            Name:         containerdisk\n            Volume Name:  containerdisk\n            Disk:\n              Bus:        virtio\n            Name:         cloudinitdisk\n            Volume Name:  cloudinitdisk\n        Machine:\n          Type:\n        Resources:\n          Requests:\n            Memory:                      64M\n      Termination Grace Period Seconds:  0\n      Volumes:\n        Name:  containerdisk\n        Registry Disk:\n          Image:  kubevirt/cirros-registry-disk-demo:latest\n        Cloud Init No Cloud:\n          User Data Base 64:  IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK\n        Name:                 cloudinitdisk\nStatus:\n  Created:  true\n  Ready:    true\nEvents:\n  Type    Reason            Age   From                              Message\n  ----    ------            ----  ----                              -------\n  Normal  SuccessfulCreate  15s   virtualmachine-controller  Created virtual machine: vm-cirros\n</code></pre>"},{"location":"architecture/#kubectl-commandline-interactions","title":"Kubectl commandline interactions","text":"<p>Whenever you want to manipulate the VirtualMachine through the commandline you can use the kubectl command. The following are examples demonstrating how to do it.</p> <pre><code>    # Define a virtual machine:\n    kubectl create -f vm.yaml\n\n    # Start the virtual machine:\n    kubectl patch virtualmachine vm --type merge -p \\\n        '{\"spec\":{\"running\":true}}'\n\n    # Look at virtual machine status and associated events:\n    kubectl describe virtualmachine vm\n\n    # Look at the now created virtual machine instance status and associated events:\n    kubectl describe virtualmachineinstance vm\n\n    # Stop the virtual machine instance:\n    kubectl patch virtualmachine vm --type merge -p \\\n        '{\"spec\":{\"running\":false}}'\n\n    # Restart the virtual machine (you delete the instance!):\n    kubectl delete virtualmachineinstance vm\n\n    # Implicit cascade delete (first deletes the virtual machine and then the virtual machine instance)\n    kubectl delete virtualmachine vm\n\n    # Explicit cascade delete (first deletes the virtual machine and then the virtual machine instance)\n    kubectl delete virtualmachine vm --cascade=true\n\n    # Orphan delete (The running virtual machine is only detached, not deleted)\n    # Recreating the virtual machine would lead to the adoption of the virtual machine instance\n    kubectl delete virtualmachine vm --cascade=false\n</code></pre>"},{"location":"contributing/","title":"Contributing","text":"<p>Welcome!! And thank you for taking the first step to contributing to the KubeVirt project. On this page you should be able to find all the information required to get started on your contirbution journey, as well as information on how to become a community member and grow into roles of responsibility. </p> <p>If you think something might be missing from this page, please help us by raising a bug!</p>"},{"location":"contributing/#prerequisites","title":"Prerequisites","text":"<p>Reviewing the following will prepare you for contributing:</p> <ul> <li>If this is your first step in the world of open source, consider reading the CNCF's Start Contributing to Open Source page for an introduction to key concepts.</li> <li>You should be comfortable with git. Most contributions follow the GitHub workflow of fork, branch, commit, open pull request, review changes, and merge to work effectively in the KubeVirt community.  If you're new to git, git-scm.com has a nice set of tutorials.</li> <li>Familiarize yourself with the various repositories of the KubeVirt GitHub organization.</li> <li>Try the one of our quick start labs on killercoda, minikube, or kind.</li> <li>See the \"Other ways to contribute\" section below.</li> </ul> <p>For code contributors:</p> <ul> <li>You need to be familiar with writing code in golang.  See the golang tour to familiarize yourself.</li> <li>To contribute to the core of the project, read the Developer contribution page and the getting started page in the kubevirt/kubevirt repo.</li> <li>Alternatively, to contribute to its storage management add-on, check out the kubevirt/containerized-data-importer (CDI) repo, and their contribution page.</li> </ul>"},{"location":"contributing/#your-first-contribution","title":"Your first contribution","text":"<p>The following will help you decide where to start:</p> <ul> <li>Check a repository issues list and label <code>good-first-issue</code> for issues that make good entry points.</li> <li>Open a pull request using GitHub to documentation. The tutorials found here can be helpful https://lab.github.com/</li> <li>Review a pull request from other community members for accuracy and language.</li> </ul>"},{"location":"contributing/#important-community-resources","title":"Important community resources","text":"<p>You should familiarize yourself with the following documents, which are critical to being a member of the community:</p> <ul> <li>Code of Conduct: Everyone is expected to abide by the CoC to ensure an open and welcoming environment. Our CoC is based off the CNCF Code of conduct which also has a variety of translations. </li> <li>Our community membership policy: How to become a member and grow into roles of responsibility.</li> <li>Project governance: Project Maintainer responsibilities.</li> </ul>"},{"location":"contributing/#other-ways-to-contribute","title":"Other ways to contribute","text":"<ul> <li>Visit the KubeVirt community page, participate on Twitter or Slack, learn about local meetups and events.</li> <li>Visit the KubeVirt website repository and submit a blog post, case study or lab.</li> <li>Visit the KubeVirt user-guide repository and find feature documentation that could use an update.</li> </ul>"},{"location":"quickstarts/","title":"Quickstarts","text":""},{"location":"quickstarts/#quickstart-guides","title":"Quickstart Guides","text":"<p>Killercoda provides an interactive environment for exploring KubeVirt scenarios:</p> <ul> <li>KubeVirt on killercoda</li> </ul> <p>Guides for deploying KubeVirt with different Kubernetes tools:</p> <ul> <li> <p>KubeVirt on minikube</p> </li> <li> <p>KubeVirt on kind</p> </li> <li> <p>KubeVirt on cloud providers</p> </li> </ul>"},{"location":"release_notes/","title":"KubeVirt release notes","text":""},{"location":"release_notes/#v130","title":"v1.3.0","text":"<p>Release on: Wed Jul 17 2024</p> <p>KubeVirt v1.3 is built for Kubernetes v1.30 and additionally supported for the previous two versions. See the KubeVirt support matrix for more information.</p> <p>To see the list of fine folks who contributed to this release, see the KubeVirt release tag for v1.3.0.</p>"},{"location":"release_notes/#api-change","title":"API change","text":"<ul> <li>[PR #11156] [nunnatsa] Move some verification from the VMI create validation webhook to the CRD</li> <li>[PR #11500] [iholder101] Support HyperV Passthrough: automatically use all available HyperV features</li> <li>[PR #11641] [alicefr] Add kubevirt.io/testWorkloadUpdateMigrationAbortion annotation and a mechanism to abort workload updates</li> <li>[PR #11700] [alicefr] Add the updateVolumeStrategy field</li> <li>[PR #11729] [lyarwood] <code>spreadOptions</code> have been introduced to preferences in order to allow for finer grain control of the <code>preferSpread</code> <code>preferredCPUTopology</code>. This includes the ability to now spread vCPUs across guest visible sockets, cores and threads.</li> <li>[PR #10545] [lyarwood] <code>ControllerRevisions</code> containing instance types and preferences are now upgraded to their latest available version when the <code>VirtualMachine</code> owning them is resync'd by <code>virt-controller</code>.</li> <li>[PR #11955] [mhenriks] Introduce snapshot.kibevirt.io/v1beta1</li> <li>[PR #11956] [mhenriks] Introduce export.kibevirt.io/v1beta1</li> <li>[PR #11533] [alicefr] Implement volume migration and introduce the migration updateVolumesStrategy field</li> <li>[PR #10490] [jschintag] Add support for building and running kubevirt on s390x.</li> <li>[PR #11330] [jean-edouard] More information in the migration state of VMI / migration objects</li> <li>[PR #11773] [jean-edouard] Persistent TPM/UEFI will use the default storage class if none is specified in the CR.</li> </ul>"},{"location":"release_notes/#bug-fix","title":"Bug fix","text":"<ul> <li>[PR #12296] [orelmisan] Network binding plugins: Enable the ability to specify compute memory overhead</li> <li>[PR #12279] [fossidhelm] Fix: persistent tpm can be used with vmis containing dots in their name</li> <li>[PR #12226] [awels] Virt export route has an edge termination of redirect</li> <li>[PR #12249] [machadovilaca] Fix missing performance metrics for VMI resources</li> <li>[PR #12237] [vladikr] Only a single vgpu display option with ramfb will be configured per VMI.</li> <li>[PR #12064] [akalenyu] BugFix: Graceful deletion skipped for any delete call to the VM (not VMI) resource</li> <li>[PR #11996] [ShellyKa13] BugFix: Fix restore panic in case of volumesnapshot missing</li> <li>[PR #11973] [fossedihelm] Bug fix: Correctly reflect RestartRequired condition</li> <li>[PR #11922] [alromeros] Bugfix: Fix VM manifest rendering in export controller</li> <li>[PR #11367] [alromeros] Bugfix: Allow vmexport download redirections by printing logs into stderr</li> <li>[PR #11219] [alromeros] Bugfix: Improve handling of IOThreads with incompatible buses</li> <li>[PR #11372] [xpivarc] Bug-fix: Fix nil panic if VM update fails</li> <li>[PR #11267] [mhenriks] BugFix: Ensure DataVolumes created by virt-controller (DataVolumeTemplates) are recreated and owned by the VM in the case of DR and backup/restore.</li> <li>[PR #10900] [KarstenB] BugFix: Fixed incorrect APIVersion of APIResourceList</li> <li>[PR #11306] [fossedihelm] fix(ksm): set the <code>kubevirt.io/ksm-enabled</code> node label to true if the ksm is managed by KubeVirt, instead of reflect the actual ksm value.</li> <li>[PR #11264] [machadovilaca] Fix perfscale buckets error</li> <li>[PR #11058] [fossedihelm] fix(vmclone): delete vmclone resource when the target vm is deleted</li> <li>[PR #11265] [xpivarc] Bug fix: VM controller doesn't corrupt its cache anymore</li> <li>[PR #11205] [akalenyu] Fix migration breaking in case the VM has an rng device after hotplugging a block volume on cgroupsv2</li> <li>[PR #11051] [alromeros] Bugfix: Improve error reporting when fsfreeze fails</li> <li>[PR #12016] [acardace] fix starting VM with Manual RunStrategy</li> <li>[PR #11963] [acardace] Fix RerunOnFailure RunStrategy</li> <li>[PR #11718] [fossedihelm] Fix: SEV methods in client-go now satisfy the proxy server configuration, if provided</li> <li>[PR #12122] [kubevirt-bot] Fix VMPools when <code>LiveUpdate</code> as <code>vmRolloutStrategy</code> is used.</li> <li>[PR #12201] [kubevirt-bot] fix RerunOnFailure stuck in Provisioning</li> <li>[PR #12151] [kubevirt-bot] Bugfix: Implement retry mechanism in export server and vmexport</li> <li>[PR #12146] [kubevirt-bot] Memory Hotplug fixes and stabilization</li> <li>[PR #12185] [kubevirt-bot] VMs with a single socket and NetworkInterfaceMultiqueue enabled require a restart to hotplug additional CPU sockets.</li> <li>[PR #12171] [kubevirt-bot] <code>PreferredDiskDedicatedIoThread</code> is now only applied to <code>virtio</code> disk devices</li> </ul>"},{"location":"release_notes/#deprecation","title":"Deprecation","text":"<ul> <li>[PR #11701] [EdDev] The SLIRP core binding is deprecated and removed.</li> <li>[PR #11901] [EdDev] The 'macvtap' core network binding is discontinued and removed.</li> <li>[PR #11915] [ormergi] The 'passt' core network binding is discontinued and removed.</li> <li>[PR #11404] [avlitman] KubeVirtComponentExceedsRequestedCPU and KubeVirtComponentExceedsRequestedMemory alerts are deprecated; they do not indicate a genuine issue.</li> </ul>"},{"location":"release_notes/#sig-compute","title":"SIG-compute","text":"<ul> <li>[PR #11498] [acardace] Allow to hotplug memory for VMs with memory limits set</li> <li>[PR #11479] [vladikr] virtual machines instance will no longer be stuck in an irrecoverable state after an interrupted postcopy migration. Instead, these will fail and could be restarted again.</li> <li>[PR #11685] [fossedihelm] Updated go version of the client-go to 1.21</li> <li>[PR #11344] [aerosouund] Refactor device plugins to use a base plugin and define a common interface</li> <li>[PR #12025] [fossedihelm] Add descheduler compatibility</li> <li>[PR #12109] [acardace] Support Memory Hotplug with Hugepages</li> <li>[PR #11883] [orelmisan] Restart of a VM is required when the CPU socket count is reduced</li> <li>[PR #11655] [acardace] Allow to hotplug vcpus for VMs with CPU requests and/or limits set</li> <li>[PR #11455] [lyarwood] <code>LiveUpdates</code>  of VMs using instance types are now supported with the same caveats as when making changes to a vanilla VM.</li> <li>[PR #11681] [lyarwood] The <code>CommonInstancetypesDeployment</code> feature and gate are retrospectively moved to Beta from the 1.2.0 release.</li> <li>[PR #11648] [kubevirt-bot] Updated common-instancetypes bundles to v1.0.0</li> <li>[PR #12240] [kubevirt-bot] Updated common-instancetypes bundles to v1.0.1</li> </ul>"},{"location":"release_notes/#sig-storage","title":"SIG-storage","text":"<ul> <li>[PR #11095] [ShellyKa13] Expose volumesnapshot error in vmsnapshot object</li> <li>[PR #11312] [alromeros] Improve handling of export resources in virtctl vmexport</li> <li>[PR #11770] [alicefr] Fix the live updates for volumes and disks</li> <li>[PR #11957] [mhenriks] snapshot: Ignore unfreeze error if VMSnapshot deleting</li> </ul>"},{"location":"release_notes/#sig-network","title":"SIG-network","text":"<ul> <li>[PR #11653] [EdDev] Build the <code>passt</code>custom CNI binary statically, for the <code>passt</code> network binding plugin.</li> <li>[PR #11678] [Vicente-Cheng] Improve the handling of ordinal pod interface name for upgrade</li> <li>[PR #11618] [AlonaKaplan] Extend network binding plugin to support device-info DownwardAPI.</li> <li>[PR #11256] [andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 10.0.0 and QEMU 8.2.0.</li> <li>[PR #11788] [ormergi] The network-info annotation is now used for mapping between SR-IOV network and the underlying device PCI address</li> <li>[PR #11659] [iholder101] Require scheduling infra components onto control-plane nodes</li> <li>[PR #12079] [EdDev] Network hotplug feature is declared as Beta.</li> </ul>"},{"location":"release_notes/#sig-scale","title":"SIG-scale","text":"<ul> <li>[PR #11272] [dharmit] Make 'image' field in hook sidecar annotation optional.</li> <li>[PR #11676] [machadovilaca] Rename rest client metrics to include kubevirt prefix</li> <li>[PR #11387] [alaypatel07] add perf-scale benchmarks for release v1.2</li> </ul>"},{"location":"release_notes/#monitoring","title":"Monitoring","text":"<ul> <li>[PR #11307] [machadovilaca] Add e2e tests for metrics</li> <li>[PR #11294] [machadovilaca] Fix kubevirt_vm_created_total being broken down by virt-api pod</li> <li>[PR #11557] [avlitman] New memory statistics added named kubevirt_memory_delta_from_requested_bytes</li> <li>[PR #11283] [assafad] Collect VMI OS info from the Guest agent as <code>kubevirt_vmi_phase_count</code> metric labels</li> <li>[PR #11906] [machadovilaca] Create kubevirt_vmi_info metric</li> <li>[PR #11934] [assafad] Add kubevirt_vmi_last_connection_timestamp_seconds metric</li> <li>[PR #12000] [machadovilaca] Create kubevirt_vmi_launcher_memory_overhead_bytes metric</li> <li>[PR #11484] [jcanocan] Reduce the downwardMetrics server maximum number of request per second to 1.</li> </ul>"},{"location":"release_notes/#uncategorized","title":"Uncategorized","text":"<ul> <li>[PR #12132] [kubevirt-bot] Introduce validatingAdmissionPolicy to restrict node patches on virt-handler</li> <li>[PR #12009] [xpivarc] By enabling NodeRestriction feature gate, Kubevirt now authorize virt-handler's requests to modify VMs.</li> <li>[PR #12089] [jean-edouard] Less privileged virt-operator ClusterRole</li> <li>[PR #11969] [iholder101] Infra components control-plane nodes NoSchedule toleration</li> <li>[PR #11835] [talcoh2x] add Intel Gaudi to adopters.</li> <li>[PR #11790] [aburdenthehand] Re-adding Cloudflare to our ADOPTERS list</li> <li>[PR #11331] [anjuls] add cloudraft to adopters.</li> <li>[PR #11942] [ido106] Update virtctl to use v1beta1 endpoint for both regular and async image upload</li> <li>[PR #11908] [victortoso] sidecar-shim: allow stderr log from binary hooks</li> <li>[PR #11846] [victortoso] SMBios sidecar can be built out-of-tree</li> <li>[PR #11482] [brianmcarey] Build KubeVirt with go v1.22.2</li> <li>[PR #11470] [brianmcarey] Build KubeVirt with Go version 1.21.8</li> <li>[PR #12097] [fossedihelm] Bump k8s deps to 0.30.0</li> <li>[PR #11416] [dhiller] emission of k8s logs when using programmatic focus with <code>FIt</code></li> <li>[PR #11183] [dhiller] Extend OWNERS for sig-buildsystem</li> <li>[PR #11149] [0xFelix] virtctl: It is possible to import volumes from GCS when creating a VM now</li> <li>[PR #11146] [RamLavi] node-labeller: Remove obsolete functionalities</li> </ul>"},{"location":"release_notes/#v120","title":"v1.2.0","text":"<p>Released on: Tue Mar 05 2024</p> <p>KubeVirt v1.2 is built for Kubernetes v1.29 and additionally supported for the previous two versions. See the KubeVirt support matrix for more information.</p>"},{"location":"release_notes/#api-change_1","title":"API change","text":"<ul> <li>[PR #11064] [AlonaKaplan] Introduce a new API to mark a binding plugin as migratable.</li> <li>[PR #10970] [alromeros] Expose fs disk information via GuestOsInfo</li> <li>[PR #10905] [tiraboschi] Aggregate DVs conditions on VMI (and so VM)</li> <li>[PR #10872] [RamLavi] IsolateEmulatorThread: Add cluster-wide parity completion setting</li> <li>[PR #10846] [RamLavi] Change vm.status.PrintableStatus default value to \"Stopped\"</li> <li>[PR #10774] [victortoso] Windows offline activation with ACPI SLIC table</li> <li>[PR #10732] [AlonaKaplan] Extend kubvirt CR by adding domain attachment option to the network binding plugin API.</li> <li>[PR #10658] [matthewei] Support \"Clone API\" to filter VirtualMachine.spec.template.annotation and VirtualMachine.spec.template.label</li> </ul>"},{"location":"release_notes/#bug-fix_1","title":"Bug fix","text":"<ul> <li>[PR #11271] [kubevirt-bot] Bug fix: VM controller doesn't corrupt its cache anymore</li> <li>[PR #11242] [kubevirt-bot] Fix migration breaking in case the VM has an rng device after hotplugging a block volume on cgroupsv2</li> <li>[PR #11069] [ormergi] Bug fix: Packet drops during the initial phase of VM live migration https://issues.redhat.com/browse/CNV-28040</li> <li>[PR #11065] [fossedihelm] fix(vmclone): Generate VM patches from vmsnapshotcontent, instead of current VM</li> <li>[PR #11050] [fossedihelm] restrict default cluster role to authenticated only users</li> <li>[PR #11047] [jschintag] Fix potential crash when trying to list USB devices on host without any</li> <li>[PR #10963] [alromeros] Bugfix: Reject volume exports when no output is specified</li> <li>[PR #10916] [orelmisan] Fix the value of VMI <code>Status.GuestOSInfo.Version</code></li> <li>[PR #10888] [fossedihelm] [Bugfix] Clone VM with WaitForFirstConsumer binding mode PVC now works.</li> <li>[PR #10860] [akalenyu] BugFix: Double cloning with filter fails  isolateEmulatorThread feature (BZ#2228103).</li> <li>[PR #10845] [orelmisan] Reject VirtualMachineClone creation when target name is equal to source name</li> <li>[PR #10753] [victortoso] Fixes  permission when using USB host passthrough</li> <li>[PR #10747] [acardace] Fix KubeVirt for CRIO 1.28 by using checksums to verify containerdisks when migrating VMIs</li> <li>[PR #10699] [qinqon] virt-launcher: fix qemu non root log path</li> <li>[PR #10689] [akalenyu] BugFix: cgroupsv2 device allowlist is bound to virt-handler internal state/block disk device overwritten on hotplug</li> <li>[PR #10593] [RamLavi] Fixes SMT Alignment Error in virt-launcher pod by optimizing</li> </ul>"},{"location":"release_notes/#deprecation_1","title":"Deprecation","text":"<ul> <li>[PR #10924] [AlonaKaplan] Deprecate macvtap</li> </ul>"},{"location":"release_notes/#sig-compute_1","title":"SIG-compute","text":"<ul> <li>[PR #11054] [jean-edouard] New cluster-wide <code>vmRolloutStrategy</code> setting to define whether changes to VMs should either be always staged or live-updated when possible.</li> <li>[PR #11001] [fossedihelm] Allow <code>kubevirt.io:default</code> clusterRole to get,list kubevirts</li> <li>[PR #10961] [jcanocan] Reduced VM rescheduling time on node failure</li> <li>[PR #10918] [orelmisan] VMClone: Emit an event in case restore creation fails</li> <li>[PR #10898] [matthewei] vmi status's guestOsInfo adds <code>Machine</code></li> <li>[PR #10840] [acardace] Requests/Limits can now be configured when using CPU/Memory hotplug</li> <li>[PR #10839] [RamLavi] Change second emulator thread assign strategy to best-effort.</li> <li>[PR #10809] [orelmisan] Source virt-launcher: Log migration info by default</li> <li>[PR #10783] [RamLavi] Support multiple CPUs in Housekeeping cgroup</li> <li>[PR #10571] [tiraboschi] vmi memory footprint increase by 35M when guest serial console logging is turned on (default on).</li> </ul>"},{"location":"release_notes/#sig-storage_1","title":"SIG-storage","text":"<ul> <li>[PR #10657] [germag] Exposing Filesystem Persistent Volumes (PVs)  to the VM using unprivilege virtiofsd.</li> <li>[PR #10529] [alromeros] Allow LUN disks to be hotplugged</li> </ul>"},{"location":"release_notes/#sig-network_1","title":"SIG-network","text":"<ul> <li>[PR #10981] [AlonaKaplan] Report IP of interfaces using network binding plugin.</li> <li>[PR #10866] [AlonaKaplan] Raise an error in case passt feature gate or API are used.</li> <li>[PR #10800] [AlonaKaplan] Support macvtap as a binding plugin</li> <li>[PR #10425] [ormergi] Introduce network binding plugin for Passt networking, interfacing with Kubevirt new network binding plugin API.</li> </ul>"},{"location":"release_notes/#sig-infra","title":"SIG-infra","text":"<ul> <li>[PR #11025] [0xFelix] Allow unprivileged users read-only access to VirtualMachineCluster{Instancetypes,Preferences} by default.</li> <li>[PR #10922] [kubevirt-bot] Updated common-instancetypes bundles to v0.4.0</li> </ul>"},{"location":"release_notes/#monitoring_1","title":"Monitoring","text":"<ul> <li>[PR #10982] [machadovilaca] Refactor monitoring metrics</li> <li>[PR #10962] [machadovilaca] Update monitoring file structure</li> <li>[PR #10853] [machadovilaca] Refactor monitoring collectors</li> <li>[PR #10700] [machadovilaca] Refactor monitoring alerts</li> <li>[PR #10693] [machadovilaca] Remove MigrateVmiDiskTransferRateMetric</li> <li>[PR #10651] [machadovilaca] Refactor monitoring  recording-rules</li> <li>[PR #10570] [machadovilaca] Fix LowKVMNodesCount not firing</li> <li>[PR #10418] [machadovilaca] Add total VMs created metric</li> </ul>"},{"location":"release_notes/#uncategorized_1","title":"Uncategorized","text":"<ul> <li>[PR #11144] [0xFelix] virtctl: Specifying size when creating a VM and using --volume-import to clone a PVC or a VolumeSnapshot is optional now</li> <li>[PR #11122] [brianmcarey] Update runc dependency to v1.1.12</li> <li>[PR #11068] [brianmcarey] Update container base image to use current stable debian 12 base</li> <li>[PR #10914] [brianmcarey] KubeVirt is now built with go 1.21.5</li> <li>[PR #10879] [brianmcarey] Built with golang 1.20.12</li> <li>[PR #10863] [dhiller] Remove year from generated code copyright</li> <li>[PR #10787] [matthewei] virtctl support to add template label and annotation filters</li> <li>[PR #10720] [awels] Restored hotplug attachment pod request/limit to original value</li> <li>[PR #10637] [dharmit] Functional tests for sidecar hook with ConfigMap</li> <li>[PR #10615] [orelmisan] Remove leftover NonRoot feature gate</li> <li>[PR #10598] [alicefr] Add PVC option to the hook sidecars for supplying additional debugging tools</li> <li>[PR #10596] [mhenriks] Disable HTTP/2 to mitigate CVE-2023-44487</li> <li>[PR #10582] [orelmisan] Remove leftover NonRootExperimental feature gate</li> <li>[PR #10567] [awels] Attachment pod creation is now rate limited</li> <li>[PR #10526] [cfilleke]  Documents steps to build the KubeVirt builder container</li> <li>[PR #10479] [dharmit] Ability to run scripts through hook sidecardevice</li> <li>[PR #10244] [hshitomi] Added \u201cadm\u201d subcommand under \u201cvirtctl\u201d, and \u201clog-verbosity\" subcommand under \u201cadm\u201d. The log-verbosity command is: to show the log verbosity of one or more components, to set the log verbosity of one or more components, and to reset the log verbosity of all components (reset to the default verbosity (2)).</li> <li>[PR #10046] [victortoso] Add v1alpha3 for hooks and fix migration when using sidecars</li> </ul>"},{"location":"release_notes/#v110","title":"v1.1.0","text":"<p>Released on: Tue Nov 07 2023</p>"},{"location":"release_notes/#api-change_2","title":"API change","text":"<ul> <li>[#10568][ormergi] Network binding plugin API support CNIs, new integration point on virt-launcher pod creation.</li> <li>[#10309][lyarwood] cluster-wide <code>common-instancetypes</code> resources can now deployed by <code>virt-operator</code> using the <code>CommonInstancetypesDeploymentGate</code> feature gate.</li> <li>[#10463][0xFelix] VirtualMachines: Introduce InferFromVolumeFailurePolicy in Instancetype- and PreferenceMatchers</li> <li>[#10447][fossedihelm] Add a Feature Gate to KV CR to automatically set memory limits when a resource quota with memory limits is associated to the creation namespace</li> <li>[#10477][jean-edouard] Dynamic KSM enabling and configuration</li> <li>[#10110][tiraboschi] Stream guest serial console logs from a dedicated container</li> <li>[#10015][victortoso] Implements USB host passthrough in permittedHostDevices of KubeVirt CRD</li> <li>[#10184][acardace] Add memory hotplug feature</li> <li>[#10231][kvaps] Propogate public-keys to cloud-init NoCloud meta-data</li> <li>[#9673][germag] DownwardMetrics: Expose DownwardMetrics through virtio-serial channel.</li> <li>[#10086][vladikr] allow live updating VM affinity and node selector</li> <li>[#10272][ormergi] Introduce network binding plugin for Slirp networking, interfacing with Kubevirt new network binding plugin API.</li> <li>[#10284][AlonaKaplan] Introduce an API for network binding plugins. The feature is behind \"NetworkBindingPlugins\" gate.</li> <li>[#10101][acardace] Deprecate <code>spec.config.machineType</code> in KubeVirt CR.</li> <li>[#9878][jean-edouard] The EFI NVRAM can now be configured to persist across reboots</li> <li>[#9932][lyarwood] <code>ControllerRevisions</code> containing <code>instancetype.kubevirt.io</code> <code>CRDs</code> are now decorated with labels detailing specific metadata of the underlying stashed object</li> <li>[#10058][alicefr] Add field errorPolicy for disks</li> <li>[#10004][AlonaKaplan] Hoyplug/unplug interfaces should be done by updating the VM spec template. virtctl and REST API endpoints were removed.</li> <li>[#9896][ormergi] The VM controller now replicates spec interfaces MAC addresses to the corresponding interfaces in the VMI spec.</li> <li>[#7708][VirrageS] <code>nodeSelector</code> and <code>schedulerName</code> fields have been added to VirtualMachineInstancetype spec.</li> <li>[#7197][vasiliy-ul] Experimantal support of SEV attestation via the new API endpoints</li> <li>[#9737][AlonaKaplan] On hotunplug - remove bridge, tap and dummy interface from virt-launcher and the caches (file and volatile) from the node.</li> </ul>"},{"location":"release_notes/#bug-fixes","title":"Bug fixes:","text":"<ul> <li>[#10515][iholder101] Bug-fix: Stop copying VMI spec to VM during snapshots</li> <li>[#10393][iholder101] [Bugfix] [Clone API] Double-cloning is now working as expected.</li> <li>[#10391][awels] BugFix: VMExport now works in a namespace with quotas defined.</li> <li>[#10380][alromeros] Bugfix: Allow image-upload to recover from PendingPopulation phase</li> <li>[#10099][iholder101] Bugfix: target virt-launcher pod hangs when migration is cancelled.</li> <li>[#10165][awels] BugFix: deleting hotplug attachment pod will no longer detach volumes that were not removed.</li> <li>[#10067][iholder101] Bug fix: <code>virtctl create clone</code> marshalling and replacement of <code>kubectl</code> with <code>kubectl virt</code></li> <li>[#9935][xpivarc] Bug fix - correct logging in container disk</li> <li>[#9872][alromeros] Bugfix: Allow lun disks to be mapped to DataVolume sources</li> <li>[#10039][simonyangcj] fix guaranteed qos of virt-launcher pod broken when use virtiofs</li> <li>[#9861][rmohr] Fix the possibility of data corruption when requesting a force-restart via \"virtctl restart\"</li> </ul>"},{"location":"release_notes/#deprecation_2","title":"Deprecation","text":"<ul> <li>[#10486][assafad] Deprecation notice for the metrics listed in the PR. Please update your systems to use the new metrics names.</li> <li>[#9821][sradco] Deprecation notice for the metrics listed in the PR. Please update your systems to use the new metrics names.</li> </ul>"},{"location":"release_notes/#sig-compute_2","title":"SIG-compute","text":"<ul> <li>[#10566][fossedihelm] Add 100Mi of memory overhead for vmi with dedicatedCPU or that wants GuaranteedQos</li> <li>[#10496][fossedihelm] Automatically set cpu limits when a resource quota with cpu limits is associated to the creation namespace and the <code>AutoResourceLimits</code> FeatureGate is enabled</li> <li>[#10543][0xFelix] Clear VM guest memory when ignoring inference failures</li> <li>[#10320][victortoso] sidecar-shim implements PreCloudInitIso hook</li> <li>[#10253][rmohr] Stop trying to create unused directory /var/run/kubevirt-ephemeral-disk in virt-controller</li> <li>[#10050][victortoso] Updating the virt stack: QEMU 8.0.0, libvirt to 9.5.0, edk2 20230524, passt 20230818, libguestfs and guestfs-tools 1.50.1, virtiofsd 1.7.2</li> <li>[#9231][victortoso] Introduces sidecar-shim container image</li> <li>[#10254][rmohr] Don't mark the KubeVirt \"Available\" condition as false on up-to-date and ready but misscheduled virt-handler pods.</li> <li>[#10182][iholder101] Stop considering nodes without <code>kubevirt.io/schedulable</code> label when finding lowest TSC frequency on the cluster</li> <li>[#10056][jean-edouard] UEFI guests now use Bochs display instead of VGA emulation</li> <li>[#10106][acardace] Add boot-menu wait time when starting the VM as paused.</li> </ul>"},{"location":"release_notes/#sig-storage_2","title":"SIG-storage","text":"<ul> <li>[#10532][alromeros] Add --volume-mode flag in image-upload</li> <li>[#10020][akalenyu] Use auth API for DataVolumes, stop importing kubevirt.io/containerized-data-importer</li> <li>[#10400][alromeros] Add new vmexport flags to download raw images, either directly (--raw) or by decompressing (--decompress) them</li> <li>[#10148][alromeros] Add port-forward functionalities to vmexport</li> <li>[#10275][awels] Ensure new hotplug attachment pod is ready before deleting old attachment pod</li> <li>[#10118][akalenyu] Change exportserver default UID to succeed exporting CDI standalone PVCs (not attached to VM)</li> <li>[#9918][ShellyKa13] Fix for hotplug with WFFC SCI storage class which uses CDI populators</li> </ul>"},{"location":"release_notes/#sig-network_2","title":"SIG-network","text":"<ul> <li>[#10366][ormergi] Kubevirt now delegates Slirp networking configuration to Slirp network binding plugin.  In case you haven't registered Slirp network binding plugin image yet (i.e.: specify in Kubevirt config) the following default image would be used: <code>quay.io/kubevirt/network-slirp-binding:20230830_638c60fc8</code>. On next release (v1.2.0) no default image will be set and registering an image would be mandatory.</li> <li>[#10185][AlonaKaplan] Add support to migration based SRIOV hotplug.</li> <li>[#10116][ormergi] Existing detached interfaces with 'absent' state will be cleared from VMI spec.</li> <li>[#9958][AlonaKaplan] Disable network interface hotplug/unplug for VMIs. It will be supported for VMs only.</li> <li>[#10489][maiqueb] Remove the network-attachment-definition <code>list</code> and <code>watch</code> verbs from virt-controller's RBAC</li> </ul>"},{"location":"release_notes/#sig-infra_1","title":"SIG-infra","text":"<ul> <li>[#10438][lyarwood] A new <code>instancetype.kubevirt.io:view</code> <code>ClusterRole</code> has been introduced that can be bound to users via a <code>ClusterRoleBinding</code> to provide read only access to the cluster scoped <code>VirtualMachineCluster{Instancetype,Preference}</code> resources.</li> </ul>"},{"location":"release_notes/#sig-scale_1","title":"SIG-scale","text":"<ul> <li>[#9989][alaypatel07] Add perf scale benchmarks for VMIs</li> </ul>"},{"location":"release_notes/#uncategorized_2","title":"Uncategorized","text":"<ul> <li>[#9590][xuzhenglun] fix embed version info of virt-operator</li> <li>[#10044][machadovilaca] Add operator-observability package</li> <li>[#10450][0xFelix] virtctl: Enable inference in create vm subcommand by default</li> <li>[#10386][liuzhen21] KubeSphere added to the adopter's file!</li> <li>[#10167][0xFelix] virtctl: Apply namespace to created manifests</li> <li>[#10173][rmohr] Move coordination/lease RBAC permissions to Roles</li> <li>[#10138][machadovilaca] Change <code>kubevirt_vmi_*_usage_seconds</code> from Gauge to Counter</li> <li>[#10107][PiotrProkop] Expose <code>kubevirt_vmi_vcpu_delay_seconds_total</code> reporting amount of seconds VM spent in waiting in the queue instead of running.</li> <li>[#10070][machadovilaca] Remove affinities label from <code>kubevirt_vmi_cpu_affinity</code> and use sum as value</li> <li>[#9982][fabiand] Introduce a support lifecycle and Kubernetes target version.</li> <li>[#10001][machadovilaca] Fix <code>kubevirt_vmi_phase_count</code> not being created</li> <li>[#9840][dhiller] Increase probability for flake checker script to find flakes</li> <li>[#9988][enp0s3] Always deploy the outdated VMI workload alert</li> <li>[#9882][dhiller] Add some context for initial contributors about automated testing and draft pull requests.</li> <li>[#9552][phoracek] gRPC client now works correctly with non-Go gRPC servers</li> <li>[#9818][akrejcir] Added \"virtctl credentials\" commands to dynamically change SSH keys in a VM, and to set user's password.</li> <li>[#9073][machadovilaca] Fix incorrect KubevirtVmHighMemoryUsage description</li> </ul>"},{"location":"release_notes/#v100","title":"v1.0.0","text":"<p>Released on: Thu Jul 11 17:39:42 2023 +0000</p>"},{"location":"release_notes/#api-changes","title":"API changes","text":"<ul> <li>[PR #9572][fossedihelm] Enable freePageReporting for new non high performance vmi</li> <li>[PR #8156][jean-edouard] TPM VM device can now be set to persistent</li> <li>[PR #8575][iholder101] QEMU-level migration parallelism (a.k.a. multifd) + Upgrade QEMU to 7.2.0-11.el9</li> <li>[PR #9322][iholder101] Add guest-to-request memory headroom ratio.</li> <li>[PR #9422][awels] Ability to specify cpu/mem request limit for supporting containers (hotplug/container disk/virtiofs/side car)</li> <li>[PR #9177][alicefr] Adding SCSI persistent reservation</li> <li>[PR #9145][awels] Show VirtualMachine name in the VMExport status</li> <li>[PR #9491][orelmisan] API, AddInterfaceOptions: Rename NetworkName to NetworkAttachmentDefinitionName and InterfaceName to Name</li> <li>[PR #9442][EdDev] Remove the VMI Status interface <code>podConfigDone</code> field in favor of a new source option in <code>infoSource</code>.</li> <li>[PR #6852][maiqueb] Dev preview: Enables network interface hotplug for VMs / VMIs</li> <li>[PR #9193][qinqon] Add annotation for live migration and bridged pod interface</li> <li>[PR #9421][lyarwood] Requests to update the target <code>Name</code> of a <code>{Instancetype,Preference}Matcher</code> without also updating the <code>RevisionName</code> are now rejected.</li> </ul>"},{"location":"release_notes/#bug-fixes_1","title":"Bug fixes","text":"<ul> <li>[PR #9591][awels] BugFix: allow multiple NFS disks to be used/hotplugged</li> <li>[PR #9536][akalenyu] BugFix: virtualmachineclusterinstancetypes/preferences show up for get all -n  <li>[PR #9300][xpivarc] Bug fix: API and virtctl invoked migration is not rejected when the VM is paused</li> <li>[PR #9189][xpivarc] Bug fix: DNS integration continues to work after migration</li> <li>[PR #9241][akalenyu] BugFix: Guestfs image url not constructed correctly</li> <li>[PR #9260][ShellyKa13] Fix bug of possible re-trigger of memory dump</li> <li>[PR #9478][xpivarc] Bug fix: Fixes case when migration is not retried if the migration Pod gets denied.</li> <li>[PR #9330][qinqon] Skip label kubevirt.io/migrationTargetNodeName from virtctl expose service selector</li> <li>[PR #9603][qinqon] Adapt node-labeller.sh script to work at non kvm envs with emulation.</li>"},{"location":"release_notes/#deprecation_3","title":"Deprecation","text":"<ul> <li>[PR #9047][machadovilaca] Deprecate VM stuck in status alerts</li> </ul>"},{"location":"release_notes/#sig-compute_3","title":"SIG-compute","text":"<ul> <li>[PR #9640][jean-edouard] TSC-enabled VMs can now migrate to a node with a non-identical (but close-enough) frequency</li> <li>[PR #9629][0xFelix] virtctl: Allow to specify the boot order of volumes when creating VMs</li> <li>[PR #9435][rmohr] Ensure existence of all PVCs attached to the VMI before creating the VM target pod.</li> <li>[PR #9470][machadovilaca] Enable libvirt GetDomainStats on paused VMs</li> <li>[PR #9163][vladikr] fixes the requests/limits CPU number mismatch for VMs with isolatedEmulatorThread</li> <li>[PR #9250][vladikr] externally created mediated devices will not be deleted by virt-handler</li> </ul>"},{"location":"release_notes/#sig-storage_3","title":"SIG-storage","text":"<ul> <li>[PR #9376][ShellyKa13] Fix vmrestore with WFFC snapshotable storage class</li> <li>[PR #9392][awels] virtctl supports retrieving vm manifest for VM export</li> <li>[PR #9188][awels] Default RBAC for clone and export</li> <li>[PR #9133][ShellyKa13] Fix addvolume not rejecting adding existing volume source, fix removevolume allowing to remove non hotpluggable volume</li> </ul>"},{"location":"release_notes/#sig-network_3","title":"SIG-network","text":"<ul> <li>[PR #9399][maiqueb] Compute the interfaces to be hotplugged based on the current domain info, rather than on the interface status.</li> <li>[PR #9220][orelmisan] client-go: Added context to VirtualMachine's methods.</li> </ul>"},{"location":"release_notes/#sig-infra_2","title":"SIG-infra","text":"<ul> <li>[PR #9651][0xFelix] virtctl: Allow to specify memory of created VMs. Default to 512Mi if no instancetype was specified or is inferred.</li> <li>[PR #9169][lyarwood] The <code>dedicatedCPUPlacement</code> attribute is once again supported within the <code>VirtualMachineInstancetype</code> and <code>VirtualMachineClusterInstancetype</code> CRDs after a recent bugfix improved <code>VirtualMachine</code> validations, ensuring defaults are applied before any attempt to validate.</li> </ul>"},{"location":"release_notes/#uncategorized_3","title":"Uncategorized","text":"<ul> <li>[PR #9632][toelke] * Add Genesis Cloud to the adopters list</li> <li>[PR #9596][iholder101] Add \"virtctl create clone\" command</li> <li>[PR #9407][assafad] Use env <code>RUNBOOK_URL_TEMPLATE</code> for the runbooks URL template</li> <li>[PR #9327][jcanocan] DownwardMetrics: Swap KubeVirt build info with qemu version in VirtProductInfo field</li> <li>[PR #9367][machadovilaca] Add VM instancetype and preference label to vmi_phase_count metric</li> <li>[PR #8906][machadovilaca] Alert if there are no available nodes to run VMs</li> <li>[PR #9320][darfux] node-labeller: Check arch on the handler side</li> <li>[PR #9127][fossedihelm] Use ECDSA instead of RSA for key generation</li> <li>[PR #9228][rumans] Bump virtiofs container limit</li> <li>[PR #9159][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 9.0.0 and QEMU 7.2.0.</li> <li>[PR #8989][rthallisey] Integrate multi-architecture container manifests into the bazel make recipes</li> <li>[PR #8937][fossedihelm] Added foreground finalizer to  virtual machine</li> </ul>"},{"location":"release_notes/#v0590","title":"v0.59.0","text":"<p>Released on: Wed Mar 1 16:49:27 2023 +0000</p> <ul> <li>[PR #9311][kubevirt-bot] fixes the requests/limits CPU number mismatch for VMs with isolatedEmulatorThread</li> <li>[PR #9276][fossedihelm] Added foreground finalizer to  virtual machine</li> <li>[PR #9295][kubevirt-bot] Fix bug of possible re-trigger of memory dump</li> <li>[PR #9270][kubevirt-bot] BugFix: Guestfs image url not constructed correctly</li> <li>[PR #9234][kubevirt-bot] The <code>dedicatedCPUPlacement</code> attribute is once again supported within the <code>VirtualMachineInstancetype</code> and <code>VirtualMachineClusterInstancetype</code> CRDs after a recent bugfix improved <code>VirtualMachine</code> validations, ensuring defaults are applied before any attempt to validate.</li> <li>[PR #9267][fossedihelm] This version of KubeVirt includes upgraded virtualization technology based on libvirt 9.0.0 and QEMU 7.2.0.</li> <li>[PR #9197][kubevirt-bot] Fix addvolume not rejecting adding existing volume source, fix removevolume allowing to remove non hotpluggable volume</li> <li>[PR #9120][0xFelix] Fix access to portforwarding on VMs/VMIs with the cluster roles kubevirt.io:admin and kubevirt.io:edit</li> <li>[PR #9116][EdDev] Allow the specification of the ACPI Index on a network interface.</li> <li>[PR #8774][avlitman] Added new Virtual machines CPU metrics:</li> <li>[PR #9087][zhuchenwang] Open <code>/dev/vhost-vsock</code> explicitly to ensure that the right vsock module is loaded</li> <li>[PR #9020][feitnomore] Adding support for status/scale subresources so that VirtualMachinePool now supports HorizontalPodAutoscaler</li> <li>[PR #9085][0xFelix] virtctl: Add options to infer instancetype and preference when creating a VM</li> <li>[PR #8917][xpivarc] Kubevirt can be configured with Seccomp profile. It now ships a custom profile for the launcher.</li> <li>[PR #9054][enp0s3] do not inject LimitRange defaults into VMI</li> <li>[PR #7862][vladikr] Store the finalized VMI migration status in the migration objects.</li> <li>[PR #8878][0xFelix] Add 'create vm' command to virtctl</li> <li>[PR #9048][jean-edouard] DisableCustomSELinuxPolicy feature gate introduced to disable our custom SELinux policy</li> <li>[PR #8953][awels] VMExport now has endpoint containing entire VM definition.</li> <li>[PR #8976][iholder101] Fix podman CRI detection</li> <li>[PR #9043][iholder101] Adjust operator functional tests to custom images specification</li> <li>[PR #8875][machadovilaca] Rename migration metrics removing 'total' keyword</li> <li>[PR #9040][lyarwood] <code>inferFromVolume</code> now uses labels instead of annotations to lookup default instance type and preference details from a referenced <code>Volume</code>. This has changed in order to provide users with a way of looking up suitably decorated resources through these labels before pointing to them within the <code>VirtualMachine</code>.</li> <li>[PR #9039][orelmisan] client-go: Added context to additional VirtualMachineInstance's methods.</li> <li>[PR #9018][orelmisan] client-go: Added context to additional VirtualMachineInstance's methods.</li> <li>[PR #9025][akalenyu] BugFix: Hotplug pods have hardcoded resource req which don't comply with LimitRange maxLimitRequestRatio of 1</li> <li>[PR #8908][orelmisan] client-go: Added context to some of VirtualMachineInstance's methods.</li> <li>[PR #6863][rmohr] The install strategy job will respect the infra node placement from now on</li> <li>[PR #8948][iholder101] Bugfix: virt-handler socket leak</li> <li>[PR #8649][acardace] KubeVirt is now able to run VMs inside restricted namespaces.</li> <li>[PR #8992][iholder101] Align with k8s fix for default limit range requirements</li> <li>[PR #8889][rmohr] Add basic TLS encryption support for vsock websocket connections</li> <li>[PR #8660][huyinhou] Fix remoteAddress field in virt-api log being truncated when it is an ipv6 address</li> <li>[PR #8961][rmohr] Bump distroless base images</li> <li>[PR #8952][rmohr] Fix read-only sata disk validation</li> <li>[PR #8657][fossedihelm] Use an increasingly exponential backoff before retrying to start the VM, when an I/O error occurs.</li> <li>[PR #8480][lyarwood] New <code>inferFromVolume</code> attributes have been introduced to the <code>{Instancetype,Preference}Matchers</code> of a <code>VirtualMachine</code>. When provided the <code>Volume</code> referenced by the attribute is checked for the following annotations with which to populate the <code>{Instancetype,Preference}Matchers</code>:</li> <li>[PR #7762][VirrageS] Service <code>kubevirt-prometheus-metrics</code> now sets <code>ClusterIP</code> to <code>None</code> to make it a headless service.</li> <li>[PR #8599][machadovilaca] Change KubevirtVmHighMemoryUsage threshold from 20MB to 50MB</li> <li>[PR #7761][VirrageS] imagePullSecrets field has been added to KubeVirt CR to support deployments form private registries</li> <li>[PR #8887][iholder101] Bugfix: use virt operator image if provided</li> <li>[PR #8750][jordigilh] Fixes an issue that prevented running real time workloads in non-root configurations due to libvirt's dependency on CAP_SYS_NICE to change the vcpu's thread's scheduling and priority to FIFO and 1. The change of priority and scheduling is now executed in the virt-launcher for both root and non-root configurations, removing the dependency in libvirt.</li> <li>[PR #8845][lyarwood] An empty <code>Timer</code> is now correctly omitted from <code>Clock</code> fixing bug #8844.</li> <li>[PR #8842][andreabolognani] The virt-launcher pod no longer needs the SYS_PTRACE capability.</li> <li>[PR #8734][alicefr] Change libguestfs-tools image using root appliance in qcow2 format</li> <li>[PR #8764][ShellyKa13] Add list of included and excluded volumes in vmSnapshot</li> <li>[PR #8811][iholder101] Custom components: support gs</li> <li>[PR #8770][dhiller] Add Ginkgo V2 Serial decorator to serial tests as preparation to simplify parallel vs. serial test run logic</li> <li>[PR #8808][acardace] Apply migration backoff only for evacuation migrations.</li> <li>[PR #8525][jean-edouard] CR option mediatedDevicesTypes is deprecated in favor of mediatedDeviceTypes</li> <li>[PR #8792][iholder101] Expose new custom components env vars to csv-generator and manifest-templator</li> <li>[PR #8701][enp0s3] Consider the ParallelOutboundMigrationsPerNode when evicting VMs</li> <li>[PR #8740][iholder101] Fix: Align Reenlightenment flows between converter.go and template.go</li> <li>[PR #8530][acardace] Use exponential backoff for failing migrations</li> <li>[PR #8720][0xFelix] The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced</li> <li>[PR #8458][iholder101] Introduce support for clones with a snapshot source (e.g. clone snapshot -&gt; VM)</li> <li>[PR #8716][rhrazdil] Add overhead of interface with Passt binding when no ports are specified</li> <li>[PR #8619][fossedihelm] virt-launcher: use <code>virtqemud</code> daemon instead of <code>libvirtd</code></li> <li>[PR #8736][knopt] Added more precise rest_client_request_latency_seconds histogram buckets</li> <li>[PR #8624][zhuchenwang] Add the REST API to be able to talk to the application in the guest VM via VSOCK.</li> <li>[PR #8625][AlonaKaplan] iptables are no longer used by masquerade binding. Nodes with iptables only won't be able to run VMs with masquerade binding.</li> <li>[PR #8673][iholder101] Allow specifying custom images for core components</li> <li>[PR #8622][jean-edouard] Built with golang 1.19</li> <li>[PR #8336][alicefr] Flag for setting the guestfs uid and gid</li> <li>[PR #8667][huyinhou] connect VM vnc failed when virt-launcher work directory is not /</li> <li>[PR #8368][machadovilaca] Use collector to set migration metrics</li> <li>[PR #8558][xpivarc] Bug-fix: LimitRange integration now works when VMI is missing namespace</li> <li>[PR #8404][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 8.7.0, QEMU 7.1.0 and CentOS Stream 9.</li> <li>[PR #8652][akalenyu] BugFix: Exporter pod does not comply with restricted PSA</li> <li>[PR #8563][xpivarc] Kubevirt now runs with nonroot user by default</li> <li>[PR #8442][kvaps] Add Deckhouse to the Adopters list</li> <li>[PR #8546][zhuchenwang] Provides the Vsock feature for KubeVirt VMs.</li> <li>[PR #8598][acardace] VMs configured with hugepages can now run using the default container_t SELinux type</li> <li>[PR #8594][kylealexlane] Fix permission denied on on selinux relabeling on some kernel versions</li> <li>[PR #8521][akalenyu] Add an option to specify a TTL for VMExport objects</li> <li>[PR #7918][machadovilaca] Add alerts for VMs unhealthy states</li> <li>[PR #8516][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> <li>[PR #7772][jean-edouard] The SELinux policy for virt-launcher is down to 4 rules, 1 for hugepages and 3 for virtiofs.</li> <li>[PR #8402][jean-edouard] Most VMIs now run under the SELinux type container_t</li> <li>[PR #8513][alromeros] [Bug-fix] Fix error handling in virtctl image-upload</li> </ul>"},{"location":"release_notes/#v0581","title":"v0.58.1","text":"<p>Released on: Thu Feb 11 00:08:46 2023 +0000</p> <ul> <li>[PR #9203][jean-edouard] Most VMIs now run under the SELinux type container_t</li> <li>[PR #9191][kubevirt-bot] Default RBAC for clone and export</li> <li>[PR #9150][kubevirt-bot] Fix access to portforwarding on VMs/VMIs with the cluster roles kubevirt.io:admin and kubevirt.io:edit</li> <li>[PR #9128][kubevirt-bot] Rename migration metrics removing 'total' keyword</li> <li>[PR #9034][akalenyu] BugFix: Hotplug pods have hardcoded resource req which don't comply with LimitRange maxLimitRequestRatio of 1</li> <li>[PR #9002][iholder101] Bugfix: virt-handler socket leak</li> <li>[PR #8907][kubevirt-bot] Bugfix: use virt operator image if provided</li> <li>[PR #8784][kubevirt-bot] Use exponential backoff for failing migrations</li> <li>[PR #8816][iholder101] Expose new custom components env vars to csv-generator, manifest-templator and gs</li> <li>[PR #8798][iholder101] Fix: Align Reenlightenment flows between converter.go and template.go</li> <li>[PR #8731][kubevirt-bot] Allow specifying custom images for core components</li> <li>[PR #8785][0xFelix] The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced</li> <li>[PR #8806][kubevirt-bot] Consider the ParallelOutboundMigrationsPerNode when evicting VMs</li> <li>[PR #8738][machadovilaca] Use collector to set migration metrics</li> <li>[PR #8747][kubevirt-bot] Add alerts for VMs unhealthy states</li> <li>[PR #8685][kubevirt-bot] BugFix: Exporter pod does not comply with restricted PSA</li> <li>[PR #8647][akalenyu] BugFix: Add an option to specify a TTL for VMExport objects</li> <li>[PR #8609][kubevirt-bot] Fix permission denied on on selinux relabeling on some kernel versions</li> <li>[PR #8578][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> </ul>"},{"location":"release_notes/#v0580","title":"v0.58.0","text":"<p>Released on: Thu Oct 13 00:24:51 2022 +0000</p> <ul> <li>[PR #8578][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> <li>[PR #8463][Barakmor1] Improve metrics documentation</li> <li>[PR #8282][akrejcir] Improves instancetype and preference controller revisions. This is a backwards incompatible change and introduces a new v1alpha2 api for instancetype and preferences.</li> <li>[PR #8272][jean-edouard] No more empty section in the kubevirt-cr manifest</li> <li>[PR #8536][qinqon] Don't show a failure if ConfigDrive cloud init has UserDataSecretRef and not NetworkDataSecretRef</li> <li>[PR #8375][xpivarc] Virtiofs can be used with Nonroot feature gate</li> <li>[PR #8465][rmohr] Add a vnc screenshot REST endpoint and a \"virtctl vnc screenshot\" command for UI and script integration</li> <li>[PR #8418][alromeros] Enable automatic token generation for VirtualMachineExport objects</li> <li>[PR #8488][0xFelix] virtctl: Be less verbose when using the local ssh client</li> <li>[PR #8396][alicefr] Add group flag for setting the gid and fsgroup in guestfs</li> <li>[PR #8476][iholder-redhat] Allow setting virt-operator log verbosity through Kubevirt CR</li> <li>[PR #8366][rthallisey] Move KubeVirt to a 15 week release cadence</li> <li>[PR #8479][arnongilboa] Enable DataVolume GC by default in cluster-deploy</li> <li>[PR #8474][vasiliy-ul] Fixed migration failure of VMs with containerdisks on systems with containerd</li> <li>[PR #8316][ShellyKa13] Fix possible race when deleting unready vmsnapshot and the vm remaining frozen</li> <li>[PR #8436][xpivarc] Kubevirt is able to run with restricted Pod Security Standard enabled with an automatic escalation of namespace privileges.</li> <li>[PR #8197][alromeros] Add vmexport command to virtctl</li> <li>[PR #8252][fossedihelm] Add <code>tlsConfiguration</code> to Kubevirt Configuration</li> <li>[PR #8431][rmohr] Fix shadow status updates and periodic status updates on VMs, performed by the snapshot controller</li> <li>[PR #8359][iholder-redhat] [Bugfix]: HyperV Reenlightenment VMIs should be able to start when TSC Frequency is not exposed</li> <li>[PR #8330][jean-edouard] Important: If you use docker with SELinux enabled, set the <code>DockerSELinuxMCSWorkaround</code> feature gate before upgrading</li> <li>[PR #8401][machadovilaca] Rename metrics to follow the naming convention</li> </ul>"},{"location":"release_notes/#v0570","title":"v0.57.0","text":"<p>Released on: Mon Sep 12 14:00:44 2022 +0000</p> <ul> <li>[PR #8129][mlhnono68] Fixes virtctl to support connection to clusters proxied by RANCHER or having special paths</li> <li>[PR #8337][0xFelix] virtctl's native SSH client is now useable in the Windows console without workarounds</li> <li>[PR #8257][awels] VirtualMachineExport now supports VM export source type.</li> <li>[PR #8367][vladikr] fix the guest memory conversion by setting it to resources.requests.memory when guest memory is not explicitly provided</li> <li>[PR #7990][ormergi] Deprecate SR-IOV live migration feature gate.</li> <li>[PR #8069][lyarwood] The VirtualMachineInstancePreset resource has been deprecated ahead of removal in a future release. Users should instead use the VirtualMachineInstancetype and VirtualMachinePreference resources to encapsulate any shared resource or preferences characteristics shared by their VirtualMachines.</li> <li>[PR #8326][0xFelix] virtctl: Do not log wrapped ssh command by default</li> <li>[PR #8325][rhrazdil] Enable route_localnet sysctl option for masquerade binding at virt-handler</li> <li>[PR #8159][acardace] Add support for USB disks</li> <li>[PR #8006][lyarwood] <code>AutoattachInputDevice</code> has been added to <code>Devices</code> allowing an <code>Input</code> device to be automatically attached to a <code>VirtualMachine</code> on start up.  <code>PreferredAutoattachInputDevice</code> has also been added to <code>DevicePreferences</code> allowing users to control this behaviour with a set of preferences.</li> <li>[PR #8134][arnongilboa] Support DataVolume garbage collection</li> <li>[PR #8157][StefanKro] TrilioVault for Kubernetes now supports KubeVirt for backup and recovery.</li> <li>[PR #8273][alaypatel07] add server-side validations for spec.topologySpreadConstraints during object creation</li> <li>[PR #8049][alicefr] Set RunAsNonRoot as default for the guestfs pod</li> <li>[PR #8107][awels] Allow VirtualMachineSnapshot as a VirtualMachineExport source</li> <li>[PR #7846][janeczku] Added support for configuring topology spread constraints for virtual machines.</li> <li>[PR #8215][alaypatel07] support validation for spec.affinity fields during vmi creation</li> <li>[PR #8071][oshoval] Relax networkInterfaceMultiqueue semantics: multi queue will configure only what it can (virtio interfaces).</li> <li>[PR #7549][akrejcir] Added new API subresources to expand instancetype and preference.</li> </ul>"},{"location":"release_notes/#v0560","title":"v0.56.0","text":"<p>Released on: Thu Aug 18 20:10:29 2022 +0000</p> <ul> <li>[PR #7599][iholder-redhat] Introduce a mechanism to abort non-running migrations - fixes \"Unable to cancel live-migration if virt-launcher pod in pending state\" bug</li> <li>[PR #8027][alaypatel07] Wait deletion to succeed all the way till objects are finalized in perfscale tests</li> <li>[PR #8198][rmohr] Improve path handling for non-root virt-launcher workloads</li> <li>[PR #8136][iholder-redhat] Fix cgroups unit tests: mock out underlying runc cgroup manager</li> <li>[PR #8047][iholder-redhat] Deprecate live migration feature gate</li> <li>[PR #7986][iholder-redhat] [Bug-fix]: Windows VM with WSL2 guest fails to migrate</li> <li>[PR #7814][machadovilaca] Add VMI filesystem usage metrics</li> <li>[PR #7849][AlonaKaplan] [TECH PREVIEW] Introducing passt - a new approach to user-mode networking for virtual machines</li> <li>[PR #7991][ShellyKa13] Virtctl memory dump with create flag to create a new pvc</li> <li>[PR #8039][lyarwood] The flavor API and associated CRDs of <code>VirtualMachine{Flavor,ClusterFlavor}</code> are renamed to instancetype and <code>VirtualMachine{Instancetype,ClusterInstancetype}</code>.</li> <li>[PR #8112][AlonaKaplan] Changing the default of <code>virtctl expose</code> <code>ip-family</code> parameter to be empty value instead of IPv4.</li> <li>[PR #8073][orenc1] Bump runc to v1.1.2</li> <li>[PR #8092][Barakmor1] Bump the version of emicklei/go-restful from 2.15.0 to 2.16.0</li> <li>[PR #8053][alromeros] [Bug-fix]: Fix mechanism to fetch fs overhead when CDI resource has a different name</li> <li>[PR #8035][0xFelix] Add option to wrap local scp client to scp command</li> <li>[PR #7981][lyarwood] Conflicts will now be raised when using flavors if the <code>VirtualMachine</code> defines any <code>CPU</code> or <code>Memory</code> resource requests.</li> <li>[PR #8068][awels] Set cache mode to match regular disks on hotplugged disks.</li> </ul>"},{"location":"release_notes/#v0550","title":"v0.55.0","text":"<p>Released on: Thu Jul 14 16:33:25 2022 +0000</p> <ul> <li>[PR #7336][iholder-redhat] Introduce clone CRD, controller and API</li> <li>[PR #7791][iholder-redhat] Introduction of an initial deprecation policy</li> <li>[PR #7875][lyarwood] <code>ControllerRevisions</code> of any <code>VirtualMachineFlavorSpec</code> or <code>VirtualMachinePreferenceSpec</code> are stored during the initial start of a <code>VirtualMachine</code> and used for subsequent restarts ensuring changes to the original <code>VirtualMachineFlavor</code> or <code>VirtualMachinePreference</code> do not modify the <code>VirtualMachine</code> and the <code>VirtualMachineInstance</code> it creates.</li> <li>[PR #8011][fossedihelm] Increase virt-launcher memory overhead</li> <li>[PR #7963][qinqon] Bump alpine_with_test_tooling</li> <li>[PR #7881][ShellyKa13] Enable memory dump to be included in VMSnapshot</li> <li>[PR #7926][qinqon] tests: Move main clean function to global AfterEach and create a VM per each infra_test.go Entry.</li> <li>[PR #7845][janeczku] Fixed a bug that caused <code>make generate</code> to fail when API code comments contain backticks. (#7844, @janeczku)</li> <li>[PR #7932][marceloamaral] Addition of kubevirt_vmi_migration_phase_transition_time_from_creation_seconds metric to monitor how long it takes to transition a VMI Migration object to a specific phase from creation time.</li> <li>[PR #7879][marceloamaral] Faster VM phase transitions thanks to an increased virt-controller QPS/Burst</li> <li>[PR #7807][acardace] make cloud-init 'instance-id' persistent across reboots</li> <li>[PR #7928][iholder-redhat] bugfix: node-labeller now removes \"host-model-cpu.node.kubevirt.io/\" and \"host-model-required-features.node.kubevirt.io/\" prefixes</li> <li>[PR #7841][jean-edouard] Non-root VMs will now migrate to root VMs after a cluster disables non-root.</li> <li>[PR #7933][akalenyu] BugFix: Fix vm restore in case of restore size bigger then PVC requested size</li> <li>[PR #7919][lyarwood] Device preferences are now applied to any default network interfaces or missing volume disks added to a <code>VirtualMachineInstance</code> at runtime.</li> <li>[PR #7910][qinqon] tests: Create the expected readiness probe instead of liveness</li> <li>[PR #7732][acardace] Prevent virt-handler from starting a migration twice</li> <li>[PR #7594][alicefr] Enable to run libguestfs-tools pod to run as noroot user</li> <li>[PR #7811][raspbeep] User now gets information about the type of commands which the guest agent does not support.</li> <li>[PR #7590][awels] VMExport allows filesystem PVCs to be exported as either disks or directories.</li> <li>[PR #7683][alicefr] Add --command and --local-ssh-opts\" options to virtctl ssh to execute remote command using local ssh method</li> </ul>"},{"location":"release_notes/#v0540","title":"v0.54.0","text":"<p>Released on: Wed Jun 8 14:15:43 2022 +0000</p> <ul> <li>[PR #7757][orenc1] new alert for excessive number of VMI migrations in a period of time.</li> <li>[PR #7517][ShellyKa13] Add virtctl Memory Dump command</li> <li>[PR #7801][VirrageS] Empty (<code>nil</code> values) of <code>Address</code> and <code>Driver</code> fields in XML will be omitted.</li> <li>[PR #7475][raspbeep] Adds the reason of a live-migration failure to a recorded event in case EvictionStrategy is set but live-migration is blocked due to its limitations.</li> <li>[PR #7739][fossedihelm] Allow <code>virtualmachines/migrate</code> subresource to admin/edit users</li> <li>[PR #7618][lyarwood] The requirement to define a <code>Disk</code> or <code>Filesystem</code> for each <code>Volume</code> associated with a <code>VirtualMachine</code> has been removed. Any <code>Volumes</code> without a <code>Disk</code> or <code>Filesystem</code> defined will have a <code>Disk</code> defined within the <code>VirtualMachineInstance</code> at runtime.</li> <li>[PR #7529][xpivarc] NoReadyVirtController and NoReadyVirtOperator should be properly fired.</li> <li>[PR #7465][machadovilaca] Add metrics for migrations and respective phases</li> <li>[PR #7592][akalenyu] BugFix: virtctl guestfs incorrectly assumes image name</li> </ul>"},{"location":"release_notes/#v0531","title":"v0.53.1","text":"<p>Released on: Tue May 17 14:55:54 2022 +0000</p> <ul> <li>[PR #7749][kubevirt-bot] NoReadyVirtController and NoReadyVirtOperator should be properly fired.</li> </ul>"},{"location":"release_notes/#v0530","title":"v0.53.0","text":"<p>Released on: Mon May 9 14:02:20 2022 +0000</p> <ul> <li>[PR #7533][akalenyu] Add several VM snapshot metrics</li> <li>[PR #7574][rmohr] Pull in cdi dependencies with minimized transitive dependencies to ease API adoption</li> <li>[PR #7318][iholder-redhat] Snapshot restores now support restoring to a target VM different than the source</li> <li>[PR #7474][borod108] Added the following metrics for live migration: kubevirt_migrate_vmi_data_processed_bytes, kubevirt_migrate_vmi_data_remaining_bytes, kubevirt_migrate_vmi_dirty_memory_rate_bytes</li> <li>[PR #7441][rmohr] Add <code>virtctl scp</code> to ease copying files from and to VMs and VMIs</li> <li>[PR #7265][rthallisey] Support steady-state job types in the load-generator tool</li> <li>[PR #7544][fossedihelm] Upgraded go version to 1.17.8</li> <li>[PR #7582][acardace] Fix failed reported migrations when actually they were successful.</li> <li>[PR #7546][0xFelix] Update virtio-container-disk to virtio-win version 0.1.217-1</li> <li>[PR #7530][iholder-redhat] [External Kernel Boot]: Disallow kernel args without providing custom kernel</li> <li>[PR #7493][davidvossel] Adds new EvictionStrategy \"External\" for blocking eviction which is handled by an external controller</li> <li>[PR #7563][akalenyu] Switch VolumeSnapshot to v1</li> <li>[PR #7406][acardace] Reject <code>LiveMigrate</code> as a workload-update strategy if the <code>LiveMigration</code> feature gate is not enabled.</li> <li>[PR #7103][jean-edouard] Non-persistent vTPM now supported. Keep in mind that the state of the TPM is wiped after each shutdown. Do not enable Bitlocker!</li> <li>[PR #7277][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 8.0.0 and QEMU 6.2.0.</li> <li>[PR #7130][Barakmor1] Add field to kubevirtCR to set Prometheus ServiceMonitor object's namespace</li> <li>[PR #7401][iholder-redhat] virt-api deployment is now scalable - replicas are determined by the number of nodes in the cluster</li> <li>[PR #7500][awels] BugFix: Fixed RBAC for admin/edit user to allow virtualmachine/addvolume and removevolume. This allows for persistent disks</li> <li>[PR #7328][apoorvajagtap] Don't ignore --identity-file when setting --local-ssh=true on <code>virtctl ssh</code></li> <li>[PR #7469][xpivarc] Users can now enable the NonRoot feature gate instead of NonRootExperimental</li> <li>[PR #7451][fossedihelm] Reduce virt-launcher memory usage by splitting monitoring and launcher processes</li> </ul>"},{"location":"release_notes/#v0520","title":"v0.52.0","text":"<p>Released on: Fri Apr 8 16:17:56 2022 +0000</p> <ul> <li>[PR #7024][fossedihelm] Add an warning message if the client and server virtctl versions are not aligned</li> <li>[PR #7486][rmohr] Move stable.txt location to a more appropriate path</li> <li>[PR #7372][saschagrunert] Fixed <code>KubeVirtComponentExceedsRequestedMemory</code> alert complaining about many-to-many matching not allowed.</li> <li>[PR #7426][iholder-redhat] Add warning for manually determining core-component replica count in Kubevirt CR</li> <li>[PR #7424][maiqueb] Provide interface binding types descriptions, which will be featured in the KubeVirt API.</li> <li>[PR #7422][orelmisan] Fixed setting custom guest pciAddress and bootOrder parameter(s) to a list of SR-IOV NICs.</li> <li>[PR #7421][rmohr] Fix knowhosts file corruption for virtctl ssh</li> <li>[PR #6854][rmohr] Make virtctl ssh work with ssh-rsa+ preauthentication</li> <li>[PR #7267][iholder-redhat] Applied migration configurations can now be found in VMI's status</li> <li>[PR #7321][iholder-redhat] [Migration Policies]: precedence to VMI labels over Namespace labels</li> <li>[PR #7326][oshoval] The Ginkgo dependency has been upgraded to v2.1.3 (major version upgrade)</li> <li>[PR #7361][SeanKnight] Fixed a bug that prevents virtctl from working with clusters accessed via Rancher authentication proxy, or any other cluster where the server URL contains a path component. (#3760)</li> <li>[PR #7255][tyleraharrison] Users are now able to specify <code>--address [ip_address]</code> when using <code>virtctl vnc</code> rather than only using 127.0.0.1</li> <li>[PR #7275][enp0s3] Add observedGeneration to virt-operator to have a race-free way to detect KubeVirt config rollouts</li> <li>[PR #7233][xpivarc] Bug fix: Successfully aborted migrations should be reported now</li> <li>[PR #7158][AlonaKaplan] Add masquerade VMs support to single stack IPv6.</li> <li>[PR #7227][rmohr] Remove VMI informer from virt-api to improve scaling characteristics of virt-api</li> <li>[PR #7288][raspbeep] Users now don't need to specify container for <code>kubectl logs &lt;vmi-pod&gt;</code> and <code>kubectl exec &lt;vmi-pod&gt;</code>.</li> <li>[PR #6709][xpivarc] Workloads will be migrated to nonroot implementation if NonRoot feature gate is set. (Except VirtioFS)</li> <li>[PR #7241][lyarwood] Fixed a bug that prevents only a unattend.xml configmap or secret being provided as contents for a sysprep disk. (#7240, @lyarwood)</li> </ul>"},{"location":"release_notes/#v0510","title":"v0.51.0","text":"<p>Released on: Tue Mar 8 21:06:59 2022 +0000</p> <ul> <li>[PR #7102][machadovilaca] Add Virtual Machine name label to virt-launcher pod</li> <li>[PR #7139][davidvossel] Fixes inconsistent VirtualMachinePool VM/VMI updates by using controller revisions</li> <li>[PR #6754][jean-edouard] New and resized disks are now always 1MiB-aligned</li> <li>[PR #7086][acardace] Add 'EvictionStrategy' as a cluster-wide setting in the KubeVirt CR</li> <li>[PR #7232][rmohr] Properly format the PDB scale event during migrations</li> <li>[PR #7223][Barakmor1] Add a name label to virt-operator pods</li> <li>[PR #7221][davidvossel] RunStrategy: Once - allows declaring a VM should run once to a finalized state</li> <li>[PR #7091][EdDev] SR-IOV interfaces are now reported in the VMI status even without an active guest-agent.</li> <li>[PR #7169][rmohr] Improve device plugin de-registration in virt-handler and some test stabilizations</li> <li>[PR #6604][alicefr] Add shareable option to identify if the disk is shared with other VMs</li> <li>[PR #7144][davidvossel] Garbage collect finalized migration objects only leaving the most recent 5 objects</li> <li>[PR #6110][xpivarc] [Nonroot] SRIOV is now available.</li> </ul>"},{"location":"release_notes/#v0500","title":"v0.50.0","text":"<p>Released on: Wed Feb 9 18:01:08 2022 +0000</p> <ul> <li>[PR #7056][fossedihelm] Update k8s dependencies to 0.23.1</li> <li>[PR #7135][davidvossel] Switch from reflects.DeepEquals to equality.Semantic.DeepEquals() across the entire project</li> <li>[PR #7052][sradco] Updated recording rule \"kubevirt_vm_container_free_memory_bytes\"</li> <li>[PR #7000][iholder-redhat] Adds a possibility to override default libvirt log filters though VMI annotations</li> <li>[PR #7064][davidvossel] Fixes issue associated with blocked uninstalls when VMIs exist during removal</li> <li>[PR #7097][iholder-redhat] [Bug fix] VMI with kernel boot stuck on \"Terminating\" status if more disks are defined</li> <li>[PR #6700][VirrageS] Simplify replacing <code>time.Ticker</code> in agent poller and fix default values for <code>qemu-*-interval</code> flags</li> <li>[PR #6581][ormergi] SRIOV network interfaces are now hot-plugged when disconnected manually or due to aborted migrations.</li> <li>[PR #6924][EdDev] Support for legacy GPU definition is removed. Please see https://kubevirt.io/user-guide/virtual_machines/host-devices on how to define host-devices.</li> <li>[PR #6735][uril] The command <code>migrate_cancel</code> was added to virtctl. It cancels an active VM migration.</li> <li>[PR #6883][rthallisey] Add instance-type to cloud-init metadata</li> <li>[PR #6999][maya-r] When expanding disk images, take the minimum between the request and the capacity - avoid using the full underlying file system on storage like NFS, local.</li> <li>[PR #6946][vladikr] Numa information of an assigned device will be presented in the devices metadata</li> <li>[PR #6042][iholder-redhat] Fully support cgroups v2, include a new cohesive package and perform major refactoring.</li> <li>[PR #6968][vladikr] Added Writeback disk cache support</li> <li>[PR #6995][sradco] Alert OrphanedVirtualMachineImages name was changed to OrphanedVirtualMachineInstances.</li> <li>[PR #6923][rhrazdil] Fix issue with ssh being unreachable on VMIs with Istio proxy</li> <li>[PR #6821][jean-edouard] Migrating VMIs that contain dedicated CPUs will now have properly dedicated CPUs on target</li> <li>[PR #6793][oshoval] Add infoSource field to vmi.status.interfaces.</li> </ul>"},{"location":"release_notes/#v0490","title":"v0.49.0","text":"<p>Released on: Tue Jan 11 17:27:09 2022 +0000</p> <ul> <li>[PR #7004][iholder-redhat] Bugfix: Avoid setting block migration for volumes used by read-only disks</li> <li>[PR #6959][enp0s3] generate event when target pod enters unschedulable phase</li> <li>[PR #6888][assafad] Added common labels into alert definitions</li> <li>[PR #6166][vasiliy-ul] Experimental support of AMD SEV</li> <li>[PR #6980][vasiliy-ul] Updated the dependencies to include the fix for CVE-2021-43565 (KubeVirt is not affected)</li> <li>[PR #6944][iholder-redhat] Remove disabling TLS configuration from Live Migration Policies</li> <li>[PR #6800][jean-edouard] CPU pinning doesn't require hardware-assisted virtualization anymore</li> <li>[PR #6501][ShellyKa13] Use virtctl image-upload to upload archive content</li> <li>[PR #6918][iholder-redhat] Bug fix: Unscheduable host-model VMI alert is now properly triggered</li> <li>[PR #6796][Barakmor1] 'kubevirt-operator' changed to 'virt-operator' on 'managed-by' label in kubevirt's components made by virt-operator</li> <li>[PR #6036][jean-edouard] Migrations can now be done over a dedicated multus network</li> <li>[PR #6933][erkanerol] Add a new lane for monitoring tests</li> <li>[PR #6949][jean-edouard] KubeVirt components should now be successfully removed on CR deletion, even when using only 1 replica for virt-api and virt-controller</li> <li>[PR #6954][maiqueb] Update the <code>virtctl</code> exposed services <code>IPFamilyPolicyType</code> default to <code>IPFamilyPolicyPreferDualStack</code></li> <li>[PR #6931][fossedihelm] added DryRun to AddVolumeOptions and RemoveVolumeOptions</li> <li>[PR #6379][nunnatsa] Fix issue https://bugzilla.redhat.com/show_bug.cgi?id=1945593</li> <li>[PR #6399][iholder-redhat] Introduce live migration policies that allow system-admins to have fine-grained control over migration configuration for different sets of VMs.</li> <li>[PR #6880][iholder-redhat] Add full Podman support for <code>make</code> and <code>make test</code></li> <li>[PR #6702][acardace] implement virt-handler canary upgrade and rollback for faster and safer rollouts</li> <li>[PR #6717][davidvossel] Introducing the VirtualMachinePools feature for managing stateful VMs at scale</li> <li>[PR #6698][rthallisey] Add tracing to the virt-controller work queue</li> <li>[PR #6762][fossedihelm] added DryRun mode to virtcl to migrate command</li> <li>[PR #6891][rmohr] Fix \"Make raw terminal failed: The handle is invalid?\" issue with \"virtctl console\" when not executed in a pty</li> <li>[PR #6783][rmohr] Skip SSH RSA auth if no RSA key was explicitly provided and not key exists at the default location</li> </ul>"},{"location":"release_notes/#v0481","title":"v0.48.1","text":"<p>Released on: Wed Dec 15 15:11:55 2021 +0000</p> <ul> <li>[PR #6900][kubevirt-bot] Skip SSH RSA auth if no RSA key was explicitly provided and not key exists at the default location</li> <li>[PR #6902][kubevirt-bot] Fix \"Make raw terminal failed: The handle is invalid?\" issue with \"virtctl console\" when not executed in a pty</li> </ul>"},{"location":"release_notes/#v0480","title":"v0.48.0","text":"<p>Released on: Mon Dec 6 18:26:51 2021 +0000</p> <ul> <li>[PR #6670][futuretea] Added 'virtctl soft-reboot' command to reboot the VMI.</li> <li>[PR #6861][orelmisan] virtctl errors are written to stderr instead of stdout</li> <li>[PR #6836][enp0s3] Added PHASE and VMI columns for the 'kubectl get vmim' CLI output</li> <li>[PR #6784][nunnatsa] kubevirt-config configMap is no longer supported for KubeVirt configuration</li> <li>[PR #6839][ShellyKa13] fix restore of VM with RunStrategy</li> <li>[PR #6533][zcahana] Paused VMIs are now marked as unready even when no readinessProbe is specified</li> <li>[PR #6858][rmohr] Fix a nil pointer in virtctl in combination with some external auth plugins</li> <li>[PR #6780][fossedihelm] Add PatchOptions to the Patch request of the VirtualMachineInstanceInterface</li> <li>[PR #6773][iholder-redhat] alert if migration for VMI with host-model CPU is stuck since no node is suitable</li> <li>[PR #6714][rhrazdil] Shorten timeout for Istio proxy detection</li> <li>[PR #6725][fossedihelm] added DryRun mode to virtcl for pause and unpause commands</li> <li>[PR #6737][davidvossel] Pending migration target pods timeout after 5 minutes when unschedulable</li> <li>[PR #6814][fossedihelm] Changed some terminology to be more inclusive</li> <li>[PR #6649][Barakmor1] Designate the apps.kubevirt.io/component label for KubeVirt components.</li> <li>[PR #6650][victortoso] Introduces support to ich9 or ac97 sound devices</li> <li>[PR #6734][Barakmor1] replacing the command that extract libvirtd's pid  to avoid this error:</li> <li>[PR #6802][rmohr] Maintain a separate api package which synchronizes to kubevirt.io/api for better third party integration with client-gen</li> <li>[PR #6730][zhhray] change kubevrit cert secret type from Opaque to kubernetes.io/tls</li> <li>[PR #6508][oshoval] Add missing domain to guest search list, in case subdomain is used.</li> <li>[PR #6664][vladikr] enable the display and ramfb for vGPUs by default</li> <li>[PR #6710][iholder-redhat] virt-launcher fix - stop logging successful shutdown when it isn't true</li> <li>[PR #6162][vladikr] KVM_HINTS_REALTIME will always be set when dedicatedCpusPlacement is requested</li> <li>[PR #6772][zcahana] Bugfix: revert #6565 which prevented upgrades to v0.47.</li> <li>[PR #6722][zcahana] Remove obsolete scheduler.alpha.kubernetes.io/critical-pod annotation</li> <li>[PR #6723][acardace] remove stale pdbs created by &lt; 0.41.1 virt-controller</li> <li>[PR #6721][iholder-redhat] Set default CPU model in VMI spec, even if not defined in KubevirtCR</li> <li>[PR #6713][zcahana] Report WaitingForVolumeBinding VM status when PVC/DV-type volumes reference unbound PVCs</li> <li>[PR #6681][fossedihelm] Users can use --dry-run flag</li> <li>[PR #6663][jean-edouard] The number of virt-api and virt-controller replicas is now configurable in the CSV</li> <li>[PR #5981][maya-r] Always resize disk.img files to the largest size at boot.</li> </ul>"},{"location":"release_notes/#v0471","title":"v0.47.1","text":"<p>Released on: Thu Nov 11 15:52:59 2021 +0000</p> <ul> <li>[PR #6775][kubevirt-bot] Bugfix: revert #6565 which prevented upgrades to v0.47.</li> <li>[PR #6703][mhenriks] Fix BZ 2018521 - On upgrade VirtualMachineSnapshots going to Failed</li> <li>[PR #6511][knopt] Fixed virt-api significant memory usage when using Cluster Profiler with large KubeVirt deployments. (#6478, @knopt)</li> <li>[PR #6629][awels] BugFix: Hotplugging more than one block device would cause IO error (#6564)</li> <li>[PR #6657][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.6.0 and QEMU 6.0.0.</li> <li>[PR #6565][Barakmor1] 'kubevirt-operator' changed to 'virt-operator' on 'managed-by' label in kubevirt's components made by virt-operator</li> <li>[PR #6642][ShellyKa13] Include hot-plugged disks in a Online VM Snapshot</li> <li>[PR #6513][brybacki] Adds force-bind flag to virtctl imageupload</li> <li>[PR #6588][erkanerol] Fix recording rules based on up metrics</li> <li>[PR #6575][davidvossel] VM controller now syncs VMI conditions to corresponding VM object</li> <li>[PR #6661][rmohr] Make the kubevirt api compatible with client-gen to make selecting compatible k8s golang dependencies easier</li> <li>[PR #6535][rmohr] Migrations use digests to reference containerDisks and kernel boot images to ensure disk consistency</li> <li>[PR #6651][ormergi] Kubevirt Conformance plugin now supports passing tests images registry.</li> <li>[PR #6589][iholder-redhat] custom kernel / initrd to boot from is now pre-pulled which improves stability</li> <li>[PR #6199][ormergi] Kubevirt Conformance plugin now supports passing image tag or digest</li> <li>[PR #6477][zcahana] Report DataVolumeError VM status when referenced a DataVolume indicates an error</li> <li>[PR #6593][rhrazdil] Removed python dependencies from virt-launcher and virt-handler containers</li> <li>[PR #6026][akrejcir] Implemented minimal VirtualMachineFlavor functionality.</li> <li>[PR #6570][erkanerol] Use honorLabels instead of labelDrop for namespace label on metrics</li> <li>[PR #6182][jordigilh] adds support for real time workloads</li> <li>[PR #6177][rmohr] Switch the node base images to centos8 stream</li> <li>[PR #6171][zcahana] Report ErrorPvcNotFound/ErrorDataVolumeNotFound VM status when PVC/DV-type volumes reference non-existent objects</li> <li>[PR #6437][VirrageS] Fix deprecated use of watch API to prevent reporting incorrect metrics.</li> <li>[PR #6482][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6375][dhiller] Rely on kubevirtci installing cdi during testing</li> </ul>"},{"location":"release_notes/#v0461","title":"v0.46.1","text":"<p>Released on: Tue Oct 19 15:41:10 2021 +0000</p> <ul> <li>[PR #6557][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> </ul>"},{"location":"release_notes/#v0460","title":"v0.46.0","text":"<p>Released on: Fri Oct 8 21:12:33 2021 +0000</p> <ul> <li>[PR #6425][awels] Hotplug disks are possible when iothreads are enabled.</li> <li>[PR #6297][acardace] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6464][awels] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6465][salanki] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> <li>[PR #6458][vladikr] Tagged SR-IOV interfaces will now appear in the config drive metadata</li> <li>[PR #6446][brybacki] Access mode for virtctl image upload is now optional. This version of virtctl now requires CDI v1.34 or greater</li> <li>[PR #6391][zcahana] Cleanup obsolete permissions from virt-operator's ClusterRole</li> <li>[PR #6419][rthallisey] Fix virt-controller panic caused by lots of deleted VMI events</li> <li>[PR #5972][kwiesmueller] Add a <code>ssh</code> command to <code>virtctl</code> that can be used to open SSH sessions to VMs/VMIs.</li> <li>[PR #6403][jrife] Removed go module pinning to an old version (v0.3.0) of github.com/go-kit/kit</li> <li>[PR #6367][brybacki] virtctl imageupload now uses DataVolume.spec.storage</li> <li>[PR #6198][iholder-redhat] Fire a Prometheus alert when a lot of REST failures are detected in virt-api</li> <li>[PR #6211][davidvossel] cluster-profiler pprof gathering tool and corresponding \"ClusterProfiler\" feature gate</li> <li>[PR #6323][vladikr] switch live migration to use unix sockets</li> <li>[PR #6374][vladikr] Fix the default setting of CPU requests on vmipods</li> <li>[PR #6283][rthallisey] Record the time it takes to delete a VMI and expose it as a metric</li> <li>[PR #6251][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6377][rmohr] Don't fail on failed selinux relabel attempts if selinux is permissive</li> <li>[PR #6308][awels] BugFix: hotplug was broken when using it with a hostpath volume that was on a separate device.</li> <li>[PR #6186][davidvossel] Add resource and verb labels to rest_client_requests_total metric</li> </ul>"},{"location":"release_notes/#v0451","title":"v0.45.1","text":"<p>Released on: Tue Oct 19 15:39:42 2021 +0000</p> <ul> <li>[PR #6537][kubevirt-bot] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> <li>[PR #6556][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6480][kubevirt-bot] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6384][kubevirt-bot] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> </ul>"},{"location":"release_notes/#v0450","title":"v0.45.0","text":"<p>Released on: Wed Sep 8 13:56:47 2021 +0000</p> <ul> <li>[PR #6191][marceloamaral] Addition of perfscale-load-generator to perform stress tests to evaluate the control plane</li> <li>[PR #6248][VirrageS] Reduced logging in hot paths</li> <li>[PR #6079][weihanglo] Hotplug volume can be unplugged at anytime and reattached after a VM restart.</li> <li>[PR #6101][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6204][sradco] This PR adds to each alert the runbook url that points to a runbook that provides additional details on each alert and how to mitigate it.</li> <li>[PR #5974][vladikr] a list of desired mdev types can now be provided in KubeVirt CR to kubevirt to configure these devices on relevant nodes</li> <li>[PR #6147][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #6161][ashleyschuett] Remove HostDevice validation on VMI creation</li> <li>[PR #6078][zcahana] Report ErrImagePull/ImagePullBackOff VM status when image pull errors occur</li> <li>[PR #6176][kwiesmueller] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #6047][ShellyKa13] Add phases to the vm snapshot api, specifically a failure phase</li> <li>[PR #6138][ansijain] NA</li> </ul>"},{"location":"release_notes/#v0443","title":"v0.44.3","text":"<p>Released on: Tue Oct 19 15:38:22 2021 +0000</p> <ul> <li>[PR #6518][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6532][kubevirt-bot] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6536][kubevirt-bot] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> </ul>"},{"location":"release_notes/#v0442","title":"v0.44.2","text":"<p>Released on: Thu Oct 7 12:55:34 2021 +0000</p> <ul> <li>[PR #6479][kubevirt-bot] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6392][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6251][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6344][kubevirt-bot] BugFix: hotplug was broken when using it with a hostpath volume that was on a separate device.</li> <li>[PR #6263][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6207][kubevirt-bot] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #6101][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6249][kubevirt-bot] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> </ul>"},{"location":"release_notes/#v0441","title":"v0.44.1","text":"<p>Released on: Thu Aug 12 12:28:02 2021 +0000</p> <ul> <li>[PR #6219][kubevirt-bot] Add phases to the vm snapshot api, specifically a failure phase</li> </ul>"},{"location":"release_notes/#v0440","title":"v0.44.0","text":"<p>Released on: Mon Aug 9 14:20:14 2021 +0000</p> <ul> <li>[PR #6058][acardace] Fix virt-launcher exit pod race condition</li> <li>[PR #6035][davidvossel] Addition of perfscale-audit tool for auditing performance of control plane during stress tests</li> <li>[PR #6145][acardace] virt-launcher: disable unencrypted TCP socket for libvirtd.</li> <li>[PR #6163][davidvossel] Handle qemu processes in defunc (zombie) state</li> <li>[PR #6105][ashleyschuett] Add VirtualMachineInstancesPerNode to KubeVirt CR under Spec.Configuration</li> <li>[PR #6104][zcahana] Report FailedUnschedulable VM status when scheduling errors occur</li> <li>[PR #5905][davidvossel] VM CrashLoop detection and Exponential Backoff</li> <li>[PR #6070][acardace] Initiate Live-Migration using a unix socket (exposed by virt-handler) instead of an additional TCP&lt;-&gt;Unix migration proxy started by virt-launcher</li> <li>[PR #5728][vasiliy-ul] Live migration of VMs with hotplug volumes is now enabled</li> <li>[PR #6109][rmohr] Fix virt-controller SCC: Reflect the need for NET_BIND_SERVICE in the virt-controller SCC.</li> <li>[PR #5942][ShellyKa13] Integrate guest agent to online VM snapshot</li> <li>[PR #6034][ashleyschuett] Go version updated to version 1.16.6</li> <li>[PR #6040][yuhaohaoyu] Improved debuggability by keeping the environment of a failed VMI alive.</li> <li>[PR #6068][dhiller] Add check that not all tests have been skipped</li> <li>[PR #6041][xpivarc] [Experimental] Virt-launcher can run as non-root user</li> <li>[PR #6062][iholder-redhat] replace dead \"stress\" binary with new, maintained, \"stress-ng\" binary</li> <li>[PR #6029][mhenriks] CDI to 1.36.0 with DataSource support</li> <li>[PR #4089][victortoso] Add support to USB Redirection with usbredir</li> <li>[PR #5946][vatsalparekh] Add guest-agent based ping probe</li> <li>[PR #6005][acardace] make containerDisk validation memory usage limit configurable</li> <li>[PR #5791][zcahana] Added a READY column to the tabular output of \"kubectl get vm/vmi\"</li> <li>[PR #6006][awels] DataVolumes created by DataVolumeTemplates will follow the associated VMs priority class.</li> <li>[PR #5982][davidvossel] Reduce vmi Update collisions (http code 409) during startup</li> <li>[PR #5891][akalenyu] BugFix: Pending VMIs when creating concurrent bulk of VMs backed by WFFC DVs</li> <li>[PR #5925][rhrazdil] Fix issue with Windows VMs not being assigned IP address configured in network-attachment-definition IPAM.</li> <li>[PR #6007][rmohr] Fix: The bandwidth limitation on migrations is no longer ignored. Caution: The default bandwidth limitation of 64Mi is changed to \"unlimited\" to not break existing installations.</li> <li>[PR #4944][kwiesmueller] Add <code>/portforward</code> subresource to <code>VirtualMachine</code> and <code>VirtualMachineInstance</code> that can tunnel TCP traffic through the API Server using a websocket stream.</li> <li>[PR #5402][alicefr] Integration of libguestfs-tools and added new command <code>guestfs</code> to virtctl</li> <li>[PR #5953][ashleyschuett] Allow Failed VMs to be stopped when using <code>--force --gracePeriod 0</code></li> <li>[PR #5876][mlsorensen] KubeVirt CR supports specifying a runtime class for virt-launcher pods via 'launcherRuntimeClass'.</li> </ul>"},{"location":"release_notes/#v0431","title":"v0.43.1","text":"<p>Released on: Tue Oct 19 15:36:32 2021 +0000</p> <ul> <li>[PR #6555][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6052][kubevirt-bot] make containerDisk validation memory usage limit configurable</li> </ul>"},{"location":"release_notes/#v0430","title":"v0.43.0","text":"<p>Released on: Fri Jul 9 15:46:22 2021 +0000</p> <ul> <li>[PR #5952][mhenriks] Use CDI beta API. CDI v1.20.0 is now the minimum requirement for kubevirt.</li> <li>[PR #5846][rmohr] Add \"spec.cpu.numaTopologyPassthrough\" which allows emulating a host-alligned virtual numa topology for high performance</li> <li>[PR #5894][rmohr] Add <code>spec.migrations.disableTLS</code> to the KubeVirt CR to allow disabling encrypted migrations. They stay secure by default.</li> <li>[PR #5649][awels] Enhancement: remove one attachment pod per disk limit (behavior on upgrade with running VM with hotplugged disks is undefined)</li> <li>[PR #5742][rmohr] VMIs which choose evictionStrategy <code>LifeMigrate</code> and request the <code>invtsc</code> cpuflag are now live-migrateable</li> <li>[PR #5911][dhiller] Bumps kubevirtci, also suppresses kubectl.sh output to avoid confusing checks</li> <li>[PR #5863][xpivarc] Fix: ioerrors don't cause crash-looping of notify server</li> <li>[PR #5867][mlsorensen] New build target added to export virt-* images as a tar archive.</li> <li>[PR #5766][davidvossel] Addition of kubevirt_vmi_phase_transition_seconds_since_creation to monitor how long it takes to transition a VMI to a specific phase from creation time.</li> <li>[PR #5823][dhiller] Change default branch to <code>main</code> for <code>kubevirt/kubevirt</code> repository</li> <li>[PR #5763][nunnatsa] Fix bug 1945589: Prevent migration of VMIs that uses virtiofs</li> <li>[PR #5827][mlsorensen] Auto-provisioned disk images on empty PVCs now leave 128KiB unused to avoid edge cases that run the volume out of space.</li> <li>[PR #5849][davidvossel] Fixes event recording causing a segfault in virt-controller</li> <li>[PR #5797][rhrazdil] Add serviceAccountDisk automatically when Istio is enabled in VMI annotations</li> <li>[PR #5723][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5806][mlsorensen] configmap, secret, and cloud-init raw disks now work when underlying node storage has 4k blocks.</li> <li>[PR #5623][iholder-redhat] [bugfix]: Allow migration of VMs with host-model CPU to migrate only for compatible nodes</li> <li>[PR #5716][rhrazdil] Fix issue with virt-launcher becoming <code>NotReady</code> after migration when Istio is used.</li> <li>[PR #5778][ashleyschuett] Update ca-bundle if it is unable to be parsed</li> <li>[PR #5787][acardace] migrated references of authorization/v1beta1 to authorization/v1</li> <li>[PR #5461][rhrazdil] Add support for Istio proxy when no explicit ports are specified on masquerade interface</li> <li>[PR #5751][acardace] EFI VMIs with secureboot disabled can now be booted even when only OVMF_CODE.secboot.fd and OVMF_VARS.fd are present in the virt-launcher image</li> <li>[PR #5629][andreyod] Support starting Virtual Machine with its guest CPU paused using <code>virtctl start --paused</code></li> <li>[PR #5725][dhiller] Generate REST API coverage report after functional tests</li> <li>[PR #5758][davidvossel] Fixes kubevirt_vmi_phase_count to include all phases, even those that occur before handler hand off.</li> <li>[PR #5745][ashleyschuett] Alert with resource usage exceeds resource requests</li> <li>[PR #5759][mhenriks] Update CDI to 1.34.1</li> <li>[PR #5038][kwiesmueller] Add exec command to VM liveness and readinessProbe executed through the qemu-guest-agent.</li> <li>[PR #5431][alonSadan] Add NFT and IPTables rules to allow port-forward to non-declared ports on the VMI. Declaring ports on VMI will limit</li> </ul>"},{"location":"release_notes/#v0422","title":"v0.42.2","text":"<p>Released on: Tue Oct 19 15:34:37 2021 +0000</p> <ul> <li>[PR #6554][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5887][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5907][kubevirt-bot] Fix: ioerrors don't cause crash-looping of notify server</li> <li>[PR #5871][maiqueb] Fix: do not override with the DHCP server advertising IP with the gateway info.</li> <li>[PR #5875][kubevirt-bot] Update ca-bundle if it is unable to be parsed</li> </ul>"},{"location":"release_notes/#v0421","title":"v0.42.1","text":"<p>Released on: Thu Jun 10 01:31:52 2021 +0000</p> <ul> <li>[PR #5738][rmohr] Stop releasing jinja2 templates of our operator. Kustomize is the preferred way for customizations.</li> <li>[PR #5691][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #5558][ormergi] Drop virt-launcher SYS_RESOURCE capability</li> <li>[PR #5694][davidvossel] Fixes null pointer dereference in migration controller</li> <li>[PR #5416][iholder-redhat] [feature] support booting VMs from a custom kernel/initrd images with custom kernel arguments</li> <li>[PR #5495][iholder-redhat] Go version updated to version 1.16.1.</li> <li>[PR #5502][rmohr] Add downwardMetrics volume to expose a limited set of hots metrics to guests</li> <li>[PR #5601][maya-r] Update libvirt-go to 7.3.0</li> <li>[PR #5661][davidvossel] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5652][rmohr] Automatically discover kube-prometheus installations and configure kubevirt monitoring</li> <li>[PR #5631][davidvossel] Expand backport policy to include logging and debug fixes</li> <li>[PR #5528][zcahana] Introduced a \"status.printableStatus\" field in the VirtualMachine CRD. This field is now displayed in the tabular output of \"kubectl get vm\".</li> <li>[PR #5200][rhrazdil] Add support for Istio proxy traffic routing with masquerade interface. nftables is required for this feature.</li> <li>[PR #5560][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5514][rhrazdil] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5583][dhiller] Reenable coverage</li> <li>[PR #5129][davidvossel] Gracefully shutdown virt-api connections and ensure zero exit code under normal shutdown conditions</li> <li>[PR #5582][dhiller] Fix flaky unit tests</li> <li>[PR #5600][davidvossel] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #5564][omeryahud] virtctl rename support is dropped</li> <li>[PR #5585][iholder-redhat] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5595][zcahana] Fixes adoption of orphan DataVolumes</li> <li>[PR #5566][davidvossel] Release branches are now cut on the first business day of the month rather than the first day.</li> <li>[PR #5108][Omar007] Fixes handling of /proc//mountpoint by working on the device information instead of mount information <li>[PR #5250][mlsorensen] Controller health checks will no longer actively test connectivity to the Kubernetes API. They will rely in health of their watches to determine if they have API connectivity.</li> <li>[PR #5563][ashleyschuett] Set KubeVirt resources flags in the KubeVirt CR</li> <li>[PR #5328][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li>"},{"location":"release_notes/#v0420","title":"v0.42.0","text":"<p>Released on: Tue Jun 8 12:09:49 2021 +0000</p> <ul> <li>[PR #5738][rmohr] Stop releasing jinja2 templates of our operator. Kustomize is the preferred way for customizations.</li> <li>[PR #5691][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #5558][ormergi] Drop virt-launcher SYS_RESOURCE capability</li> <li>[PR #5694][davidvossel] Fixes null pointer dereference in migration controller</li> <li>[PR #5416][iholder-redhat] [feature] support booting VMs from a custom kernel/initrd images with custom kernel arguments</li> <li>[PR #5495][iholder-redhat] Go version updated to version 1.16.1.</li> <li>[PR #5502][rmohr] Add downwardMetrics volume to expose a limited set of hots metrics to guests</li> <li>[PR #5601][maya-r] Update libvirt-go to 7.3.0</li> <li>[PR #5661][davidvossel] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5652][rmohr] Automatically discover kube-prometheus installations and configure kubevirt monitoring</li> <li>[PR #5631][davidvossel] Expand backport policy to include logging and debug fixes</li> <li>[PR #5528][zcahana] Introduced a \"status.printableStatus\" field in the VirtualMachine CRD. This field is now displayed in the tabular output of \"kubectl get vm\".</li> <li>[PR #5200][rhrazdil] Add support for Istio proxy traffic routing with masquerade interface. nftables is required for this feature.</li> <li>[PR #5560][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5514][rhrazdil] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5583][dhiller] Reenable coverage</li> <li>[PR #5129][davidvossel] Gracefully shutdown virt-api connections and ensure zero exit code under normal shutdown conditions</li> <li>[PR #5582][dhiller] Fix flaky unit tests</li> <li>[PR #5600][davidvossel] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #5564][omeryahud] virtctl rename support is dropped</li> <li>[PR #5585][iholder-redhat] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5595][zcahana] Fixes adoption of orphan DataVolumes</li> <li>[PR #5566][davidvossel] Release branches are now cut on the first business day of the month rather than the first day.</li> <li>[PR #5108][Omar007] Fixes handling of /proc//mountpoint by working on the device information instead of mount information <li>[PR #5250][mlsorensen] Controller health checks will no longer actively test connectivity to the Kubernetes API. They will rely in health of their watches to determine if they have API connectivity.</li> <li>[PR #5563][ashleyschuett] Set KubeVirt resources flags in the KubeVirt CR</li> <li>[PR #5328][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li>"},{"location":"release_notes/#v0414","title":"v0.41.4","text":"<p>Released on: Tue Oct 19 15:31:59 2021 +0000</p> <ul> <li>[PR #6573][acardace] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6517][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6333][acardace] Fix virt-launcher exit pod race condition</li> <li>[PR #6401][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #6147][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #5673][kubevirt-bot] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #6227][kwiesmueller] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> </ul>"},{"location":"release_notes/#v0413","title":"v0.41.3","text":"<p>Released on: Thu Aug 12 16:35:43 2021 +0000</p> <ul> <li>[PR #6196][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #6194][kubevirt-bot] Allow Failed VMs to be stopped when using <code>--force --gracePeriod 0</code></li> <li>[PR #6039][akalenyu] BugFix: Pending VMIs when creating concurrent bulk of VMs backed by WFFC DVs</li> <li>[PR #5917][davidvossel] Fixes event recording causing a segfault in virt-controller</li> <li>[PR #5886][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5866][xpivarc] Fix: Kubevirt build with golang 1.14+ will not fail on validation of container disk with memory allocation error</li> <li>[PR #5873][kubevirt-bot] Update ca-bundle if it is unable to be parsed</li> <li>[PR #5822][kubevirt-bot] migrated references of authorization/v1beta1 to authorization/v1</li> <li>[PR #5704][davidvossel] Fix virt-controller clobbering in progress vmi migration state during virt handler handoff</li> <li>[PR #5707][kubevirt-bot] Fixes null pointer dereference in migration controller</li> <li>[PR #5685][stu-gott] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5670][stu-gott] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5653][kubevirt-bot] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5644][kubevirt-bot] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5646][kubevirt-bot] virtctl rename support is dropped</li> </ul>"},{"location":"release_notes/#v0412","title":"v0.41.2","text":"<p>Released on: Wed Jul 28 12:13:19 2021 -0400</p>"},{"location":"release_notes/#v0411","title":"v0.41.1","text":"<p>Released on: Wed Jul 28 12:08:42 2021 -0400</p>"},{"location":"release_notes/#v0410","title":"v0.41.0","text":"<p>Released on: Wed May 12 14:30:49 2021 +0000</p> <ul> <li>[PR #5586][kubevirt-bot] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li> <li>[PR #5344][ashleyschuett] Reconcile PrometheusRules and ServiceMonitor resources</li> <li>[PR #5542][andreyod] Add startStrategy field to VMI spec to allow Virtual Machine start in paused state.</li> <li>[PR #5459][ashleyschuett] Reconcile service resource</li> <li>[PR #5520][ashleyschuett] Reconcile required labels and annotations on ConfigMap resources</li> <li>[PR #5533][rmohr] Fix <code>docker save</code> and <code>docker push</code> issues with released kubevirt images</li> <li>[PR #5428][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5410][ashleyschuett] Reconcile ServiceAccount resources</li> <li>[PR #5109][Omar007] Add support for specifying a logical and physical block size for disk devices</li> <li>[PR #5471][ashleyschuett] Reconcile APIService resources</li> <li>[PR #5513][ashleyschuett] Reconcile Secret resources</li> <li>[PR #5496][davidvossel] Improvements to migration proxy logging</li> <li>[PR #5376][ashleyschuett] Reconcile CustomResourceDefinition resources</li> <li>[PR #5435][AlonaKaplan] Support dual stack service on \"virtctl expose\"-</li> <li>[PR #5425][davidvossel] Fixes VM restart during eviction when EvictionStrategy=LiveMigrate</li> <li>[PR #5423][ashleyschuett] Add resource requests to virt-controller, virt-api, virt-operator and virt-handler</li> <li>[PR #5343][erkanerol] Some cleanups and small additions to the storage metrics</li> <li>[PR #4682][stu-gott] Updated Guest Agent Version compatibility check. The new approach is much more accurate.</li> <li>[PR #5485][rmohr] Fix fallback to iptables if nftables is not used on the host on arm64</li> <li>[PR #5426][rmohr] Fix fallback to iptables if nftables is not used on the host</li> <li>[PR #5403][tiraboschi] Added a kubevirt_ prefix to several recording rules and metrics</li> <li>[PR #5241][stu-gott] Introduced Duration and RenewBefore parameters for cert rotation. Previous values are now deprecated.</li> <li>[PR #5463][acardace] Fixes upgrades from KubeVirt v0.36</li> <li>[PR #5456][zhlhahaha] Enable arm64 cross-compilation</li> <li>[PR #3310][davidvossel] Doc outlines our Kubernetes version compatibility commitment</li> <li>[PR #3383][EdDev] Add <code>vmIPv6NetworkCIDR</code> under <code>NetworkSource.pod</code> to support custom IPv6 CIDR for the vm network when using masquerade binding.</li> <li>[PR #3415][zhlhahaha] Make kubevirt code fit for arm64 support. No testing is at this stage performed against arm64 at this point.</li> <li>[PR #5147][xpivarc] Remove CAP_NET_ADMIN from the virt-launcher pod(second take).</li> <li>[PR #5351][awels] Support hotplug with virtctl using addvolume and removevolume commands</li> <li>[PR #5050][ashleyschuett] Fire Prometheus Alert when a vmi is orphaned for more than an hour</li> </ul>"},{"location":"release_notes/#v0401","title":"v0.40.1","text":"<p>Released on: Tue Oct 19 13:33:33 2021 +0000</p> <ul> <li>[PR #6598][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6287][kubevirt-bot] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #5559][kubevirt-bot] Fix <code>docker save</code> issues with kubevirt images</li> <li>[PR #5500][kubevirt-bot] Support hotplug with virtctl using addvolume and removevolume commands</li> </ul>"},{"location":"release_notes/#v0400","title":"v0.40.0","text":"<p>Released on: Mon Apr 19 12:25:41 2021 +0000</p> <ul> <li>[PR #5467][rmohr] Fixes upgrades from KubeVirt v0.36</li> <li>[PR #5350][jean-edouard] Removal of entire <code>permittedHostDevices</code> section will now remove all user-defined host device plugins.</li> <li>[PR #5242][jean-edouard] Creating more than 1 migration at the same time for a given VMI will now fail</li> <li>[PR #4907][vasiliy-ul] Initial cgroupv2 support</li> <li>[PR #5324][jean-edouard] Default feature gates can now be defined in the provider configuration.</li> <li>[PR #5006][alicefr] Add discard=unmap option</li> <li>[PR #5022][davidvossel] Fixes race condition between operator adding service and webhooks that can result in installs/uninstalls failing</li> <li>[PR #5310][ashleyschuett] Reconcile CRD resources</li> <li>[PR #5102][iholder-redhat] Go version updated to 1.14.14</li> <li>[PR #4746][ashleyschuett] Reconcile Deployments, DaemonSets, MutatingWebhookConfigurations and ValidatingWebhookConfigurations</li> <li>[PR #5037][ormergi] Hot-plug SR-IOV VF interfaces to VM's post a successful migration.</li> <li>[PR #5269][mlsorensen] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> <li>[PR #5138][davidvossel] virt-handler now waits up to 5 minutes for all migrations on the node to complete before shutting down.</li> <li>[PR #5191][yuvalturg] Added a metric for monitoring CPU affinity</li> <li>[PR #5215][xphyr] Enable detection of Intel GVT-g vGPU.</li> <li>[PR #4760][rmohr] Make virt-handler heartbeat more efficient and robust: Only one combined PATCH and no need to detect different cluster types anymore.</li> <li>[PR #5091][iholder-redhat] QEMU SeaBios debug logs are being seen as part of virt-launcher log.</li> <li>[PR #5221][rmohr] Remove  workload placement validation webhook which blocks placement updates when VMIs are running</li> <li>[PR #5128][yuvalturg] Modified memory related metrics by adding several new metrics and splitting the swap traffic bytes metric</li> <li>[PR #5084][ashleyschuett] Add validation to CustomizeComponents object on the KubeVirt resource</li> <li>[PR #5182][davidvossel] New [release-blocker] functional test marker to signify tests that can never be disabled before making a release</li> <li>[PR #5137][davidvossel] Added our policy around release branch backporting in docs/release-branch-backporting.md</li> <li>[PR #5096][yuvalturg] Modified networking metrics by adding new metrics, splitting existing ones by rx/tx and using the device alias for the interface name when available</li> <li>[PR #5088][awels] Hotplug works with hostpath storage.</li> <li>[PR #4908][dhiller] Move travis tag and master builds to kubevirt prow.</li> <li>[PR #4741][EdDev] Allow live migration for SR-IOV VM/s without preserving the VF interfaces.</li> </ul>"},{"location":"release_notes/#v0392","title":"v0.39.2","text":"<p>Released on: Tue Oct 19 13:29:33 2021 +0000</p> <ul> <li>[PR #6597][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5854][rthallisey] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> <li>[PR #5561][kubevirt-bot] Fix <code>docker save</code> issues with kubevirt images</li> </ul>"},{"location":"release_notes/#v0391","title":"v0.39.1","text":"<p>Released on: Tue Apr 13 12:10:13 2021 +0000</p>"},{"location":"release_notes/#v0390","title":"v0.39.0","text":"<p>Released on: Wed Mar 10 14:51:58 2021 +0000</p> <ul> <li>[PR #5010][jean-edouard] Migrated VMs stay persistent and can therefore survive S3, among other things.</li> <li>[PR #4952][ashleyschuett] Create warning NodeUnresponsive event if a node is running a VMI pod but not a virt-handler pod</li> <li>[PR #4686][davidvossel] Automated workload updates via new KubeVirt WorkloadUpdateStrategy API</li> <li>[PR #4886][awels] Hotplug support for WFFC datavolumes.</li> <li>[PR #5026][AlonaKaplan] virt-launcher, masquerade binding - prefer nft over iptables.</li> <li>[PR #4921][borod108] Added support for Sysprep in the API. A user can now add a answer file through a ConfigMap or a Secret. The User Guide is updated accordingly. /kind feature</li> <li>[PR #4874][ormergi] Add new feature-gate SRIOVLiveMigration,</li> <li>[PR #4917][iholder-redhat] Now it is possible to enable QEMU SeaBios debug logs setting virt-launcher log verbosity to be greater than 5.</li> <li>[PR #4966][arnongilboa] Solve virtctl \"Error when closing file ... file already closed\" that shows after successful image upload</li> <li>[PR #4489][salanki] Fix a bug where a disk.img file was created on filesystems mounted via Virtio-FS</li> <li>[PR #4982][xpivarc] Fixing handling of transient domain</li> <li>[PR #4984][ashleyschuett] Change customizeComponents.patches such that '*' resourceName or resourceType matches all, all fields of a patch (type, patch, resourceName, resourceType) are now required.</li> <li>[PR #4972][vladikr] allow disabling pvspinlock to support older guest kernels</li> <li>[PR #4927][yuhaohaoyu] Fix of XML and JSON marshalling/unmarshalling for user defined device alias names which can make migrations fail.</li> <li>[PR #4552][rthallisey] VMs using bridged networking will survive a kubelet restart by having kubevirt create a dummy interface on the virt-launcher pods, so that some Kubernetes CNIs, that have implemented the <code>CHECK</code> RPC call, will not cause VMI pods to enter a failed state.</li> <li>[PR #4883][iholder-redhat] Bug fixed: Enabling libvirt debug logs only if debugLogs label value is \"true\", disabling otherwise.</li> <li>[PR #4840][alicefr] Generate k8s events on IO errors</li> <li>[PR #4940][vladikr] permittedHostDevices will support both upper and lowercase letters in the device ID</li> </ul>"},{"location":"release_notes/#v0382","title":"v0.38.2","text":"<p>Released on: Tue Oct 19 13:24:57 2021 +0000</p> <ul> <li>[PR #6596][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5853][rthallisey] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> </ul>"},{"location":"release_notes/#v0381","title":"v0.38.1","text":"<p>Released on: Mon Feb 8 19:00:24 2021 +0000</p> <ul> <li>[PR #4870][qinqon] Bump k8s deps to 0.20.2</li> <li>[PR #4571][yuvalturg] Added os, workflow and flavor labels to the kubevirt_vmi_phase_count metric</li> <li>[PR #4659][salanki] Fixed an issue where non-root users inside a guest could not write to a Virtio-FS mount.</li> <li>[PR #4844][xpivarc] Fixed limits/requests to accept int again</li> <li>[PR #4850][rmohr] virtio-scsi now respects the useTransitionalVirtio flag instead of assigning a virtio version depending on the machine layout</li> <li>[PR #4672][vladikr] allow increasing logging verbosity of infra components in KubeVirt CR</li> <li>[PR #4838][rmohr] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> <li>[PR #4806][rmohr] Make the mutating webhooks for VMIs and VMs  required to avoid letting entities into the cluster which are not properly defaulted</li> <li>[PR #4779][brybacki] Error message on virtctl image-upload to WaitForFirstConsumer DV</li> <li>[PR #4749][davidvossel] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> <li>[PR #4772][jean-edouard] Faster VMI phase transitions thanks to an increased number of VMI watch threads in virt-controller</li> <li>[PR #4730][rmohr] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> </ul>"},{"location":"release_notes/#v0380","title":"v0.38.0","text":"<p>Released on: Mon Feb 8 13:15:32 2021 +0000</p> <ul> <li>[PR #4870][qinqon] Bump k8s deps to 0.20.2</li> <li>[PR #4571][yuvalturg] Added os, workflow and flavor labels to the kubevirt_vmi_phase_count metric</li> <li>[PR #4659][salanki] Fixed an issue where non-root users inside a guest could not write to a Virtio-FS mount.</li> <li>[PR #4844][xpivarc] Fixed limits/requests to accept int again</li> <li>[PR #4850][rmohr] virtio-scsi now respects the useTransitionalVirtio flag instead of assigning a virtio version depending on the machine layout</li> <li>[PR #4672][vladikr] allow increasing logging verbosity of infra components in KubeVirt CR</li> <li>[PR #4838][rmohr] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> <li>[PR #4806][rmohr] Make the mutating webhooks for VMIs and VMs  required to avoid letting entities into the cluster which are not properly defaulted</li> <li>[PR #4779][brybacki] Error message on virtctl image-upload to WaitForFirstConsumer DV</li> <li>[PR #4749][davidvossel] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> <li>[PR #4772][jean-edouard] Faster VMI phase transitions thanks to an increased number of VMI watch threads in virt-controller</li> <li>[PR #4730][rmohr] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> </ul>"},{"location":"release_notes/#v0372","title":"v0.37.2","text":"<p>Released on: Wed Jan 27 17:49:36 2021 +0000</p> <ul> <li>[PR #4872][kubevirt-bot] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> <li>[PR #4855][kubevirt-bot] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> </ul>"},{"location":"release_notes/#v0371","title":"v0.37.1","text":"<p>Released on: Thu Jan 21 16:20:52 2021 +0000</p> <ul> <li>[PR #4842][kubevirt-bot] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> </ul>"},{"location":"release_notes/#v0370","title":"v0.37.0","text":"<p>Released on: Mon Jan 18 17:57:03 2021 +0000</p> <ul> <li>[PR #4654][AlonaKaplan] Introduce virt-launcher DHCPv6 server.</li> <li>[PR #4669][kwiesmueller] Add nodeSelector to kubevirt components restricting them to run on linux nodes only.</li> <li>[PR #4648][davidvossel] Update libvirt base container to be based of packages in rhel-av 8.3</li> <li>[PR #4653][qinqon] Allow configure cloud-init with networkData only.</li> <li>[PR #4644][ashleyschuett] Operator validation webhook will deny updates to the workloads object of the KubeVirt CR if there are running VMIs</li> <li>[PR #3349][davidvossel] KubeVirt v1 GA api</li> <li>[PR #4645][maiqueb] Re-introduce the CAP_NET_ADMIN, to allow migration of VMs already having it.</li> <li>[PR #4546][yuhaohaoyu] Failure detection and handling for VM with EFI Insecure Boot in KubeVirt environments where EFI Insecure Boot is not supported by design.</li> <li>[PR #4625][awels] virtctl upload now shows error when specifying access mode of ReadOnlyMany</li> <li>[PR #4396][xpivarc] KubeVirt is now explainable!</li> <li>[PR #4517][danielBelenky] Fix guest agent reporting.</li> </ul>"},{"location":"release_notes/#v0362","title":"v0.36.2","text":"<p>Released on: Mon Feb 22 10:20:40 2021 -0500</p>"},{"location":"release_notes/#v0361","title":"v0.36.1","text":"<p>Released on: Tue Jan 19 12:30:33 2021 +0100</p>"},{"location":"release_notes/#v0360","title":"v0.36.0","text":"<p>Released on: Wed Dec 16 14:30:37 2020 +0000</p> <ul> <li>[PR #4667][kubevirt-bot] Update libvirt base container to be based of packages in rhel-av 8.3</li> <li>[PR #4634][kubevirt-bot] Failure detection and handling for VM with EFI Insecure Boot in KubeVirt environments where EFI Insecure Boot is not supported by design.</li> <li>[PR #4647][kubevirt-bot] Re-introduce the CAP_NET_ADMIN, to allow migration of VMs already having it.</li> <li>[PR #4627][kubevirt-bot] Fix guest agent reporting.</li> <li>[PR #4458][awels] It is now possible to hotplug DataVolume and PVC volumes into a running Virtual Machine.</li> <li>[PR #4025][brybacki] Adds a special handling for DataVolumes in WaitForFirstConsumer state to support CDI's delayed binding mode.</li> <li>[PR #4217][mfranczy] Set only an IP address for interfaces reported by qemu-guest-agent. Previously that was CIDR.</li> <li>[PR #4195][davidvossel] AccessCredentials API for dynamic user/password and ssh public key injection</li> <li>[PR #4335][oshoval] VMI status displays SRIOV interfaces with their network name only when they have originally</li> <li>[PR #4408][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 6.6.0 and QEMU 5.1.0.</li> <li>[PR #4514][ArthurSens] <code>domain</code> label removed from metric <code>kubevirt_vmi_memory_unused_bytes</code></li> <li>[PR #4542][danielBelenky] Fix double migration on node evacuation</li> <li>[PR #4506][maiqueb] Remove CAP_NET_ADMIN from the virt-launcher pod.</li> <li>[PR #4501][AlonaKaplan] CAP_NET_RAW removed from virt-launcher.</li> <li>[PR #4488][salanki] Disable Virtio-FS metadata cache to prevent OOM conditions on the host.</li> <li>[PR #3937][vladikr] Generalize host devices assignment. Provides an interface between kubevirt and external device plugins. Provides a mechanism for accesslisting host devices.</li> <li>[PR #4443][rmohr] All kubevirt webhooks support now dry-runs.</li> </ul>"},{"location":"release_notes/#v0350","title":"v0.35.0","text":"<p>Released on: Mon Nov 9 13:08:27 2020 +0000</p> <ul> <li>[PR #4409][vladikr] Increase the static memory overhead by 10Mi</li> <li>[PR #4272][maiqueb] Add <code>ip-family</code> to the <code>virtctl expose</code> command.</li> <li>[PR #4398][rmohr] VMIs reflect deleted stuck virt-launcher pods with the \"PodTerminating\" reason in the ready condition. The VMIRs detects this reason and immediately creates replacement VMIs.</li> <li>[PR #4393][salanki] Disable legacy service links in <code>virt-launcher</code> Pods to speed up Pod instantiation and decrease Kubelet load in namespaces with many services.</li> <li>[PR #2935][maiqueb] Add the macvtap bind mechanism.</li> <li>[PR #4132][mstarostik] fixes a bug that prevented unique device name allocation when configuring both SCSI and SATA drives</li> <li>[PR #3257][xpivarc] Added support of <code>kubectl explain</code> for Kubevirt resources.</li> <li>[PR #4288][ezrasilvera] Adding DownwardAPI volumes type</li> <li>[PR #4233][maya-r] Update base image used for pods to Fedora 31.</li> <li>[PR #4192][xpivarc] We now run gosec in Kubevirt</li> <li>[PR #4328][stu-gott] Version 2.x QEMU guest agents are supported.</li> <li>[PR #4289][AlonaKaplan] Masquerade binding - set the virt-launcher pod interface MTU on the bridge.</li> <li>[PR #4300][maiqueb] Update the NetworkInterfaceMultiqueue openAPI documentation to better specify its semantics within KubeVirt.</li> <li>[PR #4277][awels] PVCs populated by DVs are now allowed as volumes.</li> <li>[PR #4265][dhiller] Fix virtctl help text when running as a plugin</li> <li>[PR #4273][dhiller] Only run Travis build for PRs against release branches</li> </ul>"},{"location":"release_notes/#v0342","title":"v0.34.2","text":"<p>Released on: Tue Nov 17 08:13:22 2020 -0500</p>"},{"location":"release_notes/#v0341","title":"v0.34.1","text":"<p>Released on: Mon Nov 16 08:22:56 2020 -0500</p>"},{"location":"release_notes/#v0340","title":"v0.34.0","text":"<p>Released on: Wed Oct 7 13:59:50 2020 +0300</p> <ul> <li>[PR #4315][kubevirt-bot] PVCs populated by DVs are now allowed as volumes.</li> <li>[PR #3837][jean-edouard] VM interfaces with no <code>bootOrder</code> will no longer be candidates for boot when using the BIOS bootloader, as documented</li> <li>[PR #3879][ashleyschuett] KubeVirt should now be configured through the KubeVirt CR <code>configuration</code> key. The usage of the kubevirt-config configMap will be deprecated in the future.</li> <li>[PR #4074][stu-gott] Fixed bug preventing non-admin users from pausing/unpausing VMs</li> <li>[PR #4252][rhrazdil] Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1853911</li> <li>[PR #4016][ashleyschuett] Allow for post copy VMI migrations</li> <li>[PR #4235][davidvossel] Fixes timeout failure that occurs when pulling large containerDisk images</li> <li>[PR #4263][rmohr] Add readiness and liveness probes to virt-handler, to clearly indicate readiness</li> <li>[PR #4248][maiqueb] always compile KubeVirt with selinux support on pure go builds.</li> <li>[PR #4012][danielBelenky] Added support for the eviction API for VMIs with eviction strategy. This enables VMIs to be live-migrated when the node is drained or when the descheduler wants to move a VMI to a different node.</li> <li>[PR #4075][ArthurSens] Metric kubevirt_vmi_vcpu_seconds' state label is now exposed as a human-readable state instead of an integer</li> <li>[PR #4162][vladikr] introduce a cpuAllocationRatio config parameter to normalize the number of CPUs requested for a pod, based on the number of vCPUs</li> <li>[PR #4177][maiqueb] Use vishvananda/netlink instead of songgao/water to create tap devices.</li> <li>[PR #4092][stu-gott] Allow specifying nodeSelectors, affinity and tolerations to control where KubeVirt components will run</li> <li>[PR #3927][ArthurSens] Adds new metric kubevirt_vmi_memory_unused_bytes</li> <li>[PR #3493][vladikr] virtio-fs is being added as experimental, protected by a feature-gate that needs to be enabled in the kubevirt config by the administrator</li> <li>[PR #4193][mhenriks] Add snapshot.kubevirt.io to admin/edit/view roles</li> <li>[PR #4149][qinqon] Bump kubevirtci to k8s-1.19</li> <li>[PR #3471][crobinso] Allow hiding that the VM is running on KVM, so that Nvidia graphics cards can be passed through</li> <li>[PR #4115][phoracek] Add conformance automation and manifest publishing</li> <li>[PR #3733][davidvossel] each PRs description.</li> <li>[PR #4082][mhenriks] VirtualMachineRestore API and implementation</li> <li>[PR #4154][davidvossel] Fixes issue with Service endpoints not being updated properly in place during KubeVirt updates.</li> <li>[PR #3289][vatsalparekh] Add option to run only VNC Proxy in virtctl</li> <li>[PR #4027][alicefr] Added memfd as default memory backend for hugepages. This introduces the new annotation kubevirt.io/memfd to disable memfd as default and fallback to the previous behavior.</li> <li>[PR #3612][ashleyschuett] Adds <code>customizeComponents</code> to the kubevirt api</li> <li>[PR #4029][cchengleo] Fix an issue which prevented virt-operator from installing monitoring resources in custom namespaces.</li> <li>[PR #4031][rmohr] Initial support for sonobuoy for conformance testing</li> </ul>"},{"location":"release_notes/#v0330","title":"v0.33.0","text":"<p>Released on: Tue Sep 15 14:46:00 2020 +0000</p> <ul> <li>[PR #3226][vatsalparekh] Added tests to verify custom pciAddress slots and function</li> <li>[PR #4048][davidvossel] Improved reliability for failed migration retries</li> <li>[PR #3585][mhenriks] \"virtctl image-upload pvc ...\" will create the PVC if it does not exist</li> <li>[PR #3945][xpivarc] KubeVirt is now being built with Go1.13.14</li> <li>[PR #3845][ArthurSens] action required: The domain label from VMI metrics is being removed and may break dashboards that use the domain label to identify VMIs. Use name and namespace labels instead</li> <li>[PR #4011][dhiller] ppc64le arch has been disabled for the moment, see https://github.com/kubevirt/kubevirt/issues/4037</li> <li>[PR #3875][stu-gott] Resources created by KubeVirt are now labelled more clearly in terms of relationship and role.</li> <li>[PR #3791][ashleyschuett] make node as kubevirt.io/schedulable=false on virt-handler restart</li> <li>[PR #3998][vladikr] the local provider is usable again.</li> <li>[PR #3290][maiqueb] Have virt-handler (KubeVirt agent) create the tap devices on behalf of the virt-launchers.</li> <li>[PR #3957][AlonaKaplan] virt-launcher support Ipv6 on dual stack cluster.</li> <li>[PR #3952][davidvossel] Fixes rare situation where vmi may not properly terminate if failure occurs before domain starts.</li> <li>[PR #3973][xpivarc] Fixes VMs with clock.timezone set.</li> <li>[PR #3923][danielBelenky] Add support to configure QEMU I/O mode for VMIs</li> <li>[PR #3889][rmohr] The status fields for our CRDs are now protected on normal PATCH and PUT operations.The /status subresource is now used where possible for status updates.</li> <li>[PR #3568][xpivarc] Guest swap metrics available</li> </ul>"},{"location":"release_notes/#v0320","title":"v0.32.0","text":"<p>Released on: Tue Aug 11 19:21:56 2020 +0000</p> <ul> <li>[PR #3921][vladikr] use correct memory units in libvirt xml</li> <li>[PR #3893][davidvossel] Adds recurring period that rsyncs virt-launcher domains with virt-handler</li> <li>[PR #3880][sgarbour] Better error message when input parameters are not the expected number of parameters for each argument. Help menu will popup in case the number of parameters is incorrect.</li> <li>[PR #3785][xpivarc] Vcpu wait metrics available</li> <li>[PR #3642][vatsalparekh] Add a way to update VMI Status with latest Pod IP for Masquerade bindings</li> <li>[PR #3636][ArthurSens] Adds kubernetes metadata.labels as VMI metrics' label</li> <li>[PR #3825][awels] Virtctl now prints error messages from the response body on upload errors.</li> <li>[PR #3830][davidvossel] Fixes re-establishing domain notify client connections when domain notify server restarts due to an error event.</li> <li>[PR #3778][danielBelenky] Do not emit a SyncFailed event if we fail to sync a VMI in a final state</li> <li>[PR #3803][andreabolognani] Not sure what to write here (see above)</li> <li>[PR #2694][rmohr] Use native go libraries for selinux to not rely on python-selinux tools like semanage, which are not always present.</li> <li>[PR #3692][victortoso] QEMU logs can now be fetched from outside the pod</li> <li>[PR #3738][enp0s3] Restrict creation of VMI if it has labels that are used internally by Kubevirt components.</li> <li>[PR #3725][danielBelenky] The tests binary is now part of the release and can be consumed from the GitHub release page.</li> <li>[PR #3684][rmohr] Log if critical devices, like kvm, which virt-handler wants to expose are not present on the node.</li> <li>[PR #3166][petrkotas] Introduce new virtctl commands:</li> <li>[PR #3708][andreabolognani] Make qemu work on GCE by pulling in a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1822682</li> </ul>"},{"location":"release_notes/#v0310","title":"v0.31.0","text":"<p>Released on: Thu Jul 9 16:08:18 2020 +0300</p> <ul> <li>[PR 3690][davidvossel] Update go-grpc dependency to v1.30.0 in order to improve stability</li> <li>[PR 3628][AlonaKaplan] Avoid virt-handler crash in case of virt-launcher network configuration error</li> <li>[PR 3635][jean-edouard] The \"HostDisk\" feature gate has to be enabled to use hostDisks</li> <li>[PR 3641][vatsalparekh] Reverts kubevirt/kubevirt#3488 because CI seems to have merged it without all tests passing</li> <li>[PR 3488][vatsalparekh] Add a way to update VMI Status with latest Pod IP for Masquerade bindings</li> <li>[PR 3406][tomob] If a PVC was created by a DataVolume, it cannot be used as a Volume Source for a VM. The owning DataVolume has to be used instead.</li> <li>[PR 3566][kraxel] added: TigerVNC support for linux &amp; windows</li> <li>[PR 3529][jean-edouard] Enabling EFI will also enable Secure Boot, which requires SMM to be enabled.</li> <li>[PR 3455][ashleyschuett] Add KubevirtConfiguration, MigrationConfiguration, DeveloperConfiguration and NetworkConfiguration to API-types</li> <li>[PR 3520][rmohr] Fix hot-looping on the  VMI sync-condition if errors happen during the Scheduled phase of a VMI</li> <li>[PR 3220][mhenriks] API and controller/webhook for VirtualMachineSnapshots</li> </ul>"},{"location":"release_notes/#v0307","title":"v0.30.7","text":"<p>Released on: Mon Oct 26 11:57:21 2020 -0400</p>"},{"location":"release_notes/#v0306","title":"v0.30.6","text":"<p>Released on: Wed Aug 12 10:55:31 2020 +0200</p>"},{"location":"release_notes/#v0305","title":"v0.30.5","text":"<p>Released on: Fri Jul 17 05:26:37 2020 -0400</p>"},{"location":"release_notes/#v0304","title":"v0.30.4","text":"<p>Released on: Fri Jul 10 07:44:00 2020 -0400</p>"},{"location":"release_notes/#v0303","title":"v0.30.3","text":"<p>Released on: Tue Jun 30 17:39:42 2020 -0400</p>"},{"location":"release_notes/#v0302","title":"v0.30.2","text":"<p>Released on: Thu Jun 25 17:05:59 2020 -0400</p>"},{"location":"release_notes/#v0301","title":"v0.30.1","text":"<p>Released on: Tue Jun 16 13:10:17 2020 -0400</p>"},{"location":"release_notes/#v0300","title":"v0.30.0","text":"<p>Released on: Fri Jun 5 12:19:57 2020 +0200</p> <ul> <li>Tests: Many more test fixes</li> <li>Security: Introduce a custom SELinux policy for virt-launcher</li> <li>More user friendly IPv6 default CIDR for IPv6 addresses</li> <li>Fix OpenAPI compatibility issues by switching to openapi-gen</li> <li>Improved support for EFI boot (configurable OVMF path and test fixes)</li> <li>Improved VMI IP reporting</li> <li>Support propagation of annotations from VMI to pods</li> <li>Support for more fine grained (NET_RAW( capability granting to virt-launcher</li> <li>Support for eventual consistency with DataVolumes</li> </ul>"},{"location":"release_notes/#v0292","title":"v0.29.2","text":"<p>Released on: Mon May 25 21:15:30 2020 +0200</p>"},{"location":"release_notes/#v0291","title":"v0.29.1","text":"<p>Released on: Tue May 19 10:03:27 2020 +0200</p>"},{"location":"release_notes/#v0290","title":"v0.29.0","text":"<p>Released on: Wed May 6 15:01:57 2020 +0200</p> <ul> <li>Tests: Many many test fixes</li> <li>Tests: Many more test fixes</li> <li>CI: Add lane with SELinux enabled</li> <li>CI: Drop PPC64 support for now</li> <li>Drop Genie support</li> <li>Drop the use of hostPaths in the virt-launcher for improved security</li> <li>Support priority classes for important components</li> <li>Support IPv6 over masquerade binding</li> <li>Support certificate rotations based on shared secrets</li> <li>Support for VM ready condition</li> <li>Support for advanced node labelling (supported CPU Families and machine types)</li> </ul>"},{"location":"release_notes/#v0280","title":"v0.28.0","text":"<p>Released on: Thu Apr 9 23:01:29 2020 +0200</p> <ul> <li>CI: Try to discover flaky tests before merge</li> <li>Fix the use of priorityClasses</li> <li>Fix guest memory overhead calculation</li> <li>Fix SR-IOV device overhead requirements</li> <li>Fix loading of tun module during virt-handler initialization</li> <li>Fixes for several test cases</li> <li>Fixes to support running with container_t</li> <li>Support for renaming a VM</li> <li>Support ioEmulator thread pinning</li> <li>Support a couple of alerts for virt-handler</li> <li>Support for filesystem listing using the guest agent</li> <li>Support for retrieving data from the guest agent</li> <li>Support for device role tagging</li> <li>Support for assigning devices to the PCI root bus</li> <li>Support for guest overhead override</li> <li>Rewrite container-disk in C to in order to reduce it's memory footprint</li> </ul>"},{"location":"release_notes/#v0270","title":"v0.27.0","text":"<p>Released on: Fri Mar 6 22:40:34 2020 +0100</p> <ul> <li>Support for more guest agent informations in the API</li> <li>Support setting priorityClasses on VMs</li> <li>Support for additional control plane alerts via prometheus</li> <li>Support for io and emulator thread pinning</li> <li>Support setting a custom SELinux type for the launcher</li> <li>Support to perform network configurations from handler instead of launcher</li> <li>Support to opt-out of auto attaching the serial console</li> <li>Support for different uninstall strategies for data protection</li> <li>Fix to let qemu run in the qemu group</li> <li>Fix guest agent connectivity check after i.e. live migrations</li> </ul>"},{"location":"release_notes/#v0265","title":"v0.26.5","text":"<p>Released on: Tue Apr 14 15:07:04 2020 -0400</p>"},{"location":"release_notes/#v0264","title":"v0.26.4","text":"<p>Released on: Mon Mar 30 03:43:48 2020 +0200</p>"},{"location":"release_notes/#v0263","title":"v0.26.3","text":"<p>Released on: Tue Mar 10 08:57:27 2020 -0400</p>"},{"location":"release_notes/#v0262","title":"v0.26.2","text":"<p>Released on: Tue Mar 3 12:31:56 2020 -0500</p>"},{"location":"release_notes/#v0261","title":"v0.26.1","text":"<p>Released on: Fri Feb 14 20:42:46 2020 +0100</p>"},{"location":"release_notes/#v0260","title":"v0.26.0","text":"<p>Released on: Fri Feb 7 09:40:07 2020 +0100</p> <ul> <li>Fix incorrect ownerReferences to avoid VMs getting GCed</li> <li>Fixes for several tests</li> <li>Fix greedy permissions around Secrets by delegating them to kubelet</li> <li>Fix OOM infra pod by increasing it's memory request</li> <li>Clarify device support around live migrations</li> <li>Support for an uninstall strategy to protect workloads during uninstallation</li> <li>Support for more prometheus metrics and alert rules</li> <li>Support for testing SRIOV connectivity in functional tests</li> <li>Update Kubernetes client-go to 1.16.4</li> <li>FOSSA fixes and status</li> </ul>"},{"location":"release_notes/#v0250","title":"v0.25.0","text":"<p>Released on: Mon Jan 13 20:37:15 2020 +0100</p> <ul> <li>CI: Support for Kubernetes 1.17</li> <li>Support emulator thread pinning</li> <li>Support virtctl restart --force</li> <li>Support virtctl migrate to trigger live migrations from the CLI</li> </ul>"},{"location":"release_notes/#v0240","title":"v0.24.0","text":"<p>Released on: Tue Dec 3 15:34:34 2019 +0100</p> <ul> <li>CI: Support for Kubernetes 1.15</li> <li>CI: Support for Kubernetes 1.16</li> <li>Add and fix a couple of test cases</li> <li>Support for pause and unpausing VMs</li> <li>Update of libvirt to 5.6.0</li> <li>Fix bug related to parallel scraping of Prometheus endpoints</li> <li>Fix to reliably test VNC</li> </ul>"},{"location":"release_notes/#v0233","title":"v0.23.3","text":"<p>Released on: Tue Jan 21 13:17:20 2020 -0500</p>"},{"location":"release_notes/#v0232","title":"v0.23.2","text":"<p>Released on: Fri Jan 10 10:36:36 2020 -0500</p>"},{"location":"release_notes/#v0231","title":"v0.23.1","text":"<p>Released on: Thu Nov 28 09:36:41 2019 +0100</p>"},{"location":"release_notes/#v0230","title":"v0.23.0","text":"<p>Released on: Mon Nov 4 16:42:54 2019 +0100</p> <ul> <li>Guest OS Information is available under the VMI status now</li> <li>Updated to Go 1.12.8 and latest bazel</li> <li>Updated go-yaml to v2.2.4, which has a ddos vulnerability fixed</li> <li>Cleaned up and fixed CRD scheme registration</li> <li>Several bug fixes</li> <li>Many CI improvements (e.g. more logs in case of test failures)</li> </ul>"},{"location":"release_notes/#v0220","title":"v0.22.0","text":"<p>Released on: Thu Oct 10 18:55:08 2019 +0200</p> <ul> <li>Support for Nvidia GPUs and vGPUs exposed by Nvidia Kubevirt Device Plugin.</li> <li>VMIs now successfully start if they get a 0xfe prefixed MAC address assigned from the pod network</li> <li>Removed dependency on host semanage in SELinux Permissive mode</li> <li>Some changes as result of entering the CNCF sandbox (DCO check, FOSSA check, best practice badge)</li> <li>Many bug fixes and improvements in several areas</li> <li>CI: Introduced a OKD 4 test lane</li> <li>CI: Many improved tests resulting in less flakiness</li> </ul>"},{"location":"release_notes/#v0210","title":"v0.21.0","text":"<p>Released on: Mon Sep 9 09:59:08 2019 +0200</p> <ul> <li>CI: Support for Kubernetes 1.14</li> <li>Many bug fixes in several areas</li> <li>Support for <code>virtctl migrate</code></li> <li>Support configurable number of controller threads</li> <li>Support to opt-out of bridge binding for podNetwork</li> <li>Support for OpenShift Prometheus monitoring</li> <li>Support for setting more SMBIOS fields</li> <li>Improved containerDisk memory usage and speed</li> <li>Fix CRI-O memory limit</li> <li>Drop spc_t from launcher</li> <li>Add feature gates to security sensitive features</li> </ul>"},{"location":"release_notes/#v0208","title":"v0.20.8","text":"<p>Released on: Thu Oct 3 12:03:40 2019 +0200</p>"},{"location":"release_notes/#v0207","title":"v0.20.7","text":"<p>Released on: Fri Sep 27 15:21:56 2019 +0200</p>"},{"location":"release_notes/#v0206","title":"v0.20.6","text":"<p>Released on: Wed Sep 11 06:09:47 2019 -0400</p>"},{"location":"release_notes/#v0205","title":"v0.20.5","text":"<p>Released on: Thu Sep 5 17:48:59 2019 +0200</p>"},{"location":"release_notes/#v0204","title":"v0.20.4","text":"<p>Released on: Mon Sep 2 18:55:35 2019 +0200</p>"},{"location":"release_notes/#v0203","title":"v0.20.3","text":"<p>Released on: Tue Aug 27 16:58:15 2019 +0200</p>"},{"location":"release_notes/#v0202","title":"v0.20.2","text":"<p>Released on: Tue Aug 20 15:51:07 2019 +0200</p>"},{"location":"release_notes/#v0201","title":"v0.20.1","text":"<p>Released on: Fri Aug 9 19:48:17 2019 +0200</p> <ul> <li>Container disks are now secure and they are not copied anymore on every start. Old container disks can still be used in the same secure way, but new container disks can't be used on older kubevirt releases</li> <li>Create specific SecurityContextConstraints on OKD instead of using the privileged SCC</li> <li>Added clone authorization check for DataVolumes with PVC source</li> <li>The sidecar feature is feature-gated now</li> <li>Use container image shasums instead of tags for KubeVirt deployments</li> <li>Protect control plane components against voluntary evictions with a PodDisruptionBudget of MinAvailable=1</li> <li>Replaced hardcoded <code>virtctl</code> by using the basename of the call, this enables nicer output when installed via krew plugin package manager</li> <li>Added RNG device to all Fedora VMs in tests and examples (newer kernels might block bootimg while waiting for entropy)</li> <li>The virtual memory is now set to match the memory limit, if memory limit is specified and guest memory is not</li> <li>Support nftable for CoreOS</li> <li>Added a block-volume flag to the virtctl image-upload command</li> <li>Improved virtctl console/vnc data flow</li> <li>Removed DataVolumes feature gate in favor of auto-detecting CDI support</li> <li>Removed SR-IOV feature gate, it is enabled by default now</li> <li>VMI-related metrics have been renamed from <code>kubevirt_vm_</code> to <code>kubevirt_vmi_</code> to better reflect their purpose</li> <li>Added metric to report the VMI count</li> <li>Improved integration with HCO by adding a CSV generator tool and modified KubeVirt CR conditions</li> <li>CI Improvements:</li> <li>Added dedicated SR-IOV test lane</li> <li>Improved log gathering</li> <li>Reduced amount of flaky tests</li> </ul>"},{"location":"release_notes/#v0200","title":"v0.20.0","text":"<p>Released on: Fri Aug 9 16:42:41 2019 +0200</p> <ul> <li>container Disks are now secure and they are not copied anymore on every start. Old container Disks can still be used in the same secure way, but new container Disks can't be used on older kubevirt releases</li> <li>Create specific SecurityContextConstraints on OKD instead of using the privileged SCC</li> <li>Added clone authorization check for DataVolumes with PVC source</li> <li>The sidecar feature is feature-gated now</li> <li>Use container image shasum's instead of tags for KubeVirt deployments</li> <li>Protect control plane components against voluntary evictions with a PodDisruptionBudget of MinAvailable=1</li> <li>Replaced hardcoded <code>virtctl</code> by using the basename of the call, this enables nicer output when installed via krew plugin package manager</li> <li>Added RNG device to all Fedora VMs in tests and examples (newer kernels might block boot img while waiting for entropy)</li> <li>The virtual memory is now set to match the memory limit, if memory limit is specified and guest memory is not</li> <li>Support nftable for CoreOS</li> <li>Added a block-volume flag to the virtctl image-upload command</li> <li>Improved virtctl console/vnc data flow</li> <li>Removed DataVolumes feature gate in favor of auto-detecting CDI support</li> <li>Removed SR-IOV feature gate, it is enabled by default now</li> <li>VMI-related metrics have been renamed from <code>kubevirt_vm_</code> to <code>kubevirt_vmi_</code> to better reflect their purpose</li> <li>Added metric to report the VMI count</li> <li>Improved integration with HCO by adding a CSV generator tool and modified KubeVirt CR conditions</li> <li>CI Improvements:</li> <li>Added dedicated SR-IOV test lane</li> <li>Improved log gathering</li> <li>Reduced amount of flaky tests</li> </ul>"},{"location":"release_notes/#v0190","title":"v0.19.0","text":"<p>Released on: Fri Jul 5 12:52:16 2019 +0200</p> <ul> <li>Fixes when run on kind</li> <li>Fixes for sub-resource RBAC</li> <li>Limit pod network interface bindings</li> <li>Many additional bug fixes in many areas</li> <li>Additional test cases for updates, disk types, live migration with NFS</li> <li>Additional test cases for memory over-commit, block storage, cpu manager, headless mode</li> <li>Improvements around HyperV</li> <li>Improved error handling for runStrategies</li> <li>Improved update procedure</li> <li>Improved network metrics reporting (packets and errors)</li> <li>Improved guest overhead calculation</li> <li>Improved SR-IOV test suite</li> <li>Support for live migration auto-converge</li> <li>Support for config-drive disks</li> <li>Support for setting a pullPolicy con container Disks</li> <li>Support for unprivileged VMs when using SR-IOV</li> <li>Introduction of a project security policy</li> </ul>"},{"location":"release_notes/#v0181","title":"v0.18.1","text":"<p>Released on: Thu Jun 13 12:00:56 2019 +0200</p>"},{"location":"release_notes/#v0180","title":"v0.18.0","text":"<p>Released on: Wed Jun 5 22:25:09 2019 +0200</p> <ul> <li>Build: Use of go modules</li> <li>CI: Support for Kubernetes 1.13</li> <li>Countless test cases fixes and additions</li> <li>Several smaller bug fixes</li> <li>Improved upgrade documentation</li> </ul>"},{"location":"release_notes/#v0174","title":"v0.17.4","text":"<p>Released on: Tue Jun 25 07:49:12 2019 -0400</p>"},{"location":"release_notes/#v0173","title":"v0.17.3","text":"<p>Released on: Wed Jun 19 12:00:45 2019 -0400</p>"},{"location":"release_notes/#v0172","title":"v0.17.2","text":"<p>Released on: Wed Jun 5 08:12:04 2019 -0400</p>"},{"location":"release_notes/#v0171","title":"v0.17.1","text":"<p>Released on: Tue Jun 4 14:41:10 2019 -0400</p>"},{"location":"release_notes/#v0170","title":"v0.17.0","text":"<p>Released on: Mon May 6 16:18:01 2019 +0200</p> <ul> <li>Several test case additions</li> <li>Improved virt-controller node distribution</li> <li>Improved support between version migrations</li> <li>Support for a configurable MachineType default</li> <li>Support for live-migration of a VM on node taints</li> <li>Support for VM swap metrics</li> <li>Support for versioned virt-launcher / virt-handler communication</li> <li>Support for HyperV flags</li> <li>Support for different VM run strategies (i.e manual and rerunOnFailure)</li> <li>Several fixes for live-migration (TLS support, protected pods)</li> </ul>"},{"location":"release_notes/#v0163","title":"v0.16.3","text":"<p>Released on: Thu May 2 23:51:08 2019 +0200</p>"},{"location":"release_notes/#v0162","title":"v0.16.2","text":"<p>Released on: Fri Apr 26 12:24:33 2019 +0200</p>"},{"location":"release_notes/#v0161","title":"v0.16.1","text":"<p>Released on: Tue Apr 23 19:31:19 2019 +0200</p>"},{"location":"release_notes/#v0160","title":"v0.16.0","text":"<p>Released on: Fri Apr 5 23:18:22 2019 +0200</p> <ul> <li>Bazel fixes</li> <li>Initial work to support upgrades (not finalized)</li> <li>Initial support for HyperV features</li> <li>Support propagation of MAC addresses to multus</li> <li>Support live migration cancellation</li> <li>Support for table input devices</li> <li>Support for generating OLM metadata</li> <li>Support for triggering VM live migration on node taints</li> </ul>"},{"location":"release_notes/#v0150","title":"v0.15.0","text":"<p>Released on: Tue Mar 5 10:35:08 2019 +0100</p> <ul> <li>CI: Several fixes</li> <li>Fix configurable number of KVM devices</li> <li>Narrow virt-handler permissions</li> <li>Use bazel for development builds</li> <li>Support for live migration with shared and non-shared disks</li> <li>Support for live migration progress tracking</li> <li>Support for EFI boot</li> <li>Support for libvirt 5.0</li> <li>Support for extra DHCP options</li> <li>Support for a hook to manipulate cloud-init metadata</li> <li>Support setting a VM serial number</li> <li>Support for exposing infra and VM metrics</li> <li>Support for a tablet input device</li> <li>Support for extra CPU flags</li> <li>Support for ignition metadata</li> <li>Support to set a default CPU model</li> <li>Update to go 1.11.5</li> </ul>"},{"location":"release_notes/#v0140","title":"v0.14.0","text":"<p>Released on: Mon Feb 4 22:04:14 2019 +0100</p> <ul> <li>CI: Several stabilizing fixes</li> <li>docs: Document the KubeVirt Razor</li> <li>build: golang update</li> <li>Update to Kubernetes 1.12</li> <li>Update CDI</li> <li>Support for Ready and Created Operator conditions</li> <li>Support (basic) EFI</li> <li>Support for generating cloud-init network-config</li> </ul>"},{"location":"release_notes/#v0137","title":"v0.13.7","text":"<p>Released on: Mon Oct 28 17:02:35 2019 -0400</p>"},{"location":"release_notes/#v0136","title":"v0.13.6","text":"<p>Released on: Wed Sep 25 17:19:44 2019 +0200</p>"},{"location":"release_notes/#v0135","title":"v0.13.5","text":"<p>Released on: Thu Aug 1 11:25:00 2019 -0400</p>"},{"location":"release_notes/#v0134","title":"v0.13.4","text":"<p>Released on: Thu Aug 1 09:52:35 2019 -0400</p>"},{"location":"release_notes/#v0133","title":"v0.13.3","text":"<p>Released on: Mon Feb 4 15:46:48 2019 -0500</p>"},{"location":"release_notes/#v0132","title":"v0.13.2","text":"<p>Released on: Thu Jan 24 23:24:06 2019 +0100</p>"},{"location":"release_notes/#v0131","title":"v0.13.1","text":"<p>Released on: Thu Jan 24 11:16:20 2019 +0100</p>"},{"location":"release_notes/#v0130","title":"v0.13.0","text":"<p>Released on: Tue Jan 15 08:26:25 2019 +0100</p> <ul> <li>CI: Fix virt-api race</li> <li>API: Remove volumeName from disks</li> </ul>"},{"location":"release_notes/#v0120","title":"v0.12.0","text":"<p>Released on: Fri Jan 11 22:22:02 2019 +0100</p> <ul> <li>Introduce a KubeVirt Operator for KubeVirt life-cycle management</li> <li>Introduce dedicated kubevirt namespace</li> <li>Support VMI ready conditions</li> <li>Support vCPU threads and sockets</li> <li>Support scale and HPA for VMIRs</li> <li>Support to pass NTP related DHCP options</li> <li>Support guest IP address reporting via qemu guest agent</li> <li>Support for live migration with shared storage</li> <li>Support scheduling of VMs based on CPU family</li> <li>Support masquerade network interface binding</li> </ul>"},{"location":"release_notes/#v0111","title":"v0.11.1","text":"<p>Released on: Thu Dec 13 10:21:56 2018 +0200</p>"},{"location":"release_notes/#v0110","title":"v0.11.0","text":"<p>Released on: Thu Dec 6 10:15:51 2018 +0100</p> <ul> <li>API: registryDisk got renamed to containerDisk</li> <li>CI: User OKD 3.11</li> <li>Fix: Tolerate if the PVC has less capacity than expected</li> <li>Aligned to use ownerReferences</li> <li>Update to libvirt-4.10.0</li> <li>Support for VNC on MAC OSX</li> <li>Support for network SR-IOV interfaces</li> <li>Support for custom DHCP options</li> <li>Support for VM restarts via a custom endpoint</li> <li>Support for liveness and readiness probes</li> </ul>"},{"location":"release_notes/#v0100","title":"v0.10.0","text":"<p>Released on: Thu Nov 8 15:21:34 2018 +0100</p> <ul> <li>Support for vhost-net</li> <li>Support for block multi-queue</li> <li>Support for custom PCI addresses for virtio devices</li> <li>Support for deploying KubeVirt to a custom namespace</li> <li>Support for ServiceAccount token disks</li> <li>Support for multus backed networks</li> <li>Support for genie backed networks</li> <li>Support for kuryr backed networks</li> <li>Support for block PVs</li> <li>Support for configurable disk device caches</li> <li>Support for pinned IO threads</li> <li>Support for virtio net multi-queue</li> <li>Support for image upload (depending on CDI)</li> <li>Support for custom entity lists with more VM details (custom columns)</li> <li>Support for IP and MAC address reporting of all vNICs</li> <li>Basic support for guest agent status reporting</li> <li>More structured logging</li> <li>Better libvirt error reporting</li> <li>Stricter CR validation</li> <li>Better ownership references</li> <li>Several test improvements</li> </ul>"},{"location":"release_notes/#v096","title":"v0.9.6","text":"<p>Released on: Thu Nov 22 17:14:18 2018 +0100</p>"},{"location":"release_notes/#v095","title":"v0.9.5","text":"<p>Released on: Thu Nov 8 09:57:48 2018 +0100</p>"},{"location":"release_notes/#v094","title":"v0.9.4","text":"<p>Released on: Wed Nov 7 08:22:14 2018 -0500</p>"},{"location":"release_notes/#v093","title":"v0.9.3","text":"<p>Released on: Mon Oct 22 09:04:02 2018 -0400</p>"},{"location":"release_notes/#v092","title":"v0.9.2","text":"<p>Released on: Thu Oct 18 12:14:09 2018 +0200</p>"},{"location":"release_notes/#v091","title":"v0.9.1","text":"<p>Released on: Fri Oct 5 09:01:51 2018 +0200</p>"},{"location":"release_notes/#v090","title":"v0.9.0","text":"<p>Released on: Thu Oct 4 14:42:28 2018 +0200</p> <ul> <li>CI: NetworkPolicy tests</li> <li>CI: Support for an external provider (use a preconfigured cluster for tests)</li> <li>Fix virtctl console issues with CRI-O</li> <li>Support to initialize empty PVs</li> <li>Support for basic CPU pinning</li> <li>Support for setting IO Threads</li> <li>Support for block volumes</li> <li>Move preset logic to mutating webhook</li> <li>Introduce basic metrics reporting using prometheus metrics</li> <li>Many stabilizing fixes in many places</li> </ul>"},{"location":"release_notes/#v080","title":"v0.8.0","text":"<p>Released on: Thu Sep 6 14:25:22 2018 +0200</p> <ul> <li>Support for DataVolume</li> <li>Support for a subprotocol for web browser terminals</li> <li>Support for virtio-rng</li> <li>Support disconnected VMs</li> <li>Support for setting host model</li> <li>Support for host CPU passthrough</li> <li>Support setting a vNICs mac and PCI address</li> <li>Support for memory over-commit</li> <li>Support booting from network devices</li> <li>Use less devices by default, aka disable unused ones</li> <li>Improved VMI shutdown status</li> <li>More logging to improve debugability</li> <li>A lot of small fixes, including typos and documentation fixes</li> <li>Race detection in tests</li> <li>Hook improvements</li> <li>Update to use Fedora 28 (includes updates of dependencies like libvirt and   qemu)</li> <li>Move CI to support Kubernetes 1.11</li> </ul>"},{"location":"release_notes/#v070","title":"v0.7.0","text":"<p>Released on: Wed Jul 4 17:41:33 2018 +0200</p> <ul> <li>CI: Move test storage to hostPath</li> <li>CI: Add support for Kubernetes 1.10.4</li> <li>CI: Improved network tests for multiple-interfaces</li> <li>CI: Drop Origin 3.9 support</li> <li>CI: Add test for testing templates on Origin</li> <li>VM to VMI rename</li> <li>VM affinity and anti-affinity</li> <li>Add awareness for multiple networks</li> <li>Add hugepage support</li> <li>Add device-plugin based kvm</li> <li>Add support for setting the network interface model</li> <li>Add (basic and initial) Kubernetes compatible networking approach (SLIRP)</li> <li>Add role aggregation for our roles</li> <li>Add support for setting a disks serial number</li> <li>Add support for specifying the CPU model</li> <li>Add support for setting an network interfaces MAC address</li> <li>Relocate binaries for FHS conformance</li> <li>Logging improvements</li> <li>Template fixes</li> <li>Fix OpenShift CRD validation</li> <li>virtctl: Improve vnc logging improvements</li> <li>virtctl: Add expose</li> <li>virtctl: Use PATCH instead of PUT</li> </ul>"},{"location":"release_notes/#v064","title":"v0.6.4","text":"<p>Released on: Tue Aug 21 17:29:28 2018 +0300</p>"},{"location":"release_notes/#v063","title":"v0.6.3","text":"<p>Released on: Mon Jul 30 16:14:22 2018 +0200</p>"},{"location":"release_notes/#v062","title":"v0.6.2","text":"<p>Released on: Wed Jul 4 17:49:37 2018 +0200</p> <ul> <li>Binary relocation for packaging</li> <li>QEMU Process detection</li> <li>Role aggregation</li> <li>CPU Model selection</li> <li>VM Rename fix</li> </ul>"},{"location":"release_notes/#v061","title":"v0.6.1","text":"<p>Released on: Mon Jun 18 17:07:48 2018 -0400</p>"},{"location":"release_notes/#v060","title":"v0.6.0","text":"<p>Released on: Mon Jun 11 09:30:28 2018 +0200</p> <ul> <li>A range of flakiness reducing test fixes</li> <li>Vagrant setup got deprecated</li> <li>Updated Docker and CentOS versions</li> <li>Add Kubernetes 1.10.3 to test matrix</li> <li>A couple of ginkgo concurrency fixes</li> <li>A couple of spelling fixes</li> <li>A range if infra updates</li> <li>Use /dev/kvm if possible, otherwise fallback to emulation</li> <li>Add default view/edit/admin RBAC Roles</li> <li>Network MTU fixes</li> <li>CD-ROM drives are now read-only</li> <li>Secrets can now be correctly referenced on VMs</li> <li>Add disk boot ordering</li> <li>Add virtctl version</li> <li>Add virtctl expose</li> <li>Fix virtual machine memory calculations</li> <li>Add basic virtual machine Network API</li> </ul>"},{"location":"release_notes/#v050","title":"v0.5.0","text":"<p>Released on: Fri May 4 18:25:32 2018 +0200</p> <ul> <li>Better controller health signaling</li> <li>Better virtctl error messages</li> <li>Improvements to enable CRI-O support</li> <li>Run CI on stable OpenShift</li> <li>Add test coverage for multiple PVCs</li> <li>Improved controller life-cycle guarantees</li> <li>Add Webhook validation</li> <li>Add tests coverage for node eviction</li> <li>OfflineVirtualMachine status improvements</li> <li>RegistryDisk API update</li> </ul>"},{"location":"release_notes/#v041","title":"v0.4.1","text":"<p>Released on: Thu Apr 12 11:46:09 2018 +0200</p> <ul> <li>VM shutdown fixes and tests</li> <li>Functional test for CRD validation</li> <li>Windows VM test</li> <li>DHCP link-local change</li> </ul>"},{"location":"release_notes/#v040","title":"v0.4.0","text":"<p>Released on: Fri Apr 6 16:40:31 2018 +0200</p> <ul> <li>Fix several networking issues</li> <li>Add and enable OpenShift support to CI</li> <li>Add conditional Windows tests (if an image is present)</li> <li>Add subresources for console access</li> <li>virtctl config alignment with kubectl</li> <li>Fix API reference generation</li> <li>Stable UUIDs for OfflineVirtualMachines</li> <li>Build virtctl for MacOS and Windows</li> <li>Set default architecture to x86_64</li> <li>Major improvement to the CI infrastructure (all containerized)</li> <li>virtctl convenience functions for starting and stopping a VM</li> </ul>"},{"location":"release_notes/#v030","title":"v0.3.0","text":"<p>Released on: Thu Mar 8 10:21:57 2018 +0100</p> <ul> <li>Kubernetes compatible networking</li> <li>Kubernetes compatible PV based storage</li> <li>VirtualMachinePresets support</li> <li>OfflineVirtualMachine support</li> <li>RBAC improvements</li> <li>Switch to q35 machine type by default</li> <li>A large number of test and CI fixes</li> <li>Ephemeral disk support</li> </ul>"},{"location":"release_notes/#v020","title":"v0.2.0","text":"<p>Released on: Fri Jan 5 16:30:45 2018 +0100</p> <ul> <li>VM launch and shutdown flow improvements</li> <li>VirtualMachine API redesign</li> <li>Removal of HAProxy</li> <li>Redesign of VNC/Console access</li> <li>Initial support for different vagrant providers</li> </ul>"},{"location":"release_notes/#v010","title":"v0.1.0","text":"<p>Released on: Fri Dec 8 20:43:06 2017 +0100</p> <ul> <li>Many API improvements for a proper OpenAPI reference</li> <li>Add watchdog support</li> <li>Drastically improve the deployment on non-vagrant setups</li> <li>Dropped nodeSelectors</li> <li>Separated inner component deployment from edge component deployment</li> <li>Created separate manifests for developer, test, and release deployments</li> <li>Moved components to kube-system namespace</li> <li>Improved and unified flag parsing</li> </ul>"},{"location":"release_notes/#v004","title":"v0.0.4","text":"<p>Released on: Tue Nov 7 11:51:45 2017 +0100</p> <ul> <li>Add support for node affinity to VM.Spec</li> <li>Add OpenAPI specification</li> <li>Drop swagger 1.2 specification</li> <li>virt-launcher refactoring</li> <li>Leader election mechanism for virt-controller</li> <li>Move from glide to dep for dependency management</li> <li>Improve virt-handler synchronization loops</li> <li>Add support for running the functional tests on oVirt infrastructure</li> <li>Several tests fixes (spice, cleanup, ...)</li> <li>Add console test tool</li> <li>Improve libvirt event notification</li> </ul>"},{"location":"release_notes/#v003","title":"v0.0.3","text":"<p>Released on: Fri Oct 6 10:21:16 2017 +0200</p> <ul> <li>Containerized binary builds</li> <li>Socket based container detection</li> <li>cloud-init support</li> <li>Container based ephemeral disk support</li> <li>Basic RBAC profile</li> <li>client-go updates</li> <li>Rename of VM to VirtualMachine</li> <li>Introduction of VirtualMachineReplicaSet</li> <li>Improved migration events</li> <li>Improved API documentation</li> </ul>"},{"location":"release_notes/#v002","title":"v0.0.2","text":"<p>Released on: Mon Sep 4 21:12:46 2017 +0200</p> <ul> <li>Usage of CRDs</li> <li>Moved libvirt to a pod</li> <li>Introduction of <code>virtctl</code></li> <li>Use glide instead of govendor</li> <li>Container based ephemeral disks</li> <li>Contributing guide improvements</li> <li>Support for Kubernetes Namespaces</li> </ul>"},{"location":"cluster_admin/activating_feature_gates/","title":"Activating feature gates","text":"<p>KubeVirt has a set of features that are not mature enough to be enabled by default. As such, they are protected by a Kubernetes concept called feature gates.</p>"},{"location":"cluster_admin/activating_feature_gates/#how-to-activate-a-feature-gate","title":"How to activate a feature gate","text":"<p>You can activate a specific feature gate directly in KubeVirt's CR, by provisioning the following yaml, which uses the <code>LiveMigration</code> feature gate as an example: <pre><code>cat &lt;&lt; END &gt; enable-feature-gate.yaml\n---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration: \n      featureGates:\n        - LiveMigration\nEND\n\nkubectl apply -f enable-feature-gate.yaml\n</code></pre></p> <p>Alternatively, the existing kubevirt CR can be altered: <pre><code>kubectl edit kubevirt kubevirt -n kubevirt\n</code></pre></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - DataVolumes\n        - LiveMigration\n</code></pre> <p>Note: the name of the feature gates is case sensitive.</p> <p>The snippet above assumes KubeVirt is installed in the <code>kubevirt</code> namespace. Change the namespace to suite your installation.</p>"},{"location":"cluster_admin/activating_feature_gates/#list-of-feature-gates","title":"List of feature gates","text":"<p>The list of feature gates (which evolve in time) can be checked directly from the source code.</p>"},{"location":"cluster_admin/annotations_and_labels/","title":"Annotations and labels","text":"<p>KubeVirt builds on and exposes a number of labels and annotations that either are used for internal implementation needs or expose useful information to API users. This page documents the labels and annotations that may be useful for regular API consumers. This page intentionally does not list labels and annotations that are merely part of internal implementation.</p> <p>Note: Annotations and labels that are not specific to KubeVirt are also documented here.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtio","title":"kubevirt.io","text":"<p>Example: <code>kubevirt.io=virt-launcher</code></p> <p>Used on: Pod</p> <p>This label marks resources that belong to KubeVirt. An optional value may indicate which specific KubeVirt component a resource belongs to. This label may be used to list all resources that belong to KubeVirt, for example, to uninstall it from a cluster.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtioschedulable","title":"kubevirt.io/schedulable","text":"<p>Example: <code>kubevirt.io/schedulable=true</code></p> <p>Used on: Node</p> <p>This label declares whether a particular node is available for scheduling virtual machine instances on it.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtioheartbeat","title":"kubevirt.io/heartbeat","text":"<p>Example: <code>kubevirt.io/heartbeat=2018-07-03T20:07:25Z</code></p> <p>Used on: Node</p> <p>This annotation is regularly updated by virt-handler to help determine if a particular node is alive and hence should be available for new virtual machine instance scheduling.</p>"},{"location":"cluster_admin/api_validation/","title":"API Validation","text":"<p>The KubeVirt VirtualMachineInstance API is implemented using a Kubernetes Custom Resource Definition (CRD). Because of this, KubeVirt is able to leverage a couple of features Kubernetes provides in order to perform validation checks on our API as objects created and updated on the cluster.</p>"},{"location":"cluster_admin/api_validation/#how-api-validation-works","title":"How API Validation Works","text":""},{"location":"cluster_admin/api_validation/#crd-openapiv3-schema","title":"CRD OpenAPIv3 Schema","text":"<p>The KubeVirt API is registered with Kubernetes at install time through a series of CRD definitions. KubeVirt includes an OpenAPIv3 schema in these definitions which indicates to the Kubernetes Apiserver some very basic information about our API, such as what fields are required and what type of data is expected for each value.</p> <p>This OpenAPIv3 schema validation is installed automatically and requires no thought on the users part to enable.</p>"},{"location":"cluster_admin/api_validation/#admission-control-webhooks","title":"Admission Control Webhooks","text":"<p>The OpenAPIv3 schema validation is limited. It only validates the general structure of a KubeVirt object looks correct. It does not however verify that the contents of that object make sense.</p> <p>With OpenAPIv3 validation alone, users can easily make simple mistakes (like not referencing a volume's name correctly with a disk) and the cluster will still accept the object. However, the VirtualMachineInstance will of course not start if these errors in the API exist. Ideally we'd like to catch configuration issues as early as possible and not allow an object to even be posted to the cluster if we can detect there's a problem with the object's Spec.</p> <p>In order to perform this advanced validation, KubeVirt implements its own admission controller which is registered with kubernetes as an admission controller webhook. This webhook is registered with Kubernetes at install time. As KubeVirt objects are posted to the cluster, the Kubernetes API server forwards Creation requests to our webhook for validation before persisting the object into storage.</p> <p>Note however that the KubeVirt admission controller requires features to be enabled on the cluster in order to be enabled.</p>"},{"location":"cluster_admin/api_validation/#enabling-kubevirt-admission-controller-on-kubernetes","title":"Enabling KubeVirt Admission Controller on Kubernetes","text":"<p>When provisioning a new Kubernetes cluster, ensure that both the MutatingAdmissionWebhook and ValidatingAdmissionWebhook values are present in the Apiserver's --admission-control cli argument.</p> <p>Below is an example of the --admission-control values we use during development</p> <pre><code>--admission-control='Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota'\n</code></pre> <p>Note that the old --admission-control flag was deprecated in 1.10 and replaced with --enable-admission-plugins. MutatingAdmissionWebhook and ValidatingAdmissionWebhook are enabled by default.</p>"},{"location":"cluster_admin/api_validation/#enabling-kubevirt-admission-controller-on-okd","title":"Enabling KubeVirt Admission Controller on OKD","text":"<p>OKD also requires the admission control webhooks to be enabled at install time. The process is slightly different though. With OKD, we enable webhooks using an admission plugin.</p> <p>These admission control plugins can be configured in openshift-ansible by setting the following value in ansible inventory file.</p> <pre><code>openshift_master_admission_plugin_config={\"ValidatingAdmissionWebhook\":{\"configuration\":{\"kind\": \"DefaultAdmissionConfig\",\"apiVersion\": \"v1\",\"disable\": false}},\"MutatingAdmissionWebhook\":{\"configuration\":{\"kind\": \"DefaultAdmissionConfig\",\"apiVersion\": \"v1\",\"disable\": false}}}\n</code></pre>"},{"location":"cluster_admin/authorization/","title":"Authorization","text":"<p>KubeVirt authorization is performed using Kubernetes's Resource Based Authorization Control system (RBAC). RBAC allows cluster admins to grant access to cluster resources by binding RBAC roles to users.</p> <p>For example, an admin creates an RBAC role that represents the permissions required to create a VirtualMachineInstance. The admin can then bind that role to users in order to grant them the permissions required to launch a VirtualMachineInstance.</p> <p>With RBAC roles, admins can grant users targeted access to various KubeVirt features.</p>"},{"location":"cluster_admin/authorization/#kubevirt-default-rbac-clusterroles","title":"KubeVirt Default RBAC ClusterRoles","text":"<p>KubeVirt comes with a set of predefined RBAC ClusterRoles that can be used to grant users permissions to access KubeVirt Resources.</p>"},{"location":"cluster_admin/authorization/#default-view-role","title":"Default View Role","text":"<p>The kubevirt.io:view ClusterRole gives users permissions to view all KubeVirt resources in the cluster. The permissions to create, delete, modify or access any KubeVirt resources beyond viewing the resource's spec are not included in this role. This means a user with this role could see that a VirtualMachineInstance is running, but neither shutdown nor gain access to that VirtualMachineInstance via console/VNC.</p>"},{"location":"cluster_admin/authorization/#default-edit-role","title":"Default Edit Role","text":"<p>The kubevirt.io:edit ClusterRole gives users permissions to modify all KubeVirt resources in the cluster. For example, a user with this role can create new VirtualMachineInstances, delete VirtualMachineInstances, and gain access to both console and VNC.</p>"},{"location":"cluster_admin/authorization/#default-admin-role","title":"Default Admin Role","text":"<p>The kubevirt.io:admin ClusterRole grants users full permissions to all KubeVirt resources, including the ability to delete collections of resources.</p> <p>The admin role also grants users access to view and modify the KubeVirt runtime config. This config exists within the Kubevirt Custom Resource under the <code>configuration</code> key in the namespace the KubeVirt operator is running.</p> <p>NOTE Users are only guaranteed the ability to modify the kubevirt runtime configuration if a ClusterRoleBinding is used. A RoleBinding will work to provide kubevirt CR access only if the RoleBinding targets the same namespace that the kubevirt CR exists in.</p>"},{"location":"cluster_admin/authorization/#binding-default-clusterroles-to-users","title":"Binding Default ClusterRoles to Users","text":"<p>The KubeVirt default ClusterRoles are granted to users by creating either a ClusterRoleBinding or RoleBinding object.</p>"},{"location":"cluster_admin/authorization/#binding-within-all-namespaces","title":"Binding within All Namespaces","text":"<p>With a ClusterRoleBinding, users receive the permissions granted by the role across all namespaces.</p>"},{"location":"cluster_admin/authorization/#binding-within-single-namespace","title":"Binding within Single Namespace","text":"<p>With a RoleBinding, users receive the permissions granted by the role only within a targeted namespace.</p>"},{"location":"cluster_admin/authorization/#extending-kubernetes-default-roles-with-kubevirt-permissions","title":"Extending Kubernetes Default Roles with KubeVirt permissions","text":"<p>The aggregated ClusterRole Kubernetes feature facilitates combining multiple ClusterRoles into a single aggregated ClusterRole. This feature is commonly used to extend the default Kubernetes roles with permissions to access custom resources that do not exist in the Kubernetes core.</p> <p>In order to extend the default Kubernetes roles to provide permission to access KubeVirt resources, we need to add the following labels to the KubeVirt ClusterRoles.</p> <pre><code>kubectl label clusterrole kubevirt.io:admin rbac.authorization.k8s.io/aggregate-to-admin=true\nkubectl label clusterrole kubevirt.io:edit rbac.authorization.k8s.io/aggregate-to-edit=true\nkubectl label clusterrole kubevirt.io:view rbac.authorization.k8s.io/aggregate-to-view=true\n</code></pre> <p>By adding these labels, any user with a RoleBinding or ClusterRoleBinding involving one of the default Kubernetes roles will automatically gain access to the equivalent KubeVirt roles as well.</p> <p>More information about aggregated cluster roles can be found here</p>"},{"location":"cluster_admin/authorization/#creating-custom-rbac-roles","title":"Creating Custom RBAC Roles","text":"<p>If the default KubeVirt ClusterRoles are not expressive enough, admins can create their own custom RBAC roles to grant user access to KubeVirt resources. The creation of a RBAC role is inclusive only, meaning there's no way to deny access. Instead access is only granted.</p> <p>Below is an example of what KubeVirt's default admin ClusterRole looks like. A custom RBAC role can be created by reducing the permissions in this example role.</p> <pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n  name: my-custom-rbac-role\n  labels:\n    kubevirt.io: \"\"\nrules:\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/console\n      - virtualmachineinstances/vnc\n    verbs:\n      - get\n  - apiGroups:\n      - kubevirt.io\n    resources:\n      - virtualmachineinstances\n      - virtualmachines\n      - virtualmachineinstancepresets\n      - virtualmachineinstancereplicasets\n    verbs:\n      - get\n      - delete\n      - create\n      - update\n      - patch\n      - list\n      - watch\n      - deletecollection\n</code></pre>"},{"location":"cluster_admin/confidential_computing/","title":"Confidential computing","text":""},{"location":"cluster_admin/confidential_computing/#amd-secure-encrypted-virtualization-sev","title":"AMD Secure Encrypted Virtualization (SEV)","text":"<p>FEATURE STATE: KubeVirt v0.49.0 (experimental support)</p> <p>Secure Encrypted Virtualization (SEV) is a feature of AMD's EPYC CPUs that allows the memory of a virtual machine to be encrypted on the fly.</p> <p>KubeVirt supports running confidential VMs on AMD EPYC hardware with SEV feature.</p>"},{"location":"cluster_admin/confidential_computing/#preconditions","title":"Preconditions","text":"<p>In order to run an SEV guest the following condition must be met:</p> <ul> <li><code>WorkloadEncryptionSEV</code> feature gate must be enabled.</li> <li>The guest must support UEFI boot</li> <li>SecureBoot must be disabled for the guest VM</li> </ul>"},{"location":"cluster_admin/confidential_computing/#running-an-sev-guest","title":"Running an SEV guest","text":"<p>SEV memory encryption can be requested by setting the <code>spec.domain.launchSecurity.sev</code> element in the VMI definition:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    launchSecurity:\n      sev: {}\n    firmware:\n      bootloader:\n        efi:\n          secureBoot: false\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre>"},{"location":"cluster_admin/confidential_computing/#current-limitations","title":"Current limitations","text":"<ul> <li>SEV-encrypted VMs cannot contain directly-accessible host devices (that is, PCI passthrough)</li> <li>Live Migration is not supported</li> <li>The VMs are not attested</li> </ul>"},{"location":"cluster_admin/customize_components/","title":"Customize components","text":""},{"location":"cluster_admin/customize_components/#customize-kubevirt-components","title":"Customize KubeVirt Components","text":""},{"location":"cluster_admin/customize_components/#customize-components-using-patches","title":"Customize components using patches","text":"<p> If the patch created is invalid KubeVirt will not be able to update or deploy the system. This is intended for special use cases and should not be used unless you know what you are doing.</p> <p>Valid resource types are: Deployment, DaemonSet, Service, ValidatingWebhookConfiguraton, MutatingWebhookConfiguration, APIService, and CertificateSecret. More information can be found in the API spec.</p> <p>Example customization patch: <pre><code>---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  certificateRotateStrategy: {}\n  configuration: {}\n  customizeComponents:\n    patches:\n    - resourceType: Deployment\n      resourceName: virt-controller\n      patch: '[{\"op\": \"remove\", \"path\": \"/spec/template/spec/containers/0/livenessProbe\"}]'\n      type: json\n    - resourceType: Deployment\n      resourceName: virt-controller\n      patch: '{\"metadata\":{\"annotations\":{\"patch\": \"true\"}}}'\n      type: strategic\n</code></pre></p> <p>The above example will update the <code>virt-controller</code> deployment to have an annotation in it's metadata that says <code>patch: true</code> and will remove the livenessProbe from the container definition.</p>"},{"location":"cluster_admin/customize_components/#customize-flags","title":"Customize Flags","text":"<p> If the flags are invalid or become invalid on update the component will not be able to run</p> <p>By using the customize flag option, whichever component the flags are to be applied to, all default flags will be removed and only the flags specified will be used. The available resources to change the flags on are <code>api</code>, <code>controller</code> and <code>handler</code>. You can find our more details about the API in the API spec.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  certificateRotateStrategy: {}\n  configuration: {}\n  customizeComponents:\n    flags:\n      api:\n        v: \"5\"\n        port: \"8443\"\n        console-server-port: \"8186\"\n        subresources-only: \"true\"\n</code></pre> <p>The above example would produce a <code>virt-api</code> pod with the following command</p> <pre><code>...\nspec:\n  ....\n  container:\n  - name: virt-api\n    command:\n    - virt-api\n    - --v\n    - \"5\"\n    - --console-server-port\n    - \"8186\"\n    - --port\n    - \"8443\"\n    - --subresources-only\n    - \"true\"\n    ...\n</code></pre>"},{"location":"cluster_admin/device_status_on_Arm64/","title":"Device Status on Arm64","text":"<p>This page is based on https://github.com/kubevirt/kubevirt/issues/8916</p> Devices Description Status on Arm64 DisableHotplug supported Disks sata/ virtio bus support virtio bus Watchdog i6300esb not supported UseVirtioTransitional virtio-transitional supported Interfaces e1000/ virtio-net-device support virtio-net-device Inputs tablet virtio/usb bus supported AutoattachPodInterface connect to /net/tun (devices.kubevirt.io/tun) supported AutoattachGraphicsDevice create a virtio-gpu device / vga device support virtio-gpu AutoattachMemBalloon virtio-balloon-pci-non-transitional supported AutoattachInputDevice auto add tablet supported Rng virtio-rng-pci-non-transitional host:/dev/urandom supported BlockMultiQueue \"driver\":\"virtio-blk-pci-non-transitional\",\"num-queues\":$cpu_number supported NetworkInterfaceMultiQueue -netdev tap,fds=21:23:24:25,vhost=on,vhostfds=26:27:28:29,id=hostua-default#fd number equals to queue number supported GPUs not verified Filesystems virtiofs, vhost-user-fs-pci, need to enable featuregate: ExperimentalVirtiofsSupport supported ClientPassthrough https://www.linaro.org/blog/kvm-pciemsi-passthrough-armarm64/on x86_64, iommu need to be enabled not verified Sound ich9/ ac97 not supported TPM tpm-tis-devicehttps://qemu.readthedocs.io/en/latest/specs/tpm.html supported Sriov vfio-pci not verified"},{"location":"cluster_admin/feature_gate_status_on_Arm64/","title":"Feature Gate Status on Arm64","text":"<p>This page is based on https://github.com/kubevirt/kubevirt/issues/9749 It records the feature gate status on Arm64 platform. Here is the explanation of the status:</p> <ul> <li>Supported: the feature gate support on Arm64 platform.</li> <li>Not supported yet: there are some dependencies of the feature gate not support Arm64, so this feature does not support for now. We may support the dependencies in the future.</li> <li>Not supported: The feature gate is not support on Arm64.</li> <li>Not verified: The feature has not been verified yet.</li> </ul> FEATURE GATE STATUS NOTES ExpandDisksGate Not supported yet CDI is needed CPUManager Supported use taskset to do CPU pinning, do not support kvm-hint-dedicated (this is only works on x86 platform) NUMAFeatureGate Not supported yet Need to support Hugepage on Arm64 IgnitionGate Supported This feature is only used for CoreOS/RhCOS LiveMigrationGate Supported Verified live migration with masquerade network SRIOVLiveMigrationGate Not verified Need two same Machine and SRIOV device HypervStrictCheckGate Not supported Hyperv does not work on Arm64 SidecarGate Supported GPUGate Not verified Need GPU device HostDevicesGate Not verified Need GPU or sound card SnapshotGate Supported Need snapshotter support https://github.com/kubernetes-csi/external-snapshotter VMExportGate Partially supported Need snapshotter support https://kubevirt.io/user-guide/operations/export_api/, support exporting pvc, not support exporting DataVolumes and MemoryDump which rely on CDI HotplugVolumesGate Not supported yet Rely on datavolume and CDI HostDiskGate Supported VirtIOFSGate Supported MacvtapGate Not supported yet quay.io/kubevirt/macvtap-cni not support Arm64, https://github.com/kubevirt/macvtap-cni#deployment PasstGate Supported VM have same ip with pods; start a process for network /usr/bin/passt --runas 107 -e -t 8080 DownwardMetricsFeatureGate need more information It used to let guest get host information, failed on both Arm64 and x86_64. The block is successfully attached and can see the following information: <code>-blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-private/downwardapi-disks/vhostmd0\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}</code>But unable to get information via <code>vm-dump-metrics</code>:<code>LIBMETRICS: read_mdisk(): Unable to read metrics disk</code><code>LIBMETRICS: get_virtio_metrics(): Unable to export metrics: open(/dev/virtio-ports/org.github.vhostmd.1) No such file or directory</code><code>LIBMETRICS: get_virtio_metrics(): Unable to read metrics</code> NonRootDeprecated Supported NonRoot Supported Root Supported ClusterProfiler Supported WorkloadEncryptionSEV Not supported SEV is only available on x86_64 VSOCKGate Supported HotplugNetworkIfacesGate Not supported yet Need to setup multus-cni and multus-dynamic-networks-controller: https://github.com/k8snetworkplumbingwg/multus-cni <code>cat ./deployments/multus-daemonset-thick.yml \\| kubectl apply -f -</code>https://github.com/k8snetworkplumbingwg/multus-dynamic-networks-controller <code>kubectl apply -f manifests/dynamic-networks-controller.yaml</code> Currently, the image ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick does not support Arm64 server. For more information please refer to https://github.com/k8snetworkplumbingwg/multus-cni/pull/1027. CommonInstancetypesDeploymentGate Not supported yet Support of common-instancetypes instancetypes needs to be tested, common-instancetypes preferences for ARM workloads are still missing"},{"location":"cluster_admin/gitops/","title":"Managing KubeVirt with GitOps","text":"<p>The GitOps way uses Git repositories as a single source of truth to deliver infrastructure as code. Automation is employed to keep the desired and the live state of clusters in sync at all times. This means any change to a repository is automatically applied to one or more clusters while changes to a cluster will be automatically reverted to the state described in the single source of truth.</p> <p>With GitOps the separation of testing and production environments, improving the availability of applications and working with multi-cluster environments becomes considerably easier.</p>"},{"location":"cluster_admin/gitops/#demo-repository","title":"Demo repository","text":"<p>A demo with detailed explanation on how to manage KubeVirt with GitOps can be found here.</p> <p>The demo is using Open Cluster Management and ArgoCD to deploy KubeVirt and virtual  machines across multiple clusters.</p>"},{"location":"cluster_admin/installation/","title":"Installation","text":"<p>KubeVirt is a virtualization add-on to Kubernetes and this guide assumes that a Kubernetes cluster is already installed.</p> <p>If installed on OKD, the web console is extended for management of virtual machines.</p>"},{"location":"cluster_admin/installation/#requirements","title":"Requirements","text":"<p>A few requirements need to be met before you can begin:</p> <ul> <li>Kubernetes cluster or derivative     (such as OpenShift)     based on a one of the latest three Kubernetes releases that are     out at the time the KubeVirt release is made.</li> <li>Kubernetes apiserver must have <code>--allow-privileged=true</code> in order to run KubeVirt's privileged DaemonSet.</li> <li><code>kubectl</code> client utility</li> </ul>"},{"location":"cluster_admin/installation/#container-runtime-support","title":"Container Runtime Support","text":"<p>KubeVirt is currently supported on the following container runtimes:</p> <ul> <li>containerd</li> <li>crio (with runv)</li> </ul> <p>Other container runtimes, which do not use virtualization features, should work too. However, the mentioned ones are the main target.</p>"},{"location":"cluster_admin/installation/#integration-with-apparmor","title":"Integration with AppArmor","text":"<p>In most of the scenarios, KubeVirt can run normally on systems with AppArmor. However, there are several known use cases that may require additional user interaction.</p> <ul> <li> <p>On a system with AppArmor enabled, the locally installed profiles     may block the execution of the KubeVirt privileged containers. That     usually results in initialization failure of the <code>virt-handler</code>     pod:</p> <pre><code>$ kubectl get pods -n kubevirt\nNAME                               READY   STATUS       RESTARTS         AGE\nvirt-api-77df5c4f87-7mqv4          1/1     Running      1 (17m ago)      27m\nvirt-api-77df5c4f87-wcq44          1/1     Running      1 (17m ago)      27m\nvirt-controller-749d8d99d4-56gb7   1/1     Running      1 (17m ago)      27m\nvirt-controller-749d8d99d4-78j6x   1/1     Running      1 (17m ago)      27m\nvirt-handler-4w99d                 0/1     Init:Error   14 (5m18s ago)   27m\nvirt-operator-564f568975-g9wh4     1/1     Running      1 (17m ago)      31m\nvirt-operator-564f568975-wnpz8     1/1     Running      1 (17m ago)      31m\n\n$ kubectl logs -n kubevirt virt-handler-4w99d virt-launcher\nerror: failed to get emulator capabilities\n\nerror: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied\n\n$ journalctl -b | grep DEN\n...\nMay 18 16:44:20 debian audit[6316]: AVC apparmor=\"DENIED\" operation=\"exec\" profile=\"libvirtd\" name=\"/usr/libexec/qemu-kvm\" pid=6316 comm=\"rpc-worker\" requested_mask=\"x\" denied_mask=\"x\" fsuid=107 ouid=0\nMay 18 16:44:20 debian kernel: audit: type=1400 audit(1652888660.539:39): apparmor=\"DENIED\" operation=\"exec\" profile=\"libvirtd\" name=\"/usr/libexec/qemu-kvm\" pid=6316 comm=\"rpc-worker\" requested_mask=\"x\" denied_mask=\"x\" fsuid=107 ouid=0\n...\n</code></pre> <p>Here, the host AppArmor profile for <code>libvirtd</code> does not allow the execution of the <code>/usr/libexec/qemu-kvm</code> binary. In the future this will hopefully work out of the box (tracking issue), but until then there are a couple of possible workarounds.</p> <p>The first (and simplest) one is to remove the libvirt package from the host: assuming the host is a dedicated Kubernetes node, you likely won't need it anyway.</p> <p>If you actually need libvirt to be present on the host, then you can add the following rule to the AppArmor profile for libvirtd (usually <code>/etc/apparmor.d/usr.sbin.libvirtd</code>):</p> <pre><code># vim /etc/apparmor.d/usr.sbin.libvirtd\n...\n/usr/libexec/qemu-kvm PUx,\n...\n# apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd # or systemctl reload apparmor.service\n</code></pre> </li> <li> <p>The default AppArmor profile used by the container runtimes usually     denies <code>mount</code> call for the workloads. That may prevent from     running VMs with VirtIO-FS.     This is a known issue.     The current workaround is to run such a VM as <code>unconfined</code> by adding the     following annotation to the VM or VMI object:</p> <pre><code>annotations:\n  container.apparmor.security.beta.kubernetes.io/compute: unconfined\n</code></pre> </li> </ul>"},{"location":"cluster_admin/installation/#validate-hardware-virtualization-support","title":"Validate Hardware Virtualization Support","text":"<p>Hardware with virtualization support is recommended. You can use virt-host-validate to ensure that your hosts are capable of running virtualization workloads:</p> <pre><code>$ virt-host-validate qemu\n  QEMU: Checking for hardware virtualization                                 : PASS\n  QEMU: Checking if device /dev/kvm exists                                   : PASS\n  QEMU: Checking if device /dev/kvm is accessible                            : PASS\n  QEMU: Checking if device /dev/vhost-net exists                             : PASS\n  QEMU: Checking if device /dev/net/tun exists                               : PASS\n...\n</code></pre>"},{"location":"cluster_admin/installation/#selinux-support","title":"SELinux support","text":"<p>SELinux-enabled nodes need Container-selinux installed. The minimum version is documented inside the kubevirt/kubevirt repository, in docs/getting-started.md, under \"SELinux support\".</p> <p>For (older) release branches that don't specify a container-selinux version, version 2.170.0 or newer is recommended.</p>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-kubernetes","title":"Installing KubeVirt on Kubernetes","text":"<p>KubeVirt can be installed using the KubeVirt operator, which manages the lifecycle of all the KubeVirt core components. Below is an example of how to install KubeVirt's latest official release. It supports to deploy KubeVirt on both x86_64 and Arm64 platforms.</p> <pre><code># Point at latest release\n$ export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)\n# Deploy the KubeVirt operator\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml\n# Create the KubeVirt CR (instance deployment request) which triggers the actual installation\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml\n# wait until all KubeVirt components are up\n$ kubectl -n kubevirt wait kv kubevirt --for condition=Available\n</code></pre> <p>If hardware virtualization is not available, then a software emulation fallback can be enabled using by setting in the KubeVirt CR <code>spec.configuration.developerConfiguration.useEmulation</code> to <code>true</code> as follows:</p> <pre><code>$ kubectl edit -n kubevirt kubevirt kubevirt\n</code></pre> <p>Add the following to the <code>kubevirt.yaml</code> file</p> <pre><code>spec:\n    ...\n    configuration:\n    developerConfiguration:\n        useEmulation: true\n</code></pre> <p>Note: Prior to release v0.20.0 the condition for the <code>kubectl wait</code> command was named \"Ready\" instead of \"Available\"</p> <p>Note: Prior to KubeVirt 0.34.2 a ConfigMap called <code>kubevirt-config</code> in the install-namespace was used to configure KubeVirt. Since 0.34.2 this method is deprecated. The configmap still has precedence over <code>configuration</code> on the CR exists, but it will not receive future updates and you should migrate any custom configurations to <code>spec.configuration</code> on the KubeVirt CR.</p> <p>All new components will be deployed under the <code>kubevirt</code> namespace:</p> <pre><code>kubectl get pods -n kubevirt\nNAME                                           READY     STATUS        RESTARTS   AGE\nvirt-api-6d4fc3cf8a-b2ere                      1/1       Running       0          1m\nvirt-controller-5d9fc8cf8b-n5trt               1/1       Running       0          1m\nvirt-handler-vwdjx                             1/1       Running       0          1m\n...\n</code></pre>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-okd","title":"Installing KubeVirt on OKD","text":"<p>The following SCC needs to be added prior KubeVirt deployment:</p> <pre><code>$ oc adm policy add-scc-to-user privileged -n kubevirt -z kubevirt-operator\n</code></pre> <p>Once privileges are granted, the KubeVirt can be deployed as described above.</p>"},{"location":"cluster_admin/installation/#web-user-interface-on-okd","title":"Web user interface on OKD","text":"<p>No additional steps are required to extend OKD's web console for KubeVirt.</p> <p>The virtualization extension is automatically enabled when KubeVirt deployment is detected.</p>"},{"location":"cluster_admin/installation/#from-service-catalog-as-an-apb","title":"From Service Catalog as an APB","text":"<p>You can find KubeVirt in the OKD Service Catalog and install it from there. In order to do that please follow the documentation in the KubeVirt APB repository.</p>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-k3os","title":"Installing KubeVirt on k3OS","text":"<p>The following configuration needs to be added to all nodes prior KubeVirt deployment:</p> <pre><code>k3os:\n  modules:\n  - kvm\n  - vhost_net\n</code></pre> <p>Once nodes are restarted with this configuration, the KubeVirt can be deployed as described above.</p>"},{"location":"cluster_admin/installation/#installing-the-daily-developer-builds","title":"Installing the Daily Developer Builds","text":"<p>KubeVirt releases daily a developer build from the current main branch. One can see when the last release happened by looking at our nightly-build-jobs.</p> <p>To install the latest developer build, run the following commands:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator.yaml\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr.yaml\n</code></pre> <p>To find out which commit this build is based on, run:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)\n$ curl https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/commit\nd358cf085b5a86cc4fa516215f8b757a4e61def2\n</code></pre>"},{"location":"cluster_admin/installation/#arm64-developer-builds","title":"ARM64 developer builds","text":"<p>ARM64 developer builds can be installed like this:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml\n</code></pre>"},{"location":"cluster_admin/installation/#deploying-from-source","title":"Deploying from Source","text":"<p>See the Developer Getting Started Guide to understand how to build and deploy KubeVirt from source.</p>"},{"location":"cluster_admin/installation/#installing-network-plugins-optional","title":"Installing network plugins (optional)","text":"<p>KubeVirt alone does not bring any additional network plugins, it just allows user to utilize them. If you want to attach your VMs to multiple networks (Multus CNI) or have full control over L2 (OVS CNI), you need to deploy respective network plugins. For more information, refer to OVS CNI installation guide.</p> <p>Note: KubeVirt Ansible network playbook installs these plugins by default.</p>"},{"location":"cluster_admin/installation/#restricting-kubevirt-components-node-placement","title":"Restricting KubeVirt components node placement","text":"<p>You can restrict the placement of the KubeVirt components across your  cluster nodes by editing the KubeVirt CR:</p> <ul> <li>The placement of the KubeVirt control plane components (virt-controller, virt-api)   is governed by the <code>.spec.infra.nodePlacement</code> field in the KubeVirt CR.</li> <li>The placement of the virt-handler DaemonSet pods (and consequently, the placement of the    VM workloads scheduled to the cluster) is governed by the <code>.spec.workloads.nodePlacement</code>   field in the KubeVirt CR.</li> </ul> <p>For each of these <code>.nodePlacement</code> objects, the <code>.affinity</code>, <code>.nodeSelector</code> and <code>.tolerations</code> sub-fields can be configured. See the description in the API reference for further information about using these fields.</p> <p>For example, to restrict the virt-controller and virt-api pods to only run on the control-plane nodes:</p> <pre><code>kubectl patch -n kubevirt kubevirt kubevirt --type merge --patch '{\"spec\": {\"infra\": {\"nodePlacement\": {\"nodeSelector\": {\"node-role.kubernetes.io/control-plane\": \"\"}}}}}'\n</code></pre> <p>To restrict the virt-handler pods to only run on nodes with the \"region=primary\" label:</p> <pre><code>kubectl patch -n kubevirt kubevirt kubevirt --type merge --patch '{\"spec\": {\"workloads\": {\"nodePlacement\": {\"nodeSelector\": {\"region\": \"primary\"}}}}}'\n</code></pre>"},{"location":"cluster_admin/ksm/","title":"KSM Management","text":"<p>Kernel Samepage Merging (KSM) allows de-duplication of memory. KSM tries to find identical Memory Pages and merge those to free memory.</p> <p>Further Information: - KSM (Kernel Samepage Merging) feature - Kernel Same-page Merging (KSM)</p>"},{"location":"cluster_admin/ksm/#enabling-ksm-through-kubevirt-cr","title":"Enabling KSM through KubeVirt CR","text":"<p>KSM can be enabled on nodes by <code>spec.configuration.ksmConfiguration</code> in the KubeVirt CR. <code>ksmConfiguration</code> instructs on which nodes KSM will be enabled, exposing a <code>nodeLabelSelector</code>. <code>nodeLabelSelector</code> is a LabelSelector and defines the filter, based on the node labels. If a node's labels match the label selector term, then on that node, KSM will be enabled.  </p> <p>NOTE If <code>nodeLabelSelector</code> is nil KSM will not be enabled on any nodes. Empty <code>nodeLabelSelector</code> will enable KSM on every node.  </p>"},{"location":"cluster_admin/ksm/#examples","title":"Examples:","text":"<ul> <li> <p>Enabling KSM on nodes in which the hostname is <code>node01</code> or <code>node03</code>: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector:\n        matchExpressions:\n          - key: kubernetes.io/hostname\n            operator: In\n            values:\n              - node01\n              - node03\n</code></pre></p> </li> <li> <p>Enabling KSM on nodes with labels <code>kubevirt.io/first-label: true</code>, <code>kubevirt.io/second-label: true</code>: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector:\n        matchLabels:\n          kubevirt.io/first-label: \"true\"\n          kubevirt.io/second-label: \"true\"\n</code></pre></p> </li> <li> <p>Enabling KSM on every node: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector: {}\n</code></pre></p> </li> </ul>"},{"location":"cluster_admin/ksm/#annotation-and-restore-mechanism","title":"Annotation and restore mechanism","text":"<p>On those nodes where KubeVirt enables the KSM via configuration, an annotation will be added (<code>kubevirt.io/ksm-handler-managed</code>). This annotation is an internal record to keep track of which nodes are currently  managed by virt-handler, so that it is possible to distinguish which nodes should be restored in case of future ksmConfiguration changes.</p> <p>Let's imagine this scenario:</p> <ol> <li>There are 3 nodes in the cluster and one of them(<code>node01</code>) has KSM externally enabled.</li> <li>An admin patches the KubeVirt CR adding a ksmConfiguration which enables ksm for <code>node02</code> and <code>node03</code>.</li> <li>After a while, an admin patches again the KubeVirt CR deleting the ksmConfiguration.</li> </ol> <p>Thanks to the annotation, the virt-handler is able to disable ksm on only those nodes where it itself had enabled it(<code>node02</code> <code>node03</code>), leaving the others unchanged (<code>node01</code>).</p>"},{"location":"cluster_admin/ksm/#node-labelling","title":"Node labelling","text":"<p>KubeVirt can discover on which nodes KSM is enabled and will mark them with a special label (<code>kubevirt.io/ksm-enabled</code>) with value <code>true</code>. This label can be used to schedule the vms in nodes with KSM enabled or not. <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: testvm\n    spec:\n      nodeSelector:\n        kubevirt.io/ksm-enabled: \"true\"\n      [...]\n</code></pre></p>"},{"location":"cluster_admin/migration_policies/","title":"Migration Policies","text":"<p>Migration policies provides a new way of applying migration configurations to Virtual Machines. The policies can refine Kubevirt CR's <code>MigrationConfiguration</code> that sets the cluster-wide migration configurations. This way, the cluster-wide settings serve as a default that can be refined (i.e. changed, removed or added) by the migration policy.</p> <p>Please bear in mind that migration policies are in version <code>v1alpha1</code>. This means that this API is not fully stable yet and that APIs may change in the future.</p>"},{"location":"cluster_admin/migration_policies/#overview","title":"Overview","text":"<p>KubeVirt supports Live Migrations of Virtual Machine workloads. Before migration policies were introduced, migration settings could be configurable only on the cluster-wide scope by editing KubevirtCR's spec or more specifically MigrationConfiguration CRD.</p> <p>Several aspects (although not all) of migration behaviour that can be customized are: - Bandwidth - Auto-convergence - Post/Pre-copy - Max number of parallel migrations - Timeout</p> <p>Migration policies generalize the concept of defining migration configurations, so it would be possible to apply different configurations to specific groups of VMs.</p> <p>Such capability can be useful for a lot of different use cases on which there is a need to differentiate between different workloads. Differentiation of different configurations could be needed because different workloads are considered to be in different priorities, security segregation, workloads with different requirements, help to converge workloads which aren't migration-friendly, and many other reasons.</p>"},{"location":"cluster_admin/migration_policies/#api-examples","title":"API Examples","text":""},{"location":"cluster_admin/migration_policies/#migration-configurations","title":"Migration Configurations","text":"<p>Currently the MigrationPolicy spec will only include the following configurations from KubevirtCR's MigrationConfiguration (in the future more configurations that aren't part of Kubevirt CR are intended to be added): <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n    allowAutoConverge: true\n    bandwidthPerMigration: 217Ki\n    completionTimeoutPerGiB: 23\n    allowPostCopy: false\n</code></pre></p> <p>All above fields are optional. When omitted, the configuration will be applied as defined in KubevirtCR's MigrationConfiguration. This way, KubevirtCR will serve as a configurable set of defaults for both VMs that are not bound to any MigrationPolicy and VMs that are bound to a MigrationPolicy that does not define all fields of the configurations.</p>"},{"location":"cluster_admin/migration_policies/#matching-policies-to-vms","title":"Matching Policies to VMs","text":"<p>Next in the spec are the selectors that define the group of VMs on which to apply the policy. The options to do so are the following.</p> <p>This policy applies to the VMs in namespaces that have all the required labels: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true       # Matches a key and a value \n</code></pre></p> <p>This policy applies for the VMs that have all the required labels: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    virtualMachineInstanceSelector:\n      workload-type: db       # Matches a key and a value \n</code></pre></p> <p>It is also possible to combine the previous two: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true\n    virtualMachineInstanceSelector:\n      workload-type: db\n</code></pre></p>"},{"location":"cluster_admin/migration_policies/#full-manifest","title":"Full Manifest:","text":"<pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\nmetadata:\n  name: my-awesome-policy\nspec:\n  # Migration Configuration\n  allowAutoConverge: true\n  bandwidthPerMigration: 217Ki\n  completionTimeoutPerGiB: 23\n  allowPostCopy: false\n\n  # Matching to VMs\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true\n    virtualMachineInstanceSelector:\n      workload-type: db\n</code></pre>"},{"location":"cluster_admin/migration_policies/#policies-precedence","title":"Policies' Precedence","text":"<p>It is possible that multiple policies apply to the same VMI. In such cases, the precedence is in the same order as the bullets above (VMI labels first, then namespace labels). It is not allowed to define two policies with the exact same selectors.</p> <p>If multiple policies apply to the same VMI: * The most detailed policy will be applied, that is, the policy with the highest number of matching labels</p> <ul> <li>If multiple policies match to a VMI with the same number of matching labels, the policies will be sorted by the lexicographic order of the matching labels keys. The first one in this order will be applied.</li> </ul>"},{"location":"cluster_admin/migration_policies/#example","title":"Example","text":"<p>For example, let's imagine a VMI with the following labels:</p> <ul> <li> <p>size: small</p> </li> <li> <p>os: fedora</p> </li> <li> <p>gpu: nvidia</p> </li> </ul> <p>And let's say the namespace to which the VMI belongs contains the following labels:</p> <ul> <li> <p>priority: high</p> </li> <li> <p>bandwidth: medium</p> </li> <li> <p>hpc-workload: true</p> </li> </ul> <p>The following policies are listed by their precedence (high to low):</p> <p>1) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high, bandwidth: medium}</code></p> <ul> <li>Matching labels: 4, First key in lexicographic order: <code>bandwidth</code>.</li> </ul> <p>2) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high, hpc-workload:true}</code></p> <ul> <li>Matching labels: 4, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>3) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>Matching labels: 3, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>4) VMI labels: <code>{size: small}</code>, Namespace labels: <code>{priority:high, hpc-workload:true}</code></p> <ul> <li>Matching labels: 3, First key in lexicographic order: <code>hpc-workload</code>.</li> </ul> <p>5) VMI labels: <code>{gpu: nvidia}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>Matching labels: 2, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>6) VMI labels: <code>{gpu: nvidia}</code>, Namespace labels: <code>{}</code></p> <ul> <li>Matching labels: 1, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>7) VMI labels: <code>{gpu: intel}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>VMI label does not match - policy cannot be applied.</li> </ul>"},{"location":"cluster_admin/node_maintenance/","title":"Node maintenance","text":"<p>Before removing a kubernetes node from the cluster, users will want to ensure that VirtualMachineInstances have been gracefully terminated before powering down the node. Since all VirtualMachineInstances are backed by a Pod, the recommended method of evicting VirtualMachineInstances is to use the kubectl drain command, or in the case of OKD the oc adm drain command.</p>"},{"location":"cluster_admin/node_maintenance/#evict-all-vms-from-a-node","title":"Evict all VMs from a Node","text":"<p>Select the node you'd like to evict VirtualMachineInstances from by identifying the node from the list of cluster nodes.</p> <p><code>kubectl get nodes</code></p> <p>The following command will gracefully terminate all VMs on a specific node. Replace <code>&lt;node-name&gt;</code> with the name of the node where the eviction should occur.</p> <p><code>kubectl drain &lt;node-name&gt; --delete-local-data --ignore-daemonsets=true --force --pod-selector=kubevirt.io=virt-launcher</code></p> <p>Below is a break down of why each argument passed to the drain command is required.</p> <ul> <li> <p><code>kubectl drain &lt;node-name&gt;</code> is selecting a specific node as a target     for the eviction</p> </li> <li> <p><code>--delete-local-data</code> is a required flag that is necessary for     removing any pod that utilizes an emptyDir volume. The     VirtualMachineInstance Pod does use emptyDir volumes, however the     data in those volumes are ephemeral which means it is safe to delete     after termination.</p> </li> <li> <p><code>--ignore-daemonsets=true</code> is a required flag because every node     running a VirtualMachineInstance will also be running our helper     DaemonSet called virt-handler. DaemonSets are not allowed to be     evicted using kubectl drain. By default, if this command     encounters a DaemonSet on the target node, the command will fail.     This flag tells the command it is safe to proceed with the eviction     and to just ignore DaemonSets.</p> </li> <li> <p><code>--force</code> is a required flag because VirtualMachineInstance pods are     not owned by a ReplicaSet or DaemonSet controller. This means     kubectl can't guarantee that the pods being terminated on the target     node will get re-scheduled replacements placed else where in the     cluster after the pods are evicted. KubeVirt has its own controllers     which manage the underlying VirtualMachineInstance pods. Each     controller behaves differently to a VirtualMachineInstance being     evicted. That behavior is outlined further down in this document.</p> </li> <li> <p><code>--pod-selector=kubevirt.io=virt-launcher</code> means only     VirtualMachineInstance pods managed by KubeVirt will be removed from     the node.</p> </li> </ul>"},{"location":"cluster_admin/node_maintenance/#evict-all-vms-and-pods-from-a-node","title":"Evict all VMs and Pods from a Node","text":"<p>By removing the <code>-pod-selector</code> argument from the previous command, we can issue the eviction of all Pods on a node. This command ensures Pods associated with VMs as well as all other Pods are evicted from the target node.</p> <p><code>kubectl drain &lt;node name&gt; --delete-local-data --ignore-daemonsets=true --force</code></p>"},{"location":"cluster_admin/node_maintenance/#evacuate-vmis-via-live-migration-from-a-node","title":"Evacuate VMIs via Live Migration from a Node","text":"<p>If the <code>LiveMigration</code> feature gate is enabled, it is possible to specify an <code>evictionStrategy</code> on VMIs which will react with live-migrations on specific taints on nodes. The following snippet on a VMI or the VMI templates in a VM ensures that the VMI is migrated during node eviction:</p> <pre><code>spec:\n  evictionStrategy: LiveMigrate\n</code></pre> <p>Here a full VMI:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  evictionStrategy: LiveMigrate\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: cloudinitdisk\n    cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n</code></pre> Behind the scenes a PodDisruptionBudget is created for each VMI which has an evictionStrategy defined. This ensures that evictions are be blocked on these VMIs and that we can guarantee that a VMI will be migrated instead of shut off.</p> <p>Note Prior to v0.34 the drain process with live migrations was detached from the <code>kubectl drain</code> itself and required in addition specifying a special taint on the nodes: <code>kubectl taint nodes foo kubevirt.io/drain=draining:NoSchedule</code>. This is no longer needed. The taint will still be respected if provided but is obsolete.</p>"},{"location":"cluster_admin/node_maintenance/#re-enabling-a-node-after-eviction","title":"Re-enabling a Node after Eviction","text":"<p>The kubectl drain will result in the target node being marked as unschedulable. This means the node will not be eligible for running new VirtualMachineInstances or Pods.</p> <p>If it is decided that the target node should become schedulable again, the following command must be run.</p> <p><code>kubectl uncordon &lt;node name&gt;</code></p> <p>or in the case of OKD.</p> <p><code>oc adm uncordon &lt;node name&gt;</code></p>"},{"location":"cluster_admin/node_maintenance/#shutting-down-a-node-after-eviction","title":"Shutting down a Node after Eviction","text":"<p>From KubeVirt's perspective, a node is safe to shutdown once all VirtualMachineInstances have been evicted from the node. In a multi-use cluster where VirtualMachineInstances are being scheduled alongside other containerized workloads, it is up to the cluster admin to ensure all other pods have been safely evicted before powering down the node.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachine-evictions","title":"VirtualMachine Evictions","text":"<p>The eviction of any VirtualMachineInstance that is owned by a VirtualMachine set to running=true will result in the VirtualMachineInstance being re-scheduled to another node.</p> <p>The VirtualMachineInstance in this case will be forced to power down and restart on another node. In the future once KubeVirt introduces live migration support, the VM will be able to seamlessly migrate to another node during eviction.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachineinstancereplicaset-eviction-behavior","title":"VirtualMachineInstanceReplicaSet Eviction Behavior","text":"<p>The eviction of VirtualMachineInstances owned by a VirtualMachineInstanceReplicaSet will result in the VirtualMachineInstanceReplicaSet scheduling replacements for the evicted VirtualMachineInstances on other nodes in the cluster.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachineinstance-eviction-behavior","title":"VirtualMachineInstance Eviction Behavior","text":"<p>VirtualMachineInstances not backed by either a VirtualMachineInstanceReplicaSet or an VirtualMachine object will not be re-scheduled after eviction.</p>"},{"location":"cluster_admin/operations_on_Arm64/","title":"Arm64 Operations","text":"<p>This page summarizes all operations that are not supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hotplug-network-interfaces","title":"Hotplug Network Interfaces","text":"<p>Hotplug Network Interfaces are not supported on Arm64, because the image ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick does not support for the Arm64 platform. For more information please refer to https://github.com/k8snetworkplumbingwg/multus-cni/pull/1027.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hotplug-volumes","title":"Hotplug Volumes","text":"<p>Hotplug Volumes are not supported on Arm64, because the Containerized Data Importer is not supported on Arm64 for now.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hugepages-support","title":"Hugepages support","text":"<p>Hugepages feature is not supported on Arm64. The hugepage mechanism differs between X86_64 and Arm64. Now we only verify KubeVirt on 4k pagesize systems.</p>"},{"location":"cluster_admin/operations_on_Arm64/#containerized-data-importer","title":"Containerized Data Importer","text":"<p>For now, we have not supported this project on Arm64, but it is in our plan.</p>"},{"location":"cluster_admin/operations_on_Arm64/#export-api","title":"Export API","text":"<p>Export API is partially supported on the Arm64 platform. As CDI is not supported yet, the export of DataVolumes and MemoryDump are not supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#virtual-machine-memory-dump","title":"Virtual machine memory dump","text":"<p>As explained above, MemoryDump requires CDI, and is not yet supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#mediated-devices-and-virtual-gpus","title":"Mediated devices and virtual GPUs","text":"<p>This is not verified on Arm64 platform.</p>"},{"location":"cluster_admin/scheduler/","title":"KubeVirt Scheduler","text":"<p>Scheduling is the process of matching Pods/VMs to Nodes. By default, the scheduler used is  kube-scheduler. Further details can be found at Kubernetes Scheduler Documentation.</p> <p>Custom schedulers can be used if the default scheduler does not satisfy your needs. For instance, you might want to schedule VMs using a load aware scheduler such as Trimaran Schedulers.</p>"},{"location":"cluster_admin/scheduler/#creating-a-custom-scheduler","title":"Creating a Custom Scheduler","text":"<p>KubeVirt is compatible with custom schedulers. The configuration steps are described in the Official Kubernetes  Documentation. Please note, the Kubernetes version KubeVirt is running on and the Kubernetes version used to build the custom scheduler have to match. To get the Kubernetes version KubeVirt is running on, you can run the following command:</p> <pre><code>$ kubectl version\nClient Version: version.Info{Major:\"1\", Minor:\"22\", GitVersion:\"v1.22.13\", GitCommit:\"a43c0904d0de10f92aa3956c74489c45e6453d6e\", GitTreeState:\"clean\", BuildDate:\"2022-08-17T18:28:56Z\", GoVersion:\"go1.16.15\", Compiler:\"gc\", Platform:\"linux/amd64\"}\nServer Version: version.Info{Major:\"1\", Minor:\"22\", GitVersion:\"v1.22.13\", GitCommit:\"a43c0904d0de10f92aa3956c74489c45e6453d6e\", GitTreeState:\"clean\", BuildDate:\"2022-08-17T18:23:45Z\", GoVersion:\"go1.16.15\", Compiler:\"gc\", Platform:\"linux/amd64\"}\n</code></pre> <p>Pay attention to the <code>Server</code> line.  In this case, the Kubernetes version is <code>v1.22.13</code>. You have to checkout the matching Kubernetes version and build the Kubernetes project:</p> <pre><code>$ cd kubernetes\n$ git checkout v1.22.13\n$ make\n</code></pre> <p>Then, you can follow the configuration steps described here. Additionally, the ClusterRole <code>system:kube-scheduler</code> needs permissions to use the verbs <code>watch</code>, <code>list</code> and <code>get</code> on StorageClasses.</p> <pre><code>- apiGroups:                                                                                                   \n  - storage.k8s.io                                                                                             \n  resources:                                                                                                   \n  - storageclasses                                                                                             \n  verbs:                                                                                                       \n  - watch                                                                                                      \n  - list                                                                                                       \n  - get \n</code></pre>"},{"location":"cluster_admin/scheduler/#scheduling-vms-with-the-custom-scheduler","title":"Scheduling VMs with the Custom Scheduler","text":"<p>The second scheduler should be up and running. You can check it with:</p> <pre><code>$ kubectl get all -n kube-system\n</code></pre> <p>The deployment <code>my-scheduler</code> should be up and running if everything is setup properly. In order to launch the VM using the custom scheduler, you need to set the <code>SchedulerName</code> in the VM's spec to <code>my-scheduler</code>. Here is an example VM definition:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\nspec:\n  running: true\n  template:\n    spec:\n      schedulerName: my-scheduler\n      domain:\n        devices:\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          rng: {}\n        resources:\n          requests:\n            memory: 1Gi\n      terminationGracePeriodSeconds: 180\n      volumes:\n        - containerDisk:\n            image: quay.io/containerdisks/fedora:latest\n          name: containerdisk\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: fedora\n              user: fedora\n          name: cloudinitdisk\n</code></pre> In case the specified <code>SchedulerName</code> does not match any existing scheduler, the <code>virt-launcher</code> pod will stay in state Pending,  until the specified scheduler can be found. You can check if the VM has been scheduled using the <code>my-scheduler</code> checking the <code>virt-launcher</code> pod events associated with the VM. The pod should have been scheduled with <code>my-scheduler</code>.</p> <pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vm-fedora-dpc87    2/2     Running   0          24m\n\n$ kubectl describe pod virt-launcher-vm-fedora-dpc87\n[...] \nEvents:\n  Type    Reason     Age   From              Message\n  ----    ------     ----  ----              -------\n  Normal  Scheduled  21m   my-scheduler  Successfully assigned default/virt-launcher-vm-fedora-dpc87 to node01\n[...]\n</code></pre>"},{"location":"cluster_admin/tekton_tasks/","title":"KubeVirt Tekton","text":""},{"location":"cluster_admin/tekton_tasks/#prerequisites","title":"Prerequisites","text":"<ul> <li>Tekton</li> <li>KubeVirt</li> <li>CDI</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#kubevirt-tekton-tasks","title":"KubeVirt Tekton Tasks","text":""},{"location":"cluster_admin/tekton_tasks/#what-are-kubevirt-tekton-tasks","title":"What are KubeVirt Tekton Tasks?","text":"<p>KubeVirt-specific Tekton Tasks, which are focused on:</p> <ul> <li>Creating and managing resources (VMs, DataVolumes)</li> <li>Executing commands in VMs</li> <li>Manipulating disk images with libguestfs tools</li> </ul> <p>KubeVirt Tekton Tasks and example Pipelines are available in artifacthub.io from where you can easily deploy them to your cluster.</p>"},{"location":"cluster_admin/tekton_tasks/#existing-tasks","title":"Existing Tasks","text":""},{"location":"cluster_admin/tekton_tasks/#create-virtual-machines","title":"Create Virtual Machines","text":"<ul> <li>create-vm-from-manifest - create a VM from provided manifest or with virtctl.</li> <li>create-vm-from-template - create a VM from template (works only on OpenShift).</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#utilize-templates","title":"Utilize Templates","text":"<ul> <li>copy-template - Copies the given template and creates a new one (works only on OpenShift).</li> <li>modify-vm-template - Modifies a template with user provided data (works only on OpenShift).</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#modify-data-objects","title":"Modify Data Objects","text":"<ul> <li>modify-data-object - Creates / modifies / deletes a datavolume / datasource</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#generate-ssh-keys","title":"Generate SSH Keys","text":"<ul> <li>generate-ssh-keys - Generates a private and public key pair, and injects it into a VM.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#execute-commands-in-virtual-machines","title":"Execute commands in Virtual Machines","text":"<ul> <li>execute-in-vm - Execute commands over SSH in a VM.</li> <li>cleanup-vm - Execute commands and/or stop/delete VMs.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#manipulate-pvcs-with-libguestfs-tools","title":"Manipulate PVCs with libguestfs tools","text":"<ul> <li>disk-virt-customize - execute virt-customize commands in PVCs.</li> <li>disk-virt-sysprep- execute virt-sysprep commands in PVCs.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#wait-for-virtual-machine-instance-status","title":"Wait for Virtual Machine Instance Status","text":"<ul> <li>wait-for-vmi-status - Waits for a VMI to be running.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#modify-windows-iso","title":"Modify Windows iso","text":"<ul> <li>modify-windows-iso-file - modifies windows iso (replaces prompt bootloader with no-prompt bootloader) and replaces original iso    in PVC with updated one. This helps with automated installation of Windows in EFI boot mode. By default Windows in EFI boot mode    uses a prompt bootloader, which will not continue with the boot process until a key is pressed. By replacing it with the non-prompt    bootloader no key press is required to boot into the Windows installer.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#example-pipeline","title":"Example Pipeline","text":"<p>All these Tasks can be used for creating Pipelines. We prepared example Pipelines which show what can you do with the KubeVirt Tasks.</p> <ul> <li> <p>Windows efi installer - This Pipeline will prepare a Windows 10/11/2k22 datavolume with virtio drivers installed. User has to provide a working link to a Windows 10/11/2k22 iso file. The Pipeline is suitable for Windows versions, which requires EFI (e.g. Windows 10/11/2k22). More information about Pipeline can be found here</p> </li> <li> <p>Windows customize - This Pipeline will install a SQL server or a VS Code in a Windows VM. More information about Pipeline can be found here</p> </li> </ul> <p>Note</p> <ul> <li>If you define a different namespace for Pipelines and a different namespace for Tasks, you will have to create a cluster resolver object. </li> <li>By default, example Pipelines create the resulting datavolume in the <code>kubevirt-os-images</code> namespace. </li> <li>In case you would like to create resulting datavolume in different namespace (by specifying <code>baseDvNamespace</code> attribute in Pipeline), additional RBAC permissions will be required (list of all required RBAC permissions can be found here). </li> <li>In case you would like to live migrate the VM while a given Pipeline is running, the following prerequisities must be met </li> </ul>"},{"location":"cluster_admin/unresponsive_nodes/","title":"Unresponsive nodes","text":"<p>KubeVirt has its own node daemon, called virt-handler. In addition to the usual k8s methods of detecting issues on nodes, the virt-handler daemon has its own heartbeat mechanism. This allows for fine-tuned error handling of VirtualMachineInstances.</p>"},{"location":"cluster_admin/unresponsive_nodes/#virt-handler-heartbeat","title":"virt-handler heartbeat","text":"<p><code>virt-handler</code> periodically tries to update the <code>kubevirt.io/schedulable</code> label and the <code>kubevirt.io/heartbeat</code> annotation on the node it is running on:</p> <pre><code>$ kubectl get nodes -o yaml\napiVersion: v1\nitems:\n- apiVersion: v1\n  kind: Node\n  metadata:\n    annotations:\n      kubevirt.io/heartbeat: 2018-11-05T09:42:25Z\n    creationTimestamp: 2018-11-05T08:55:53Z\n    labels:\n      beta.kubernetes.io/arch: amd64\n      beta.kubernetes.io/os: linux\n      cpumanager: \"false\"\n      kubernetes.io/hostname: node01\n      kubevirt.io/schedulable: \"true\"\n      node-role.kubernetes.io/control-plane: \"\"\n</code></pre> <p>If a <code>VirtualMachineInstance</code> gets scheduled, the scheduler is only considering nodes where <code>kubevirt.io/schedulable</code> is <code>true</code>. This can be seen when looking on the corresponding pod of a <code>VirtualMachineInstance</code>:</p> <pre><code>$ kubectl get pods  virt-launcher-vmi-nocloud-ct6mr -o yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  [...]\nspec:\n  [...]\n  nodeName: node01\n  nodeSelector:\n    kubevirt.io/schedulable: \"true\"\n  [...]\n</code></pre> <p>In case there is a communication issue or the host goes down, <code>virt-handler</code> can't update its labels and annotations any-more. Once the last <code>kubevirt.io/heartbeat</code> timestamp is older than five minutes, the KubeVirt node-controller kicks in and sets the <code>kubevirt.io/schedulable</code> label to <code>false</code>. As a consequence no more VMIs will be schedule to this node until virt-handler is connected again.</p>"},{"location":"cluster_admin/unresponsive_nodes/#deleting-stuck-vmis-when-virt-handler-is-unresponsive","title":"Deleting stuck VMIs when virt-handler is unresponsive","text":"<p>In cases where <code>virt-handler</code> has some issues but the node is in general fine, a <code>VirtualMachineInstance</code> can be deleted as usual via <code>kubectl delete vmi &lt;myvm&gt;</code>. Pods of a <code>VirtualMachineInstance</code> will be told by the cluster-controllers they should shut down. As soon as the Pod is gone, the <code>VirtualMachineInstance</code> will be moved to <code>Failed</code> state, if <code>virt-handler</code> did not manage to update it's heartbeat in the meantime. If <code>virt-handler</code> could recover in the meantime, <code>virt-handler</code> will move the <code>VirtualMachineInstance</code> to failed state instead of the cluster-controllers.</p>"},{"location":"cluster_admin/unresponsive_nodes/#deleting-stuck-vmis-when-the-whole-node-is-unresponsive","title":"Deleting stuck VMIs when the whole node is unresponsive","text":"<p>If the whole node is unresponsive, deleting a <code>VirtualMachineInstance</code> via <code>kubectl delete vmi &lt;myvmi&gt;</code> alone will never remove the <code>VirtualMachineInstance</code>. In this case all pods on the unresponsive node need to be force-deleted: First make sure that the node is really dead. Then delete all pods on the node via a force-delete: <code>kubectl delete pod --force --grace-period=0 &lt;mypod&gt;</code>.</p> <p>As soon as the pod disappears and the heartbeat from virt-handler timed out, the VMIs will be moved to <code>Failed</code> state. If they were already marked for deletion they will simply disappear. If not, they can be deleted and will disappear almost immediately.</p>"},{"location":"cluster_admin/unresponsive_nodes/#timing-considerations","title":"Timing considerations","text":"<p>It takes up to five minutes until the KubeVirt cluster components can detect that virt-handler is unhealthy. During that time-frame it is possible that new VMIs are scheduled to the affected node. If virt-handler is not capable of connecting to these pods on the node, the pods will sooner or later go to failed state. As soon as the cluster finally detects the issue, the VMIs will be set to failed by the cluster.</p>"},{"location":"cluster_admin/updating_and_deletion/","title":"Updating and deletion","text":""},{"location":"cluster_admin/updating_and_deletion/#updating-kubevirt-control-plane","title":"Updating KubeVirt Control Plane","text":"<p>Zero downtime rolling updates are supported starting with release <code>v0.17.0</code> onward. Updating from any release prior to the KubeVirt <code>v0.17.0</code> release is not supported.</p> <p>Note: Updating is only supported from N-1 to N release.</p> <p>Updates are triggered one of two ways.</p> <ol> <li> <p>By changing the imageTag value in the KubeVirt CR's spec.</p> <p>For example, updating from <code>v0.17.0-alpha.1</code> to <code>v0.17.0</code> is as simple as patching the KubeVirt CR with the <code>imageTag: v0.17.0</code> value. From there the KubeVirt operator will begin the process of rolling out the new version of KubeVirt. Existing VM/VMIs will remain uninterrupted both during and after the update succeeds.</p> <pre><code>$ kubectl patch kv kubevirt -n kubevirt --type=json -p '[{ \"op\": \"add\", \"path\": \"/spec/imageTag\", \"value\": \"v0.17.0\" }]'\n</code></pre> </li> <li> <p>Or, by updating the kubevirt operator if no imageTag value is set.</p> <p>When no imageTag value is set in the kubevirt CR, the system assumes that the version of KubeVirt is locked to the version of the operator. This means that updating the operator will result in the underlying KubeVirt installation being updated as well.</p> <pre><code>$ export RELEASE=v0.26.0\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml\n</code></pre> </li> </ol> <p>The first way provides a fine granular approach where you have full control over what version of KubeVirt is installed independently of what version of the KubeVirt operator you might be running. The second approach allows you to lock both the operator and operand to the same version.</p> <p>Newer KubeVirt may require additional or extended RBAC rules. In this case, the #1 update method may fail, because the virt-operator present in the cluster doesn't have these RBAC rules itself. In this case, you need to update the <code>virt-operator</code> first, and then proceed to update kubevirt. See this issue for more details.</p>"},{"location":"cluster_admin/updating_and_deletion/#updating-kubevirt-workloads","title":"Updating KubeVirt Workloads","text":"<p>Workload updates are supported as an opt in feature starting with <code>v0.39.0</code></p> <p>By default, when KubeVirt is updated this only involves the control plane components. Any existing VirtualMachineInstance (VMI) workloads that are running before an update occurs remain 100% untouched. The workloads continue to run and are not interrupted as part of the default update process.</p> <p>It's important to note that these VMI workloads do involve components such as libvirt, qemu, and virt-launcher, which can optionally be updated during the KubeVirt update process as well. However that requires opting in to having virt-operator perform automated actions on workloads.</p> <p>Opting in to VMI updates involves configuring the <code>workloadUpdateStrategy</code> field on the KubeVirt CR. This field controls the methods virt-operator will use to when updating the VMI workload pods.</p> <p>There are two methods supported.</p> <p>LiveMigrate: Which results in VMIs being updated by live migrating the virtual machine guest into a new pod with all the updated components enabled.</p> <p>Evict:  Which results in the VMI's pod being shutdown. If the VMI is controlled by a higher level VirtualMachine object with <code>runStrategy: always</code>, then a new VMI will spin up in a new pod with updated components.</p> <p>The least disruptive way to update VMI workloads is to use LiveMigrate. Any VMI workload that is not live migratable will be left untouched. If live migration is not enabled in the cluster, then the only option available for virt-operator managed VMI updates is the Evict method.</p> <p>Example: Enabling VMI workload updates via LiveMigration</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - LiveMigrate\n</code></pre> <p>Example: Enabling VMI workload updates via Evict with batch tunings</p> <p>The batch tunings allow configuring how quickly VMI's are evicted. In large clusters, it's desirable to ensure that VMI's are evicted in batches in order to distribute load.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - Evict\n    batchEvictionSize: 10\n    batchEvictionInterval: \"1m\"\n</code></pre> <p>Example: Enabling VMI workload updates with both LiveMigrate and Evict</p> <p>When both LiveMigrate and Evict are specified, then any workloads which are live migratable will be guaranteed to be live migrated. Only workloads which are not live migratable will be evicted.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - LiveMigrate\n      - Evict\n    batchEvictionSize: 10\n    batchEvictionInterval: \"1m\"\n</code></pre>"},{"location":"cluster_admin/updating_and_deletion/#deleting-kubevirt","title":"Deleting KubeVirt","text":"<p>To delete the KubeVirt you should first to delete <code>KubeVirt</code> custom resource and then delete the KubeVirt operator.</p> <pre><code>$ export RELEASE=v0.17.0\n$ kubectl delete -n kubevirt kubevirt kubevirt --wait=true # --wait=true should anyway be default\n$ kubectl delete apiservices v1.subresources.kubevirt.io # this needs to be deleted to avoid stuck terminating namespaces\n$ kubectl delete mutatingwebhookconfigurations virt-api-mutator # not blocking but would be left over\n$ kubectl delete validatingwebhookconfigurations virt-operator-validator # not blocking but would be left over\n$ kubectl delete validatingwebhookconfigurations virt-api-validator # not blocking but would be left over\n$ kubectl delete -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml --wait=false\n</code></pre> <p>Note: If by mistake you deleted the operator first, the KV custom resource will get stuck in the <code>Terminating</code> state, to fix it, delete manually finalizer from the resource.</p> <p>Note: The <code>apiservice</code> and the <code>webhookconfigurations</code> need to be deleted manually due to a bug.</p> <pre><code>$ kubectl -n kubevirt patch kv kubevirt --type=json -p '[{ \"op\": \"remove\", \"path\": \"/metadata/finalizers\" }]'\n</code></pre>"},{"location":"cluster_admin/virtual_machines_on_Arm64/","title":"Virtual Machines on Arm64","text":"<p>This page summaries all unsupported Virtual Machines configurations and different default setups on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#virtual-hardware","title":"Virtual hardware","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#machine-type","title":"Machine Type","text":"<p>Currently, we only support one machine type, <code>virt</code>, which is set by default.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#biosuefi","title":"BIOS/UEFI","text":"<p>On Arm64 platform, we only support UEFI boot which is set by default. UEFI secure boot is not supported.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#cpu","title":"CPU","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#node-labeller","title":"Node-labeller","text":"<p>Currently, Node-labeller is partially supported on Arm64 platform. It does not yet support parsing virsh_domcapabilities.xml and capabilities.xml, and extracting related information such as CPU features.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#model","title":"Model","text":"<p><code>host-passthrough</code> is the only model that supported on Arm64. The CPU model is set by default on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#clock","title":"Clock","text":"<p><code>kvm</code> and <code>hyperv</code> timers are not supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#video-and-graphics-device","title":"Video and Graphics Device","text":"<p>We do not support vga devices but use virtio-gpu by default.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#hugepages","title":"Hugepages","text":"<p>Hugepages are not supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#resources-requests-and-limits","title":"Resources Requests and Limits","text":"<p>CPU pinning is supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#numa","title":"NUMA","text":"<p>As Hugepages are a precondition of the NUMA feature, and Hugepages are not enabled on the Arm64 platform, the NUMA feature does not work on Arm64.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#disks-and-volumes","title":"Disks and Volumes","text":"<p>Arm64 only supports virtio and scsi disk bus types.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#interface-and-networks","title":"Interface and Networks","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#macvlan","title":"macvlan","text":"<p>We do not support <code>macvlan</code> network because the project https://github.com/kubevirt/macvtap-cni does not support Arm64.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#sriov","title":"SRIOV","text":"<p>This class of devices is not verified on the Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#liveness-and-readiness-probes","title":"Liveness and Readiness Probes","text":"<p><code>Watchdog</code> device is not supported on Arm64 platform.</p>"},{"location":"compute/client_passthrough/","title":"Client Passthrough","text":"<p>KubeVirt included support for redirecting devices from the client's machine to the VMI with the support of virtctl command.</p>"},{"location":"compute/client_passthrough/#usb-redirection","title":"USB Redirection","text":"<p>Support for redirection of client's USB device was introduced in release v0.44. This feature is not enabled by default. To enable it, add an empty <code>clientPassthrough</code> under devices, as such:</p> <pre><code>spec:\n  domain:\n    devices:\n      clientPassthrough: {}\n</code></pre> <p>This configuration currently adds 4 USB slots to the VMI that can only be used with virtctl.</p> <p>There are two ways of redirecting the same USB devices: Either using its device's vendor and product information or the actual bus and device address information. In Linux, you can gather this info with <code>lsusb</code>, a redacted example below:</p> <pre><code>&gt; lsusb\nBus 002 Device 008: ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2/50\nBus 001 Device 003: ID 13d3:5406 IMC Networks Integrated Camera\nBus 001 Device 010: ID 0781:55ae SanDisk Corp. Extreme 55AE\n</code></pre>"},{"location":"compute/client_passthrough/#using-vendor-and-product","title":"Using Vendor and Product","text":"<p>Redirecting the Kingston storage device. <pre><code>virtctl usbredir 0951:1666 vmi-name\n</code></pre></p>"},{"location":"compute/client_passthrough/#using-bus-and-device-address","title":"Using Bus and Device address","text":"<p>Redirecting the integrated camera <pre><code>virtctl usbredir 01-03 vmi-name\n</code></pre></p>"},{"location":"compute/client_passthrough/#requirements-for-virtctl-usbredir","title":"Requirements for <code>virtctl usbredir</code>","text":"<p>The <code>virtctl</code> command uses an application called <code>usbredirect</code> to handle client's USB device by unplugging the device from the Client OS and channeling the communication between the device and the VMI.</p>"},{"location":"compute/client_passthrough/#usbredirect","title":"usbredirect","text":"<p>The <code>usbredirect</code> binary comes from the usbredir project and is supported by most Linux distros. You can either fetch the latest release or MSI installer for Windows support.</p>"},{"location":"compute/client_passthrough/#permissions","title":"Permissions","text":"<p>Managing USB devices requires privileged access in most Operation Systems. The user running <code>virtctl usbredir</code> would need to be privileged or run it in a privileged manner (e.g: with <code>sudo</code>)</p>"},{"location":"compute/client_passthrough/#windows-support","title":"Windows support","text":"<ul> <li>Redirecting USB devices on Windows requires the installation of UsbDk.</li> <li>Be sure to have <code>usbredirect</code> included in the PATH Enviroment Variable.</li> </ul>"},{"location":"compute/cpu_hotplug/","title":"CPU Hotplug","text":"<p>The CPU hotplug feature was introduced in KubeVirt v1.0, making it possible to configure the VM workload to allow for adding or removing virtual CPUs while the VM is running.</p>"},{"location":"compute/cpu_hotplug/#abstract","title":"Abstract","text":"<p>A virtual CPU (vCPU) is the CPU that is seen to the Guest VM OS. A VM owner can manage the amount of vCPUs from the VM spec template using the CPU topology fields (<code>spec.template.spec.domain.cpu</code>). The <code>cpu</code> object has the integers <code>cores,sockets,threads</code> so that the virtual CPU is calculated by the following formula: <code>cores * sockets * threads</code>. </p> <p>Before CPU hotplug was introduced, the VM owner could change these integers in the VM template while the VM is running, and they were staged until the next boot cycle. With CPU hotplug, it is possible to patch the <code>sockets</code> integer in the VM template and the change will take effect right away. </p> <p>Per each new socket that is hot-plugged, the amount of new vCPUs that would be seen by the guest is <code>cores * threads</code>, since the overall calculation of vCPUs is <code>cores * sockets * threads</code>. </p>"},{"location":"compute/cpu_hotplug/#configuration","title":"Configuration","text":""},{"location":"compute/cpu_hotplug/#enable-feature-gate","title":"Enable feature-gate","text":"<p>In order to enable CPU hotplug we need to add the <code>VMLiveUpdateFeatures</code> feature gate in Kubevirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"compute/cpu_hotplug/#configure-the-workload-update-strategy","title":"Configure the workload update strategy","text":"<p>Current implementation of the hotplug process requires the VM to live-migrate. The migration will be triggered automatically by the workload updater. The workload update strategy in the KubeVirt CR must be configured with <code>LiveMigrate</code>, as follows:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre>"},{"location":"compute/cpu_hotplug/#configure-the-vm-rollout-strategy","title":"Configure the VM rollout strategy","text":"<p>Hotplug requires a VM rollout strategy of <code>LiveUpdate</code>, so that the changes made to the VM object propagate to the VMI without a restart. This is also done in the KubeVirt CR configuration:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre> <p>More information can be found on the VM Rollout Strategies page</p>"},{"location":"compute/cpu_hotplug/#optional-set-maximum-sockets-or-hotplug-ratio","title":"[OPTIONAL] Set maximum sockets or hotplug ratio","text":"<p>You can explicitly set the maximum amount of sockets in three ways:</p> <ol> <li>with a value VM level</li> <li>with a value at the cluster level</li> <li>with a ratio at the cluster level (<code>maxSockets = ratio * sockets</code>).</li> </ol> <p>Note: the third way (cluster-level ratio) will also affect other quantitative hotplug resources like memory.</p> <p> VM level </p> <p> Cluster level value </p> <p> Cluster level ratio </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nspec:\n  template:\n    spec:\n      domain:\n        cpu:\n          maxSockets: 8\n</code></pre> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxCpuSockets: 8\n</code></pre> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxHotplugRatio: 4\n</code></pre> <p>The VM-level configuration will take precedence over the cluster-wide configuration.</p>"},{"location":"compute/cpu_hotplug/#hotplug-process","title":"Hotplug process","text":"<p>Let's assume we have a running VM with the 4 vCPUs, which were configured with <code>sockets:4 cores:1 threads:1</code> In the VMI status we can observe the current CPU topology the VM is running with:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\n...\nstatus:\n  currentCPUTopology:\n    cores: 1\n    sockets: 4\n    threads: 1\n</code></pre> Now we want to hotplug another socket, by patching the VM object:</p> <p><pre><code>kubectl patch vm vm-cirros --type='json' \\\n-p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/domain/cpu/sockets\", \"value\": 5}]'\n</code></pre> We can observe the CPU hotplug process in the VMI status:</p> <pre><code>status:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: null\n    status: \"True\"\n    type: LiveMigratable\n  - lastProbeTime: null\n    lastTransitionTime: null\n    status: \"True\"\n    type: HotVCPUChange\n  currentCPUTopology:\n    cores: 1\n    sockets: 4\n    threads: 1\n</code></pre> <p>Please note the condition <code>HotVCPUChange</code> that indicates the hotplug process is taking place. Also you can notice the VirtualMachineInstanceMigration object that was created for the VM in subject:</p> <p><pre><code>NAME                             PHASE     VMI\nkubevirt-workload-update-kflnl   Running   vm-cirros\n</code></pre> When the hotplug process has completed, the <code>currentCPUTopology</code> will be updated with the new number of sockets and the migration is marked as successful.</p> <pre><code>#kubectl get vmi vm-cirros -oyaml\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vm-cirros\nspec:\n  domain:\n    cpu:\n      cores: 1\n      sockets: 5\n      threads: 1\n...\n...\nstatus:\n  currentCPUTopology:\n    cores: 1\n    sockets: 5\n    threads: 1\n\n\n#kubectl get vmim -l kubevirt.io/vmi-name=vm-cirros\nNAME                             PHASE       VMI\nkubevirt-workload-update-cgdgd   Succeeded   vm-cirros\n</code></pre>"},{"location":"compute/cpu_hotplug/#limitations","title":"Limitations","text":"<ul> <li>VPCU hotplug is currently not supported by ARM64 architecture.</li> <li>Current hotplug implementation involves live-migration of the VM workload.</li> </ul>"},{"location":"compute/dedicated_cpu_resources/","title":"Dedicated CPU resources","text":"<p>Certain workloads, requiring a predictable latency and enhanced performance during its execution would benefit from obtaining dedicated CPU resources. KubeVirt, relying on the Kubernetes CPU manager, is able to pin guest's vCPUs to the host's pCPUs.</p>"},{"location":"compute/dedicated_cpu_resources/#kubernetes-cpu-manager","title":"Kubernetes CPU manager","text":"<p>Kubernetes CPU manager is a mechanism that affects the scheduling of workloads, placing it on a host which can allocate <code>Guaranteed</code> resources and pin certain Pod's containers to host pCPUs, if the following requirements are met:</p> <ul> <li>Pod's QoS is Guaranteed<ul> <li>resources requests and limits are equal</li> <li>all containers in the Pod express CPU and memory requirements</li> </ul> </li> <li>Requested number of CPUs is an Integer</li> </ul> <p>Additional information: </p> <ul> <li>Enabling the CPU manager on Kubernetes</li> <li>Enabling the CPU manager on OKD</li> <li>Kubernetes blog explaining the feature</li> </ul>"},{"location":"compute/dedicated_cpu_resources/#requesting-dedicated-cpu-resources","title":"Requesting dedicated CPU resources","text":"<p>Setting <code>spec.domain.cpu.dedicatedCpuPlacement</code> to <code>true</code> in a VMI spec will indicate the desire to allocate dedicated CPU resource to the VMI</p> <p>Kubevirt will verify that all the necessary conditions are met, for the Kubernetes CPU manager to pin the virt-launcher container to dedicated host CPUs. Once, virt-launcher is running, the VMI's vCPUs will be pinned to the pCPUS that has been dedicated for the virt-launcher container.</p> <p>Expressing the desired amount of VMI's vCPUs can be done by either setting the guest topology in <code>spec.domain.cpu</code> (<code>sockets</code>, <code>cores</code>, <code>threads</code>) or <code>spec.domain.resources.[requests/limits].cpu</code> to a whole number integer ([1-9]+) indicating the number of vCPUs requested for the VMI. Number of vCPUs is counted as <code>sockets * cores * threads</code> or if <code>spec.domain.cpu</code> is empty then it takes value from <code>spec.domain.resources.requests.cpu</code> or <code>spec.domain.resources.limits.cpu</code>.</p> <p>Note: Users should not specify both <code>spec.domain.cpu</code> and <code>spec.domain.resources.[requests/limits].cpu</code></p> <p>Note: <code>spec.domain.resources.requests.cpu</code> must be equal to <code>spec.domain.resources.limits.cpu</code></p> <p>Note: Multiple cpu-bound microbenchmarks show a significant performance advantage when using <code>spec.domain.cpu.sockets</code> instead of <code>spec.domain.cpu.cores</code>.</p> <p>All inconsistent requirements will be rejected.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      sockets: 2\n      cores: 1\n      threads: 1\n      dedicatedCpuPlacement: true\n    resources:\n      limits:\n        memory: 2Gi\n[...]\n</code></pre> <p>OR</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      dedicatedCpuPlacement: true\n    resources:\n      limits:\n        cpu: 2\n        memory: 2Gi\n[...]\n</code></pre>"},{"location":"compute/dedicated_cpu_resources/#requesting-dedicated-cpu-for-qemu-emulator","title":"Requesting dedicated CPU for QEMU emulator","text":"<p>A number of QEMU threads, such as QEMU main event loop, async I/O operation completion, etc., also execute on the same physical CPUs as the VMI's vCPUs. This may affect the expected latency of a vCPU. In order to enhance the real-time support in KubeVirt and provide improved latency, KubeVirt will allocate an additional dedicated CPU, exclusively for the emulator thread, to which it will be pinned. This will effectively \"isolate\" the emulator thread from the vCPUs of the VMI. In case <code>ioThreadsPolicy</code> is set to <code>auto</code> IOThreads will also be \"isolated\" and placed on the same physical CPU as the QEMU emulator thread.</p> <p>This functionality can be enabled by specifying <code>isolateEmulatorThread: true</code> inside VMI spec's <code>Spec.Domain.CPU</code> section. Naturally, this setting has to be specified in a combination with a <code>dedicatedCpuPlacement: true</code>.</p> <p>Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      dedicatedCpuPlacement: true\n      isolateEmulatorThread: true\n    resources:\n      limits:\n        cpu: 2\n        memory: 2Gi\n</code></pre>"},{"location":"compute/dedicated_cpu_resources/#compute-nodes-with-smt-enabled","title":"Compute Nodes with SMT Enabled","text":"<p>When the following conditions are met:</p> <ul> <li>The compute node has SMT enabled</li> <li>Kubelet's CPUManager policy is set to static - full-pcpus-only</li> <li>The VM is configured to have an even number of CPUs</li> <li><code>dedicatedCpuPlacement</code> and <code>isolateEmulatorThread</code> are enabled</li> </ul> <p>The VM is scheduled, but rejected by the kubelet with the following event: <pre><code>SMT Alignment Error: requested 3 cpus not multiple cpus per core = 2\n</code></pre></p> <p>In order to address this issue:</p> <ol> <li>Enable the <code>AlignCPUs</code> feature gate in the KubeVirt CR.</li> <li>Add the following annotation to the Kubevirt CR:</li> </ol> <pre><code>alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity:\n</code></pre> <p>KubeVirt will then add one or two dedicated CPUs for the emulator threads, in a way that completes the total CPU count to be even.</p>"},{"location":"compute/dedicated_cpu_resources/#identifying-nodes-with-a-running-cpu-manager","title":"Identifying nodes with a running CPU manager","text":"<p>At this time, Kubernetes doesn't label the nodes that has CPU manager running on it.</p> <p>KubeVirt has a mechanism to identify which nodes has the CPU manager running and manually add a <code>cpumanager=true</code> label. This label will be removed when KubeVirt will identify that CPU manager is no longer running on the node. This automatic identification should be viewed as a temporary workaround until Kubernetes will provide the required functionality. Therefore, this feature should be manually enabled by activating the <code>CPUManager</code> feature gate to the KubeVirt CR.</p> <p>When automatic identification is disabled, cluster administrator may manually add the above label to all the nodes when CPU Manager is running.</p> <ul> <li> <p>Nodes' labels are view-able: <code>kubectl describe nodes</code></p> </li> <li> <p>Administrators may manually label a missing node:     <code>kubectl label node [node_name] cpumanager=true</code></p> </li> </ul>"},{"location":"compute/dedicated_cpu_resources/#sidecar-containers-and-cpu-allocation-overhead","title":"Sidecar containers and CPU allocation overhead","text":"<p>Note: In order to run sidecar containers, KubeVirt requires the <code>Sidecar</code> feature gate to be enabled in KubeVirt's CR.</p> <p>According to the Kubernetes CPU manager model, in order the POD would reach the required QOS level <code>Guaranteed</code>, all containers in the POD must express CPU and memory requirements. At this time, Kubevirt often uses a sidecar container to mount VMI's registry disk. It also uses a sidecar container of it's hooking mechanism. These additional resources can be viewed as an overhead and should be taken into account when calculating a node capacity.</p> <p>Note: The current defaults for sidecar's resources: <code>CPU: 200m</code> <code>Memory: 64M</code> As the CPU resource is not expressed as a whole number, CPU manager will not attempt to pin the sidecar container to a host CPU.</p>"},{"location":"compute/host-devices/","title":"Host Devices Assignment","text":"<p>KubeVirt provides a mechanism for assigning host devices to a virtual machine. This mechanism is generic and allows various types of PCI devices, such as accelerators (including GPUs) or any other devices attached to a PCI bus, to be assigned. It also allows Linux Mediated devices, such as pre-configured virtual GPUs to be assigned using the same mechanism.</p>"},{"location":"compute/host-devices/#host-preparation-for-pci-passthrough","title":"Host preparation for PCI Passthrough","text":"<ul> <li> <p>Host Devices passthrough requires the virtualization extension and the IOMMU extension (Intel VT-d or AMD IOMMU) to be enabled in the BIOS.</p> </li> <li> <p>To enable IOMMU, depending on the CPU type, a host should be booted with an additional kernel parameter, <code>intel_iommu=on</code> for Intel and <code>amd_iommu=on</code> for AMD.</p> </li> </ul> <p>Append these parameters to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.</p> <pre><code># vi /etc/default/grub\n...\nGRUB_CMDLINE_LINUX=\"nofb splash=quiet console=tty0 ... intel_iommu=on\n...\n\n# grub2-mkconfig -o /boot/grub2/grub.cfg\n\n# reboot\n</code></pre> <ul> <li>The vfio-pci kernel module should be enabled on the host. <pre><code># modprobe vfio-pci\n</code></pre></li> </ul>"},{"location":"compute/host-devices/#preparation-of-pci-devices-for-passthrough","title":"Preparation of PCI devices for passthrough","text":"<p>At this time, KubeVirt is only able to assign PCI devices that are using the <code>vfio-pci</code> driver. To prepare a specific device for device assignment, it should first be unbound from its original driver and bound to the <code>vfio-pci</code> driver.</p> <ul> <li>Find the PCI address of the desired device:</li> </ul> <pre><code>$ lspci -DD|grep NVIDIA\n0000.65:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)\n</code></pre> <ul> <li>Bind that device to the <code>vfio-pci</code> driver: <pre><code>echo 0000:65:00.0 &gt; /sys/bus/pci/drivers/nvidia/unbind\necho \"vfio-pci\" &gt; /sys/bus/pci/devices/0000\\:65\\:00.0/driver_override\necho 0000:65:00.0 &gt; /sys/bus/pci/drivers/vfio-pci/bind\n</code></pre></li> </ul>"},{"location":"compute/host-devices/#preparation-of-mediated-devices-such-as-vgpu","title":"Preparation of mediated devices such as vGPU","text":"<p>In general, configuration of a Mediated devices (mdevs), such as vGPUs, should be done according to the vendor directions.  KubeVirt can now facilitate the creation of the mediated devices / vGPUs on the cluster nodes. This assumes that the required vendor driver is already installed on the nodes. See the Mediated devices and virtual GPUs to learn more about this functionality.</p> <p>Once the mdev is configured, KubeVirt will be able to discover and use it for device assignment.</p>"},{"location":"compute/host-devices/#listing-permitted-devices","title":"Listing permitted devices","text":"<p>Administrators can control which host devices are exposed and permitted to be used in the cluster. Permitted host devices in the cluster will need to be allowlisted in KubeVirt CR by its <code>vendor:product</code> selector for PCI devices or mediated device names.</p> <pre><code>configuration:\n  permittedHostDevices:\n    pciHostDevices:\n    - pciVendorSelector: \"10DE:1EB8\"\n      resourceName: \"nvidia.com/TU104GL_Tesla_T4\"\n      externalResourceProvider: true\n    - pciVendorSelector: \"8086:6F54\"\n      resourceName: \"intel.com/qat\"\n    mediatedDevices:\n    - mdevNameSelector: \"GRID T4-1Q\"\n      resourceName: \"nvidia.com/GRID_T4-1Q\"\n</code></pre> <ul> <li> <p><code>pciVendorSelector</code> is a PCI vendor ID and product ID tuple in the form <code>vendor_id:product_id</code>.  This tuple can identify specific types of devices on a host. For example, the identifier <code>10de:1eb8</code>, shown above, can be found using <code>lspci</code>.</p> <pre><code>$ lspci -nnv|grep -i nvidia\n65:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)\n</code></pre> </li> <li> <p><code>mdevNameSelector</code> is a name of a Mediated device type that can identify specific types of Mediated devices on a host.</p> <p>You can see what mediated types a given PCI device supports by examining the contents of <code>/sys/bus/pci/devices/SLOT:BUS:DOMAIN.FUNCTION/mdev_supported_types/TYPE/name</code>. For example, if you have an NVIDIA T4 GPU on your system, and you substitute in the <code>SLOT</code>, <code>BUS</code>, <code>DOMAIN</code>, and <code>FUNCTION</code> values that are correct for your system into the above path name, you will see that a <code>TYPE</code> of <code>nvidia-226</code> contains the selector string <code>GRID T4-2A</code> in its <code>name</code> file.</p> <p>Taking <code>GRID T4-2A</code> and specifying it as the <code>mdevNameSelector</code> allows KubeVirt to find a corresponding mediated device by matching it against <code>/sys/class/mdev_bus/SLOT:BUS:DOMAIN.FUNCTION/$mdevUUID/mdev_type/name</code> for some values of <code>SLOT:BUS:DOMAIN.FUNCTION</code> and <code>$mdevUUID</code>.</p> </li> <li> <p>External providers: <code>externalResourceProvider</code> field indicates that this resource is being provided by an external device plugin. In this case, KubeVirt will only permit the usage of this device in the cluster but will leave the allocation and monitoring to an external device plugin.</p> </li> </ul>"},{"location":"compute/host-devices/#starting-a-virtual-machine","title":"Starting a Virtual Machine","text":"<p>Host devices can be assigned to virtual machines via the <code>gpus</code> and <code>hostDevices</code> fields.  The <code>deviceNames</code> can reference both PCI and Mediated device resource names.</p> <pre><code>kind: VirtualMachineInstance\nspec:\n  domain:\n    devices:\n      gpus:\n      - deviceName: nvidia.com/TU104GL_Tesla_T4\n        name: gpu1\n      - deviceName: nvidia.com/GRID_T4-1Q\n        name: gpu2\n      hostDevices:\n      - deviceName: intel.com/qat\n        name: quickaccess1\n</code></pre>"},{"location":"compute/host-devices/#nvme-pci-passthrough","title":"NVMe PCI passthrough","text":"<p>In order to passthrough an NVMe device the procedure is very similar to the gpu case. The device needs to be listed under the <code>permittedHostDevice</code> and under <code>hostDevices</code> in the VM declaration. </p> <p>Currently, the KubeVirt device plugin doesn't allow the user to select a specific device by specifying the address. Therefore, if multiple NVMe devices with the same vendor and product id exist in the cluster, they could be randomly assigned to a VM. If the devices are not on the same node, then the nodeSelector mitigates the issue.</p> <p>Example:</p> <p>Modify the <code>permittedHostDevice</code></p> <pre><code>configuration:\n  permittedHostDevices:\n    pciHostDevices:\n    - pciVendorSelector: 8086:5845\n      resourceName: devices.kubevirt.io/nvme\n</code></pre> <p>VMI declaration: <pre><code>kind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-nvme\n  name: vmi-nvme\nspec:\n  nodeSelector: \n    kubernetes.io/hostname: node03   # &lt;--\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      hostDevices:  # &lt;--\n      - name: nvme  # &lt;--\n        deviceName: devices.kubevirt.io/nvme  # &lt;--\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p>"},{"location":"compute/host-devices/#usb-host-passthrough","title":"USB Host Passthrough","text":"<p>Since KubeVirt v1.1, we can provide USB devices that are plugged in a Node to the VM running in the same Node.</p>"},{"location":"compute/host-devices/#requirements","title":"Requirements","text":"<p>Cluster admin privilege to edit the KubeVirt CR in order to:</p> <ul> <li>Enable the <code>HostDevices</code> feature gate</li> <li>Edit the <code>permittedHostDevices</code> configuration to expose node USB devices to the cluster</li> </ul>"},{"location":"compute/host-devices/#exposing-usb-devices","title":"Exposing USB Devices","text":"<p>In order to assign USB devices to your VMI, you'll need to expose those devices to the cluster under a resource name. The device allowlist can be edited in KubeVirt CR under <code>configuration.permittedHostDevices.usb</code>.</p> <p>For this example, we will use the <code>kubevirt.io/storage</code> resource name for the device with <code>vendor: \"46f4\"</code> and <code>product: \"0001\"</code> <sup>1</sup>.</p> <pre><code>spec:\n  configuration:\n    permittedHostDevices:\n      usb:\n        - resourceName: kubevirt.io/storage\n          selectors:\n            - vendor: \"46f4\"\n              product: \"0001\"\n</code></pre> <p>After adding the <code>usb</code> configuration under <code>permittedHostDevices</code> to the KubeVirt CR, KubeVirt's device-plugin will expose this resource name and you can use it in your VMI.</p>"},{"location":"compute/host-devices/#adding-usb-to-your-vm","title":"Adding USB to your VM","text":"<p>Now, in the VMI configuration, you can add the <code>devices.hostDevices.deviceName</code> and reference the resource name provided in the previous step, and also give it a local <code>name</code>, for example:</p> <pre><code>spec:\n  domain:\n    devices:\n      hostDevices:\n      - deviceName: kubevirt.io/storage\n        name: usb-storage\n</code></pre> <p>You can find a working example, which uses QEMU's emulated USB storage, under examples/vmi-usb.yaml.</p>"},{"location":"compute/host-devices/#bundle-of-usb-devices","title":"Bundle of USB devices","text":"<p>You might be interested to redirect more than one USB device to a VMI, for example, a keyboard, a mouse and a smartcard device. The KubeVirt CR supports assigning multiple USB devices under the same resource name, so you could do:</p> <pre><code>spec:\n  configuration:\n    permittedHostDevices:\n      usb:\n        - resourceName: kubevirt.io/peripherals\n          selectors:\n            - vendor: \"045e\"\n              product: \"07a5\"\n            - vendor: \"062a\"\n              product: \"4102\"\n            - vendor: \"072f\"\n              product: \"b100\"\n</code></pre> <p>Adding to the VMI configuration:</p> <pre><code>spec:\n  domain:\n    devices:\n      hostDevices:\n      - deviceName: kubevirt.io/peripherals\n        name: local-peripherals \n</code></pre> <p>Note that all USB devices need to be present in order for the assignment to work.</p> <ol> <li> <p>Note that you can easily find the <code>vendor:product</code> value with the <code>lsusb</code> command.\u00a0\u21a9</p> </li> </ol>"},{"location":"compute/hugepages/","title":"Hugepages support","text":"<p>For hugepages support you need at least Kubernetes version <code>1.9</code>.</p>"},{"location":"compute/hugepages/#enable-feature-gate","title":"Enable feature-gate","text":"<p>To enable hugepages on Kubernetes, check the official documentation.</p> <p>To enable hugepages on OKD, check the official documentation.</p>"},{"location":"compute/hugepages/#pre-allocate-hugepages-on-a-node","title":"Pre-allocate hugepages on a node","text":"<p>To pre-allocate hugepages on boot time, you will need to specify hugepages under kernel boot parameters <code>hugepagesz=2M hugepages=64</code> and restart your machine.</p> <p>You can find more about hugepages under official documentation.</p>"},{"location":"compute/live_migration/","title":"Live Migration","text":"<p>Live migration is a process during which a running Virtual Machine Instance moves to another compute node while the guest workload continues to run and remain accessible.</p>"},{"location":"compute/live_migration/#enabling-the-live-migration-support","title":"Enabling the live-migration support","text":"<p>Live migration is enabled by default in recent versions of KubeVirt. Versions prior to v0.56, it must be enabled in the feature gates. The feature gates field in the KubeVirt CR must be expanded by adding the <code>LiveMigration</code> to it.</p>"},{"location":"compute/live_migration/#limitations","title":"Limitations","text":"<ul> <li> <p>Virtual machines using a PersistentVolumeClaim (PVC) must have a   shared ReadWriteMany (RWX) access mode to be live migrated.</p> </li> <li> <p>Live migration is not allowed with a pod network binding of bridge   interface type   ()</p> </li> <li> <p>Live migration requires ports <code>49152, 49153</code> to be available in the virt-launcher pod.   If these ports are explicitly specified in masquarade interface, live migration will not function.</p> </li> </ul>"},{"location":"compute/live_migration/#initiate-live-migration","title":"Initiate live migration","text":"<p>Live migration is initiated by posting a VirtualMachineInstanceMigration (VMIM) object to the cluster. The example below starts a migration process for a virtual machine instance <code>vmi-fedora</code></p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\n</code></pre>"},{"location":"compute/live_migration/#using-virtctl-to-initiate-live-migration","title":"Using virtctl to initiate live migration","text":"<p>Live migration can also be initiated using virtctl <pre><code>    virtctl migrate vmi-fedora\n</code></pre></p>"},{"location":"compute/live_migration/#migration-status-reporting","title":"Migration Status Reporting","text":""},{"location":"compute/live_migration/#condition-and-migration-method","title":"Condition and migration method","text":"<p>When starting a virtual machine instance, it has also been calculated whether the machine is live migratable. The result is being stored in the VMI <code>VMI.status.conditions</code>. The calculation can be based on multiple parameters of the VMI, however, at the moment, the calculation is largely based on the <code>Access Mode</code> of the VMI volumes. Live migration is only permitted when the volume access mode is set to <code>ReadWriteMany</code>. Requests to migrate a non-LiveMigratable VMI will be rejected.</p> <p>The reported <code>Migration Method</code> is also being calculated during VMI start. <code>BlockMigration</code> indicates that some of the VMI disks require copying from the source to the destination. <code>LiveMigration</code> means that only the instance memory will be copied.</p> <pre><code>Status:\n  Conditions:\n    Status: True\n    Type: LiveMigratable\n  Migration Method: BlockMigration\n</code></pre>"},{"location":"compute/live_migration/#migration-status","title":"Migration Status","text":"<p>The migration progress status is being reported in the VMI <code>VMI.status</code>. Most importantly, it indicates whether the migration has been <code>Completed</code> or if it <code>Failed</code>.</p> <p>Below is an example of a successful migration.</p> <pre><code>Migration State:\n    Completed:        true\n    End Timestamp:    2019-03-29T03:37:52Z\n    Migration Config:\n      Completion Timeout Per GiB:  800\n      Progress Timeout:             150\n    Migration UID:                  c64d4898-51d3-11e9-b370-525500d15501\n    Source Node:                    node02\n    Start Timestamp:                2019-03-29T04:02:47Z\n    Target Direct Migration Node Ports:\n      35001:                      0\n      41068:                      49152\n      38284:                      49153\n    Target Node:                  node01\n    Target Node Address:          10.128.0.46\n    Target Node Domain Detected:  true\n    Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq\n</code></pre>"},{"location":"compute/live_migration/#canceling-a-live-migration","title":"Canceling a live migration","text":"<p>Live migration can also be canceled by simply deleting the migration object. A successfully aborted migration will indicate that the abort has been requested <code>Abort Requested</code>, and that it succeeded: <code>Abort Status: Succeeded</code>. The migration in this case will be <code>Completed</code> and <code>Failed</code>.</p> <pre><code>Migration State:\n    Abort Requested:  true\n    Abort Status:     Succeeded\n    Completed:        true\n    End Timestamp:    2019-03-29T04:02:49Z\n    Failed:           true\n    Migration Config:\n      Completion Timeout Per GiB:  800\n      Progress Timeout:             150\n    Migration UID:                  57a693d6-51d7-11e9-b370-525500d15501\n    Source Node:                    node02\n    Start Timestamp:                2019-03-29T04:02:47Z\n    Target Direct Migration Node Ports:\n      39445:                      0\n      43345:                      49152\n      44222:                      49153\n    Target Node:                  node01\n    Target Node Address:          10.128.0.46\n    Target Node Domain Detected:  true\n    Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq\n</code></pre>"},{"location":"compute/live_migration/#using-virtctl-to-cancel-a-live-migration","title":"Using virtctl to cancel a live migration","text":"<p>Live migration can also be canceled using virtctl, by specifying the name of a VMI which is currently being migrated <pre><code>    virtctl migrate-cancel vmi-fedora\n</code></pre></p>"},{"location":"compute/live_migration/#changing-cluster-wide-migration-limits","title":"Changing Cluster Wide Migration Limits","text":"<p>KubeVirt puts some limits in place, so that migrations don't overwhelm the cluster. By default, it is configured to only run <code>5</code> migrations in parallel with an additional limit of a maximum of <code>2</code> outbound migrations per node. Finally, every migration is limited to a bandwidth of <code>64MiB/s</code>.</p> <p>These values can be changed in the <code>kubevirt</code> CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    migrations:\n      parallelMigrationsPerCluster: 5\n      parallelOutboundMigrationsPerNode: 2\n      bandwidthPerMigration: 64Mi\n      completionTimeoutPerGiB: 800\n      progressTimeout: 150\n      disableTLS: false\n      nodeDrainTaintKey: \"kubevirt.io/drain\"\n      allowAutoConverge: false\n      allowPostCopy: false\n      unsafeMigrationOverride: false\n</code></pre> <p>Bear in mind that most of these configuration can be overridden and fine-tuned to a specified group of VMs. For more information, please see Migration Policies.</p>"},{"location":"compute/live_migration/#understanding-different-migration-strategies","title":"Understanding different migration strategies","text":"<p>Live migration is a complex process. During a migration, the source VM needs to transfer its whole state (mainly RAM) to the target VM. If there are enough resources available, such as network bandwidth and CPU power, migrations should converge nicely. If this is not the scenario, however, the migration might get stuck without an ability to progress.</p> <p>The main factor that affects migrations from the guest perspective is its <code>dirty rate</code>, which is the rate by which the VM dirties memory. Guests with high dirty rate lead to a race during migration. On the one hand, memory would be transferred continuously to the target, and on the other, the same memory would get dirty by the guest. On such scenarios, one could consider to use more advanced migration strategies.</p> <p>Let's explain the 3 supported migration strategies as of today.</p>"},{"location":"compute/live_migration/#pre-copy","title":"Pre-copy","text":"<p>Pre-copy is the default strategy. It should be used for most cases.</p> <p>The way it works is as following:</p> <ol> <li>The target VM is created, but the guest keeps running on the source VM.</li> <li>The source starts sending chunks of VM state (mostly memory) to the target. This continues until   all of the state has been transferred to the target.</li> <li>The guest starts executing on the target VM.</li> <li>The source VM is being removed.</li> </ol> <p>Pre-copy is the safest and fastest strategy for most cases. Furthermore, it can be easily cancelled, can utilize multithreading, and more. If there is no real reason to use another strategy, this is definitely the strategy to go with.</p> <p>However, on some cases migrations might not converge easily, that is, by the time the chunk of source VM state would be received by the target VM, it would already be mutated by the source VM (which is the VM the guest executes on). There are many reasons for migrations to fail converging, such as a high dirty-rate or low resources like network bandwidth and CPU. On such scenarios, see the following alternative strategies below.</p>"},{"location":"compute/live_migration/#post-copy","title":"Post-copy","text":"<p>The way post-copy migrations work is as following:</p> <ol> <li>The target VM is created.</li> <li>The guest is being run on the target VM.</li> <li>The source starts sending chunks of VM state (mostly memory) to the target.</li> <li>When the guest, running on the target VM, would access memory:<ol> <li>If the memory exists on the target VM, the guest can access it.</li> <li>Otherwise, the target VM asks for a chunk of memory from the source VM.</li> </ol> </li> <li>Once all of the memory state is updated at the target VM, the source VM is being removed.</li> </ol> <p>The main idea here is that the guest starts to run immediately on the target VM. This approach has advantages and disadvantages:</p> <p>advantages:</p> <ul> <li>The same memory chunk is never being transferred twice. This is possible due to the fact that with post-copy it doesn't matter that a page had been dirtied since the guest is already running on the target VM.</li> <li>This means that a high dirty-rate has much less effect.</li> <li>Consumes less network bandwidth.</li> </ul> <p>disadvantages:</p> <ul> <li>When using post-copy, the VM state has no one source of truth. When the guest (running on the target VM) writes to memory, this memory is one part of the guest's state, but some other parts of it may still be updated only at the source VM. This situation is generally dangerous, since, for  example, if either the target or guest VMs crash the state cannot be recovered.</li> <li>Slow warmup: when the guest starts executing, no memory is present at the target VM. Therefore, the guest would have to wait for a lot of memory in a short period of time.</li> <li>Slower than pre-copy on most cases.</li> <li>Harder to cancel a migration.</li> </ul>"},{"location":"compute/live_migration/#auto-converge","title":"Auto-converge","text":"<p>Auto-converge is a technique to help pre-copy migrations converge faster without changing the core algorithm of how the migration works.</p> <p>Since a high dirty-rate is usually the most significant factor for migrations to not converge, auto-converge simply throttles the guest's CPU. If the migration would converge fast enough, the guest's CPU would not be throttled or throttled negligibly. But, if the migration would not converge fast enough, the CPU would be throttled more and more as time goes.</p> <p>This technique dramatically increases the probability of the migration converging eventually.</p>"},{"location":"compute/live_migration/#using-a-different-network-for-migrations","title":"Using a different network for migrations","text":"<p>Live migrations can be configured to happen on a different network than the one Kubernetes is configured to use. That potentially allows for more determinism, control and/or bandwidth, depending on use-cases.</p>"},{"location":"compute/live_migration/#creating-a-migration-network-on-a-cluster","title":"Creating a migration network on a cluster","text":"<p>A separate physical network is required, meaning that every node on the cluster has to have at least 2 NICs, and the NICs that will be used for migrations need to be interconnected, i.e. all plugged to the same switch. The examples below assume that <code>eth1</code> will be used for migrations.</p> <p>It is also required for the Kubernetes cluster to have multus installed.</p> <p>If the desired network doesn't include a DHCP server, then whereabouts will be needed as well.</p> <p>Finally, a NetworkAttachmentDefinition needs to be created in the namespace where KubeVirt is installed. Here is an example: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: migration-network\n  namespace: kubevirt\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"migration-bridge\",\n      \"type\": \"macvlan\",\n      \"master\": \"eth1\",\n      \"mode\": \"bridge\",\n      \"ipam\": {\n        \"type\": \"whereabouts\",\n        \"range\": \"10.1.1.0/24\"\n      }\n    }'\n</code></pre></p>"},{"location":"compute/live_migration/#configuring-kubevirt-to-migrate-vmis-over-that-network","title":"Configuring KubeVirt to migrate VMIs over that network","text":"<p>This is just a matter of adding the name of the NetworkAttachmentDefinition to the KubeVirt CR, like so: <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n      - LiveMigration\n    migrations:\n      network: migration-network\n</code></pre></p> <p>That change will trigger a restart of the virt-handler pods, as they get connected to that new network.</p> <p>From now on, migrations will happen over that network.</p>"},{"location":"compute/live_migration/#configuring-kubevirtci-for-testing-migration-networks","title":"Configuring KubeVirtCI for testing migration networks","text":"<p>Developers and people wanting to test the feature before deploying it on a real cluster might want to configure a dedicated migration network in KubeVirtCI.</p> <p>KubeVirtCI can simply be configured to include a virtual secondary network, as well as automatically install multus and whereabouts. The following environment variables just have to be declared before running <code>make cluster-up</code>: <pre><code>export KUBEVIRT_NUM_NODES=2;\nexport KUBEVIRT_NUM_SECONDARY_NICS=1;\nexport KUBEVIRT_DEPLOY_ISTIO=true;\nexport KUBEVIRT_WITH_CNAO=true\n</code></pre></p>"},{"location":"compute/live_migration/#migration-timeouts","title":"Migration timeouts","text":"<p>Depending on the type, the live migration process will copy virtual machine memory pages and disk blocks to the destination. During this process non-locked pages and blocks are being copied and become free for the instance to use again. To achieve a successful migration, it is assumed that the instance will write to the free pages and blocks (pollute the pages) at a lower rate than these are being copied.</p>"},{"location":"compute/live_migration/#completion-time","title":"Completion time","text":"<p>In some cases the virtual machine can write to different memory pages / disk blocks at a higher rate than these can be copied, which will prevent the migration process from completing in a reasonable amount of time. In this case, live migration will be aborted if it is running for a long period of time. The timeout is calculated base on the size of the VMI, it's memory and the ephemeral disks that are needed to be copied. The configurable parameter <code>completionTimeoutPerGiB</code>, which defaults to 800s is the time for GiB of data to wait for the migration to be completed before aborting it. A VMI with 8Gib of memory will time out after 6400 seconds.</p>"},{"location":"compute/live_migration/#progress-timeout","title":"Progress timeout","text":"<p>Live migration will also be aborted when it will be noticed that copying memory doesn't make any progress. The time to wait for live migration to make progress in transferring data is configurable by <code>progressTimeout</code> parameter, which defaults to 150s</p>"},{"location":"compute/live_migration/#disabling-secure-migrations","title":"Disabling secure migrations","text":"<p>FEATURE STATE: KubeVirt v0.43</p> <p>Sometimes it may be desirable to disable TLS encryption of migrations to improve performance. Use <code>disableTLS</code> to do that:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"LiveMigration\"\n    migrationConfiguration:\n      disableTLS: true\n</code></pre> <p>Note: While this increases performance it may allow MITM attacks. Be careful.</p>"},{"location":"compute/mediated_devices_configuration/","title":"Mediated devices and virtual GPUs","text":""},{"location":"compute/mediated_devices_configuration/#configuring-mediated-devices-and-virtual-gpus","title":"Configuring mediated devices and virtual GPUs","text":"<p>KubeVirt aims to facilitate the configuration of mediated devices on large clusters. Administrators can use the <code>mediatedDevicesConfiguration</code> API in the KubeVirt CR to create or remove mediated devices in a declarative way, by providing a list of the desired mediated device types that they expect to be configured in the cluster.</p> <p>You can also include the <code>nodeMediatedDeviceTypes</code> option to provide a more specific configuration that targets a specific node or a group of nodes directly with a node selector. The <code>nodeMediatedDeviceTypes</code> option must be used in combination with <code>mediatedDevicesTypes</code> in order to override the global configuration set in the <code>mediatedDevicesTypes</code> section.</p> <p>KubeVirt will use the provided configuration to automatically create the relevant mdev/vGPU devices on nodes that can support it.</p> <p>Currently, a single mdev type per card will be configured. The maximum amount of instances of the selected mdev type will be configured per card.</p> <p>Note: Some vendors, such as NVIDIA, require a driver to be installed on the nodes to provide mediated devices, including vGPUs.</p> <p>Example snippet of a KubeVirt CR configuration that includes both <code>nodeMediatedDeviceTypes</code> and <code>mediatedDevicesTypes</code>: <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-222\n      - nvidia-228\n      nodeMediatedDeviceTypes:\n      - nodeSelector:\n          kubernetes.io/hostname: nodeName\n        mediatedDevicesTypes:\n        - nvidia-234\n</code></pre></p>"},{"location":"compute/mediated_devices_configuration/#configuration-scenarios","title":"Configuration scenarios","text":""},{"location":"compute/mediated_devices_configuration/#example-large-cluster-with-multiple-cards-on-each-node","title":"Example: Large cluster with multiple cards on each node","text":"<p>On nodes with multiple cards that can support similar vGPU types, the relevant desired types will be created in a round-robin manner.</p> <p>For example, considering the following KubeVirt CR configuration:</p> <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-222\n      - nvidia-228\n      - nvidia-105\n      - nvidia-108\n</code></pre> <p>This cluster has nodes with two different PCIe cards:</p> <ol> <li> <p>Nodes with 3 Tesla T4 cards, where each card can support multiple devices types:</p> <ul> <li>nvidia-222</li> <li>nvidia-223</li> <li>nvidia-228</li> <li>...</li> </ul> </li> <li> <p>Nodes with 2 Tesla V100 cards, where each card can support multiple device types:</p> <ul> <li>nvidia-105</li> <li>nvidia-108</li> <li>nvidia-217</li> <li>nvidia-299</li> <li>...</li> </ul> </li> </ol> <p>KubeVirt will then create the following devices:</p> <ol> <li>Nodes with 3 Tesla T4 cards will be configured with:<ul> <li>16 vGPUs of type nvidia-222 on card 1</li> <li>2 vGPUs of type nvidia-228 on card 2</li> <li>16 vGPUs of type nvidia-222 on card 3</li> </ul> </li> <li>Nodes with 2 Tesla V100 cards will be configured with:<ul> <li>16 vGPUs of type nvidia-105 on card 1</li> <li>2 vGPUs of type nvidia-108 on card 2</li> </ul> </li> </ol>"},{"location":"compute/mediated_devices_configuration/#example-single-card-on-a-node-multiple-desired-vgpu-types-are-supported","title":"Example: Single card on a node, multiple desired vGPU types are supported","text":"<p>When nodes only have a single card, the first supported type from the list will be configured.</p> <p>For example, consider the following list of desired types, where nvidia-223 and nvidia-224 are supported:</p> <p><pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-223\n      - nvidia-224\n</code></pre> In this case, nvidia-223 will be configured on the node because it is the first supported type in the list.</p>"},{"location":"compute/mediated_devices_configuration/#overriding-configuration-on-a-specifc-node","title":"Overriding configuration on a specifc node","text":"<p>To override the global configuration set by <code>mediatedDevicesTypes</code>, include the <code>nodeMediatedDeviceTypes</code> option, specifying the node selector and the <code>mediatedDevicesTypes</code> that you want to override for that node.</p>"},{"location":"compute/mediated_devices_configuration/#example-overriding-the-configuration-for-a-specific-node-in-a-large-cluster-with-multiple-cards-on-each-node","title":"Example: Overriding the configuration for a specific node in a large cluster with multiple cards on each node","text":"<p>In this example, the KubeVirt CR includes the <code>nodeMediatedDeviceTypes</code> option to override the global configuration specifically for node 2, which will only use the nvidia-234 type.</p> <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-230\n      - nvidia-223\n      - nvidia-224\n    nodeMediatedDeviceTypes:\n    - nodeSelector:\n        kubernetes.io/hostname: node2  \n      mediatedDevicesTypes:\n      - nvidia-234\n</code></pre> <p>The cluster has two nodes that both have 3 Tesla T4 cards.</p> <ul> <li>Each card can support a long list of types, including:<ul> <li>nvidia-222</li> <li>nvidia-223</li> <li>nvidia-224</li> <li>nvidia-230</li> <li>...</li> </ul> </li> </ul> <p>KubeVirt will then create the following devices:</p> <ol> <li>Node 1<ul> <li>type nvidia-230 on card 1</li> <li>type nvidia-223 on card 2</li> </ul> </li> <li>Node 2<ul> <li>type nvidia-234 on card 1 and card 2</li> </ul> </li> </ol> <p>Node 1 has been configured in a round-robin manner based on the global configuration but node 2 only uses the nvidia-234 that was specified for it.</p>"},{"location":"compute/mediated_devices_configuration/#updating-and-removing-vgpu-types","title":"Updating and Removing vGPU types","text":"<p>Changes made to the <code>mediatedDevicesTypes</code> section of the KubeVirt CR will trigger a re-evaluation of the configured mdevs/vGPU types on the cluster nodes.</p> <p>Any change to the node labels that match the <code>nodeMediatedDeviceTypes</code> nodeSelector in the KubeVirt CR will trigger a similar re-evaluation.</p> <p>Consequently, mediated devices will be reconfigured or entirely removed based on the updated configuration.</p>"},{"location":"compute/mediated_devices_configuration/#assigning-vgpumdev-to-a-virtual-machine","title":"Assigning vGPU/MDEV to a Virtual Machine","text":"<p>See the Host Devices Assignment to learn how to consume the newly created mediated devices/vGPUs.</p>"},{"location":"compute/memory_dump/","title":"Virtual machine memory dump","text":"<p>Kubevirt now supports getting a VM memory dump for analysis purposes. The Memory dump can be used to diagnose, identify and resolve issues in the VM. Typically providing information about the last state of the programs, applications and system before they were terminated or crashed.</p> <p>Note This memory dump is not used for saving VM state and resuming it later.</p>"},{"location":"compute/memory_dump/#prerequisites","title":"Prerequisites","text":""},{"location":"compute/memory_dump/#hot-plug-feature-gate","title":"Hot plug Feature Gate","text":"<p>The memory dump process mounts a PVC to the virt-launcher in order to get the output in that PVC, hence the hot plug volumes feature gate must be enabled. The feature gates field in the KubeVirt CR must be expanded by adding the <code>HotplugVolumes</code> to it.</p>"},{"location":"compute/memory_dump/#virtctl-support","title":"Virtctl support","text":""},{"location":"compute/memory_dump/#get-memory-dump","title":"Get memory dump","text":"<p>Now lets assume we have a running VM and the name of the VM is 'my-vm'. We can either dump to an existing pvc, or request one to be created.</p>"},{"location":"compute/memory_dump/#existing-pvc","title":"Existing PVC","text":"<p>The size of the PVC must be big enough to hold the memory dump. The calculation is (VMMemorySize + 100Mi) * FileSystemOverhead, Where <code>VMMemorySize</code> is the memory size, 100Mi is reserved space for the memory dump overhead and <code>FileSystemOverhead</code> is the value used to adjust requested PVC size with the filesystem overhead. also the PVC must have a <code>FileSystem</code> volume mode.</p> <p>Example for such PVC:</p> <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: my-pvc\nspec:\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 2Gi\n  storageClassName: rook-ceph-block\n  volumeMode: Filesystem\n</code></pre> <p>We can get a memory dump of the VM to the PVC by using the 'memory-dump get' command available with virtctl</p> <pre><code>$ virtctl memory-dump get my-vm --claim-name=my-pvc\n</code></pre>"},{"location":"compute/memory_dump/#on-demand-pvc","title":"On demand PVC","text":"<p>For on demand PVC, we need to add <code>--create-claim</code> flag to the virtctl request:</p> <pre><code>$ virtctl memory-dump get my-vm --claim-name=new-pvc --create-claim\n</code></pre> <p>A PVC with size big enough for the dump will be created. We can also request specific storage class and access mode with appropriate flags.</p>"},{"location":"compute/memory_dump/#download-memory-dump","title":"Download memory dump","text":"<p>By adding the <code>--output</code> flag, the memory will be dumped to the PVC and then downloaded to the given output path.</p> <pre><code>$ virtctl memory-dump get myvm --claim-name=memoryvolume --create-claim --output=memoryDump.dump.gz\n</code></pre> <p>For downloading the last memory dump from the PVC associated with the VM, without triggering another memory dump, use the memory dump download command.</p> <pre><code>$ virtctl memory-dump download myvm --output=memoryDump.dump.gz\n</code></pre> <p>For downloading a memory dump from a PVC already disassociated from the VM you can use the virtctl vmexport command</p>"},{"location":"compute/memory_dump/#monitoring-the-memory-dump","title":"Monitoring the memory dump","text":"<p>Information regarding the memory dump process will be available on the VM's status section <pre><code>memoryDumpRequest:\n  claimName: memory-dump\n  phase: Completed\n  startTimestamp: \"2022-03-29T11:00:04Z\"\n  endTimestamp: \"2022-03-29T11:00:09Z\"\n  fileName: my-vm-my-pvc-20220329-110004\n</code></pre></p> <p>During the process the volumeStatus on the VMI will be updated with the process information such as the attachment pod information and messages, if all goes well once the process is completed, the PVC is unmounted from the virt-launcher pod and the volumeStatus is deleted. A memory dump annotation will be added to the PVC with the memory dump file name.</p>"},{"location":"compute/memory_dump/#retriggering-the-memory-dump","title":"Retriggering the memory dump","text":"<p>Getting a new memory dump to the same PVC is possible without the need to use any flag: <pre><code>$ virtctl memory-dump get my-vm\n</code></pre></p> <p>Note Each memory-dump command will delete the previous dump in that PVC.</p> <p>In order to get a memory dump to a different PVC you need to 'remove' the current memory-dump PVC and then do a new get with the new PVC name.</p>"},{"location":"compute/memory_dump/#remove-memory-dump","title":"Remove memory dump","text":"<p>As mentioned in order to remove the associated memory dump PVC you need to run a 'memory-dump remove' command. This will allow you to replace the current PVC and get the memory dump to a new one.</p> <pre><code>$ virtctl memory-dump remove my-vm\n</code></pre>"},{"location":"compute/memory_dump/#handle-the-memory-dump","title":"Handle the memory dump","text":"<p>Once the memory dump process is completed the PVC will hold the output. You can manage the dump in one of the following ways: - Download the memory dump - Create a pod with troubleshooting tools that will mount the PVC and inspect it within the pod. - Include the memory dump in the VM Snapshot (will include both the memory dump and the disks) to save a snapshot of the VM in that point of time and inspect it when needed. (The VM Snapshot can be exported and downloaded).</p> <p>The output of the memory dump can be inspected with memory analysis tools for example Volatility3</p>"},{"location":"compute/memory_hotplug/","title":"Memory Hotplug","text":"<p>Memory hotplug was introduced in KubeVirt version 1.1, enabling the dynamic resizing of the amount of memory available to a running VM.</p>"},{"location":"compute/memory_hotplug/#limitations","title":"Limitations","text":"<ul> <li>Memory hotplug is currently only supported on the x86_64 architecture.</li> <li>Current hotplug implementation involves live-migration of the VM workload.</li> </ul>"},{"location":"compute/memory_hotplug/#configuration","title":"Configuration","text":""},{"location":"compute/memory_hotplug/#enable-feature-gate","title":"Enable feature-gate","text":"<p>To use memory hotplug we need to add the <code>VMLiveUpdateFeatures</code> feature gate in the KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"compute/memory_hotplug/#configure-the-workload-update-strategy","title":"Configure the Workload Update Strategy","text":"<p>Configure <code>LiveMigrate</code> as <code>workloadUpdateStrategy</code> in the KubeVirt CR, since the current implementation of the hotplug process requires the VM to live-migrate.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre>"},{"location":"compute/memory_hotplug/#configure-the-vm-rollout-strategy","title":"Configure the VM rollout strategy","text":"<p>Finally, set the VM rollout strategy to <code>LiveUpdate</code>, so that the changes made to the VM object propagate to the VMI without a restart. This is also done in the KubeVirt CR configuration:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre> <p>NOTE: If memory hotplug is enabled/disabled on an already running VM, a reboot is necessary for the changes to take effect.</p> <p>More information can be found on the VM Rollout Strategies page.</p>"},{"location":"compute/memory_hotplug/#optional-set-a-cluster-wide-maximum-amount-of-memory","title":"[OPTIONAL] Set a cluster-wide maximum amount of memory","text":"<p>You can set the maximum amount of memory for the guest using a cluster level setting in the KubeVirt CR.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxGuest: 8Gi\n</code></pre> <p>The VM-level configuration will take precedence over the cluster-wide one.</p>"},{"location":"compute/memory_hotplug/#memory-hotplug-in-action","title":"Memory Hotplug in Action","text":"<p>First we enable the <code>VMLiveUpdateFeatures</code> feature gate, set the rollout strategy to <code>LiveUpdate</code> and set <code>LiveMigrate</code> as <code>workloadUpdateStrategy</code> in the KubeVirt CR.</p> <pre><code>$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates\", \"value\": [\"VMLiveUpdateFeatures\"]}]' --type='json'\n$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/vmRolloutStrategy\", \"value\": \"LiveUpdate\"}]' --type='json'\n$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\": [\"LiveMigrate\"]}]' --type='json'\n</code></pre> <p>Now we create a VM with memory hotplug enabled.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-cirros\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        memory:\n          maxGuest: 2Gi\n          guest: 128Mi\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/alpine-container-disk-demo:devel\n        name: containerdisk\n</code></pre> <p>The Virtual Machine will automatically start and once booted it will report the currently available memory to the guest in the <code>status.memory</code> field inside the VMI.</p> <p><pre><code>$ kubectl get vmi vm-cirros -o json | jq .status.memory\n</code></pre> <pre><code>{\n  \"guestAtBoot\": \"128Mi\",\n  \"guestCurrent\": \"128Mi\",\n  \"guestRequested\": \"128Mi\"\n}\n</code></pre></p> <p>Since the Virtual Machine is now running we can patch the VM object to double the available guest memory so that we'll go from 128Mi to 256Mi.</p> <pre><code>$ kubectl patch vm vm-cirros -p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/domain/memory/guest\", \"value\": \"256Mi\"}]' --type='json'\n</code></pre> <p>After the hotplug request is processed and the Virtual Machine is live migrated, the new amount of memory should be available to the guest and visible in the VMI object.</p> <p><pre><code>$ kubectl get vmi vm-cirros -o json | jq .status.memory\n</code></pre> <pre><code>{\n  \"guestAtBoot\": \"128Mi\",\n  \"guestCurrent\": \"256Mi\",\n  \"guestRequested\": \"256Mi\"\n}\n</code></pre></p>"},{"location":"compute/node_assignment/","title":"Node assignment","text":"<p>You can constrain the VM to only run on specific nodes or to prefer running on specific nodes:</p> <ul> <li>nodeSelector</li> <li>Affinity and anti-affinity</li> <li>Taints and Tolerations</li> </ul>"},{"location":"compute/node_assignment/#nodeselector","title":"nodeSelector","text":"<p>Setting <code>spec.nodeSelector</code> requirements, constrains the scheduler to only schedule VMs on nodes, which contain the specified labels. In the following example the vmi contains the labels <code>cpu: slow</code> and <code>storage: fast</code>:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>Thus the scheduler will only schedule the vmi to nodes which contain these labels in their metadata. It works exactly like the Pods <code>nodeSelector</code>. See the Pod nodeSelector Documentation for more examples.</p>"},{"location":"compute/node_assignment/#affinity-and-anti-affinity","title":"Affinity and anti-affinity","text":"<p>The <code>spec.affinity</code> field allows specifying hard- and soft-affinity for VMs. It is possible to write matching rules against workloads (VMs and Pods) and Nodes. Since VMs are a workload type based on Pods, Pod-affinity affects VMs as well.</p> <p>An example for <code>podAffinity</code> and <code>podAntiAffinity</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  affinity:\n    podAffinity:\n      requiredDuringSchedulingIgnoredDuringExecution:\n      - labelSelector:\n          matchExpressions:\n          - key: security\n            operator: In\n            values:\n            - S1\n        topologyKey: failure-domain.beta.kubernetes.io/zone\n    podAntiAffinity:\n      preferredDuringSchedulingIgnoredDuringExecution:\n      - weight: 100\n        podAffinityTerm:\n          labelSelector:\n            matchExpressions:\n            - key: security\n              operator: In\n              values:\n              - S2\n          topologyKey: kubernetes.io/hostname\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>Affinity and anti-affinity works exactly like the Pods <code>affinity</code>. This includes <code>podAffinity</code>, <code>podAntiAffinity</code>, <code>nodeAffinity</code> and <code>nodeAntiAffinity</code>. See the Pod affinity and anti-affinity Documentation for more examples and details.</p>"},{"location":"compute/node_assignment/#taints-and-tolerations","title":"Taints and Tolerations","text":"<p>Affinity as described above, is a property of VMs that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite - they allow a node to repel a set of VMs.</p> <p>Taints and tolerations work together to ensure that VMs are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any VMs that do not tolerate the taints. Tolerations are applied to VMs, and allow (but do not require) the VMs to schedule onto nodes with matching taints.</p> <p>You add a taint to a node using kubectl taint. For example,</p> <pre><code>kubectl taint nodes node1 key=value:NoSchedule\n</code></pre> <p>An example for <code>tolerations</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  tolerations:\n  - key: \"key\"\n    operator: \"Equal\"\n    value: \"value\"\n    effect: \"NoSchedule\"\n</code></pre>"},{"location":"compute/node_assignment/#node-balancing-with-descheduler","title":"Node balancing with Descheduler","text":"<p>In some cases we might need to rebalance the cluster on current scheduling policy and load conditions. Descheduler can find pods, which violates e.g. scheduling decisions and evict them based on descheduler policies. Kubevirt VMs are handled as pods with local storage, so by default, descheduler will not evict them. But it can be easily overridden by adding special annotation to the VMI template in the VM:</p> <pre><code>spec:\n  template:\n    metadata:\n      annotations:\n        descheduler.alpha.kubernetes.io/evict: true\n</code></pre> <p>This annotation will cause, that the descheduler will be able to evict the VM's pod which can then be scheduled by scheduler on different nodes. A VirtualMachine will never restart or re-create a VirtualMachineInstance until the current instance of the VirtualMachineInstance is deleted from the cluster.</p>"},{"location":"compute/node_assignment/#live-update","title":"Live update","text":"<p>When the VM rollout strategy is set to <code>LiveUpdate</code>, changes to a VM's node selector or affinities will dynamically propagate to the VMI (unless the <code>RestartRequired</code> condition is set). Changes to tolerations will not dynamically propagate, and will trigger a <code>RestartRequired</code> condition if changed on a running VM.</p> <p>Modifications of the node selector / affinities will only take effect on next migration, the change alone will not trigger one.</p>"},{"location":"compute/node_overcommit/","title":"Node overcommit","text":"<p>KubeVirt does not yet support classical Memory Overcommit Management or Memory Ballooning. In other words VirtualMachineInstances can't give back memory they have allocated. However, a few other things can be tweaked to reduce the memory footprint and overcommit the per-VMI memory overhead.</p>"},{"location":"compute/node_overcommit/#remove-the-graphical-devices","title":"Remove the Graphical Devices","text":"<p>First the safest option to reduce the memory footprint, is removing the graphical device from the VMI by setting <code>spec.domain.devices.autottachGraphicsDevice</code> to <code>false</code>. See the video and graphics device documentation for further details and examples.</p> <p>This will save a constant amount of <code>16MB</code> per VirtualMachineInstance but also disable VNC access.</p>"},{"location":"compute/node_overcommit/#overcommit-the-guest-overhead","title":"Overcommit the Guest Overhead","text":"<p>Before you continue, make sure you make yourself comfortable with the Out of Resource Management of Kubernetes.</p> <p>Every VirtualMachineInstance requests slightly more memory from Kubernetes than what was requested by the user for the Operating System. The additional memory is used for the per-VMI overhead consisting of our infrastructure which is wrapping the actual VirtualMachineInstance process.</p> <p>In order to increase the VMI density on the node, it is possible to not request the additional overhead by setting <code>spec.domain.resources.overcommitGuestOverhead</code> to <code>true</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      overcommitGuestOverhead: true\n      requests:\n        memory: 1024M\n[...]\n</code></pre> <p>This will work fine for as long as most of the VirtualMachineInstances will not request the whole memory. That is especially the case if you have short-lived VMIs. But if you have long-lived VirtualMachineInstances or do extremely memory intensive tasks inside the VirtualMachineInstance, your VMIs will use all memory they are granted sooner or later.</p>"},{"location":"compute/node_overcommit/#overcommit-guest-memory","title":"Overcommit Guest Memory","text":"<p>The third option is real memory overcommit on the VMI. In this scenario the VMI is explicitly told that it has more memory available than what is requested from the cluster by setting <code>spec.domain.memory.guest</code> to a value higher than <code>spec.domain.resources.requests.memory</code>.</p> <p>The following definition requests <code>1024MB</code> from the cluster but tells the VMI that it has <code>2048MB</code> of memory available:</p> <pre><code>apiVersion: kubevirt.io/v1alpha3\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      overcommitGuestOverhead: true\n      requests:\n        memory: 1024M\n    memory:\n      guest: 2048M\n[...]\n</code></pre> <p>For as long as there is enough free memory available on the node, the VMI can happily consume up to <code>2048MB</code>. This VMI will get the <code>Burstable</code> resource class assigned by Kubernetes (See QoS classes in Kubernetes for more details). The same eviction rules like for Pods apply to the VMI in case the node gets under memory pressure.</p> <p>Implicit memory overcommit is disabled by default. This means that when memory request is not specified, it is set to match <code>spec.domain.memory.guest</code>. However, it can be enabled using <code>spec.configuration.developerConfiguration.memoryOvercommit</code> in the <code>kubevirt</code> CR. For example, by setting <code>memoryOvercommit: \"150\"</code> we define that when memory request is not explicitly set, it will be implicitly set to achieve memory overcommit of 150%. For instance, when <code>spec.domain.memory.guest: 3072M</code>, memory request is set to 2048M, if omitted. Note that the actual memory request depends on additional configuration options like OvercommitGuestOverhead.</p>"},{"location":"compute/node_overcommit/#configuring-the-memory-pressure-behavior-of-nodes","title":"Configuring the memory pressure behavior of nodes","text":"<p>If the node gets under memory pressure, depending on the <code>kubelet</code> configuration the virtual machines may get killed by the OOM handler or by the <code>kubelet</code> itself. It is possible to tweak that behaviour based on the requirements of your VirtualMachineInstances by:</p> <ul> <li>Configuring Soft Eviction     Thresholds</li> <li>Configuring Hard Eviction     Thresholds</li> <li>Requesting the right QoS class for VirtualMachineInstances</li> <li>Setting <code>--system-reserved</code> and <code>--kubelet-reserved</code></li> <li>Enabling KSM</li> <li>Enabling swap</li> </ul>"},{"location":"compute/node_overcommit/#configuring-soft-eviction-thresholds","title":"Configuring Soft Eviction Thresholds","text":"<p>Note: Soft Eviction will effectively shutdown VirtualMachineInstances. They are not paused, hibernated or migrated. Further, Soft Eviction is disabled by default.</p> <p>If configured, VirtualMachineInstances get evicted once the available memory falls below the threshold specified via <code>--eviction-soft</code> and the VirtualmachineInstance is given the chance to perform a shutdown of the VMI within a timespan specified via <code>--eviction-max-pod-grace-period</code>. The flag <code>--eviction-soft-grace-period</code> specifies for how long a soft eviction condition must be held before soft evictions are triggered.</p> <p>If set properly according to the demands of the VMIs, overcommitting should only lead to soft evictions in rare cases for some VMIs. They may even get re-scheduled to the same node with less initial memory demand. For some workload types, this can be perfectly fine and lead to better overall memory-utilization.</p>"},{"location":"compute/node_overcommit/#configuring-hard-eviction-thresholds","title":"Configuring Hard Eviction Thresholds","text":"<p>Note: If unspecified, the kubelet will do hard evictions for Pods once <code>memory.available</code> falls below <code>100Mi</code>.</p> <p>Limits set via <code>--eviction-hard</code> will lead to immediate eviction of VirtualMachineInstances or Pods. This stops VMIs without a grace period and is comparable with power-loss on a real computer.</p> <p>If the hard limit is hit, VMIs may from time to time simply be killed. They may be re-scheduled to the same node immediately again, since they start with less memory consumption again. This can be a simple option, if the memory threshold is only very seldom hit and the work performed by the VMIs is reproducible or it can be resumed from some checkpoints.</p>"},{"location":"compute/node_overcommit/#requesting-the-right-qos-class-for-virtualmachineinstances","title":"Requesting the right QoS Class for VirtualMachineInstances","text":"<p>Different QoS classes get assigned to Pods and VirtualMachineInstances based on the <code>requests.memory</code> and <code>limits.memory</code>. KubeVirt right now supports the QoS classes <code>Burstable</code> and <code>Guaranteed</code>. <code>Burstable</code> VMIs are evicted before <code>Guaranteed</code> VMIs.</p> <p>This allows creating two classes of VMIs:</p> <ul> <li>One type can have equal <code>requests.memory</code> and <code>limits.memory</code> set     and therefore gets the <code>Guaranteed</code> class assigned. This one will     not get evicted and should never run into memory issues, but is more     demanding.</li> <li>One type can have no <code>limits.memory</code> or a <code>limits.memory</code> which is     greater than <code>requests.memory</code> and therefore gets the <code>Burstable</code>     class assigned. These VMIs will be evicted first.</li> </ul>"},{"location":"compute/node_overcommit/#setting-system-reserved-and-kubelet-reserved","title":"Setting <code>--system-reserved</code> and <code>--kubelet-reserved</code>","text":"<p>It may be important to reserve some memory for other daemons (not DaemonSets) which are running on the same node (ssh, dhcp servers, etc). The reservation can be done with the <code>--system reserved</code> switch. Further for the Kubelet and Docker a special flag called <code>--kubelet-reserved</code> exists.</p>"},{"location":"compute/node_overcommit/#enabling-ksm","title":"Enabling KSM","text":"<p>The KSM (Kernel same-page merging) daemon can be started on the node. Depending on its tuning parameters it can more or less aggressively try to merge identical pages between applications and VirtualMachineInstances. The more aggressive it is configured the more CPU it will use itself, so the memory overcommit advantages comes with a slight CPU performance hit.</p> <p>Config file tuning allows changes to scanning frequency (how often will KSM activate) and aggressiveness (how many pages per second will it scan).</p>"},{"location":"compute/node_overcommit/#enabling-swap","title":"Enabling Swap","text":"<p>Note: This will definitely make sure that your VirtualMachines can't crash or get evicted from the node but it comes with the cost of pretty unpredictable performance once the node runs out of memory and the kubelet may not detect that it should evict Pods to increase the performance again.</p> <p>Enabling swap is in general not recommended on Kubernetes right now. However, it can be useful in combination with KSM, since KSM merges identical pages over time. Swap allows the VMIs to successfully allocate memory which will then effectively never be used because of the later de-duplication done by KSM.</p>"},{"location":"compute/node_overcommit/#node-cpu-allocation-ratio","title":"Node CPU allocation ratio","text":"<p>KubeVirt runs Virtual Machines in a Kubernetes Pod. This pod requests a certain amount of CPU time from the host. On the other hand, the Virtual Machine is being created with a certain amount of vCPUs. The number of vCPUs may not necessarily correlate to the number of requested CPUs by the POD.  Depending on the QOS of the POD, vCPUs can be scheduled on a variable amount of physical CPUs; this depends on the available CPU resources on a node. When there are fewer available CPUs on the node as the requested vCPU, vCPU will be over committed.</p> <p>By default, each pod requests 100mil of CPU time. The CPU requested on the pod sets the cgroups cpu.shares which serves as a priority for the scheduler to provide CPU time for vCPUs in this POD. As the number of vCPUs increases, this will reduce the amount of CPU time each vCPU may get when competing with other processes on the node or other Virtual Machine Instances with a lower amount of vCPUs.</p> <p>The <code>cpuAllocationRatio</code> comes to normalize the amount of CPU time the POD will request based on the number of vCPUs. For example, POD CPU request = number of vCPUs * 1/cpuAllocationRatio When cpuAllocationRatio is set to 1, a full amount of vCPUs will be requested for the POD.</p> <p>Note: In Kubernetes, one full core is 1000 of CPU time More Information</p> <p>Administrators can change this ratio by updating the KubeVirt CR</p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      cpuAllocationRatio: 10\n</code></pre>"},{"location":"compute/numa/","title":"NUMA","text":"<p>FEATURE STATE: KubeVirt v0.43</p> <p>NUMA support in KubeVirt is at this stage limited to a small set of special use-cases and will improve over time together with improvements made to Kubernetes.</p> <p>In general, the goal is to map the host NUMA topology as efficiently as possible to the Virtual Machine topology to improve the performance.</p> <p>The following NUMA mapping strategies can be used:</p> <ul> <li>GuestMappingPassthrough</li> </ul>"},{"location":"compute/numa/#preconditions","title":"Preconditions","text":"<p>In order to use current NUMA support, the following preconditions must be met:</p> <ul> <li>Dedicated CPU Resources must be configured.</li> <li>Hugepages need to be allocatable on target   nodes.</li> <li>The <code>NUMA</code> feature gate   must be enabled.</li> </ul>"},{"location":"compute/numa/#guestmappingpassthrough","title":"GuestMappingPassthrough","text":"<p>GuestMappingPassthrough will pass through the node numa topology to the guest. The topology is based on the dedicated CPUs which the VMI got assigned from the kubelet via the CPU Manager. It can be requested by setting <code>spec.domain.cpu.numa.guestMappingPassthrough</code> on the VMI.</p> <p>Since KubeVirt does not know upfront which exclusive CPUs the VMI will get from the kubelet, there are some limitations:</p> <ul> <li>Guests may see different NUMA topologies when being rescheduled.</li> <li>The resulting NUMA topology may be asymmetrical.</li> <li>The VMI may fail to start on the node if not enough hugepages are available on   the assigned NUMA nodes.</li> </ul> <p>While this NUMA modelling strategy has its limitations, aligning the guest's NUMA architecture with the node's can be critical for high-performance applications.</p> <p>An example VMI may look like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: numavm\nspec:\n  domain:\n    cpu:\n      cores: 4\n      dedicatedCpuPlacement: true\n      numa:\n        guestMappingPassthrough: { }\n    devices:\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n        - disk:\n            bus: virtio\n          name: cloudinitdisk\n    resources:\n      requests:\n        memory: 64Mi\n    memory:\n      hugepages:\n        pageSize: 2Mi\n  volumes:\n    - containerDisk:\n        image: quay.io/kubevirt/cirros-container-disk-demo\n      name: containerdisk\n    - cloudInitNoCloud:\n        userData: |\n          #!/bin/sh\n          echo 'printed from cloud-init userdata'\n      name: cloudinitdisk\n</code></pre>"},{"location":"compute/numa/#running-real-time-workloads","title":"Running real-time workloads","text":""},{"location":"compute/numa/#overview","title":"Overview","text":"<p>It is possible to deploy Virtual Machines that run a real-time kernel and make use of libvirtd's guest cpu and memory optimizations that improve the overall latency. These changes leverage mostly on already available settings in KubeVirt, as we will see shortly, but the VMI manifest now exposes two new settings that instruct KubeVirt to configure the generated libvirt XML with the recommended tuning settings for running real-time workloads.</p> <p>To make use of the optimized settings, two new settings have been added to the VMI schema:</p> <ul> <li> <p><code>spec.domain.cpu.realtime</code>: When defined, it instructs KubeVirt to configure the linux scheduler for the VCPUS to run processes in FIFO scheduling policy (SCHED_FIFO) with priority 1. This setting guarantees that all processes running in the host will be executed with real-time priority.</p> </li> <li> <p><code>spec.domain.cpu.realtime.mask</code>: It defines which VCPUs assigned to the VM are used for real-time. If not defined, libvirt will define all VCPUS assigned to run processes in FIFO scheduling and in the highest priority (1).</p> </li> </ul>"},{"location":"compute/numa/#preconditions_1","title":"Preconditions","text":"<p>A prerequisite to running real-time workloads include locking resources in the cluster to allow the real-time VM exclusive usage. This translates into nodes, or node, that have been configured with a dedicated set of CPUs and also provides support for NUMA with a free number of hugepages of 2Mi or 1Gi size (depending on the configuration in the VMI). Additionally, the node must be configured to allow the scheduler to run processes with real-time policy.</p>"},{"location":"compute/numa/#nodes-capable-of-running-real-time-workloads","title":"Nodes capable of running real-time workloads","text":"<p>When the KubeVirt pods are deployed in a node, it will check if it is capable of running processes in real-time scheduling policy and label the node as real-time capable (kubevirt.io/realtime). If, on the other hand, the node is not able to deliver such capability, the label is not applied. To check which nodes are able to host real-time VM workloads run this command:</p> <pre><code>$&gt;kubectl get nodes -l kubevirt.io/realtime\nNAME         STATUS   ROLES              AGE    VERSION\nworker-0-0   Ready    worker             12d    v1.20.0+df9c838\n</code></pre> <p>Internally, the KubeVirt pod running in each node checks if the kernel setting <code>kernel.sched_rt_runtime_us</code> equals to -1, which grants processes to run in real-time scheduling policy for an unlimited amount of time.</p>"},{"location":"compute/numa/#configuring-a-vm-manifest","title":"Configuring a VM Manifest","text":"<p>Here is an example of a VM manifest that runs a custom fedora container disk configured to run with a real-time kernel. The settings have been configured for optimal efficiency.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: fedora-realtime\n  name: fedora-realtime\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: fedora-realtime\n    spec:\n      domain:\n        devices:\n          autoattachSerialConsole: true\n          autoattachMemBalloon: false\n          autoattachGraphicsDevice: false\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk \n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 1Gi\n            cpu: 2\n          limits:\n            memory: 1Gi\n            cpu: 2\n        cpu:\n          model: host-passthrough\n          dedicatedCpuPlacement: true\n          isolateEmulatorThread: true\n          ioThreadsPolicy: auto\n          features:\n            - name: tsc-deadline\n              policy: require\n          numa:\n            guestMappingPassthrough: {}\n          realtime: {}\n        memory:\n          hugepages:\n            pageSize: 1Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-realtime-container-disk:v20211008-22109a3\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n            bootcmd:\n              - tuned-adm profile realtime\n        name: cloudinitdisk\n</code></pre> <p>Breaking down the tuned sections, we have the following configuration:</p> <p>Devices:  - Disable the guest's memory balloon capability - Avoid attaching a graphics device, to reduce the number of interrupts to the kernel.</p> <pre><code>spec:\n  domain:\n    devices:\n      autoattachSerialConsole: true\n      autoattachMemBalloon: false\n      autoattachGraphicsDevice: false\n</code></pre> <p>CPU: - model: <code>host-passthrough</code> to allow the guest to see host CPU without masking any capability. - dedicated CPU Placement: The VM needs to have dedicated CPUs assigned to it. The Kubernetes CPU Manager takes care of this aspect. - isolatedEmulatorThread: to request an additional CPU to run the emulator on it, thus avoid using CPU cycles from the workload CPUs. - ioThreadsPolicy: Set to auto to let the dedicated IO thread to run in the same CPU as the emulator thread. - NUMA: defining <code>guestMappingPassthrough</code> enables NUMA support for this VM. - realtime: instructs the virt-handler to configure this VM for real-time workloads, such as configuring the VCPUS to use FIFO scheduler policy and set priority to 1. cpu: <pre><code>cpu:\n  model: host-passthrough\n  dedicatedCpuPlacement: true\n  isolateEmulatorThread: true\n  ioThreadsPolicy: auto\n  features:\n    - name: tsc-deadline\n      policy: require\n  numa:\n    guestMappingPassthrough: {}\n  realtime: {}\n</code></pre></p> <p>Memory - pageSize: allocate the pod's memory in hugepages of the given size, in this case of 1Gi. <pre><code>memory:\n  hugepages:\n    pageSize: 1Gi\n</code></pre></p>"},{"location":"compute/numa/#how-to-dedicate-vcpus-for-real-time-only","title":"How to dedicate VCPUS for real-time only","text":"<p>It is possible to pass a regular expression of the VCPUs to isolate to use real-time scheduling policy, by using the <code>realtime.mask</code> setting.</p> <pre><code>cpu:\n  numa:\n    guestMappingPassthrough: {}\n  realtime:\n    mask: \"0\"\n</code></pre> <p>When applied this configuration, KubeVirt will only set the first VCPU for real-time scheduler policy, leaving the remaining VCPUS to use the default scheduler policy. Other examples of valid masks are: - <code>0-3</code>: Use cores 0 to 3 for real-time scheduling, assuming that the VM has requested at least 3 cores. - <code>0-3,^1</code>: Use cores 0, 2 and 3 for real-time scheduling only, assuming that the VM has requested at least 3 cores.</p>"},{"location":"compute/numa/#additional-reading","title":"Additional Reading","text":"<p>Kubernetes provides additional NUMA components that may be relevant to your use-case but typically are not enabled by default. Please consult the Kubernetes documentation for details on configuration of these components.</p>"},{"location":"compute/numa/#topology-manager","title":"Topology Manager","text":"<p>Topology Manager provides optimizations related to CPU isolation, memory and device locality. It is useful, for example, where an SR-IOV network adaptor VF allocation needs to be aligned with a NUMA node.</p> <p>https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/</p>"},{"location":"compute/numa/#memory-manager","title":"Memory Manager","text":"<p>Memory Manager is analogous to CPU Manager. It is useful, for example, where you want to align hugepage allocations with a NUMA node. It works in conjunction with Topology Manager.</p> <p>The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.</p> <p>https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/</p>"},{"location":"compute/persistent_tpm_and_uefi_state/","title":"Persistent TPM and UEFI state","text":"<p>FEATURE STATE: KubeVirt v1.0.0</p> <p>For both TPM and UEFI, libvirt supports persisting data created by a virtual machine as files on the virtualization host. In KubeVirt, the virtualization host is the virt-launcher pod, which is ephemeral (created on VM start and destroyed on VM stop). As of v1.0.0, KubeVirt supports using a PVC to persist those files. KubeVirt usually refers to that storage area as \"backend storage\".</p>"},{"location":"compute/persistent_tpm_and_uefi_state/#backend-storage","title":"Backend storage","text":"<p>KubeVirt automatically creates backend storage PVCs for VMs that need it. However, the admin must first enable the <code>VMPersistentState</code> feature gate, and tell KubeVirt which storage class to use by setting the <code>vmStateStorageClass</code> configuration parameter in the KubeVirt Custom Resource (CR). The storage class must support read-write-many (RWX) in filesystem mode (FS). Here's an example of KubeVirt CR that sets both: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmStateStorageClass: \"nfs-csi\"\n    developerConfiguration:\n      featureGates:\n      - VMPersistentState\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#limitations","title":"Limitations","text":"<ul> <li>As mentioned above, the backend storage PVC can only be created using a storage class that supports RWX FS. There is ongoing work to support block storage in future versions of KubeVirt.</li> <li>Backend storage is currently incompatible with VM snapshot. It is planned to add snapshot support in the future.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#tpm-with-persistent-state","title":"TPM with persistent state","text":"<p>Since KubeVirt v0.53.0, a TPM device can be added to a VM (with just <code>tpm: {}</code>). However, the data stored in it does not persist across reboots. Support for persistence was added in v1.0.0 using a simple <code>persistent</code> boolean parameter that default to false, to preserve previous behavior. Of course, backend storage must first be configured before adding a persistent TPM to a VM. See above. Here's a portion of a VM definition that includes a persistent TPM: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm\nspec:\n  template:\n    spec:\n      domain:\n        devices:\n          tpm:\n            persistent: true\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#uses","title":"Uses","text":"<ul> <li>The Microsoft Windows 11 installer requires the presence of a TPM device, even though it doesn't use this. Persistence is not required in this case however.</li> <li>Some disk encryption software have optional/mandatory TPM support. For example, Bitlocker requires a persistent TPM device.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#notes","title":"Notes","text":"<ul> <li>The TPM device exposed to the virtual machine is fully emulated (vTPM). The worker nodes do not need to have a TPM device.</li> <li>When TPM persistence is enabled, the <code>tpm-crb</code> model is used (instead of <code>tpm-tis</code> for non-persistent vTPMs)</li> <li>A virtual TPM does not provide the same security guarantees as a physical one.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#efi-with-persistent-vars","title":"EFI with persistent VARS","text":"<p>EFI support is handled by libvirt using OVMF. OVMF data usually consists of 2 files, CODE and VARS. VARS is where persistent data from the guest can be stored. When EFI persistence is enabled on a VM, the VARS file will be persisted inside the backend storage. Of course, backend storage must first be configured before enabling EFI persistence on a VM. See above. Here's a portion of a VM definition that includes a persistent EFI: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm\nspec:\n  template:\n    spec:\n      domain:\n        firmware:\n          bootloader:\n            efi:\n              persistent: true\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#uses_1","title":"Uses","text":"<ul> <li>Preserving user-created Secure Boot certificates.</li> <li>Preserving EFI firmware settings, like language or display resolution.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#notes_1","title":"Notes","text":"<ul> <li>The boot entries/order can, and most likely will, get overriden by libvirt. This is to satisfy the VM specfications. Do not expect manual boot setting changes to persist.</li> </ul>"},{"location":"compute/resources_requests_and_limits/","title":"Resources requests and limits","text":"<p>In this document, we are talking about the resources values set on the virt-launcher compute container, referred to as \"the container\" below for simplicity.</p>"},{"location":"compute/resources_requests_and_limits/#cpu","title":"CPU","text":"<p>Note: dedicated CPUs (and isolated emulator thread) are ignored here as they have a dedicated page.</p>"},{"location":"compute/resources_requests_and_limits/#cpu-requests-on-the-container","title":"CPU requests on the container","text":"<ul> <li>By default, the container requests (1/cpuAllocationRatio) CPU per vCPU. The number of vCPUs is socketscoresthreads, defaults to 1.</li> <li>cpuAllocationRatio defaults to 10 but can be changed in the CR.</li> <li>If a CPU limit is manually set on the VM(I) and no CPU request is, the CPU requests on the container will match the CPU limits</li> <li>Manually setting CPU requests on the VM(I) will override all of the above and be the CPU requests for the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#cpu-limits-on-the-container","title":"CPU limits on the container","text":"<ul> <li>By default, no CPU limit is set on the container</li> <li>If auto CPU limits is enabled (see next section), then the container will have a CPU limit of 1 per vCPU</li> <li>Manually setting CPU limits on the VM(I) will override all of the above and be the CPU limits for the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#auto-cpu-limits","title":"Auto CPU limits","text":"<p>KubeVirt provides two ways to automatically set CPU limits on VM(I)s:</p> <ul> <li>Enable the <code>AutoResourceLimitsGate</code> feature gate.</li> <li>Add the namespaceLabelSelector in the KubeVirt CR.</li> </ul> <p>In both cases, the VM(I) created will have a CPU limit of 1 per vCPU.</p>"},{"location":"compute/resources_requests_and_limits/#autoresourcelimitsgate-feature-gate","title":"AutoResourceLimitsGate feature gate","text":"<p>By enabling this feature gate, cpu limits will be added to the vmi if all the following conditions are true:</p> <ul> <li>The namespace where the VMI will be created has a ResourceQuota containing cpu limits.</li> <li>The VMI has no manually set cpu limits.</li> <li>The VMI is not requesting dedicated CPU.</li> </ul>"},{"location":"compute/resources_requests_and_limits/#autocpulimitnamespacelabelselector-configuration","title":"autoCPULimitNamespaceLabelSelector configuration","text":"<p>Cluster admins can define a label selector in the KubeVirt CR. Once that label selector is defined, if the creation namespace matches the selector, all VM(I)s created in it will have a CPU limits set.</p> <p>Example:</p> <ul> <li> <p>CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    autoCPULimitNamespaceLabelSelector:\n      matchLabels:\n        autoCpuLimit: \"true\"\n</code></pre></p> </li> <li> <p>Namespace: <pre><code>apiVersion: v1\nkind: Namespace\nmetadata:\n  labels:\n    autoCpuLimit: \"true\"\n    kubernetes.io/metadata.name: default\n  name: default\n</code></pre></p> </li> </ul>"},{"location":"compute/resources_requests_and_limits/#memory","title":"Memory","text":""},{"location":"compute/resources_requests_and_limits/#memory-requests-on-the-container","title":"Memory requests on the container","text":"<ul> <li>VM(I)s must specify a desired amount of memory, in either spec.domain.memory.guest or spec.domain.resources.requests.memory (ignoring hugepages, see the dedicated page). If both are set, the memory requests take precedence. A calculated amount of overhead will be added to it, forming the memory request value for the container.</li> </ul>"},{"location":"compute/resources_requests_and_limits/#memory-limits-on-the-container","title":"Memory limits on the container","text":"<ul> <li>By default, no memory limit is set on the container</li> <li>If auto memory limits is enabled (see next section), then the container will have a limit of 2x the requested memory.</li> <li>Manually setting a memory limit on the VM(I) will set the same value on the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#warnings","title":"Warnings","text":"<ul> <li>Memory limits have to be more than memory requests + overhead, otherwise the container will have memory requests &gt; limits and be rejected by Kubernetes.</li> <li>Memory usage bursts could lead to VM crashes when memory limits are set</li> </ul>"},{"location":"compute/resources_requests_and_limits/#auto-memory-limits","title":"Auto memory limits","text":"<p>KubeVirt provides a feature gate(<code>AutoResourceLimitsGate</code>) to automatically set memory limits on VM(I)s. By enabling this feature gate, memory limits will be added to the vmi if all the following conditions are true:</p> <ul> <li>The namespace where the VMI will be created has a ResourceQuota containing memory limits.</li> <li>The VMI has no manually set memory limits.</li> <li>The VMI is not requesting dedicated CPU.</li> </ul> <p>If all the previous conditions are true, the memory limits will be set to a value (<code>2x</code>) of the memory requests. This ratio can be adjusted, per namespace, by adding the annotation <code>alpha.kubevirt.io/auto-memory-limits-ratio</code>, with the desired custom value. For example, with <code>alpha.kubevirt.io/auto-memory-limits-ratio: 1.2</code>, the memory limits set will be equal to (<code>1.2x</code>) of the memory requests.</p>"},{"location":"compute/run_strategies/","title":"Run Strategies","text":""},{"location":"compute/run_strategies/#overview","title":"Overview","text":"<p>VirtualMachines have a <code>Running</code> setting that determines whether or not there should be a guest running or not. Because KubeVirt will always immediately restart a VirtualMachineInstance for VirtualMachines with <code>spec.running: true</code>, a simple boolean is not always enough to fully describe desired behavior. For instance, there are cases when a user would like the ability to shut down a guest from inside the virtual machine. With <code>spec.running: true</code>, KubeVirt would immediately restart the VirtualMachineInstance.</p>"},{"location":"compute/run_strategies/#runstrategy","title":"RunStrategy","text":"<p>To allow for greater variation of user states, the <code>RunStrategy</code> field has been introduced. This is mutually exclusive with <code>Running</code> as they have somewhat overlapping conditions. There are currently four RunStrategies defined:</p> <ul> <li> <p>Always: The system is tasked with keeping the VM in a running     state.     This is achieved by respawning a VirtualMachineInstance whenever     the current one terminated in a controlled (e.g. shutdown from     inside the guest) or uncontrolled (e.g. crash) way.     This behavior is equal to <code>spec.running: true</code>.</p> </li> <li> <p>RerunOnFailure: Similar to <code>Always</code>, except that the VM is only     restarted if it terminated in an uncontrolled way (e.g. crash)     and due to an infrastructure reason (i.e. the node crashed,     the KVM related process OOMed).     This allows a user to determine when the VM should be shut down     by initiating the shut down inside the guest.     Note: Guest sided crashes (i.e. BSOD) are not covered by this.     In such cases liveness checks or the use of a watchdog can help.</p> </li> <li> <p>Manual: The system will not automatically turn the VM on or off,     instead the user manually controlls the VM status by issuing     start, stop, and restart commands on the VirtualMachine     subresource endpoints.</p> </li> <li> <p>Halted: The system is asked to ensure that no VM is running.     This is achieved by stopping any VirtualMachineInstance that is     associated ith the VM. If a guest is already running, it will be     stopped.     This behavior is equal to <code>spec.running: false</code>.</p> </li> </ul> <p>Note: <code>RunStrategy</code> and <code>running</code> are mutually exclusive, because they can be contradictory. The API server will reject VirtualMachine resources that define both.</p>"},{"location":"compute/run_strategies/#virtctl","title":"Virtctl","text":"<p>The <code>start</code>, <code>stop</code> and <code>restart</code> methods of virtctl will invoke their respective subresources of VirtualMachines. This can have an effect on the runStrategy of the VirtualMachine as below:</p> RunStrategy start stop restart <p>Always</p> <p><code>-</code></p> <p><code>Halted</code></p> <p><code>Always</code></p> <p>RerunOnFailure</p> <p><code>RerunOnFailure</code></p> <p><code>RerunOnFailure</code></p> <p><code>RerunOnFailure</code></p> <p>Manual</p> <p><code>Manual</code></p> <p><code>Manual</code></p> <p><code>Manual</code></p> <p>Halted</p> <p><code>Always</code></p> <p><code>-</code></p> <p><code>-</code></p> <p>Table entries marked with <code>-</code> don't make sense, so won't have an effect on RunStrategy.</p>"},{"location":"compute/run_strategies/#runstrategy-examples","title":"RunStrategy Examples","text":""},{"location":"compute/run_strategies/#always","title":"Always","text":"<p>An example usage of the Always RunStrategy.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-cirros\n  name: vm-cirros\nspec:\n  runStrategy: Always\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-cirros\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n        name: containerdisk\n</code></pre>"},{"location":"compute/virtual_hardware/","title":"Virtual hardware","text":"<p>Fine-tuning different aspects of the hardware which are not device related (BIOS, mainboard, etc.) is sometimes necessary to allow guest operating systems to properly boot and reboot.</p>"},{"location":"compute/virtual_hardware/#machine-type","title":"Machine Type","text":"<p>QEMU is able to work with two different classes of chipsets for x86_64, so called machine types. The x86_64 chipsets are i440fx (also called pc) and q35. They are versioned based on qemu-system-${ARCH}, following the format <code>pc-${machine_type}-${qemu_version}</code>, e.g.<code>pc-i440fx-2.10</code> and <code>pc-q35-2.10</code>.</p> <p>KubeVirt defaults to QEMU's newest q35 machine type. If a custom machine type is desired, it is configurable through the following structure:</p> <p><pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    machine:\n      # This value indicates QEMU machine type.\n      type: pc-q35-2.10\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> Comparison of the machine types' internals can be found at QEMU wiki.</p>"},{"location":"compute/virtual_hardware/#biosuefi","title":"BIOS/UEFI","text":"<p>All virtual machines use BIOS by default for booting.</p> <p>It is possible to utilize UEFI/OVMF by setting a value via <code>spec.firmware.bootloader</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-alpine-efi\n  name: vmi-alpine-efi\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    features:\n      smm:\n        enabled: true\n    firmware:\n      # this sets the bootloader type\n      bootloader:\n        efi: {}\n</code></pre> <p>Enabling EFI automatically enables Secure Boot, unless the <code>secureBoot</code> field under <code>efi</code> is set to <code>false</code>. Secure Boot itself requires the SMM CPU feature to be enabled as above, which does not happen automatically, for security reasons.</p>"},{"location":"compute/virtual_hardware/#smbios-firmware","title":"SMBIOS Firmware","text":"<p>In order to provide a consistent view on the virtualized hardware for the guest OS, the SMBIOS UUID can be set to a constant value via <code>spec.firmware.uuid</code>:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    firmware:\n      # this sets the UUID\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n      serial: e4686d2c-6e8d-4335-b8fd-81bee22f4815\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p>In addition, the SMBIOS serial number can be set to a constant value via <code>spec.firmware.serial</code>, as demonstrated above.</p>"},{"location":"compute/virtual_hardware/#cpu","title":"CPU","text":"<p>Note: This is not related to scheduling decisions or resource assignment.</p>"},{"location":"compute/virtual_hardware/#topology","title":"Topology","text":"<p>Setting the number of CPU cores is possible via <code>spec.domain.cpu.cores</code>. The following VM will have a CPU with <code>3</code> cores:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the cores\n      cores: 3\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#labeling-nodes-with-cpu-models-and-cpu-features","title":"Labeling nodes with cpu models and cpu features","text":"<p>KubeVirt can create node selectors based on VM cpu models and features. With these node selectors, VMs will be scheduled on the nodes that support the matching VM cpu model and features.</p> <p>To properly label the node, user can use Kubevirt Node-labeller, which creates all  necessary labels or create node labels by himself.</p> <p>Kubevirt node-labeller creates 3 types of labels: cpu models, cpu features and kvm info. It uses libvirt to get all supported cpu models and cpu features on host and then Node-labeller creates labels from cpu models. </p> <p>Node-labeller supports obsolete list of cpu models and minimal baseline cpu model for features. Both features can be set via KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    obsoleteCPUModels:\n      486: true\n      pentium: true\n...\n</code></pre> <p>Obsolete cpus will not be inserted in labels. If KubeVirt CR doesn't  contain <code>obsoleteCPUModels</code> variable, Labeller sets default values (\"pentium, pentium2, pentium3, pentiumpro, coreduo, n270,  core2duo, Conroe, athlon, phenom, kvm32, kvm64, qemu32 and qemu64\").</p> <p>User can change obsoleteCPUModels by adding / removing cpu model in config map. Kubevirt then update nodes with new labels.</p> <p>For homogenous cluster / clusters without live migration enabled it's possible to disable the node labeler and avoid adding labels to the nodes by adding the  following annotation to the nodes:</p> <p><code>node-labeller.kubevirt.io/skip-node</code>.</p>"},{"location":"compute/virtual_hardware/#model","title":"Model","text":"<p>Note: If CPU model wasn't defined, the VM will have CPU model closest to one that used on the node where the VM is running.</p> <p>Note: CPU model is case sensitive.</p> <p>Setting the CPU model is possible via <code>spec.domain.cpu.model</code>. The following VM will have a CPU with the <code>Conroe</code> model:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the CPU model\n      model: Conroe\n...\n</code></pre> <p>You can check list of available models here.</p> <p>When CPUNodeDiscovery feature-gate is enabled and VM has cpu model, Kubevirt creates node selector with format: <code>cpu-model.node.kubevirt.io/&lt;cpuModel&gt;</code>, e.g. <code>cpu-model.node.kubevirt.io/Conroe</code>. When VM doesn\u2019t have cpu model, then no node selector is created.</p>"},{"location":"compute/virtual_hardware/#enabling-default-cluster-cpu-model","title":"Enabling default cluster cpu model","text":"<p>To enable the default cpu model, user may add the <code>cpuModel</code> field in the KubeVirt CR.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    cpuModel: \"EPYC\"\n...\n</code></pre> <p>Default CPU model is set when vmi doesn't have any cpu model. When vmi has cpu model set, then vmi's cpu model is preferred. When default cpu model is not set and vmi's cpu model is not set too, <code>host-model</code> will be set. Default cpu model can be changed when kubevirt is running. When CPUNodeDiscovery feature gate is enabled Kubevirt creates node selector with default cpu model.</p>"},{"location":"compute/virtual_hardware/#cpu-model-special-cases","title":"CPU model special cases","text":"<p>As special cases you can set <code>spec.domain.cpu.model</code> equals to: - <code>host-passthrough</code> to passthrough CPU from the node to the VM</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this passthrough the node CPU to the VM\n      model: host-passthrough\n...\n</code></pre> <ul> <li><code>host-model</code> to get CPU on the VM close to the node one</li> </ul> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this set the VM CPU close to the node one\n      model: host-model\n...\n</code></pre> <p>See the CPU API reference for more details.</p>"},{"location":"compute/virtual_hardware/#features","title":"Features","text":"<p>Setting CPU features is possible via <code>spec.domain.cpu.features</code> and can contain zero or more CPU features :</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the CPU features\n      features:\n      # this is the feature's name\n      - name: \"apic\"\n      # this is the feature's policy\n       policy: \"require\"\n...\n</code></pre> <p>Note: Policy attribute can either be omitted or contain one of the following policies: force, require, optional, disable, forbid.</p> <p>Note: In case a policy is omitted for a feature, it will default to require.</p> <p>Behaviour according to Policies:</p> <ul> <li>All policies will be passed to libvirt during virtual machine     creation.</li> <li>In case the feature gate \"CPUNodeDiscovery\" is enabled and the     policy is omitted or has \"require\" value, then the virtual machine     could be scheduled only on nodes that support this feature.</li> <li>In case the feature gate \"CPUNodeDiscovery\" is enabled and the     policy has \"forbid\" value, then the virtual machine would not be     scheduled on nodes that support this feature.</li> </ul> <p>Full description about features and policies can be found here.</p> <p>When CPUNodeDiscovery feature-gate is enabled Kubevirt creates node selector from cpu features with format: <code>cpu-feature.node.kubevirt.io/&lt;cpuFeature&gt;</code>, e.g. <code>cpu-feature.node.kubevirt.io/apic</code>. When VM doesn\u2019t have cpu feature, then no node selector is created.</p>"},{"location":"compute/virtual_hardware/#clock","title":"Clock","text":""},{"location":"compute/virtual_hardware/#guest-time","title":"Guest time","text":"<p>Sets the virtualized hardware clock inside the VM to a specific time. Available options are</p> <ul> <li> <p>utc</p> </li> <li> <p>timezone</p> </li> </ul> <p>See the Clock API Reference for all possible configuration options.</p>"},{"location":"compute/virtual_hardware/#utc","title":"utc","text":"<p>If <code>utc</code> is specified, the VM's clock will be set to UTC.</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      utc: {}\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#timezone","title":"timezone","text":"<p>If <code>timezone</code> is specified, the VM's clock will be set to the specified local time.</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      timezone: \"America/New York\"\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#timers","title":"Timers","text":"<ul> <li> <p>pit</p> </li> <li> <p>rtc</p> </li> <li> <p>kvm</p> </li> <li> <p>hyperv</p> </li> </ul> <p>A pretty common timer configuration for VMs looks like this:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      utc: {}\n      # here are the timer\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p><code>hpet</code> is disabled,<code>pit</code> and <code>rtc</code> are configured to use a specific <code>tickPolicy</code>. Finally, <code>hyperv</code> is made available too.</p> <p>See the Timer API Reference for all possible configuration options.</p> <p>Note: Timer can be part of a machine type. Thus it may be necessary to explicitly disable them. We may in the future decide to add them via cluster-level defaulting, if they are part of a QEMU machine definition.</p>"},{"location":"compute/virtual_hardware/#random-number-generator-rng","title":"Random number generator (RNG)","text":"<p>You may want to use entropy collected by your cluster nodes inside your guest. KubeVirt allows to add a <code>virtio</code> RNG device to a virtual machine as follows.</p> <pre><code>metadata:\n  name: vmi-with-rng\nspec:\n  domain:\n    devices:\n      rng: {}\n</code></pre> <p>For Linux guests, the <code>virtio-rng</code> kernel module should be loaded early in the boot process to acquire access to the entropy source. Other systems may require similar adjustments to work with the <code>virtio</code> RNG device.</p> <p>Note: Some guest operating systems or user payloads may require the RNG device with enough entropy and may fail to boot without it. For example, fresh Fedora images with newer kernels (4.16.4+) may require the <code>virtio</code> RNG device to be present to boot to login.</p>"},{"location":"compute/virtual_hardware/#video-and-graphics-device","title":"Video and Graphics Device","text":"<p>By default a minimal Video and Graphics device configuration will be applied to the VirtualMachineInstance. The video device is <code>vga</code> compatible and comes with a memory size of 16 MB. This device allows connecting to the OS via <code>vnc</code>.</p> <p>It is possible not attach it by setting <code>spec.domain.devices.autoattachGraphicsDevice</code> to <code>false</code>:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    devices:\n      autoattachGraphicsDevice: false\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p>VMIs without graphics and video devices are very often referenced as <code>headless</code> VMIs.</p> <p>If using a huge amount of small VMs this can be helpful to increase the VMI density per node, since no memory needs to be reserved for video.</p>"},{"location":"compute/virtual_hardware/#features_1","title":"Features","text":"<p>KubeVirt supports a range of virtualization features which may be tweaked in order to allow non-Linux based operating systems to properly boot. Most noteworthy are</p> <ul> <li> <p>acpi</p> </li> <li> <p>apic</p> </li> <li> <p>hyperv</p> </li> </ul> <p>A common feature configuration is shown by the following example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    # typical features\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre> <p>See the Features API Reference for all available features and configuration options.</p>"},{"location":"compute/virtual_hardware/#resources-requests-and-limits","title":"Resources Requests and Limits","text":"<p>An optional resource request can be specified by the users to allow the scheduler to make a better decision in finding the most suitable Node to place the VM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    resources:\n      requests:\n        memory: \"1Gi\"\n        cpu: \"1\"\n      limits:\n        memory: \"2Gi\"\n        cpu: \"2\"\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#cpu_1","title":"CPU","text":"<p>Specifying CPU limits will determine the amount of cpu shares set on the control group the VM is running in, in other words, the amount of time the VM's CPUs can execute on the assigned resources when there is a competition for CPU resources.</p> <p>For more information please refer to how Pods with resource limits are run.</p>"},{"location":"compute/virtual_hardware/#memory-overhead","title":"Memory Overhead","text":"<p>Various VM resources, such as a video adapter, IOThreads, and supplementary system software, consume additional memory from the Node, beyond the requested memory intended for the guest OS consumption. In order to provide a better estimate for the scheduler, this memory overhead will be calculated and added to the requested memory.</p> <p>Please see how Pods with resource requests are scheduled for additional information on resource requests and limits.</p>"},{"location":"compute/virtual_hardware/#hugepages","title":"Hugepages","text":"<p>KubeVirt give you possibility to use hugepages as backing memory for your VM. You will need to provide desired amount of memory <code>resources.requests.memory</code> and size of hugepages to use <code>memory.hugepages.pageSize</code>, for example for x86_64 architecture it can be <code>2Mi</code>.</p> <pre><code>apiVersion: kubevirt.io/v1alpha1\nkind: VirtualMachine\nmetadata:\n  name: myvm\nspec:\n  domain:\n    resources:\n      requests:\n        memory: \"64Mi\"\n    memory:\n      hugepages:\n        pageSize: \"2Mi\"\n    disks:\n    - name: myimage\n      disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre> <p>In the above example the VM will have <code>64Mi</code> of memory, but instead of regular memory it will use node hugepages of the size of <code>2Mi</code>.</p>"},{"location":"compute/virtual_hardware/#limitations","title":"Limitations","text":"<ul> <li> <p>a node must have pre-allocated hugepages</p> </li> <li> <p>hugepages size cannot be bigger than requested memory</p> </li> <li> <p>requested memory must be divisible by hugepages size</p> </li> <li> <p>hugepages uses by default memfd. Memfd is supported from kernel &gt;= 4.14. If you run on an older host (e.g centos 7.9), it is required to disable memfd with the annotation <code>kubevirt.io/memfd: \"false\"</code> in the VMI metadata annotation.</p> </li> </ul>"},{"location":"compute/virtual_hardware/#input-devices","title":"Input Devices","text":""},{"location":"compute/virtual_hardware/#tablet","title":"Tablet","text":"<p>Kubevirt supports input devices. The only type which is supported is <code>tablet</code>. Tablet input device supports only <code>virtio</code> and <code>usb</code> bus. Bus can be empty. In that case, <code>usb</code> will be selected.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: myvm\nspec:\n  domain:\n    devices:\n      inputs:\n      - type: tablet\n        bus: virtio\n        name: tablet1\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre>"},{"location":"compute/vsock/","title":"VSOCK","text":"<p>VM Sockets (vsock) is a fast and efficient guest-host communication mechanism.</p>"},{"location":"compute/vsock/#background","title":"Background","text":"<p>Right now KubeVirt uses virtio-serial for local guest-host communication. Currently it used in KubeVirt by libvirt and qemu to communicate with the qemu-guest-agent. Virtio-serial can also be used by other agents, but it is a little bit cumbersome due to:</p> <ul> <li>A small set of ports on the virtio-serial device</li> <li>Low bandwidth</li> <li>No socket based communication possible, which requires every agent to establish their own protocols, or work with translation layers like SLIP to be able to use protocols like gRPC for reliable communication.</li> <li>No easy and supportable way to get a virtio-serial socket assigned and being able to access it without entering the virt-launcher pod.</li> <li>Due to the point above, privileges are required for services.</li> </ul> <p>With virtio-vsock we get support for easy guest-host communication which solves the above issues from a user/admin perspective.</p>"},{"location":"compute/vsock/#usage","title":"Usage","text":""},{"location":"compute/vsock/#feature-gate","title":"Feature Gate","text":"<p>To enable VSOCK in KubeVirt cluster, the user may expand the <code>featureGates</code> field in the KubeVirt CR by adding the <code>VSOCK</code> to it.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"VSOCK\"\n</code></pre> <p>Alternatively, users can edit an existing kubevirt CR:</p> <p><code>kubectl edit kubevirt kubevirt -n kubevirt</code></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"VSOCK\"\n</code></pre>"},{"location":"compute/vsock/#virtual-machine-instance","title":"Virtual Machine Instance","text":"<p>To attach VSOCK device to a Virtual Machine, the user has to add <code>autoattachVSOCK: true</code> in a <code>devices</code> section of Virtual Machine Instance specification:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-vsock\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      autoattachVSOCK: true\n</code></pre> <p>This will expose VSOCK device to the VM. The <code>CID</code> will be assigned randomly by <code>virt-controller</code>, and exposed to the Virtual Machine Instance status:</p> <pre><code>status:\n  VSOCKCID: 123\n</code></pre>"},{"location":"compute/vsock/#security","title":"Security","text":"<p>NOTE:  The <code>/dev/vhost-vsock</code> device is NOT NEEDED to connect or bind to a VSOCK socket.</p> <p>To make VSOCK feature secure, following measures are put in place:</p> <ul> <li>The whole VSOCK features will live behind a feature gate.</li> <li>By default the first 1024 ports of a vsock device are privileged. Services trying to bind to those require <code>CAP_NET_BIND_SERVICE</code> capability.</li> <li><code>AF_VSOCK</code> socket syscall gets blocked in containerd 1.7+ (containerd/containerd#7442). It is right now the responsibility of the vendor to ensure that the used CRI selects a default seccomp policy which blocks VSOCK socket calls in a similar way like it was done for containerd.</li> <li>CIDs are assigned by <code>virt-controller</code> and are unique per Virtual Machine Instance to ensure that <code>virt-handler</code> has an easy way of tracking the identity without races. While this still allows <code>virt-launcher</code> to fake-use an assigned CID, it eliminates possible assignment races which attackers could make use-of to redirect VSOCK calls.</li> </ul>"},{"location":"compute/windows_virtio_drivers/","title":"Windows virtio drivers","text":"<p>Purpose of this document is to explain how to install virtio drivers for Microsoft Windows running in a fully virtualized guest.</p>"},{"location":"compute/windows_virtio_drivers/#do-i-need-virtio-drivers","title":"Do I need virtio drivers?","text":"<p>Yes. Without the virtio drivers, you cannot use paravirtualized hardware properly. It would either not work, or will have a severe performance penalty.</p> <p>For more information about VirtIO and paravirtualization, see VirtIO and paravirtualization</p> <p>For more details on configuring your VirtIO driver please refer to Installing VirtIO driver on a new Windows virtual machine and Installing VirtIO driver on an existing Windows virtual machine.</p>"},{"location":"compute/windows_virtio_drivers/#which-drivers-i-need-to-install","title":"Which drivers I need to install?","text":"<p>There are usually up to 8 possible devices that are required to run Windows smoothly in a virtualized environment. KubeVirt currently supports only:</p> <ul> <li> <p>viostor, the block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>viorng, the entropy source driver, applies to PCI Device in the     Other devices group.</p> </li> <li> <p>NetKVM, the network driver, applies to Ethernet Controller in     the Other devices group. Available only if a virtio NIC is     configured.</p> </li> </ul> <p>Other virtio drivers, that exists and might be supported in the future:</p> <ul> <li> <p>Balloon, the balloon driver, applies to PCI Device in the Other     devices group</p> </li> <li> <p>vioserial, the paravirtual serial driver, applies to PCI Simple     Communications Controller in the Other devices group.</p> </li> <li> <p>vioscsi, the SCSI block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>qemupciserial, the emulated PCI serial driver, applies to PCI Serial     Port in the Other devices group.</p> </li> <li> <p>qxl, the paravirtual video driver, applied to Microsoft Basic     Display Adapter in the Display adapters group.</p> </li> <li> <p>pvpanic, the paravirtual panic driver, applies to Unknown device in     the Other devices group.</p> </li> </ul> <p>Note</p> <p>Some drivers are required in the installation phase. When you are installing Windows onto the virtio block storage you have to provide an appropriate virtio driver. Namely, choose viostor driver for your version of Microsoft Windows, eg. does not install XP driver when you run Windows 10.</p> <p>Other drivers can be installed after the successful windows installation. Again, please install only drivers matching your Windows version.</p>"},{"location":"compute/windows_virtio_drivers/#how-to-install-during-windows-install","title":"How to install during Windows install?","text":"<p>To install drivers before the Windows starts its install, make sure you have virtio-win package attached to your VirtualMachine as SATA CD-ROM. In the Windows installation, choose advanced install and load driver. Then please navigate to loaded Virtio CD-ROM and install one of viostor or vioscsi, depending on whichever you have set up.</p> <p>Step by step screenshots:</p> <p></p> <p></p> <p></p> <p></p> <p></p> <p></p>"},{"location":"compute/windows_virtio_drivers/#how-to-install-after-windows-install","title":"How to install after Windows install?","text":"<p>After windows install, please go to Device Manager. There you should see undetected devices in \"available devices\" section. You can install virtio drivers one by one going through this list.</p> <p></p> <p></p> <p></p> <p></p> <p>For more details on how to choose a proper driver and how to install the driver, please refer to the Windows Guest Virtual Machines on Red Hat Enterprise Linux 7.</p>"},{"location":"compute/windows_virtio_drivers/#how-to-obtain-virtio-drivers","title":"How to obtain virtio drivers?","text":"<p>The virtio Windows drivers are distributed in a form of containerDisk, which can be simply mounted to the VirtualMachine. The container image, containing the disk is located at: https://quay.io/repository/kubevirt/virtio-container-disk?tab=tags and the image be pulled as any other docker container:</p> <pre><code>docker pull quay.io/kubevirt/virtio-container-disk\n</code></pre> <p>However, pulling image manually is not required, it will be downloaded if not present by Kubernetes when deploying VirtualMachine.</p>"},{"location":"compute/windows_virtio_drivers/#attaching-to-virtualmachine","title":"Attaching to VirtualMachine","text":"<p>KubeVirt distributes virtio drivers for Microsoft Windows in a form of container disk. The package contains the virtio drivers and QEMU guest agent. The disk was tested on Microsoft Windows Server 2012. Supported Windows version is XP and up.</p> <p>The package is intended to be used as CD-ROM attached to the virtual machine with Microsoft Windows. It can be used as SATA CDROM during install phase or to provide drivers in an existing Windows installation.</p> <p>Attaching the virtio-win package can be done simply by adding ContainerDisk to you VirtualMachine.</p> <pre><code>spec:\n  domain:\n    devices:\n      disks:\n        - name: virtiocontainerdisk\n          # Any other disk you want to use, must go before virtioContainerDisk.\n          # KubeVirt boots from disks in order ther are defined.\n          # Therefore virtioContainerDisk, must be after bootable disk.\n          # Other option is to choose boot order explicitly:\n          #  - https://kubevirt.io/api-reference/v0.13.2/definitions.html#_v1_disk\n          # NOTE: You either specify bootOrder explicitely or sort the items in\n          #       disks. You can not do both at the same time.\n          # bootOrder: 2\n          cdrom:\n            bus: sata\nvolumes:\n  - containerDisk:\n      image: quay.io/kubevirt/virtio-container-disk\n    name: virtiocontainerdisk\n</code></pre> <p>Once you are done installing virtio drivers, you can remove virtio container disk by simply removing the disk from yaml specification and restarting the VirtualMachine.</p>"},{"location":"debug_virt_stack/debug/","title":"Debug","text":"<p>This page contains instructions on how to debug KubeVirt.</p> <p>This is useful to both KubeVirt developers and advanced users that would like to gain deep understanding on what's happening behind the scenes.</p>"},{"location":"debug_virt_stack/debug/#log-verbosity","title":"Log Verbosity","text":"<p>KubeVirt produces a lot of logging throughout its codebase. Some log entries have a verbosity level defined to them. The verbosity level that's defined for a log entry determines the minimum verbosity level in order to expose the log entry.</p> <p>In code, the log entry looks similar to: <code>log.Log.V(verbosity).Infof(\"...\")</code> while <code>verbosity</code> is the minimum verbosity level for this entry.</p> <p>For example, if the log verbosity for some log entry is <code>3</code>, then the log would be exposed only if the log verbosity is defined to be equal or greater than <code>3</code>, or else it would be filtered out.</p> <p>Currently, log verbosity can be defined per-component or per-node. The most updated API is detailed here.</p>"},{"location":"debug_virt_stack/debug/#setting-verbosity-per-kubevirt-component","title":"Setting verbosity per KubeVirt component","text":"<p>One way of raising log verbosity is to manually determine it for the different components in <code>KubeVirt</code> CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      logVerbosity:\n        virtLauncher: 2\n        virtHandler: 3\n        virtController: 4\n        virtAPI: 5\n        virtOperator: 6\n</code></pre></p> <p>This option is best for debugging specific components.</p>"},{"location":"debug_virt_stack/debug/#libvirt-virtqemudconf-set-log_filters-according-to-virt-launcher-log-verbosity","title":"libvirt virtqemud.conf set log_filters according to virt-launcher log Verbosity","text":"Verbosity level log_filters in virtqemud.conf 5 log_filters=\"3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 3:qemu.qemu_monitor 3:qemu.qemu_monitor_json 3:conf.domain_addr 1:*\" 6 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 3:qemu.qemu_monitor 1:* 7 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 1:* 8 and above 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:* <p>User can set self-defined log-filters via the annotations tag <code>kubevirt.io/libvirt-log-filters</code> in VMI configuration. e.g. <pre><code>kind: VirtualMachineInstance\nmetadata:\n  name: my-vmi\n  annotations:\n    kubevirt.io/libvirt-log-filters: \"3:remote 4:event 1:*\"\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#setting-verbosity-per-nodes","title":"Setting verbosity per nodes","text":"<p>Another way is to set verbosity level per node: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      nodeVerbosity:\n        \"node01\": 4\n        \"otherNodeName\": 6\n</code></pre></p> <p><code>nodeVerbosity</code> is essentially a map from string to int where the key is the node name and the value is the verbosity level. The verbosity level would be defined for all the different components in that node (e.g. <code>virt-handler</code>, <code>virt-launcher</code>, etc).</p>"},{"location":"debug_virt_stack/debug/#how-to-retrieve-kubevirt-components-logs","title":"How to retrieve KubeVirt components' logs","text":"<p>In Kubernetes, logs are defined at the Pod level. Therefore, first it's needed to list the Pods of KubeVirt's core components. In order to do that we can first list the Pods under KubeVirt's install namespace.</p> <p>For example: <pre><code>$&gt; kubectl get pods -n &lt;KubeVirt Install Namespace&gt;\nNAME                               READY   STATUS    RESTARTS   AGE\ndisks-images-provider-7gqbc        1/1     Running   0          32m\ndisks-images-provider-vg4kx        1/1     Running   0          32m\nvirt-api-57fcc4497b-7qfmc          1/1     Running   0          31m\nvirt-api-57fcc4497b-tx9nc          1/1     Running   0          31m\nvirt-controller-76c784655f-7fp6m   1/1     Running   0          30m\nvirt-controller-76c784655f-f4pbd   1/1     Running   0          30m\nvirt-handler-2m86x                 1/1     Running   0          30m\nvirt-handler-9qs6z                 1/1     Running   0          30m\nvirt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m\nvirt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m\n</code></pre></p> <p>Then, we can pick one of the pods and fetch its logs. For example: <pre><code>$&gt; kubectl logs -n &lt;KubeVirt Install Namespace&gt; virt-handler-2m86x | head -n8\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"set verbosity to 2\",\"pos\":\"virt-handler.go:453\",\"timestamp\":\"2022-04-17T08:58:37.373695Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"set verbosity to 2\",\"pos\":\"virt-handler.go:453\",\"timestamp\":\"2022-04-17T08:58:37.373726Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"setting rate limiter to 5 QPS and 10 Burst\",\"pos\":\"virt-handler.go:462\",\"timestamp\":\"2022-04-17T08:58:37.373782Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]\",\"pos\":\"cpu_plugin.go:96\",\"timestamp\":\"2022-04-17T08:58:37.390221Z\"}\n{\"component\":\"virt-handler\",\"level\":\"warning\",\"msg\":\"host model mode is expected to contain only one model\",\"pos\":\"cpu_plugin.go:103\",\"timestamp\":\"2022-04-17T08:58:37.390263Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"node-labeller is running\",\"pos\":\"node_labeller.go:94\",\"timestamp\":\"2022-04-17T08:58:37.391011Z\"}\n</code></pre></p> <p>Obviously, for both examples above, <code>&lt;KubeVirt Install Namespace&gt;</code> needs to be replaced with the actual namespace KubeVirt is installed in.</p>"},{"location":"debug_virt_stack/debug/#kubevirt-pprof-profiler","title":"KubeVirt PProf Profiler","text":"<p>Using the <code>cluster-profiler</code> client tool, a developer can get the PProf profiling data for every component in the Kubevirt Control plane. Here is a user guide:</p>"},{"location":"debug_virt_stack/debug/#compile-cluster-profiler","title":"Compile <code>cluster-profiler</code>","text":"<p>Build from source code <pre><code>$ git clone https://github.com/kubevirt/kubevirt.git\n$ cd kubevirt/tools/cluster-profiler\n$ go build\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#enable-the-feature-gate","title":"Enable the feature gate","text":"<p>Add <code>ClusterProfiler</code> in KubeVirt config <pre><code>$ cat &lt;&lt; END &gt; enable-feature-gate.yaml\n\n---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - ClusterProfiler\nEND\n\n$ kubectl apply -f enable-feature-gate.yaml\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#do-the-profiling","title":"Do the profiling","text":"<p>Start CPU profiling <pre><code>$ cluster-profiler --cmd start\n\n2023/05/17 09:31:09 SUCCESS: started cpu profiling KubeVirt control plane\n</code></pre> Stop CPU profiling <pre><code>$ cluster-profiler --cmd stop\n\n2023/05/17 09:31:14 SUCCESS: stopped cpu profiling KubeVirt control plane\n</code></pre> Dump the pprof result <pre><code>$ cluster-profiler --cmd dump\n\n2023/05/17 09:31:18 Moving already existing \"cluster-profiler-results\" =&gt; \"cluster-profiler-results-old-67fq\"\nSUCCESS: Dumped PProf 6 results for KubeVirt control plane to [cluster-profiler-results]\n</code></pre> The PProf result can be found in the folder <code>cluster-profiler-results</code> <pre><code>$ tree cluster-profiler-results\n\ncluster-profiler-results\n\u251c\u2500\u2500 virt-api-5f96f84dcb-lkpb7\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-controller-5bbd9554d9-2f8j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-controller-5bbd9554d9-qct2w\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-handler-ccq6c\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-operator-cdc677b7-pg9j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u2514\u2500\u2500 virt-operator-cdc677b7-pjqdx\n    \u251c\u2500\u2500 allocs.pprof\n    \u251c\u2500\u2500 block.pprof\n    \u251c\u2500\u2500 cpu.pprof\n    \u251c\u2500\u2500 goroutine.pprof\n    \u251c\u2500\u2500 heap.pprof\n    \u251c\u2500\u2500 mutex.pprof\n    \u2514\u2500\u2500 threadcreate.pprof\n</code></pre></p>"},{"location":"debug_virt_stack/launch-qemu-gdb/","title":"Launch QEMU with gdb and connect locally with gdb client","text":"<p>This guide is for cases where QEMU counters very early failures and it is hard to synchronize it in a later point in time.</p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#image-creation-and-pvc-population","title":"Image creation and PVC population","text":"<p>This scenario is a slight variation of the guide about starting strace, hence some of the details on the image build and the PVC population are simply skipped and explained in the other section.</p> <p>In this example, QEMU will be launched with <code>gdbserver</code> and later we will connect to it using a local <code>gdb</code> client.</p> <p>The wrapping script looks like: <pre><code>#!/bin/bash\n\nLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/gdbserver \\\n    localhost:1234 \\\n    /usr/libexec/qemu-kvm $@ &amp;\nprintf \"%d\" $(pgrep gdbserver) &gt; /run/libvirt/qemu/run/default_vmi-debug-tools.pid\n</code></pre></p> <p>First, we need to build and push the image with the wrapping script and the gdbserver: <pre><code>FROM quay.io/centos/centos:stream9 as build\n\nENV DIR /debug-tools\nENV DEBUGINFOD_URLS https://debuginfod.centos.org/\nRUN mkdir -p ${DIR}/logs\n\nRUN yum  install --installroot=${DIR} -y gdb-gdbserver &amp;&amp; yum clean all\n\nCOPY ./wrap_qemu_gdb.sh $DIR/wrap_qemu_gdb.sh\nRUN chmod 0755 ${DIR}/wrap_qemu_gdb.sh\nRUN chown 107:107 ${DIR}/wrap_qemu_gdb.sh\nRUN chown 107:107 ${DIR}/logs\n</code></pre></p> <p>Then, we can create and populate the <code>debug-tools</code> PVC as with did in the strace example: <pre><code>$ k apply -f debug-tools-pvc.yaml\npersistentvolumeclaim/debug-tools created\n$ kubectl  apply -f populate-job-pvc.yaml\njob.batch/populate-pvc created\n$ $ kubectl  get jobs\nNAME           COMPLETIONS   DURATION   AGE\npopulate-pvc   1/1           7s         2m12s\n</code></pre></p> <p>Configmap: <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;emulator&gt;/usr/libexec/qemu-kvm&lt;/emulator&gt;|&lt;emulator&gt;/var/run/debug/wrap_qemu_gdb.sh&lt;/emulator&gt;|\" $tempFile\n    cat $tempFile\n</code></pre></p> <p>As last step, we need to create the configmaps to modify the VM XML: <pre><code>$ kubectl apply -f configmap.yaml\nconfigmap/my-config-map created\n</code></pre></p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#build-client-image","title":"Build client image","text":"<p>In this scenario, we use an additional container image containing <code>gdb</code> and the same qemu binary as the target process to debug. This image will be run locally with <code>podman</code>.</p> <p>In order to build this image, we need to identify the image of the <code>virt-launcher</code> container we want to debug. Based on the KubeVirt installation, the namespace and the name of the KubeVirt CR could vary. In this example, we'll assume that KubeVirt CR is called <code>kubevirt</code> and installed in the <code>kubevirt</code> namespace.</p> <p>You can easily find out the right names in your cluster by searching with: <pre><code>$ kubectl get kubevirt -A\nNAMESPACE   NAME       AGE     PHASE\nkubevirt    kubevirt   3h11m   Deployed\n</code></pre></p> <p>The steps to build the image are:</p> <ol> <li> <p>Get the registry of the images of the KubeVirt installation: <pre><code>$ export registry=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.registry'|tr -d \"\\\"\")\n$ echo $registry\n\"registry:5000/kubevirt\"\n</code></pre></p> </li> <li> <p>Get the shasum of the virt-launcher image: <pre><code>$ export tag=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.virtLauncherSha'|tr -d \"\\\"\")\n$ echo $tag\n\"sha256:6c8b85eed8e83a4c70779836b246c057d3e882eb513f3ded0a02e0a4c4bda837\"\n</code></pre></p> </li> </ol> <p>Example of Dockerfile: <pre><code>ARG registry\nARG tag\nFROM ${registry}/kubevirt/virt-launcher${tag} AS launcher\nFROM quay.io/centos/centos:stream9 as build\n\nRUN yum  install -y gdb &amp;&amp; yum clean all\n\nCOPY --from=launcher /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm\n</code></pre></p> <ol> <li>Build the image by using the <code>registry</code> and the <code>tag</code> retrieved in the previous steps: <pre><code>$ podman build \\\n    -t gdb-client \\\n    --build-arg registry=$registry  \\\n    --build-arg tag=@$tag \\\n    -f Dockerfile.client .\n</code></pre></li> </ol> <p>Podman will replace the registry and tag arguments provided on the command line. In this way, we can specify the image registry and shasum for the KubeVirt version to debug.</p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#run-the-vm-to-troubleshoot","title":"Run the VM to troubleshoot","text":"<p>For this example, we add an annotation to keep the virt-launcher pod running even if any errors occur: <pre><code>metadata:\n  annotations:\n    kubevirt.io/keep-launcher-alive-after-failure: \"true\"\n</code></pre></p> <p>Then, we can launch the VM: <pre><code>$ kubectl apply -f debug-vmi.yaml\nvirtualmachineinstance.kubevirt.io/vmi-debug-tools created\n$ kubectl  get vmi\nNAME              AGE   PHASE       IP    NODENAME   READY\nvmi-debug-tools   28s   Scheduled         node01     False\n$ kubectl  get po\nNAME                                  READY   STATUS      RESTARTS   AGE\npopulate-pvc-dnxld                    0/1     Completed   0          4m17s\nvirt-launcher-vmi-debug-tools-tfh28   4/4     Running     0          25s\n</code></pre></p> <p>The wrapping script starts the <code>gdbserver</code> and expose in the port <code>1234</code> inside the container. In order to be able to connect remotely to the gdbserver, we can use the command <code>kubectl port-forward</code> to expose the gdb port on our machine.</p> <pre><code>$ kubectl  port-forward virt-launcher-vmi-debug-tools-tfh28 1234\nForwarding from 127.0.0.1:1234 -&gt; 1234\nForwarding from [::1]:1234 -&gt; 1234\n</code></pre> <p>Finally, we can start the gbd client in the container: <pre><code>$ podman run -ti --network host gdb-client:latest\n$ gdb /usr/libexec/qemu-kvm -ex 'target remote localhost:1234'\nGNU gdb (GDB) Red Hat Enterprise Linux 10.2-12.el9\nCopyright (C) 2021 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\nType \"show copying\" and \"show warranty\" for details.\nThis GDB was configured as \"x86_64-redhat-linux-gnu\".\nType \"show configuration\" for configuration details.\nFor bug reporting instructions, please see:\n&lt;https://www.gnu.org/software/gdb/bugs/&gt;.\nFind the GDB manual and other documentation resources online at:\n    &lt;http://www.gnu.org/software/gdb/documentation/&gt;.\n\nFor help, type \"help\".\n--Type &lt;RET&gt; for more, q to quit, c to continue without paging--\nType \"apropos word\" to search for commands related to \"word\"...\nReading symbols from /usr/libexec/qemu-kvm...\n\nReading symbols from /root/.cache/debuginfod_client/26221a84fabd219a68445ad0cc87283e881fda15/debuginfo...\nRemote debugging using localhost:1234\nReading /lib64/ld-linux-x86-64.so.2 from remote target...\nwarning: File transfers from remote targets can be slow. Use \"set sysroot\" to access files locally instead.\nReading /lib64/ld-linux-x86-64.so.2 from remote target...\nReading symbols from target:/lib64/ld-linux-x86-64.so.2...\nDownloading separate debug info for /system-supplied DSO at 0x7ffc10eff000...\n0x00007f1a70225e70 in _start () from target:/lib64/ld-linux-x86-64.so.2\n</code></pre></p> <p>For simplicity, we started podman with the option <code>--network host</code> in this way, the container is able to access any port mapped on the host.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/","title":"Launch QEMU with strace","text":"<p>This guide explains how launch QEMU with a debugging tool in virt-launcher pod. This method can be useful to debug early failures or starting QEMU as a child of the debug tool relying on ptrace. The second point is particularly relevant when a process is operating in a non-privileged environment since otherwise, it would need root access to be able to ptrace the process.</p> <p>Ephemeral containers are among the emerging techniques to overcome the lack of debugging tool inside the original image. This solution does, however, come with a number of limitations.  For example, it is possible to spawn a new container inside the same pod of the application to debug and share the same PID namespace. Though they share the same PID namespace, KubeVirt's usage of unprivileged containers makes it, for example, impossible to ptrace a running container. Therefore, this technique isn't appropriate for our needs.</p> <p>Due to its security and image size reduction, KubeVirt container images are based on distroless containers. These kinds of images are extremely beneficial for deployments, but they are challenging to troubleshoot because there is no package management, which prevents the installation of additional tools on the flight.</p> <p>Wrapping the QEMU binary in a script is one practical method for debugging QEMU launched by Libvirt. This script launches the QEMU as a child of this process together with the debugging tool (such as strace or valgrind).</p> <p>The final part that needs to be added is the configuration for Libvirt to use the wrapped script rather than calling the QEMU program directly.</p> <p>It is possible to alter the generated XML with the help of KubeVirt sidecars. This allows us to use the wrapping script in place of the built-in emulator.</p> <p>The primary concept behind this configuration is that all of the additional tools, scripts, and final output files will be stored in a PerstistentVolumeClaim (PVC) that this guide refers to as <code>debug-tools</code>. The virt-launcher pod that we wish to debug will have this PVC attached to it.</p> <p>PVC: <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: debug-tools\nspec:\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Filesystem\n  resources:\n    requests:\n      storage: 1Gi\n</code></pre></p> <p>In this guide, we'll apply the above concepts to debug QEMU inside virt-launcher using strace without the need of build a custom virt-launcher image.</p> <p>You can see a full demo of this setup: </p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-bring-the-debug-tools-and-wrapping-script-into-distroless-containers","title":"How to bring the debug tools and wrapping script into distroless containers","text":"<p>This section provides an example of how to provide extra tools into the distroless container that will be supplied as a PVC using a Dockerfile. Although there are several ways to accomplish this, this covers a relatively simple technique. Alternatively, you could run a pod and manually populate the PVC by execing into the pod.</p> <p>Dockerfile: <pre><code>FROM quay.io/centos/centos:stream9 as build\n\nENV DIR /debug-tools\nRUN mkdir -p ${DIR}/logs\n\nRUN  yum install  --installroot=${DIR} -y strace &amp;&amp; yum clean all\n\nCOPY ./wrap_qemu_strace.sh $DIR/wrap_qemu_strace.sh\nRUN chmod 0755 ${DIR}/wrap_qemu_strace.sh\nRUN chown 107:107 ${DIR}/wrap_qemu_strace.sh\nRUN chown 107:107 ${DIR}/logs\n</code></pre></p> <p>The directory <code>debug-tools</code> stores the content that will be later copied inside the <code>debug-tools</code> PVC. We are essentially adding the missing utilities in the custom directory with <code>yum install --installroot=${DIR}}</code>, and the parent image matches with the parent images of virt-launcher.</p> <p>The <code>wrap_qemu_strace.sh</code> is the wrapping script that will be used to launch QEMU with <code>strace</code> similarly as the example with <code>valgrind</code>. <pre><code>#!/bin/bash\n\nLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/strace \\\n        -o /var/run/debug/logs/strace.out \\\n        /usr/libexec/qemu-kvm $@\n</code></pre></p> <p>It is important to set the dynamic library path <code>LD_LIBRARY_PATH</code> to the path where the PVC will be mounted in the virt-launcher container.</p> <p>Then, you will simply need to build the image and your debug setup is ready. The Dockerfle and the script <code>wrap_qemu_strace.sh</code> need to be in the same directory where you run the command. <pre><code>$ podman build -t debug .\n</code></pre></p> <p>The second step is to populate the PVC. This can be easily achieved using a kubernetes <code>Job</code> like: <pre><code>apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: populate-pvc\nspec:\n  template:\n    spec:\n      volumes:\n        - name: populate\n          persistentVolumeClaim:\n            claimName: debug-tools\n      containers:\n        - name: populate\n          image: registry:5000/debug:latest\n          command: [\"sh\", \"-c\", \"cp -r /debug-tools/* /vol\"]\n          imagePullPolicy: Always\n          volumeMounts:\n            - mountPath: \"/vol\"\n              name: populate\n      restartPolicy: Never\n  backoffLimit: 4\n</code></pre></p> <p>The image referenced in the <code>Job</code> is the image we built in the previous step. Once applied this and the job completed, the<code>debug-tools</code> PVC is ready to be used.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-start-qemu-launched-by-a-debugging-tool-eg-strace","title":"How to start qemu launched by a debugging tool (e.g strace)","text":"<p>This part is achieved by using ConfigMaps and a KubeVirt sidecar (more details in the section Using ConfigMap to run custom script).</p> <p>Configmap: <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;emulator&gt;/usr/libexec/qemu-kvm&lt;/emulator&gt;|&lt;emulator&gt;/var/run/debug/wrap_qemu_strace.sh&lt;/emulator&gt;|\" $tempFile\n    cat $tempFile\n</code></pre></p> <p>The script that replaces the QEMU binary with the wrapping script in the XML is stored in the configmap <code>my-config-map</code>. This script will run as a hook, as explained in full in the documentation for the KubeVirt sidecar.</p> <p>Once all the objects created, we can finally run the guest to debug.</p> <p>VMI: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    hooks.kubevirt.io/hookSidecars: '[{\"args\": [\"--version\", \"v1alpha2\"],\n    \"image\":\"registry:5000/kubevirt/sidecar-shim:devel\",\n    \"pvc\": {\"name\": \"debug-tools\",\"volumePath\": \"/debug\", \"sharedComputePath\": \"/var/run/debug\"},\n    \"configMap\": {\"name\": \"my-config-map\",\"key\": \"my_script.sh\", \"hookPath\": \"/usr/bin/onDefineDomain\"}}]'\n  labels:\n    special: vmi-debug-tools\n  name: vmi-debug-tools\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p> <p>The VMI example is a simply VM instance declaration and the interesting parts are the annotations for the hook: * <code>image</code> refers to the sidecar-shim already built and shipped with KubeVirt * <code>pvc</code> refers to the PVC populated with the debug setup. The <code>name</code> refers to the claim name, the <code>volumePath</code> is the path inside the sidecar container where the volume is mounted while the <code>sharedComputePath</code> is the path of the same volume inside the compute container. * <code>configMap</code> refers to the confimap containing the script to modify the XML for the wrapping script</p> <p>Once the VM is declared, the hook will modify the emulator section and Libvirt will call the wrapping script instead of QEMU directly.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-fetch-the-output","title":"How to fetch the output","text":"<p>The wrapping script configures <code>strace</code> to store the output in the PVC. In this way, it is possible to retrieve the output file in a later time, for example using an additional pod like:</p> <pre><code>apiVersion: v1\nkind: Pod\nmetadata:\n  name: fetch-logs\nspec:\n  securityContext:\n    runAsUser: 107\n    fsGroup: 107\n  volumes:\n    - name: populate\n      persistentVolumeClaim:\n        claimName: debug-tools\n  containers:\n    - name: populate\n      image: busybox:latest\n      command: [\"tail\", \"-f\", \"/dev/null\"]\n      volumeMounts:\n        - mountPath: \"/vol\"\n          name: populate\n</code></pre> <p>It is then possible to copy the file locally with: <pre><code>$ kubectl cp fetch-logs:/vol/logs/strace.out strace.out\n</code></pre></p>"},{"location":"debug_virt_stack/logging/","title":"Control libvirt logging for each component","text":"<p>Generally, cluster admins can control the log verbosity of each KubeVirt component in KubeVirt CR. For more details, please, check the KubeVirt documentation.</p> <p>Nonetheless, regular users can also adjust the qemu component logging to have a finer control over it. The annotation <code>kubevirt.io/libvirt-log-filters</code> enables you to modify each component's log level.</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    kubevirt.io/libvirt-log-filters: \"2:qemu.qemu_monitor 3:*\"\n  labels:\n    special: vmi-debug-tools\n  name: vmi-debug-tools\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p> <p>Then, it is possible to obtain the logs from the virt-launcher output:</p> <pre><code>$ kubectl get pods\nNAME                                  READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-debug-tools-fk64q   3/3     Running   0          64s\n$ kubectl  logs virt-launcher-vmi-debug-tools-fk64q\n[..]\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324640, \\\"microseconds\\\": 523652}, \\\"event\\\": \\\"NIC_RX_FILTER_CHANGED\\\", \\\"data\\\": {\\\"name\\\": \\\"ua-default\\\", \\\"path\\\": \\\"/machine/peripheral/ua-default/virtio-backend\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:40.523000Z\"}\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324644, \\\"microseconds\\\": 165626}, \\\"event\\\": \\\"VSERPORT_CHANGE\\\", \\\"data\\\": {\\\"open\\\": true, \\\"id\\\": \\\"channel0\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:44.165000Z\"}\n[..]\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324646, \\\"microseconds\\\": 707666}, \\\"event\\\": \\\"RTC_CHANGE\\\", \\\"data\\\": {\\\"offset\\\": 0, \\\"qom-path\\\": \\\"/machine/unattached/device[8]\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:46.708000Z\"}\n[..]\n</code></pre> <p>The annotation enables the filter from the container creation. However, in certain cases you might desire to change the logging level dynamically once the container and libvirt have already been started. In this case, <code>virt-admin</code> comes to the rescue.</p> <p>Example: <pre><code>$ kubectl get pods\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters  \"1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util\"\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters\n Logging filters: 1:*libvirt* 1:*qemu* 1:*conf* 1:*security* 3:*event* 3:*json* 3:*file* 3:*object* 1:*util*\n</code></pre></p> <p>Otherwise, if you prefer to redirect the output to a file and fetch it later, you can rely on <code>kubectl cp</code> to retrieve the file. In this case, we are saving the file in the <code>/var/run/libvirt</code> directory because the compute container has the permissions to write there.</p> <p>Example: <pre><code>$ kubectl get pods\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session daemon-log-outputs \"1:file:/var/run/libvirt/libvirtd.log\"\n$ kubectl cp virt-launcher-vmi-ephemeral-nqcld:/var/run/libvirt/libvirtd.log libvirt-kubevirt.log\ntar: Removing leading `/' from member names\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/","title":"Privileged debugging on the node","text":"<p>This article describes the scenarios in which you can create privileged pods and have root access to the cluster nodes.</p> <p>With privileged pods, you may access devices in <code>/dev</code>, utilize host namespaces and ptrace processes that are running on the node, and use the <code>hostPath</code> volume to mount node directories in the container.</p> <p>A quick way to verify if you are allowed to create privileged pods is to create a sample pod with the <code>--dry-run=server</code> option, like:</p> <pre><code>$ kubectl apply -f debug-pod.yaml --dry-run=server\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#build-the-container-image","title":"Build the container image","text":"<p>KubeVirt uses distroless containers and those images don't have a package manager, for this reason it isn't possible to use the image as parent for installing additional packages.</p> <p>In certain debugging scenarios, the tools require to have exactly the same binary available. However, if the debug tools are operating in a different container, this can be especially difficult as the filesystems of the containers are isolated.</p> <p>This section will cover how to build a container image with the debug tools plus binaries of the KubeVirt version you want to debug.</p> <p>Based on your installation the namespace and the name of the KubeVirt CR could vary. In this example, we'll assume that KubeVirt CR is called <code>kubevirt</code> and installed in the <code>kubevirt</code> namespace. You can easily find out how it is called in your cluster by searching with <code>kubectl get kubevirt -A</code>. This is necessary as we need to retrieve the original <code>virt-launcher</code> image to have exactly the same QEMU binary we want to debug.</p> <p>Get the registry of the images of the KubeVirt installation: <pre><code>$ export registry=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.registry'|tr -d \"\\\"\")\n$ echo $registry\n\"registry:5000/kubevirt\"\n</code></pre></p> <p>Get the shasum of the virt-launcher image: <pre><code>$ export tag=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.virtLauncherSha'|tr -d \"\\\"\")\n$ echo $tag\n\"sha256:6c8b85eed8e83a4c70779836b246c057d3e882eb513f3ded0a02e0a4c4bda837\"\n</code></pre></p> <p>Dockerfile: <pre><code>ARG registry\nARG tag\nFROM ${registry}/kubevirt/virt-launcher${tag} AS launcher\n\nFROM quay.io/centos/centos:stream9\n\nRUN yum install -y \\\n        gdb \\\n        kernel-devel \\\n        qemu-kvm-tools \\\n        strace \\\n        systemtap-client \\\n        systemtap-devel \\\n    &amp;&amp; yum clean all\nCOPY --from=launcher / /\n</code></pre></p> <p>Then, we can build the image by using the <code>registry</code> and the <code>tag</code> retrieved in the previous steps: <pre><code>$ podman build \\\n    -t debug-tools \\\n    --build-arg registry=$registry  \\\n    --build-arg tag=@$tag \\\n    -f Dockerfile .\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/#deploy-the-privileged-debug-pod","title":"Deploy the privileged debug pod","text":"<p>This is an example that gives you a couple of suggestions how you can define your debugging pod:</p> <pre><code>kind: Pod\nmetadata:\n  name: node01-debug\nspec:\n  containers:\n  - command:\n    - /bin/sh\n    image: registry:5000/debug-tools:latest\n    imagePullPolicy: Always\n    name: debug\n    securityContext:\n      privileged: true\n      runAsUser: 0\n    stdin: true\n    stdinOnce: true\n    tty: true\n    volumeMounts:\n    - mountPath: /host\n      name: host\n    - mountPath: /usr/lib/modules\n      name: modules\n    - mountPath: /sys/kernel\n      name: sys-kernel\n  hostNetwork: true\n  hostPID: true\n  nodeName: node01\n  restartPolicy: Never\n  volumes:\n  - hostPath:\n      path: /\n      type: Directory\n    name: host\n  - hostPath:\n      path: /usr/lib/modules\n      type: Directory\n    name: modules\n  - hostPath:\n      path: /sys/kernel\n      type: Directory\n    name: sys-kernel\n</code></pre> <p>The <code>privileged</code> option is required to have access to mostly all the resources on the node.</p> <p>The <code>nodeName</code> ensures that the debugging pod will be scheduled on the desired node. In order to select the right now, you can use the <code>-owide</code> option with <code>kubectl get po</code> and this will report the nodes where the pod is running.</p> <p>Example: <pre><code> k get pods -owide\nNAME                                READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES\nlocal-volume-provisioner-4jtkb      1/1     Running   0          152m    10.244.196.129   node01   &lt;none&gt;           &lt;none&gt;\nnode01-debug                        1/1     Running   0          44m     192.168.66.101   node01   &lt;none&gt;           &lt;none&gt;\nvirt-launcher-vmi-ephemeral-xg98p   3/3     Running   0          2m54s   10.244.196.148   node01   &lt;none&gt;           1/1\n</code></pre></p> <p>In the <code>volumes</code> section, you can specify the directories you want to be directly mounted in the debugging container. For example, <code>/usr/lib/modules</code> is particularly useful if you need to load some kernel modules.</p> <p>Sharing the host pid namespace with the option <code>hostPID</code> allows you to see all the processes on the node and attach to it with tools like <code>gdb</code> and <code>strace</code>.</p> <p><code>exec</code>-ing into the pod gives you a shell with  privileged access to the node plus the tooling you installed into the image:</p> <pre><code>$ kubectl exec -ti debug -- bash\n</code></pre> <p>The following examples assume you have already execed into the <code>node01-debug</code> pod.</p>"},{"location":"debug_virt_stack/privileged-node-debugging/#validating-the-host-for-virtualization","title":"Validating the host for virtualization","text":"<p>The tool <code>vist-host-validate</code> is utility to validate the host to run libvirt hypervisor. This, for example, can be used to check if a particular node is kvm capable.</p> <p>Example: <pre><code>$  virt-host-validate\n  QEMU: Checking for hardware virtualization                                 : PASS\n  QEMU: Checking if device /dev/kvm exists                                   : PASS\n  QEMU: Checking if device /dev/kvm is accessible                            : PASS\n  QEMU: Checking if device /dev/vhost-net exists                             : PASS\n  QEMU: Checking if device /dev/net/tun exists                               : PASS\n  QEMU: Checking for cgroup 'cpu' controller support                         : PASS\n  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS\n  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS\n  QEMU: Checking for cgroup 'memory' controller support                      : PASS\n  QEMU: Checking for cgroup 'devices' controller support                     : PASS\n  QEMU: Checking for cgroup 'blkio' controller support                       : PASS\n  QEMU: Checking for device assignment IOMMU support                         : PASS\n  QEMU: Checking if IOMMU is enabled by kernel                               : PASS\n  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/#run-a-command-directly-on-the-node","title":"Run a command directly on the node","text":"<p>The debug container has in the volume section the host filesystem mounted under <code>/host</code>. This can be particularly useful if you want to access the node filesystem or execute a command directly on the host. However, the tool needs already to be present on the node.</p> <pre><code># chroot /host\nsh-5.1# cat /etc/os-release\nNAME=\"CentOS Stream\"\nVERSION=\"9\"\nID=\"centos\"\nID_LIKE=\"rhel fedora\"\nVERSION_ID=\"9\"\nPLATFORM_ID=\"platform:el9\"\nPRETTY_NAME=\"CentOS Stream 9\"\nANSI_COLOR=\"0;31\"\nLOGO=\"fedora-logo-icon\"\nCPE_NAME=\"cpe:/o:centos:centos:9\"\nHOME_URL=\"https://centos.org/\"\nBUG_REPORT_URL=\"https://bugzilla.redhat.com/\"\nREDHAT_SUPPORT_PRODUCT=\"Red Hat Enterprise Linux 9\"\nREDHAT_SUPPORT_PRODUCT_VERSION=\"CentOS Stream\"\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#attach-to-a-running-process-eg-strace-or-gdb","title":"Attach to a running process (e.g strace or gdb)","text":"<p>This requires the field <code>hostPID: true</code> in this way you are able to list all the processes running on the node.</p> <pre><code>$ ps -ef |grep qemu-kvm\nqemu       50122   49850  0 12:34 ?        00:00:25 /usr/libexec/qemu-kvm -name guest=default_vmi-ephemeral,debug-threads=on -S -object {\"qom-type\":\"secret\",\"id\":\"masterKey0\",\"format\":\"raw\",\"file\":\"/var/run/kubevirt-private/libvirt/qemu/lib/domain-1-default_vmi-ephemera/master-key.aes\"} -machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on -accel kvm -cpu Skylake-Client-IBRS,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,xsaves=on,pdpe1gb=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,fb-clear=on,hle=off,rtm=off -m size=131072k -object {\"qom-type\":\"memory-backend-ram\",\"id\":\"pc.ram\",\"size\":134217728} -overcommit mem-lock=off -smp 1,sockets=1,dies=1,cores=1,threads=1 -object {\"qom-type\":\"iothread\",\"id\":\"iothread1\"} -uuid b56f06f0-07e9-4fe5-8913-18a14e83a4d1 -smbios type=1,manufacturer=KubeVirt,product=None,uuid=b56f06f0-07e9-4fe5-8913-18a14e83a4d1,family=KubeVirt -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=21,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device {\"driver\":\"pcie-root-port\",\"port\":16,\"chassis\":1,\"id\":\"pci.1\",\"bus\":\"pcie.0\",\"multifunction\":true,\"addr\":\"0x2\"} -device {\"driver\":\"pcie-root-port\",\"port\":17,\"chassis\":2,\"id\":\"pci.2\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x1\"} -device {\"driver\":\"pcie-root-port\",\"port\":18,\"chassis\":3,\"id\":\"pci.3\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x2\"} -device {\"driver\":\"pcie-root-port\",\"port\":19,\"chassis\":4,\"id\":\"pci.4\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x3\"} -device {\"driver\":\"pcie-root-port\",\"port\":20,\"chassis\":5,\"id\":\"pci.5\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x4\"} -device {\"driver\":\"pcie-root-port\",\"port\":21,\"chassis\":6,\"id\":\"pci.6\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x5\"} -device {\"driver\":\"pcie-root-port\",\"port\":22,\"chassis\":7,\"id\":\"pci.7\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x6\"} -device {\"driver\":\"pcie-root-port\",\"port\":23,\"chassis\":8,\"id\":\"pci.8\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x7\"} -device {\"driver\":\"pcie-root-port\",\"port\":24,\"chassis\":9,\"id\":\"pci.9\",\"bus\":\"pcie.0\",\"addr\":\"0x3\"} -device {\"driver\":\"virtio-scsi-pci-non-transitional\",\"id\":\"scsi0\",\"bus\":\"pci.5\",\"addr\":\"0x0\"} -device {\"driver\":\"virtio-serial-pci-non-transitional\",\"id\":\"virtio-serial0\",\"bus\":\"pci.6\",\"addr\":\"0x0\"} -blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt/container-disks/disk_0.img\",\"node-name\":\"libvirt-2-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"} -blockdev {\"node-name\":\"libvirt-2-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-2-storage\"} -blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-ephemeral-disks/disk-data/containerdisk/disk.qcow2\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"} -blockdev {\"node-name\":\"libvirt-1-format\",\"read-only\":false,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":\"libvirt-2-format\"} -device {\"driver\":\"virtio-blk-pci-non-transitional\",\"bus\":\"pci.7\",\"addr\":\"0x0\",\"drive\":\"libvirt-1-format\",\"id\":\"ua-containerdisk\",\"bootindex\":1,\"write-cache\":\"on\",\"werror\":\"stop\",\"rerror\":\"stop\"} -netdev {\"type\":\"tap\",\"fd\":\"22\",\"vhost\":true,\"vhostfd\":\"24\",\"id\":\"hostua-default\"} -device {\"driver\":\"virtio-net-pci-non-transitional\",\"host_mtu\":1480,\"netdev\":\"hostua-default\",\"id\":\"ua-default\",\"mac\":\"7e:cb:ba:c3:71:88\",\"bus\":\"pci.1\",\"addr\":\"0x0\",\"romfile\":\"\"} -add-fd set=0,fd=20,opaque=serial0-log -chardev socket,id=charserial0,fd=18,server=on,wait=off,logfile=/dev/fdset/0,logappend=on -device {\"driver\":\"isa-serial\",\"chardev\":\"charserial0\",\"id\":\"serial0\",\"index\":0} -chardev socket,id=charchannel0,fd=19,server=on,wait=off -device {\"driver\":\"virtserialport\",\"bus\":\"virtio-serial0.0\",\"nr\":1,\"chardev\":\"charchannel0\",\"id\":\"channel0\",\"name\":\"org.qemu.guest_agent.0\"} -audiodev {\"id\":\"audio1\",\"driver\":\"none\"} -vnc vnc=unix:/var/run/kubevirt-private/3a8f7774-7ec7-4cfb-97ce-581db52ee053/virt-vnc,audiodev=audio1 -device {\"driver\":\"VGA\",\"id\":\"video0\",\"vgamem_mb\":16,\"bus\":\"pcie.0\",\"addr\":\"0x1\"} -global ICH9-LPC.noreboot=off -watchdog-action reset -device {\"driver\":\"virtio-balloon-pci-non-transitional\",\"id\":\"balloon0\",\"free-page-reporting\":true,\"bus\":\"pci.8\",\"addr\":\"0x0\"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on\n$ gdb -p 50122 /usr/libexec/qemu-kvm\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#debugging-using-crictl","title":"Debugging using <code>crictl</code>","text":"<p><code>Crictl</code> is a cli for CRI runtimes and can be particularly useful to troubleshoot container failures (for a more detailed guide, please refer to this Kubernetes article).</p> <p>In this example, we'll concentrate to find where libvirt creates the files and directory in the <code>compute</code> container of the virt-launcher pod.</p> <pre><code>$ crictl ps |grep compute\n67bc7be3222da       5ef5ba25a087a80e204f28be6c9250bbf378fd87fa927085abd516188993d695                                                       25 minutes ago      Running             compute                   0                   7b045ea9f485f       virt-launcher-vmi-ephemeral-xg98p\n$ crictl inspect 67bc7be3222da\n[..]\n    \"mounts\": [\n      {\n      {\n        \"containerPath\": \"/var/run/libvirt\",\n        \"hostPath\": \"/var/lib/kubelet/pods/2ccc3e93-d1c3-4f22-bb31-321bfa74edf6/volumes/kubernetes.io~empty-dir/libvirt-runtime\",\n        \"propagation\": \"PROPAGATION_PRIVATE\",\n        \"readonly\": false,\n        \"selinuxRelabel\": true\n      },\n[..]\n$ ls /var/lib/kubelet/pods/2ccc3e93-d1c3-4f22-bb31-321bfa74edf6/volumes/kubernetes.io~empty-dir/libvirt-runtime/\ncommon      qemu         virtlogd-sock  virtqemud-admin-sock  virtqemud.conf\nhostdevmgr  virtlogd-admin-sock  virtlogd.pid   virtqemud-sock        virtqemud.pid\n</code></pre>"},{"location":"debug_virt_stack/virsh-commands/","title":"Execute virsh commands in virt-launcher pod","text":"<p>A powerful utility to check and troubleshoot the VM state is <code>virsh</code> and the utility is already installed in the <code>compute</code> container on the virt-launcher pod.</p> <p>For example, it possible to run any QMP commands.</p> <p>For a full list of QMP command, please refer to the QEMU documentation.</p> <pre><code>$ kubectl get po\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-xg98p   3/3     Running   0          44m\n$ kubectl exec -ti virt-launcher-vmi-debug-tools-fk64q -- bash\nbash-5.1$ virsh list\n Id   Name                      State\n-----------------------------------------\n 1    default_vmi-debug-tools   running\nbash-5.1$ virsh qemu-monitor-command default_vmi-debug-tools query-status --pretty\n{\n  \"return\": {\n    \"status\": \"running\",\n    \"singlestep\": false,\n    \"running\": true\n  },\n  \"id\": \"libvirt-439\"\n}\n$ virsh qemu-monitor-command default_vmi-debug-tools query-kvm --pretty\n{\n  \"return\": {\n    \"enabled\": true,\n    \"present\": true\n  },\n  \"id\": \"libvirt-438\"\n}\n</code></pre> <p>Another useful virsh command is the <code>qemu-monitor-event</code>. Once invoked, it observes and reports the QEMU events.</p> <p>The following example shows the events generated for pausing and unpausing the guest.</p> <pre><code>$ kubectl get po\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          57m\n$ kubectl exec -ti virt-launcher-vmi-ephemeral-nqcld -- virsh qemu-monitor-event --pretty --loop\n</code></pre> <p>Then, you can, for example, pause and then unpause the guest and check the triggered events: <pre><code>$ virtctl pause vmi vmi-ephemeral\nVMI vmi-ephemeral was scheduled to pause\n $ virtctl unpause vmi vmi-ephemeral\nVMI vmi-ephemeral was scheduled to unpause\n</code></pre></p> <p>From the monitored events: <pre><code>$ kubectl exec -ti virt-launcher-vmi-ephemeral-nqcld -- virsh qemu-monitor-event --pretty --loop\nevent STOP at 1698405797.422823 for domain 'default_vmi-ephemeral': &lt;null&gt;\nevent RESUME at 1698405823.162458 for domain 'default_vmi-ephemeral': &lt;null&gt;\n</code></pre></p>"},{"location":"network/dns/","title":"DNS records","text":"<p>In order to create unique DNS records per VirtualMachineInstance, it is possible to set <code>spec.hostname</code> and <code>spec.subdomain</code>. If a subdomain is set and a headless service with a name, matching the subdomain, exists, kube-dns will create unique DNS entries for every VirtualMachineInstance which matches the selector of the service. Have a look at the DNS for Services and Pods documentation for additional information.</p> <p>The following example consists of a VirtualMachine and a headless Service which matches the labels and the subdomain of the VirtualMachineInstance:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vmi-fedora\n  labels:\n    expose: me\nspec:\n  hostname: \"myvmi\"\n  subdomain: \"mysubdomain\"\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-registry-disk-demo:latest\n  - cloudInitNoCloud:\n      userDataBase64: IyEvYmluL2Jhc2gKZWNobyAiZmVkb3JhOmZlZG9yYSIgfCBjaHBhc3N3ZAo=\n    name: cloudinitdisk\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: mysubdomain\nspec:\n  selector:\n    expose: me\n  clusterIP: None\n  ports:\n  - name: foo # Actually, no port is needed.\n    port: 1234\n    targetPort: 1234\n</code></pre> <p>As a consequence, when we enter the VirtualMachineInstance via e.g. <code>virtctl console vmi-fedora</code> and ping <code>myvmi.mysubdomain</code> we see that we find a DNS entry for <code>myvmi.mysubdomain.default.svc.cluster.local</code> which points to <code>10.244.0.57</code>, which is the IP of the VirtualMachineInstance (not of the Service):</p> <pre><code>[fedora@myvmi ~]$ ping myvmi.mysubdomain\nPING myvmi.mysubdomain.default.svc.cluster.local (10.244.0.57) 56(84) bytes of data.\n64 bytes from myvmi.mysubdomain.default.svc.cluster.local (10.244.0.57): icmp_seq=1 ttl=64 time=0.029 ms\n[fedora@myvmi ~]$ ip a\n2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000\n    link/ether 0a:58:0a:f4:00:39 brd ff:ff:ff:ff:ff:ff\n    inet 10.244.0.57/24 brd 10.244.0.255 scope global dynamic eth0\n       valid_lft 86313556sec preferred_lft 86313556sec\n    inet6 fe80::858:aff:fef4:39/64 scope link\n       valid_lft forever preferred_lft forever\n</code></pre> <p>So <code>spec.hostname</code> and <code>spec.subdomain</code> get translated to a DNS A-record of the form <code>&lt;vmi.spec.hostname&gt;.&lt;vmi.spec.subdomain&gt;.&lt;vmi.metadata.namespace&gt;.svc.cluster.local</code>. If no <code>spec.hostname</code> is set, then we fall back to the VirtualMachineInstance name itself. The resulting DNS A-record looks like this then: <code>&lt;vmi.metadata.name&gt;.&lt;vmi.spec.subdomain&gt;.&lt;vmi.metadata.namespace&gt;.svc.cluster.local</code>.</p>"},{"location":"network/hotplug_interfaces/","title":"Hotplug Network Interfaces","text":"<p>Release: - v1.1.0: Alpha - v1.3.0: Beta</p> <p>KubeVirt supports hotplugging and unplugging network interfaces into a running Virtual Machine (VM). </p> <p>Hotplug is supported for interfaces using the <code>virtio</code> model connected through bridge binding  or SR-IOV binding.</p> <p>Hot-unplug is supported only for interfaces connected through bridge binding.</p>"},{"location":"network/hotplug_interfaces/#requirements","title":"Requirements","text":"<p>Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:</p> <ul> <li>Multus Dynamic Networks Controller:   this daemon will listen to annotation changes, and trigger Multus to configure   a new attachment for this pod.</li> <li>Multus CNI running as a thick plugin:   this Multus version exposes an endpoint to create attachments for a given pod   on demand.</li> </ul>"},{"location":"network/hotplug_interfaces/#enabling-network-interface-hotplug-support","title":"Enabling network interface hotplug support","text":"<p>Network interface hotplug support must be enabled via a feature gate. The feature gates array in the KubeVirt CR must feature <code>HotplugNICs</code>.</p>"},{"location":"network/hotplug_interfaces/#adding-an-interface-to-a-running-vm","title":"Adding an interface to a running VM","text":"<p>First start a VM. You can refer to the following example: <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          interfaces:\n          - masquerade: {}\n            name: defaultnetwork\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: defaultnetwork\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n</code></pre></p> <p>You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one: <pre><code>apiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: new-fancy-net\nspec:\n    config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"type\": \"bridge\",\n      \"mtu\": 1300,\n      \"name\":\"new-fancy-net\"\n    }'\n</code></pre> Please refer to the Multus documentation for more information.</p> <p>Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation by  editing the VM spec template and adding the desired interface and network: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: dyniface1\n          bridge: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: dyniface1\n      multus:\n        networkName: new-fancy-net\n ...\n</code></pre></p> <p>Note: <code>virtctl</code> <code>addinterface</code> and <code>removeinterface</code> commands are no longer available, hotplug/unplug interfaces is done by editing the VM spec template.</p> <p>The interface and network will be added to the corresponding VMI object as well by Kubevirt.</p> <p>You can now check the VMI status for the presence of this new interface: <pre><code>kubectl get vmi vm-fedora -ojsonpath=\"{ @.status.interfaces }\"\n</code></pre></p>"},{"location":"network/hotplug_interfaces/#removing-an-interface-from-a-running-vm","title":"Removing an interface from a running VM","text":"<p>Following the example above, the user can request an interface unplug operation by editing the VM spec template and set the desired interface state to <code>absent</code>: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n          - name: defaultnetwork\n            masquerade: {}\n          # set the interface state to absent \n          - name: dyniface1\n            state: absent\n            bridge: {}\n    networks:\n      - name: defaultnetwork\n        pod: {}\n      - name: dyniface1\n        multus:\n          networkName: new-fancy-net\n</code></pre> The interface in the corresponding VMI object will be set with state 'absent' as well by Kubevirt.</p> <p>Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.</p>"},{"location":"network/hotplug_interfaces/#migration-based-hotplug","title":"Migration based hotplug","text":"<p>In case your cluster doesn't run Multus as  thick plugin  and  Multus Dynamic Networks controller,  it's possible to hotplug an interface by migrating the VM.</p> <p>The actual attachment won't take place immediately, and the new interface will be available in the guest once the migration is completed.</p>"},{"location":"network/hotplug_interfaces/#add-new-interface","title":"Add new interface","text":"<p>Add the desired interface and network to the VM spec template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: dyniface1\n          bridge: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: dyniface1\n      multus:\n        networkName: new-fancy-net\n ...\n</code></pre></p> <p>At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the migration is completed the VM will have the new interface attached.</p> <p>Note: It is recommended to avoid performing migrations in parallel to a hotplug operation.  It is safer to assure hotplug succeeded or at least reached the VMI specification before issuing a migration.</p>"},{"location":"network/hotplug_interfaces/#remove-interface","title":"Remove interface","text":"<p>Set the desired interface state to <code>absent</code> in the VM spec template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n          - name: defaultnetwork\n            masquerade: {}\n          # set the interface state to absent \n          - name: dyniface1\n            state: absent\n            bridge: {}\n    networks:\n      - name: defaultnetwork\n        pod: {}\n      - name: dyniface1\n        multus:\n          networkName: new-fancy-net\n</code></pre></p> <p>At this point the subject interface should be detached from the guest but exist in the pod.</p> <p>Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm_1","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the VM is migrated, the interface will not exist in the migration target pod.</p> <p>Note: It is recommended to avoid performing migrations in parallel to an unplug operation. It is safer to assure unplug succeeded or at least reached the VMI specification before issuing a migration.</p>"},{"location":"network/hotplug_interfaces/#sr-iov-interfaces","title":"SR-IOV interfaces","text":"<p>Kubevirt supports hot-plugging of SR-IOV interfaces to running VMs.</p> <p>Similar to bridge binding interfaces, edit the VM spec template and add the desired SR-IOV interface and network: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: sriov-net\n          sriov: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: sriov-net\n      multus:\n        networkName: sriov-net-1\n ...\n</code></pre> Please refer to the Interface and Networks documentation for more information about SR-IOV networking.</p> <p>At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm_2","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the VM is migrated, the interface will not exist in the migration target pod. Due to limitation of Kubernetes device plugin API to allocate resources dynamically, the SR-IOV device plugin cannot allocate additional SR-IOV resources for Kubevirt to hotplug. Thus, SR-IOV interface hotplug is limited to migration based hotplug only, regardless of Multus \"thick\" version.</p>"},{"location":"network/hotplug_interfaces/#virtio-limitations","title":"Virtio Limitations","text":"<p>The hotplugged interfaces have <code>model: virtio</code>. This imposes several limitations: each interface will consume a PCI slot in the VM, and there are a total maximum of 32. Furthermore, other devices will also use these PCI slots (e.g. disks, guest-agent, etc).</p> <p>Kubevirt reserves resources for 4 interface to allow later hotplug operations. The actual maximum amount of available resources depends on the machine type (e.g. q35 adds another PCI slot). For more information on maximum limits, see libvirt documentation.</p> <p>Yet, upon a VM restart, the hotplugged interface will become part of the standard networks; this mitigates the maximum hotplug interfaces (per machine type) limitation.</p> <p>Note: The user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.</p>"},{"location":"network/interfaces_and_networks/","title":"Interfaces and Networks","text":"<p>Connecting a virtual machine to a network consists of two parts. First, networks are specified in <code>spec.networks</code>. Then, interfaces backed by the networks are added to the VM by specifying them in <code>spec.domain.devices.interfaces</code>.</p> <p>Each interface must have a corresponding network with the same name.</p> <p>An <code>interface</code> defines a virtual network interface of a virtual machine (also called a frontend). A <code>network</code> specifies the backend of an <code>interface</code> and declares which logical or physical device it is connected to (also called as backend).</p> <p>There are multiple ways of configuring an <code>interface</code> as well as a <code>network</code>.</p> <p>All possible configuration options are available in the Interface API Reference and Network API Reference.</p>"},{"location":"network/interfaces_and_networks/#backend","title":"Backend","text":"<p>Network backends are configured in <code>spec.networks</code>. A network must have a unique name. Additional fields declare which logical or physical device the network relates to.</p> <p>Each network should declare its type by defining one of the following fields:</p> Type Description <p><code>pod</code></p> <p>Default Kubernetes network</p> <p><code>multus</code></p> <p>Secondary network provided using Multus</p>"},{"location":"network/interfaces_and_networks/#pod","title":"pod","text":"<p>A <code>pod</code> network represents the default pod <code>eth0</code> interface configured by cluster network solution that is present in each pod.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n  networks:\n  - name: default\n    pod: {} # Stock pod network\n</code></pre>"},{"location":"network/interfaces_and_networks/#multus","title":"multus","text":"<p>It is also possible to connect VMIs to secondary networks using Multus. This assumes that multus is installed across your cluster and a corresponding <code>NetworkAttachmentDefinition</code> CRD was created.</p> <p>The following example defines a network which uses the bridge CNI plugin, which will connect the VMI to Linux bridge <code>br1</code>. Other CNI plugins such as ptp, ovs-cni, or Flannel might be used as well. For their installation and usage refer to the respective project documentation.</p> <p>First the <code>NetworkAttachmentDefinition</code> needs to be created. That is usually done by an administrator. Users can then reference the definition.</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: bridge-test\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"bridge-test\",\n      \"type\": \"bridge\",\n      \"bridge\": \"br1\",\n      \"disableContainerInterface\": true\n    }'\n</code></pre> <p>With following definition, the VMI will be connected to the default pod network and to the secondary Open vSwitch network.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n          bootOrder: 1   # attempt to boot from an external tftp server\n          dhcpOptions:\n            bootFileName: default_image.bin\n            tftpServerName: tftp.example.com\n        - name: ovs-net\n          bridge: {}\n          bootOrder: 2   # if first attempt failed, try to PXE-boot from this L2 networks\n  networks:\n  - name: default\n    pod: {} # Stock pod network\n  - name: ovs-net\n    multus: # Secondary multus network\n      networkName: ovs-vlan-100\n</code></pre> <p>It is also possible to define a multus network as the default pod network with Multus. A version of multus after this Pull Request is required (currently master).</p> <p>Note the following:</p> <ul> <li> <p>A multus default network and a pod network type are mutually     exclusive.</p> </li> <li> <p>The virt-launcher pod that starts the VMI will not have the pod     network configured.</p> </li> <li> <p>The multus delegate chosen as default must return at least one     IP address.</p> </li> </ul> <p>Create a <code>NetworkAttachmentDefinition</code> with IPAM.</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: bridge-test\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"bridge-test\",\n      \"type\": \"bridge\",\n      \"bridge\": \"br1\",\n      \"ipam\": {\n        \"type\": \"host-local\",\n        \"subnet\": \"10.250.250.0/24\"\n      }\n    }'\n</code></pre> <p>Define a VMI with a Multus network as the default.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: test1\n          bridge: {}\n  networks:\n  - name: test1\n    multus: # Multus network as default\n      default: true\n      networkName: bridge-test\n</code></pre>"},{"location":"network/interfaces_and_networks/#invalid-cnis-for-secondary-networks","title":"Invalid CNIs for secondary networks","text":"<p>The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces.</p> <ul> <li> <p>macvlan</p> </li> <li> <p>ipvlan</p> </li> </ul> <p>The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address. The aforementioned CNIs require the pod interface to have the original MAC address.</p> <p>These issues are tracked individually:</p> <ul> <li> <p>macvlan</p> </li> <li> <p>ipvlan</p> </li> </ul> <p>Feel free to discuss and / or propose fixes for them; we'd like to have these plugins as valid options on our ecosystem.</p>"},{"location":"network/interfaces_and_networks/#frontend","title":"Frontend","text":"<p>Network interfaces are configured in <code>spec.domain.devices.interfaces</code>. They describe properties of virtual interfaces as \"seen\" inside guest instances. The same network backend may be connected to a virtual machine in multiple different ways, each with their own connectivity guarantees and characteristics.</p> <p>Each interface should declare its type by defining on of the following fields:</p> Type Description <p><code>bridge</code></p> <p>Connect using a linux bridge</p> <p><code>slirp</code></p> <p>Connect using QEMU user networking mode</p> <p><code>sriov</code></p> <p>Pass through a SR-IOV PCI device via <code>vfio</code></p> <p><code>masquerade</code></p> <p>Connect using Iptables rules to nat the traffic</p> <p>Each interface may also have additional configuration fields that modify properties \"seen\" inside guest instances, as listed below:</p> Name Format Default value Description <p><code>model</code></p> <p>One of: <code>e1000</code>, <code>e1000e</code>, <code>ne2k_pci</code>, <code>pcnet</code>, <code>rtl8139</code>, <code>virtio</code></p> <p><code>virtio</code></p> <p>NIC type</p> <p>macAddress</p> <p><code>ff:ff:ff:ff:ff:ff</code> or <code>FF-FF-FF-FF-FF-FF</code></p> <p>MAC address as seen inside the guest system, for example: <code>de:ad:00:00:be:af</code></p> <p>ports</p> <p>empty</p> <p>List of ports to be forwarded to the virtual machine.</p> <p>pciAddress</p> <p><code>0000:81:00.1</code></p> <p>Set network interface PCI address, for example: <code>0000:81:00.1</code></p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          model: e1000 # expose e1000 NIC to the guest\n          masquerade: {} # connect through a masquerade\n          ports:\n           - name: http\n             port: 80\n  networks:\n  - name: default\n    pod: {}\n</code></pre> <p>Note: For secondary interfaces, when a MAC address is specified for a virtual machine interface, it is passed to the underlying CNI plugin which is, in turn, expected to configure the backend to allow for this particular MAC. Not every plugin has native support for custom MAC addresses.</p> <p>Note: For some CNI plugins without native support for custom MAC addresses, there is a workaround, which is to use the <code>tuning</code> CNI plugin to adjust pod interface MAC address. This can be used as follows:</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: ptp-mac\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"ptp-mac\",\n      \"plugins\": [\n        {\n          \"type\": \"ptp\",\n          \"ipam\": {\n            \"type\": \"host-local\",\n            \"subnet\": \"10.1.1.0/24\"\n          }\n        },\n        {\n          \"type\": \"tuning\"\n        }\n      ]\n    }'\n</code></pre> <p>This approach may not work for all plugins. For example, OKD SDN is not compatible with <code>tuning</code> plugin.</p> <ul> <li> <p>Plugins that handle custom MAC addresses natively: <code>ovs</code>, <code>bridge</code>.</p> </li> <li> <p>Plugins that are compatible with <code>tuning</code> plugin: <code>flannel</code>, <code>ptp</code>.</p> </li> <li> <p>Plugins that don't need special MAC address treatment: <code>sriov</code> (in     <code>vfio</code> mode). </p> </li> </ul>"},{"location":"network/interfaces_and_networks/#ports","title":"Ports","text":"<p>Declare ports listen by the virtual machine</p> <p>Note: When using the slirp interface only the configured ports will be forwarded to the virtual machine.</p> Name Format Required Description <p><code>name</code></p> <p>no</p> <p>Name</p> <p><code>port</code></p> <p>1 - 65535</p> <p>yes</p> <p>Port to expose</p> <p><code>protocol</code></p> <p>TCP,UDP</p> <p>no</p> <p>Connection protocol</p> <p>Tip: Use <code>e1000</code> model if your guest image doesn't ship with virtio drivers.</p> <p>If <code>spec.domain.devices.interfaces</code> is omitted, the virtual machine is connected using the default pod network interface of <code>bridge</code> type. If you'd like to have a virtual machine instance without any network connectivity, you can use the <code>autoattachPodInterface</code> field as follows:</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      autoattachPodInterface: false\n</code></pre>"},{"location":"network/interfaces_and_networks/#mtu","title":"MTU","text":"<p>There are two methods for the MTU to be propagated to the guest interface.</p> <ul> <li>Libvirt - for this the guest machine needs new enough virtio network driver that understands the data passed into the guest via a PCI config register in the emulated device.</li> <li>DHCP - for this the guest DHCP client should be able to read the MTU from the DHCP server response.</li> </ul> <p>On Windows guest non virtio interfaces, MTU has to be set manually using <code>netsh</code> or other tool since the Windows DHCP client doesn't request/read the MTU.</p> <p>The table below is summarizing the MTU propagation to the guest.</p> masquerade bridge with CNI IP bridge with no CNI IP Windows virtio DHCP &amp; libvirt DHCP &amp; libvirt libvirt libvirt non-virtio DHCP DHCP X X <ul> <li>bridge with CNI IP - means the CNI gives IP to the pod interface and bridge binding is used to bind the pod interface to the guest.</li> </ul>"},{"location":"network/interfaces_and_networks/#bridge","title":"bridge","text":"<p>In <code>bridge</code> mode, virtual machines are connected to the network backend through a linux \"bridge\". The pod network IPv4 address (if exists) is delegated to the virtual machine via DHCPv4. The virtual machine should be configured to use DHCP to acquire IPv4 addresses.</p> <p>Note: If a specific MAC address is not configured in the virtual machine interface spec the MAC address from the relevant pod interface is delegated to the virtual machine.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          bridge: {} # connect through a bridge\n  networks:\n  - name: red\n    multus:\n      networkName: red\n</code></pre> <p>At this time, <code>bridge</code> mode doesn't support additional configuration fields.</p> <p>Note: due to IPv4 address delegation, in <code>bridge</code> mode the pod doesn't have an IP address configured, which may introduce issues with third-party solutions that may rely on it. For example, Istio may not work in this mode.</p> <p>Note: admin can forbid using <code>bridge</code> interface type for pod networks via a designated configuration flag. To achieve it, the admin should set the following option to <code>false</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    network:\n      permitBridgeInterfaceOnPodNetwork: false\n</code></pre> <p>Note: binding the pod network using <code>bridge</code> interface type may cause issues. Other than the third-party issue mentioned in the above note, live migration is not allowed with a pod network binding of <code>bridge</code> interface type, and also some CNI plugins might not allow to use a custom MAC address for your VM instances. If you think you may be affected by any of issues mentioned above, consider changing the default interface type to <code>masquerade</code>, and disabling the <code>bridge</code> type for pod network, as shown in the example above.</p>"},{"location":"network/interfaces_and_networks/#slirp","title":"slirp","text":"<p>In <code>slirp</code> mode, virtual machines are connected to the network backend using QEMU user networking mode. In this mode, QEMU allocates internal IP addresses to virtual machines and hides them behind NAT.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          slirp: {} # connect using SLIRP mode\n  networks:\n  - name: red\n    pod: {}\n</code></pre> <p>At this time, <code>slirp</code> mode doesn't support additional configuration fields.</p> <p>Note: in <code>slirp</code> mode, the only supported protocols are TCP and UDP. ICMP is not supported.</p> <p>More information about SLIRP mode can be found in QEMU Wiki.</p> <p>Note: Since v1.1.0, Kubevirt delegates Slirp network configuration to the Slirp network binding plugin by default. In case the binding plugin is not registered, Kubevirt will use the following default image: <code>quay.io/kubevirt/network-slirp-binding:20230830_638c60fc8</code>.</p> <p>Note: In the next release (v1.2.0) no default image will be set by Kubevirt, registering an image will be mandatory.</p> <p>Note: On disconnected clusters it will be necessary to mirror Slirp binding plugin image to the cluster registry.</p>"},{"location":"network/interfaces_and_networks/#masquerade","title":"masquerade","text":"<p>In <code>masquerade</code> mode, KubeVirt allocates internal IP addresses to virtual machines and hides them behind NAT. All the traffic exiting virtual machines is \"source NAT'ed\" using pod IP addresses; thus, cluster workloads should use the pod's IP address to contact the VM over this interface. This IP address is reported in the VMI's <code>spec.status.interface</code>. A guest operating system should be configured to use DHCP to acquire IPv4 addresses.</p> <p>To allow the VM to live-migrate or hard restart (both cause the VM to run on a different pod, with a different IP address) and still be reachable, it should be exposed by a Kubernetes service.</p> <p>To allow traffic of specific ports into virtual machines, the template <code>ports</code> section of the interface should be configured as follows. If the <code>ports</code> section is missing, all ports forwarded into the VM.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          masquerade: {} # connect using masquerade mode\n          ports:\n            - port: 80 # allow incoming traffic on port 80 to get into the virtual machine\n  networks:\n  - name: red\n    pod: {}\n</code></pre> <p>Note: Masquerade is only allowed to connect to the pod network.</p> <p>Note: The network CIDR can be configured in the pod network section using the <code>vmNetworkCIDR</code> attribute.</p>"},{"location":"network/interfaces_and_networks/#masquerade-ipv4-and-ipv6-dual-stack-support","title":"masquerade - IPv4 and IPv6 dual-stack support","text":"<p><code>masquerade</code> mode can be used in IPv4 and IPv6 dual-stack clusters to provide a VM with an IP connectivity over both protocols.</p> <p>As with the IPv4 <code>masquerade</code> mode, the VM can be contacted using the pod's IP address - which will be in this case two IP addresses, one IPv4 and one IPv6. Outgoing traffic is also \"NAT'ed\" to the pod's respective IP address from the given family.</p> <p>Unlike in IPv4, the configuration of the IPv6 address and the default route is not automatic; it should be configured via cloud init, as shown below:</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      disks:\n        - disk:\n          bus: virtio\n          name: cloudinitdisk\n      interfaces:\n        - name: red\n          masquerade: {} # connect using masquerade mode\n          ports:\n            - port: 80 # allow incoming traffic on port 80 to get into the virtual machine\n  networks:\n  - name: red\n    pod: {}\n  volumes:\n  - cloudInitNoCloud:\n      networkData: |\n        version: 2\n        ethernets:\n          eth0:\n            dhcp4: true\n            addresses: [ fd10:0:2::2/120 ]\n            gateway6: fd10:0:2::1\n      userData: |-\n        #!/bin/bash\n        echo \"fedora\" |passwd fedora --stdin\n</code></pre> <p>Note: The IPv6 address for the VM and default gateway must be the ones shown above.</p>"},{"location":"network/interfaces_and_networks/#masquerade-ipv6-single-stack-support","title":"masquerade - IPv6 single-stack support","text":"<p><code>masquerade</code> mode can be used in IPv6 single stack clusters to provide a VM with an IPv6 only connectivity.</p> <p>As with the IPv4 <code>masquerade</code> mode, the VM can be contacted using the pod's IP address - which will be in this case the IPv6 one. Outgoing traffic is also \"NAT'ed\" to the pod's respective IPv6 address.</p> <p>As with the dual-stack cluster, the configuration of the IPv6 address and the default route is not automatic; it should be configured via cloud init, as shown in the dual-stack section.</p> <p>Unlike the dual-stack cluster, which has a DHCP server for IPv4, the IPv6 single stack cluster has no DHCP server at all. Therefore, the VM won't have the search domains information and reaching a destination using its FQDN is not possible. Tracking issue - https://github.com/kubevirt/kubevirt/issues/7184</p>"},{"location":"network/interfaces_and_networks/#passt","title":"passt","text":"<p>Warning: The core binding is being deprecated and targeted for removal in v1.3 . As an alternative, the same functionality is introduced and available as a binding plugin.</p> <p><code>passt</code> is a new approach for user-mode networking which can be used as a simple replacement for Slirp (which is practically dead).</p> <p><code>passt</code> is a universal tool which implements a translation layer between a Layer-2 network interface and native Layer -4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host.</p> <p>Its main benefits are: - doesn't require extra network capabilities as CAP_NET_RAW and CAP_NET_ADMIN. - allows integration with service meshes (which expect applications to run locally) out of the box. - supports IPv6 out of the box (in contrast to the existing bindings which require configuring IPv6 manually).</p> Masquerade Bridge Passt Supports migration Yes No No(will be supported in the future) VM uses Pod IP No Yes Yes(in the future it will be possible to configure the VM IP. Currently the default is the pod IP) Service Mesh out of the box No(only ISTIO is supported, adjustmets on both ISTIO and kubevirt had to be done to make it work) No Yes Doesn\u2019t require extra capabilities on the virt-launcher pod Yes(multiple workarounds had to be added to kuebivrt to make it work) No(Multiple workarounds had to be added to kuebivrt to make it work) Yes Doesn't require extra network devices on the virt-launcher pod No(bridge and tap device are created) No(bridge and tap device are created) Yes Supports IPv6 Yes(requires manual configuration on the VM) No Yes <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          passt: {} # connect using passt mode\n          ports:\n            - port: 8080 # allow incoming traffic on port 8080 to get into the virtual machine\n  networks:\n    - name: red\n      pod: {}\n</code></pre>"},{"location":"network/interfaces_and_networks/#requirementsrecommendations","title":"Requirements/Recommendations:","text":"<ol> <li>To get better performance the node should be configured with: <pre><code>sysctl -w net.core.rmem_max = 33554432\nsysctl -w net.core.wmem_max = 33554432\n</code></pre></li> <li>To run multiple passt VMs with no explicit ports, the node's <code>fs.file-max</code> should be increased    (for a VM forwards all IPv4 and IPv6 ports, for TCP and UDP, passt needs to create ~2^18 sockets): <pre><code>sysctl -w fs.file-max = 9223372036854775807\n</code></pre></li> </ol> <p>NOTE: To achieve optimal memory consumption with Passt binding, specify ports required for your workload. When no ports are explicitly specified, all ports are forwarded, leading to memory overhead of up to 800 Mi.</p>"},{"location":"network/interfaces_and_networks/#temporary-restrictions","title":"Temporary restrictions:","text":"<ol> <li><code>passt</code> currently only supported as primary network and doesn't allow extra multus networks to be configured on the VM.</li> </ol> <p>passt interfaces are feature gated; to enable the feature, follow these instructions, in order to activate the <code>Passt</code> feature gate (case sensitive).</p> <p>More information about passt mode can be found in passt Wiki.</p>"},{"location":"network/interfaces_and_networks/#virtio-net-multiqueue","title":"virtio-net multiqueue","text":"<p>Setting the <code>networkInterfaceMultiqueue</code> to <code>true</code> will enable the multi-queue functionality, increasing the number of vhost queue, for interfaces configured with a <code>virtio</code> model.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      networkInterfaceMultiqueue: true\n</code></pre> <p>Users of a Virtual Machine with multiple vCPUs may benefit of increased network throughput and performance.</p> <p>Currently, the number of queues is being determined by the number of vCPUs of a VM. This is because multi-queue support optimizes RX interrupt affinity and TX queue selection in order to make a specific queue private to a specific vCPU.</p> <p>Without enabling the feature, network performance does not scale as the number of vCPUs increases. Guests cannot transmit or retrieve packets in parallel, as virtio-net has only one TX and RX queue.</p> <p>Virtio interfaces advertise on their status.interfaces.interface entry a field named queueCount. The queueCount field indicates how many queues were assigned to the interface. Queue count value is derived from the domain XML. In case the number of queues can't be determined (i.e interface that is reported by quest-agent only), it will be omitted.</p> <p>NOTE: Although the virtio-net multiqueue feature provides a performance benefit, it has some limitations and therefore should not be unconditionally enabled</p>"},{"location":"network/interfaces_and_networks/#some-known-limitations","title":"Some known limitations","text":"<ul> <li> <p>Guest OS is limited to ~200 MSI vectors. Each NIC queue requires a     MSI vector, as well as any virtio device or assigned PCI device.     Defining an instance with multiple virtio NICs and vCPUs might lead     to a possibility of hitting the guest MSI limit.</p> </li> <li> <p>virtio-net multiqueue works well for incoming traffic, but can     occasionally cause a performance degradation, for outgoing traffic.     Specifically, this may occur when sending packets under 1,500 bytes     over the Transmission Control Protocol (TCP) stream.</p> </li> <li> <p>Enabling virtio-net multiqueue increases the total network     throughput, but in parallel it also increases the CPU consumption.</p> </li> <li> <p>Enabling virtio-net multiqueue in the host QEMU config, does not     enable the functionality in the guest OS. The guest OS administrator     needs to manually turn it on for each guest NIC that requires this     feature, using ethtool.</p> </li> <li> <p>MSI vectors would still be consumed (wasted), if multiqueue was     enabled in the host, but has not been enabled in the guest OS by the     administrator.</p> </li> <li> <p>In case the number of vNICs in a guest instance is proportional to     the number of vCPUs, enabling the multiqueue feature is less     important.</p> </li> <li> <p>Each virtio-net queue consumes 64 KiB of kernel memory for the vhost     driver.</p> </li> </ul> <p>NOTE: Virtio-net multiqueue should be enabled in the guest OS manually, using ethtool. For example: <code>ethtool -L &lt;NIC&gt; combined #num_of_queues</code></p> <p>More information please refer to KVM/QEMU MultiQueue.</p>"},{"location":"network/interfaces_and_networks/#sriov","title":"sriov","text":"<p>In <code>sriov</code> mode, virtual machines are directly exposed to an SR-IOV PCI device, usually allocated by Intel SR-IOV device plugin. The device is passed through into the guest operating system as a host device, using the vfio userspace interface, to maintain high networking performance.</p>"},{"location":"network/interfaces_and_networks/#how-to-expose-sr-iov-vfs-to-kubevirt","title":"How to expose SR-IOV VFs to KubeVirt","text":"<p>To simplify procedure, please use SR-IOV network operator to deploy and configure SR-IOV components in your cluster. On how to use the operator, please refer to their respective documentation.</p> <p>Note: KubeVirt relies on VFIO userspace driver to pass PCI devices into VMI guest. Because of that, when configuring SR-IOV operator policies, make sure you define a pool of VF resources that uses <code>deviceType: vfio-pci</code>.</p> <p>Once the operator is deployed, an SriovNetworkNodePolicy  must be provisioned, in which the list of SR-IOV devices to expose (with respective configurations) is defined.</p> <p>Please refer to the following <code>SriovNetworkNodePolicy</code> for an example:</p> <pre><code>apiVersion: sriovnetwork.openshift.io/v1\nkind: SriovNetworkNodePolicy\nmetadata:\n  name: policy-1\n  namespace: sriov-network-operator\nspec:\n  deviceType: vfio-pci\n  mtu: 9000\n  nicSelector:\n    pfNames:\n    - ens1f0\n  nodeSelector:\n    sriov: \"true\"\n  numVfs: 8\n  priority: 90\n  resourceName: sriov-nic\n</code></pre> <p>The policy above will configure the <code>SR-IOV</code> device plugin, allowing the PF named <code>ens1f0</code> to be exposed in the SRIOV capable nodes as a resource named <code>sriov-nic</code>.</p>"},{"location":"network/interfaces_and_networks/#start-an-sr-iov-vm","title":"Start an SR-IOV VM","text":"<p>Once all the SR-IOV components are deployed, it is needed to indicate how to configure the SR-IOV network. Refer to the following <code>SriovNetwork</code> for an example:</p> <pre><code>apiVersion: sriovnetwork.openshift.io/v1\nkind: SriovNetwork\nmetadata:\n  name: sriov-net\n  namespace: sriov-network-operator\nspec:\n  ipam: |\n    {}\n  networkNamespace: default\n  resourceName: sriov-nic\n  spoofChk: \"off\"\n</code></pre> <p>Finally, to create a VM that will attach to the aforementioned Network, refer to the following VMI spec:</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-perf\n  name: vmi-perf\nspec:\n  domain:\n    cpu:\n      sockets: 2\n      cores: 1\n      threads: 1\n      dedicatedCpuPlacement: true\n    resources:\n      requests:\n        memory: \"4Gi\"\n      limits:\n        memory: \"4Gi\"\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      interfaces:\n      - masquerade: {}\n        name: default\n      - name: sriov-net\n        sriov: {}\n      rng: {}\n    machine:\n      type: \"\"\n  networks:\n  - name: default\n    pod: {}\n  - multus:\n      networkName: default/sriov-net\n    name: sriov-net\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: docker.io/kubevirt/fedora-cloud-container-disk-demo:latest\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |\n        #!/bin/bash\n        echo \"centos\" |passwd centos --stdin\n        dhclient eth1\n    name: cloudinitdisk\n</code></pre> <p>Note: for some NICs (e.g. Mellanox), the kernel module needs to be installed in the guest VM.</p> <p>Note: Placement on dedicated CPUs can only be achieved if the Kubernetes CPU manager is running on the SR-IOV capable workers. For further details please refer to the dedicated cpu resources documentation.</p>"},{"location":"network/interfaces_and_networks/#macvtap","title":"Macvtap","text":"<p>Note: The core binding will be deprecated soon. As an alternative, the same functionality is introduced and available as a binding plugin.</p> <p>In <code>macvtap</code> mode, virtual machines are directly exposed to the Kubernetes nodes L2 network. This is achieved by 'extending' an existing network interface with a virtual device that has its own MAC address.</p> <p>Macvtap interfaces are feature gated; to enable the feature, follow these instructions, in order to activate the <code>Macvtap</code> feature gate (case sensitive).</p> <p>Note: On KinD clusters, the user needs to adjust the cluster configuration, mounting <code>dev</code> of the running host onto the KinD nodes, because of a known issue.</p>"},{"location":"network/interfaces_and_networks/#limitations","title":"Limitations","text":"<ul> <li>Live migration is not seamless, see issue #5912</li> </ul>"},{"location":"network/interfaces_and_networks/#how-to-expose-host-interface-to-the-macvtap-device-plugin","title":"How to expose host interface to the macvtap device plugin","text":"<p>To simplify the procedure, please use the Cluster Network Addons Operator to deploy and configure the macvtap components in your cluster.</p> <p>The aforementioned operator effectively deploys the macvtap-cni cni / device plugin combo.</p> <p>There are two different alternatives to configure which host interfaces get exposed to the user, enabling them to create macvtap interfaces on top of:</p> <ul> <li>select the host interfaces: indicates which host interfaces are exposed.</li> <li>expose all interfaces: all interfaces of all hosts are exposed.</li> </ul> <p>Both options are configured via the <code>macvtap-deviceplugin-config</code> ConfigMap, and more information on how to configure it can be found in the macvtap-cni repo.</p> <p>You can find a minimal example, in which the <code>eth0</code> interface of the Kubernetes nodes is exposed, via the <code>lowerDevice</code> attribute. <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: |\n    [\n        {\n            \"name\"       : \"dataplane\",\n            \"lowerDevice\": \"eth0\",\n            \"mode\"       : \"bridge\",\n            \"capacity\"   : 50\n        }\n    ]\n</code></pre></p> <p>This step can be omitted, since the default configuration of the aforementioned <code>ConfigMap</code> is to expose all host interfaces (which is represented by the following configuration): <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: '[]'\n</code></pre></p>"},{"location":"network/interfaces_and_networks/#start-a-vm-with-macvtap-interfaces","title":"Start a VM with macvtap interfaces","text":"<p>Once the macvtap components are deployed, it is needed to indicate how to configure the macvtap network. Refer to the following <code>NetworkAttachmentDefinition</code> for a simple example:</p> <p><pre><code>---\nkind: NetworkAttachmentDefinition\napiVersion: k8s.cni.cncf.io/v1\nmetadata:\n  name: macvtapnetwork\n  annotations:\n    k8s.v1.cni.cncf.io/resourceName: macvtap.network.kubevirt.io/eth0\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"macvtapnetwork\",\n      \"type\": \"macvtap\",\n      \"mtu\": 1500\n    }'\n</code></pre> The requested <code>k8s.v1.cni.cncf.io/resourceName</code> annotation must point to an exposed host interface (via the <code>lowerDevice</code> attribute, on the <code>macvtap-deviceplugin-config</code> <code>ConfigMap</code>).</p> <p>Finally, to create a VM that will attach to the aforementioned Network, refer to the following VMI spec:</p> <p><pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-host-network\n  name: vmi-host-network\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      interfaces:\n      - macvtap: {}\n        name: hostnetwork\n      rng: {}\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  networks:\n  - multus:\n      networkName: macvtapnetwork\n    name: hostnetwork\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #!/bin/bash\n        echo \"fedora\" |passwd fedora --stdin\n    name: cloudinitdisk\n</code></pre> The requested <code>multus</code> <code>networkName</code> - i.e. <code>macvtapnetwork</code> - must match the name of the provisioned <code>NetworkAttachmentDefinition</code>.</p> <p>Note: VMIs with macvtap interfaces can be migrated, but their MAC addresses must be statically set.</p>"},{"location":"network/interfaces_and_networks/#security","title":"Security","text":""},{"location":"network/interfaces_and_networks/#mac-spoof-check","title":"MAC spoof check","text":"<p>MAC spoofing refers to the ability to generate traffic with an arbitrary source MAC address. An attacker may use this option to generate attacks on the network.</p> <p>In order to protect against such scenarios, it is possible to enable the mac-spoof-check support in CNI plugins that support it.</p> <p>The pod primary network which is served by the cluster network provider is not covered by this documentation. Please refer to the relevant provider to check how to enable spoofing check. The following text refers to the secondary networks, served using multus.</p> <p>There are two known CNI plugins that support mac-spoof-check:</p> <ul> <li>sriov-cni:   Through the <code>spoofchk</code> parameter .</li> <li>bridge-cni:   Through the <code>macspoofchk</code> parameter.</li> </ul> <p>The configuration is to be done on the  NetworkAttachmentDefinition by the operator and any interface that refers to it, will have this feature enabled.</p> <p>Below is an example of using the <code>bridge</code> CNI with <code>macspoofchk</code> enabled: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: br-spoof-check\nspec:\n  config: '{\n            \"cniVersion\": \"0.3.1\",\n            \"name\": \"br-spoof-check\",\n            \"type\": \"bridge\",\n            \"bridge\": \"br10\",\n            \"disableContainerInterface\": true,\n            \"macspoofchk\": true\n        }'\n</code></pre></p> <p>On the VMI, the network section should point to this NetworkAttachmentDefinition by name: <pre><code>networks:\n- name: default\n  pod: {}\n- multus:\n    networkName: br-spoof-check\n  name: br10\n</code></pre></p>"},{"location":"network/interfaces_and_networks/#limitations_1","title":"Limitations","text":"<ul> <li>The <code>bridge</code> CNI supports mac-spoof-check through nftables, therefore the node must support nftables and have the <code>nft</code> binary deployed.</li> </ul>"},{"location":"network/istio_service_mesh/","title":"Istio service mesh","text":"<p>Service mesh allows to monitor, visualize and control traffic between pods. Kubevirt supports running VMs as a part of Istio service mesh.</p>"},{"location":"network/istio_service_mesh/#limitations","title":"Limitations","text":"<ul> <li> <p>Istio service mesh is only supported with a pod network masquerade or passt binding.</p> </li> <li> <p>Istio uses a list of ports for its own purposes, these ports must not be explicitly specified in a VMI interface.</p> </li> <li> <p>Istio only supports IPv4.</p> </li> </ul>"},{"location":"network/istio_service_mesh/#prerequisites","title":"Prerequisites","text":"<ul> <li> <p>This guide assumes that Istio is already deployed and uses Istio CNI Plugin. See Istio documentation for more information.</p> </li> <li> <p>Optionally, <code>istioctl</code> binary for troubleshooting. See Istio installation inctructions.</p> </li> <li> <p>The target namespace where the VM is created must be labelled with <code>istio-injection=enabled</code> label.</p> </li> <li> <p>If Multus is used to manage CNI, the following <code>NetworkAttachmentDefinition</code> is required in the application namespace: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: istio-cni\n</code></pre></p> </li> </ul>"},{"location":"network/istio_service_mesh/#create-a-virtualmachineinstance-with-enabled-istio-proxy-injecton","title":"Create a VirtualMachineInstance with enabled Istio proxy injecton","text":"<p>The example below specifies a VMI with masquerade network interface and <code>sidecar.istio.io/inject</code> annotation to register the VM to the service mesh. </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    sidecar.istio.io/inject: \"true\"\n  labels:\n    app: vmi-istio\n  name: vmi-istio\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    resources:\n      requests:\n        memory: 1024M\n  networks:\n    - name: default\n      pod: {}\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: registry:5000/kubevirt/fedora-cloud-container-disk-demo:devel\n</code></pre> <p>Istio expects each application to be associated with at least one Kubernetes service. Create the following Service exposing port 8080:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vmi-istio\nspec:\n  selector:\n    app: vmi-istio\n  ports:\n    - port: 8080\n      name: http\n      protocol: TCP\n</code></pre> <p>Note: Each Istio enabled VMI must feature the <code>sidecar.istio.io/inject</code> annotation instructing KubeVirt to perform necessary network configuration.</p>"},{"location":"network/istio_service_mesh/#verification","title":"Verification","text":"<p>Verify istio-proxy sidecar is deployed and able to synchronize with Istio control plane using <code>istioctl proxy-status</code> command. See Istio Debbuging Envoy and Istiod documentation section for more information about <code>proxy-status</code> subcommand.</p> <pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-ncx7r    3/3     Running   0          7s\n\n$ kubectl get pods virt-launcher-vmi-istio-ncx7r -o jsonpath='{.spec.containers[*].name}'\ncompute volumecontainerdisk istio-proxy\n\n$ istioctl proxy-status\nNAME                                    CDS        LDS        EDS        RDS          ISTIOD                      VERSION\n...\nvirt-launcher-vmi-istio-ncx7r.default   SYNCED     SYNCED     SYNCED     SYNCED       istiod-7c4d8c7757-hshj5     1.10.0\n</code></pre>"},{"location":"network/istio_service_mesh/#troubleshooting","title":"Troubleshooting","text":""},{"location":"network/istio_service_mesh/#istio-sidecar-is-not-deployed","title":"Istio sidecar is not deployed","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-jnw6p    2/2     Running   0          37s\n\n$ kubectl get pods virt-launcher-vmi-istio-jnw6p -o jsonpath='{.spec.containers[*].name}'\ncompute volumecontainerdisk\n</code></pre> <p>Resolution: Make sure the <code>istio-injection=enabled</code> is added to the target namespace. If the issue persists, consult relevant part of Istio documentation.</p>"},{"location":"network/istio_service_mesh/#istio-sidecar-is-not-ready","title":"Istio sidecar is not ready","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-lg5gp    2/3     Running   0          90s\n\n$ kubectl describe pod virt-launcher-vmi-istio-lg5gp\n  ...\n  Warning  Unhealthy  2d8h (x3 over 2d8h)  kubelet            Readiness probe failed: Get \"http://10.244.186.222:15021/healthz/ready\": dial tcp 10.244.186.222:15021: connect: no route to host\n  Warning  Unhealthy  2d8h (x4 over 2d8h)  kubelet            Readiness probe failed: Get \"http://10.244.186.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n</code></pre> <p>Resolution: Make sure the <code>sidecar.istio.io/inject: \"true\"</code> annotation is defined in the created VMI and that masquerade or passt binding is used for pod network interface.</p>"},{"location":"network/istio_service_mesh/#virt-launcher-pod-for-vmi-is-stuck-at-initialization-phase","title":"Virt-launcher pod for VMI is stuck at initialization phase","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS     RESTARTS   AGE\nvirt-launcher-vmi-istio-44mws    0/3     Init:0/3   0          29s\n\n$ kubectl describe pod virt-launcher-vmi-istio-44mws\n  ...\n  Multus: [default/virt-launcher-vmi-istio-44mws]: error loading k8s delegates k8s args: TryLoadPodDelegates: error in getting k8s network for pod: GetNetworkDelegates: failed getting the delegate: getKubernetesDelegate: cannot find a network-attachment-definition (istio-cni) in namespace (default): network-attachment-definitions.k8s.cni.cncf.io \"istio-cni\" not found\n</code></pre> <p>Resolution: Make sure the <code>istio-cni</code> NetworkAttachmentDefinition (provided in the Prerequisites section) is created in the target namespace.</p>"},{"location":"network/network_binding_plugins/","title":"Network Binding Plugins","text":"<p>[v1.1.0, Alpha feature]</p> <p>A modular plugin which integrates with Kubevirt to implement a network binding.</p>"},{"location":"network/network_binding_plugins/#overview","title":"Overview","text":""},{"location":"network/network_binding_plugins/#network-connectivity","title":"Network Connectivity","text":"<p>In order for a VM to have access to external network(s), several layers need to be defined and configured, depending on the connectivity characteristics needs.</p> <p>These layers include:</p> <ul> <li>Host connectivity: Network provider.</li> <li>Host to Pod connectivity: CNI.</li> <li>Pod to domain connectivity: Network Binding.</li> </ul> <p>This guide focuses on the Network Binding portion.</p>"},{"location":"network/network_binding_plugins/#network-binding","title":"Network Binding","text":"<p>The network binding defines how the domain (VM) network interface is wired in the VM pod through the domain to the guest.</p> <p>The network binding includes:</p> <ul> <li>Domain vNIC configuration.</li> <li>Pod network configuration (optional).</li> <li>Services to deliver network details to the guest (optional).   E.g. DHCP server to pass the IP configuration to the guest.</li> </ul>"},{"location":"network/network_binding_plugins/#plugins","title":"Plugins","text":"<p>The network bindings have been part of Kubevirt core API and codebase. With the increase of the number of network bindings added and frequent requests to tweak and change the existing network bindings, a decision has been made to create a network binding plugin infrastructure.</p> <p>The plugin infrastructure provides means to compose a network binding plugin and integrate it into Kubevirt in a modular manner.</p> <p>Kubevirt is providing several network binding plugins as references. The following plugins are available:</p> <ul> <li>passt [v1.1.0]</li> <li>macvtap [v1.1.1]</li> <li>slirp [v1.1.0]</li> </ul>"},{"location":"network/network_binding_plugins/#definition-flow","title":"Definition &amp; Flow","text":"<p>A network binding plugin configuration consist of the following steps:</p> <ul> <li> <p>Deploy network binding optional components:</p> </li> <li> <p>Binding CNI plugin.</p> </li> <li>Binding NetworkAttachmentDefinition manifest.</li> <li>Access to the sidecar image.</li> <li> <p>Enable <code>NetworkBindingPlugins</code> Feature Gate (FG).</p> </li> <li> <p>Register network binding.</p> </li> <li>Assign VM network interface binding.</li> </ul>"},{"location":"network/network_binding_plugins/#deployment","title":"Deployment","text":"<p>Depending on the plugin, some components need to be deployed in the cluster. Not all network binding plugins require all these components, therefore these steps are optional.</p> <ul> <li>Binding CNI plugin: When it is required to change the pod network stack   (and a core domain-attachment is not a fit), a custom CNI plugin is   composed to serve the network binding plugin.</li> </ul> <p>This binary needs to be deployed on each node of the cluster, like any   other CNI plugin.</p> <p>The binary can be built from source or consumed from an existing artifact.</p> <p>Note: The location of the CNI plugins binaries depends on the platform used and its configuration. A frequently used path for such binaries is <code>/opt/cni/bin/</code>.</p> <ul> <li>Binding NetworkAttachmentDefinition: It references the binding CNI plugin,   with optional configuration settings.   The manifest needs to be deployed on the cluster at a namespace which   is accessible by the VM and its pod.</li> </ul> <p>Example: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: netbindingpasst\nspec:\n  config: '{\n            \"cniVersion\": \"1.0.0\",\n            \"name\": \"netbindingpasst\",\n            \"plugins\": [\n              {\n                \"type\": \"cni-passt-binding-plugin\"\n              }\n            ]\n  }'\n</code></pre></p> <p>Note: It is possible to deploy the NetworkAttachmentDefinition on the <code>default</code> namespace, where all other namespaces can access it. Nevertheless, it is recommended (for security reasons) to define the NetworkAttachmentDefinition in the same namespace the VM resides.</p> <ul> <li> <p>Multus: In order   for the network binding CNI and the NetworkAttachmentDefinition to operate,   there is a need to have Multus deployed on the cluster.   For more information, check the   Quickstart Intallation Guide.</p> </li> <li> <p>Sidecar image: When a core domain-attachment is not a fit, a sidecar is   used to configure the vNIC domain configuration.   In a more complex scenarios, the sidecar also runs services like DHCP to   deliver IP information to the guest.</p> </li> </ul> <p>The sidecar image is built and usually pushed to an image registry for   consumption. Therefore, the cluster needs to have access to the image.</p> <p>The image can be built from source and pushed to an accessible registry   or used from a given registry that already contains it.</p> <ul> <li>Feature Gate   The network binding plugin is currently (v1.1.0) in Alpha stage, protected   by a feature gate (FG) named <code>NetworkBindingPlugins</code>.</li> </ul> <p>It is therefore necessary to set the FG in the Kubevirt CR.</p> <p>Example (valid when the FG subtree is already defined): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p>"},{"location":"network/network_binding_plugins/#register","title":"Register","text":"<p>In order to use a network binding plugin, the cluster admin needs to register the binding. Registration includes the addition of the binding name with all its parameters to the Kubevirt CR.</p> <p>The following (optional) parameters are currently supported:</p> <ul> <li>networkAttachmentDefinition</li> <li>sidecarImage</li> <li>domainAttachmentType</li> <li>migration</li> </ul>"},{"location":"network/network_binding_plugins/#networkattachmentdefinition","title":"networkAttachmentDefinition","text":"<p>From: v1.1.0</p> <p>Use the  format to specify the NetworkAttachementDefinition that defines the CNI plugin and the configuration the binding plugin uses. Used when the binding plugin needs to change the pod network namespace."},{"location":"network/network_binding_plugins/#sidecarimage","title":"sidecarImage","text":"<p>From: v1.1.0</p> <p>Specify a container image in a registry. Used when the binding plugin needs to modify the domain vNIC configuration or when a service needs to be executed (e.g. DHCP server). </p>"},{"location":"network/network_binding_plugins/#domainattachmenttype","title":"domainAttachmentType","text":"<p>From: v1.1.1</p> <p>The Domain Attachment type is a pre-defined core kubevirt method to attach an interface to the domain.</p> <p>Specify the name of a core domain attachment type. A possible alternative to a sidecar, to configure the domain vNIC.</p> <p>Supported types:</p> <ul> <li><code>tap</code> (from v1.1.1): The domain configuration is set to use an existing   tap device. It also supports existing <code>macvtap</code> devices.</li> </ul> <p>When both the <code>domainAttachmentType</code> and <code>sidecarImage</code> are specified, the domain will first be configured according to the <code>domainAttachmentType</code> and then the <code>sidecarImage</code> may modify it.</p>"},{"location":"network/network_binding_plugins/#migration","title":"migration","text":"<p>From: v1.2.0</p> <p>Specify whether the network binding plugin supports migration. It is possible to specify a migration method. Supported migration method types: - <code>link-refresh</code> (from v1.2.0): after migration, the guest nic will be deactivated and then activated again.    It can be useful to renew the DHCP lease.</p> <p>Note: In some deployments the Kubevirt CR is controlled by an external controller (e.g. HCO). In such cases, make sure to configure the wrapper operator/controller so the changes will get preserved.</p> <p>Example (the <code>passt</code> binding): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\"\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    }\n                }\n            }\n        }}]'\n</code></pre></p>"},{"location":"network/network_binding_plugins/#compute-resource-overhead","title":"Compute Resource Overhead","text":"<p>From: v1.3.0</p> <p>Some plugins may need additional resources to be added to the compute container of the virt-launcher pod.</p> <p>It is possible to specify compute resource overhead that will be added to the <code>compute</code> container of virt-launcher pods derived from virtual machines using the plugin.</p> <p>Note: At the moment, only memory overhead requests are supported.</p> <p>Note: In some deployments the Kubevirt CR is controlled by an external controller (e.g. HCO). In such cases, make sure to configure the wrapper operator/controller so the changes will get preserved.</p> <p>Example (the <code>passt</code> binding): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\",\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    },\n                    \"computeResourceOverhead\": {\n                      \"requests\": {\n                        \"memory\": \"500Mi\",\n                      }\n                    }\n                }\n            }\n        }}]'\n</code></pre></p> <p>Every compute container in a virt-launcher pod derived from a VM using the passt network binding plugin, will have an additional <code>500Mi</code> memory overhead.</p>"},{"location":"network/network_binding_plugins/#vm-network-interface","title":"VM Network Interface","text":"<p>When configuring the VM/VMI network interface, the binding plugin name can be specified. If it exists in the Kubevirt CR, it will be used to setup the network interface.</p> <p>Example (<code>passt</code> binding): <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-passt\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-passt\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: passtnet\n            binding:\n              name: passt\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: passtnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre></p>"},{"location":"network/networkpolicy/","title":"NetworkPolicy","text":"<p>Before creating NetworkPolicy objects, make sure you are using a networking solution which supports NetworkPolicy. Network isolation is controlled entirely by NetworkPolicy objects. By default, all vmis in a namespace are accessible from other vmis and network endpoints. To isolate one or more vmis in a project, you can create NetworkPolicy objects in that namespace to indicate the allowed incoming connections.</p> <p>Note: vmis and pods are treated equally by network policies, since labels are passed through to the pods which contain the running vmi. With other words, labels on vmis can be matched by <code>spec.podSelector</code> on the policy.</p>"},{"location":"network/networkpolicy/#create-networkpolicy-to-deny-all-traffic","title":"Create NetworkPolicy to Deny All Traffic","text":"<p>To make a project \"deny by default\" add a NetworkPolicy object that matches all vmis but accepts no traffic.</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: deny-by-default\nspec:\n  podSelector: {}\n  ingress: []\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-only-accept-connections-from-vmis-within-namespaces","title":"Create NetworkPolicy to only Accept connections from vmis within namespaces","text":"<p>To make vmis accept connections from other vmis in the same namespace, but reject all other connections from vmis in other namespaces:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: allow-same-namespace\nspec:\n  podSelector: {}\n  ingress:\n  - from:\n    - podSelector: {}\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-only-allow-http-and-https-traffic","title":"Create NetworkPolicy to only allow HTTP and HTTPS traffic","text":"<p>To enable only HTTP and HTTPS access to the vmis, add a NetworkPolicy object similar to:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: allow-http-https\nspec:\n  podSelector: {}\n  ingress:\n  - ports:\n    - protocol: TCP\n      port: 8080\n    - protocol: TCP\n      port: 8443\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-deny-traffic-by-labels","title":"Create NetworkPolicy to deny traffic by labels","text":"<p>To make one specific vmi with a label <code>type: test</code> to reject all traffic from other vmis, create:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: deny-by-label\nspec:\n  podSelector:\n    matchLabels:\n      type: test\n  ingress: []\n</code></pre> <p>Kubernetes NetworkPolicy Documentation can be found here: Kubernetes NetworkPolicy</p>"},{"location":"network/service_objects/","title":"Service objects","text":"<p>Once the VirtualMachineInstance is started, in order to connect to a VirtualMachineInstance, you can create a <code>Service</code> object for a VirtualMachineInstance. Currently, three types of service are supported: <code>ClusterIP</code>, <code>NodePort</code> and <code>LoadBalancer</code>. The default type is <code>ClusterIP</code>.</p> <p>Note: Labels on a VirtualMachineInstance are passed through to the pod, so simply add your labels for service creation to the VirtualMachineInstance. From there on it works like exposing any other k8s resource, by referencing these labels in a service.</p>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-clusterip-service","title":"Expose VirtualMachineInstance as a ClusterIP Service","text":"<p>Give a VirtualMachineInstance with the label <code>special: key</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vmi-ephemeral\n  labels:\n    special: key\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/cirros-registry-disk-demo:latest\n</code></pre> <p>we can expose its SSH port (22) by creating a <code>ClusterIP</code> service:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vmiservice\nspec:\n  ports:\n  - port: 27017\n    protocol: TCP\n    targetPort: 22\n  selector:\n    special: key\n  type: ClusterIP\n</code></pre> <p>You just need to create this <code>ClusterIP</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl create -f vmiservice.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>Notes: * If <code>--target-port</code> is not set, it will be take the same value as <code>--port</code> * The cluster IP is usually allocated automatically, but it may also be forced into a value using the <code>--cluster-ip</code> flag (assuming value is in the valid range and not taken)</p> <p>Query the service object:</p> <pre><code>$ kubectl get service\nNAME        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE\nvmiservice   ClusterIP   172.30.3.149   &lt;none&gt;        27017/TCP   2m\n</code></pre> <p>You can connect to the VirtualMachineInstance by service IP and service port inside the cluster network:</p> <pre><code>$ ssh cirros@172.30.3.149 -p 27017\n</code></pre>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-nodeport-service","title":"Expose VirtualMachineInstance as a NodePort Service","text":"<p>Expose the SSH port (22) of a VirtualMachineInstance running on KubeVirt by creating a <code>NodePort</code> service:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: nodeport\nspec:\n  externalTrafficPolicy: Cluster\n  ports:\n  - name: nodeport\n    nodePort: 30000\n    port: 27017\n    protocol: TCP\n    targetPort: 22\n  selector:\n    special: key\n  type: NodePort\n</code></pre> <p>You just need to create this <code>NodePort</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl -f nodeport.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name nodeport --type NodePort --port 27017 --target-port 22 --node-port 30000\n</code></pre> <p>Notes: * If <code>--node-port</code> is not set, its value will be allocated dynamically (in the range above 30000) * If the <code>--node-port</code> value is set, it must be unique across all services</p> <p>The service can be listed by querying for the service objects:</p> <pre><code>$ kubectl get service\nNAME           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE\nnodeport       NodePort   172.30.232.73   &lt;none&gt;        27017:30000/TCP   5m\n</code></pre> <p>Connect to the VirtualMachineInstance by using a node IP and node port outside the cluster network:</p> <pre><code>$ ssh cirros@$NODE_IP -p 30000\n</code></pre>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-loadbalancer-service","title":"Expose VirtualMachineInstance as a LoadBalancer Service","text":"<p>Expose the RDP port (3389) of a VirtualMachineInstance running on KubeVirt by creating <code>LoadBalancer</code> service. Here is an example:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: lbsvc\nspec:\n  externalTrafficPolicy: Cluster\n  ports:\n  - port: 27017\n    protocol: TCP\n    targetPort: 3389\n  selector:\n    special: key\n  type: LoadBalancer\n</code></pre> <p>You could create this <code>LoadBalancer</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl -f lbsvc.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name lbsvc --type LoadBalancer --port 27017 --target-port 3389\n</code></pre> <p>Note that the external IP of the service could be forced to a value using the <code>--external-ip</code> flag (no validation is performed on this value).</p> <p>The service can be listed by querying for the service objects:</p> <pre><code>$ kubectl get svc\nNAME      TYPE           CLUSTER-IP       EXTERNAL-IP                   PORT(S)           AGE\nlbsvc     LoadBalancer   172.30.27.5      172.29.10.235,172.29.10.235   27017:31829/TCP   5s\n</code></pre> <p>Use <code>vinagre</code> client to connect your VirtualMachineInstance by using the public IP and port.</p> <p>Note that here the external port here (31829) was dynamically allocated.</p>"},{"location":"network/net_binding_plugins/macvtap/","title":"Macvtap binding","text":""},{"location":"network/net_binding_plugins/macvtap/#overview","title":"Overview","text":"<p>With the <code>macvtap</code> binding plugin, virtual machines are directly exposed to the Kubernetes nodes L2 network. This is achieved by 'extending' an existing network interface with a virtual device that has its own MAC address.</p> <p>Its main benefits are:</p> <ul> <li>Direct connection to the node nic with no intermediate bridges.</li> </ul>"},{"location":"network/net_binding_plugins/macvtap/#functionality-support","title":"Functionality support","text":"Functionality Support Run without extra capabilities (on pod) Yes Migration support No IPAM support (on pod) No Primary network (pod network) No Secondary network Yes"},{"location":"network/net_binding_plugins/macvtap/#known-issues","title":"Known Issues","text":"<ul> <li>Live migration is not fully supported, see issue #5912</li> </ul> <p>Warning: On KinD clusters, the user needs to adjust the cluster configuration, mounting <code>dev</code> of the running host onto the KinD nodes, because of a known issue.</p>"},{"location":"network/net_binding_plugins/macvtap/#deployment","title":"Deployment","text":"<p>The <code>macvtap</code> solution consists of a CNI and a DP.</p> <p>In order to use <code>macvtap</code>, the following points need to be covered:</p> <ul> <li>Deploy the CNI plugin binary on the nodes.</li> <li>Deploy the Device Plugin daemon on the nodes.</li> <li>Configure which node interfaces are exposed.</li> <li>Define a NetworkAttachmentDefinition that points to the CNI plugin.</li> </ul>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-cni-and-dp-deployment-on-nodes","title":"Macvtap CNI and DP deployment on nodes","text":"<p>To simplify the procedure, use the Cluster Network Addons Operator to deploy and configure the macvtap components in your cluster.</p> <p>The aforementioned operator effectively deploys the macvtap cni and device plugin.</p>"},{"location":"network/net_binding_plugins/macvtap/#expose-node-interface-to-the-macvtap-device-plugin","title":"Expose node interface to the macvtap device plugin","text":"<p>There are two different alternatives to configure which host interfaces get exposed to the user, enabling them to create macvtap interfaces on top of:</p> <ul> <li>select the host interfaces: indicates which host interfaces are exposed.</li> <li>expose all interfaces: all interfaces of all hosts are exposed.</li> </ul> <p>Both options are configured via the <code>macvtap-deviceplugin-config</code> ConfigMap, and more information on how to configure it can be found in the macvtap-cni repo.</p> <p>This is a minimal example, in which the <code>eth0</code> interface of the Kubernetes nodes is exposed, via the <code>lowerDevice</code> attribute.</p> <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: |\n    [\n        {\n            \"name\"        : \"dataplane\",\n            \"lowerDevice\" : \"eth0\",\n            \"mode\"        : \"bridge\",\n            \"capacity\"    : 50\n        },\n    ]\n</code></pre> <p>This step can be omitted, since the default configuration of the aforementioned <code>ConfigMap</code> is to expose all host interfaces (which is represented by the following configuration):</p> <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: '[]'\n</code></pre>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-networkattachmentdefinition","title":"Macvtap NetworkAttachmentDefinition","text":"<p>The configuration needed for a macvtap network attachment can be minimalistic:</p> <pre><code>kind: NetworkAttachmentDefinition\napiVersion: k8s.cni.cncf.io/v1\nmetadata:\n  name: macvtapnetwork\n  annotations:\n    k8s.v1.cni.cncf.io/resourceName: macvtap.network.kubevirt.io/eth0\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"macvtapnetwork\",\n      \"type\": \"macvtap\",\n      \"mtu\": 1500\n    }'\n</code></pre> <p>The object should be created in a \"default\" namespace where all other namespaces can access, or, in the same namespace the VMs reside in.</p> <p>The requested <code>k8s.v1.cni.cncf.io/resourceName</code> annotation must point to an exposed host interface (via the <code>lowerDevice</code> attribute, on the <code>macvtap-deviceplugin-config</code> <code>ConfigMap</code>).</p>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-network-binding-plugin","title":"Macvtap network binding plugin","text":"<p>[v1.1.1]</p> <p>The binding plugin replaces the experimental core macvtap binding implementation (including its API).</p> <p>Note: The network binding plugin infrastructure and the macvtap plugin specifically are in Alpha stage. Please use them with care, preferably on a non-production deployment.</p> <p>The macvtap binding plugin consists of the following components:</p> <ul> <li>Macvtap CNI plugin.</li> </ul> <p>The plugin needs to:</p> <ul> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>And in detail:</p>"},{"location":"network/net_binding_plugins/macvtap/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\n  \"op\": \"add\",\n  \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",\n  \"value\": \"NetworkBindingPlugins\"\n}]'\n</code></pre></p> <p>Note: The specific macvtap plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster. The macvtap binding is still in evaluation, use it with care.</p>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-registration","title":"Macvtap Registration","text":"<p>The macvtap binding plugin configuration needs to be added to the kubevirt CR in order to be used by VMs.</p> <p>To register the macvtap binding, patch the kubevirt CR as follows:</p> <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"macvtap\": {\n                    \"domainAttachmentType\": \"tap\"\n                }\n            }\n        }}]'\n</code></pre>"},{"location":"network/net_binding_plugins/macvtap/#vm-macvtap-network-interface","title":"VM Macvtap Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-macvtap\n  name: vm-net-binding-macvtap\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-macvtap\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: podnet\n            masquerade: {}\n          - name: hostnetwork\n            binding:\n              name: macvtap\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: podnet\n        pod: {}\n      - name: hostnetwork\n        multus:\n          networkName: macvtapnetwork\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre> <p>The multus <code>networkName</code> value should correspond with the name used in the network attachment definition section.</p> <p>The <code>binding</code> value should correspond with the name used in the registration.</p>"},{"location":"network/net_binding_plugins/passt/","title":"Passt binding","text":""},{"location":"network/net_binding_plugins/passt/#overview","title":"Overview","text":"<p>Plug A Simple Socket Transport is an enhanced alternative to SLIRP, providing user-space network connectivity.</p> <p><code>passt</code> is a universal tool which implements a translation layer between a Layer-2 network interface and native Layer -4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host.</p> <p>Its main benefits are:</p> <ul> <li>Doesn't require extra network capabilities as CAP_NET_RAW and CAP_NET_ADMIN.</li> <li>Allows integration with service meshes (which expect applications to run locally) out of the box.</li> <li>Supports IPv6 out of the box (in contrast to the existing bindings which require configuring IPv6   manually).</li> </ul>"},{"location":"network/net_binding_plugins/passt/#functionality-support","title":"Functionality support","text":"Functionality Support Migration support Yes Service Mesh support Yes Pod IP in guest Yes Custom CIDR in guest No Require extra capabilities (on pod) to operate No Primary network (pod network) Yes Secondary network No"},{"location":"network/net_binding_plugins/passt/#node-optimization-requirementsrecommendations","title":"Node optimization requirements/recommendations:","text":"<ol> <li>To get better performance the node should be configured with: <pre><code>sysctl -w net.core.rmem_max = 33554432\nsysctl -w net.core.wmem_max = 33554432\n</code></pre></li> <li>To run multiple passt VMs with no explicit ports, the node's <code>fs.file-max</code> should be increased    (for a VM forwards all IPv4 and IPv6 ports, for TCP and UDP, passt needs to create ~2^18 sockets): <pre><code>sysctl -w fs.file-max = 9223372036854775807\n</code></pre></li> </ol> <p>NOTE: To achieve optimal memory consumption with Passt binding, specify ports required for your workload. When no ports are explicitly specified, all ports are forwarded, leading to memory overhead of up to 800 Mi.</p>"},{"location":"network/net_binding_plugins/passt/#passt-network-binding-plugin","title":"Passt network binding plugin","text":"<p>[v1.1.0]</p> <p>The binding plugin replaces the experimental core passt binding implementation (including its API).</p> <p>Note: The network binding plugin infrastructure and the passt plugin specifically are in Alpha stage. Please use them with care, preferably on a non-production deployment.</p> <p>The passt binding plugin consists of the following components:</p> <ul> <li>Passt CNI plugin.</li> <li>Sidecar image.</li> </ul> <p>As described in the definition &amp; flow section, the passt plugin needs to:</p> <ul> <li>Deploy the CNI plugin binary on the nodes.</li> <li>Define a NetworkAttachmentDefinition that points to the CNI plugin.</li> <li>Assure access to the sidecar image.</li> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>And in detail:</p>"},{"location":"network/net_binding_plugins/passt/#passt-cni-deployment-on-nodes","title":"Passt CNI deployment on nodes","text":"<p>The CNI plugin binary can be retrieved directly from the kubevirt release assets (on GitHub) or to be built from its sources.</p> <p>Note: The kubevirt project uses Bazel to build the binaries and container images. For more information in how to build the whole project, visit the developer getting started guide.</p> <p>Once the binary is ready, you may rename it to a meaningful name (e.g. <code>kubevirt-passt-binding</code>). This name is used in the NetworkAttachmentDefinition configuration.</p> <p>Copy the binary to each node in your cluster. The location of the CNI plugins may vary between platforms and versions. One common path is <code>/opt/cni/bin/</code>.</p>"},{"location":"network/net_binding_plugins/passt/#passt-networkattachmentdefinition","title":"Passt NetworkAttachmentDefinition","text":"<p>The configuration needed for passt is minimalistic:</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: netbindingpasst\nspec:\n  config: '{\n            \"cniVersion\": \"1.0.0\",\n            \"name\": \"netbindingpasst\",\n            \"plugins\": [\n              {\n                \"type\": \"kubevirt-passt-binding\"\n              }\n            ]\n  }'\n</code></pre> <p>The object should be created in a \"default\" namespace where all other namespaces can access, or, in the same namespace the VMs reside in.</p>"},{"location":"network/net_binding_plugins/passt/#passt-sidecar-image","title":"Passt sidecar image","text":"<p>Passt sidecar image is built and pushed to kubevirt quay repository.</p> <p>The sidecar sources can be found here.</p> <p>The relevant sidecar image needs to be accessible by the cluster and specified in the Kubevirt CR when registering the network binding plugin.</p>"},{"location":"network/net_binding_plugins/passt/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p> <p>Note: The specific passt plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster. The passt binding is still in evaluation, use it with care.</p>"},{"location":"network/net_binding_plugins/passt/#passt-registration","title":"Passt Registration","text":"<p>As described in the registration section, passt binding plugin configuration needs to be added to the kubevirt CR.</p> <p>To register the passt binding, patch the kubevirt CR as follows: <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\",\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    },\n                    \"computeResourceOverhead\": {\n                      \"requests\": {\n                        \"memory\": \"500Mi\",\n                      }\n                    }\n                }\n            }\n        }}]'\n</code></pre></p> <p>The NetworkAttachmentDefinition and sidecarImage values should correspond with the names used in the previous sections, here and here.</p> <p>When using the plugin, additional memory overhead of <code>500Mi</code> will be requested for the compute container in the virt-launcher pod.</p>"},{"location":"network/net_binding_plugins/passt/#vm-passt-network-interface","title":"VM Passt Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-passt\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-passt\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: passtnet\n            binding:\n              name: passt\n            ports:\n            - name: http\n              port: 80\n              protocol: TCP\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: passtnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre>"},{"location":"network/net_binding_plugins/slirp/","title":"Slirp","text":""},{"location":"network/net_binding_plugins/slirp/#overview","title":"Overview","text":"<p>SLIRP provides user-space network connectivity.</p> <p>Note: in <code>slirp</code> mode, the only supported protocols are TCP and UDP. ICMP is not supported.</p>"},{"location":"network/net_binding_plugins/slirp/#slirp-network-binding-plugin","title":"Slirp network binding plugin","text":"<p>[v1.1.0]</p> <p>The binding plugin replaces the core <code>slirp</code> binding API.</p> <p>Note: The network binding plugin infrastructure is in Alpha stage. Please use them with care.</p> <p>The slirp binding plugin consists of the following components:</p> <ul> <li>Sidecar image.</li> </ul> <p>As described in the definition &amp; flow section, the slirp plugin needs to:</p> <ul> <li>Assure access to the sidecar image.</li> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>Note: In order for the core slirp binding to use the network binding plugin the registered name for this binding should be <code>slirp</code>.</p>"},{"location":"network/net_binding_plugins/slirp/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p> <p>Note: The specific slirp plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster.</p>"},{"location":"network/net_binding_plugins/slirp/#slirp-registration","title":"Slirp Registration","text":"<p>As described in the registration section, slirp binding plugin configuration needs to be added to the kubevirt CR.</p> <p>To register the slirp binding, patch the kubevirt CR as follows: <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"slirp\": {\n                    \"sidecarImage\": \"quay.io/kubevirt/network-slirp-binding:v1.1.0\"\n                }\n            }\n        }}]'\n</code></pre></p>"},{"location":"network/net_binding_plugins/slirp/#vm-slirp-network-interface","title":"VM Slirp Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-slirp\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-slirp\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: slirpnet\n            binding:\n              name: slirp\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: slirpnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre>"},{"location":"storage/clone_api/","title":"Clone API","text":"<p>The <code>clone.kubevirt.io</code> API Group defines resources for cloning KubeVirt objects. Currently, the only supported cloning type is <code>VirtualMachine</code>, but more types are planned to be supported in the future (see future roadmap below).</p> <p>Please bear in mind that the clone API is in version <code>v1alpha1</code>. This means that this API is not fully stable yet and that APIs may change in the future.</p>"},{"location":"storage/clone_api/#prerequesites","title":"Prerequesites","text":""},{"location":"storage/clone_api/#snapshot-restore","title":"Snapshot / Restore","text":"<p>Under the hood, the clone API relies upon Snapshot &amp; Restore APIs. Therefore, in order to be able to use the clone API, please see Snapshot &amp; Restore prerequesites.</p>"},{"location":"storage/clone_api/#snapshot-feature-gate","title":"Snapshot Feature Gate","text":"<p>Currently, clone API is guarded by Snapshot feature gate. The feature gates field in the KubeVirt CR must be expanded by adding the <code>Snapshot</code> to it.</p>"},{"location":"storage/clone_api/#the-clone-object","title":"The clone object","text":"<p>Firstly, as written above, the clone API relies upon Snapshot &amp; Restore APIs under the hood. Therefore, it might be helpful to look at Snapshot &amp; Restore user-guide page for more info.</p>"},{"location":"storage/clone_api/#virtualmachineclone-object-overview","title":"VirtualMachineClone object overview","text":"<p>In order to initiate cloning, a <code>VirtualMachineClone</code> object (CRD) needs to be created on the cluster. An example for such an object is: <pre><code>kind: VirtualMachineClone\napiVersion: \"clone.kubevirt.io/v1alpha1\"\nmetadata:\n  name: testclone\n\nspec:\n  # source &amp; target definitions\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: vm-cirros\n  target:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: vm-clone-target\n\n  # labels &amp; annotations definitions\n  labelFilters:\n    - \"*\"\n    - \"!someKey/*\"\n  annotationFilters:\n    - \"anotherKey/*\"\n\n  # template labels &amp; annotations definitions\n  template:\n    labelFilters:\n      - \"*\"\n      - \"!someKey/*\"\n    annotationFilters:\n      - \"anotherKey/*\"\n\n  # other identity stripping specs:\n  newMacAddresses:\n    interfaceName: \"00-11-22\"\n  newSMBiosSerial: \"new-serial\"\n</code></pre></p> <p>In the next section I will go through the different settings to elaborate them.</p>"},{"location":"storage/clone_api/#source-target","title":"Source &amp; Target","text":"<p>The source and target indicate the source/target API group, kind and name. A few important notes:</p> <ul> <li> <p>Currently, the only supported kinds are <code>VirtualMachine</code> (of <code>kubevirt.io</code> api group) and <code>VirtualMachineSnapshot</code> ( of <code>snapshot.kubevirt.io</code> api group), but more types are expected to be supported in the future. See \"future roadmap\" below for more info.</p> </li> <li> <p>The target name is optional. If unspecified, the clone controller will generate a name for the target automatically.</p> </li> <li> <p>The target and source must reside in the same namespace.</p> </li> </ul>"},{"location":"storage/clone_api/#label-annotation-filters","title":"Label &amp; Annotation filters","text":"<p>These spec fields are intended to determine which labels / annotations are being copied to the target or stripped away.</p> <p>The filters are a list of strings. Each string represents a key that may exist at the source. Every source key that matches to one of these values is being copied to the cloned target. In addition, special regular-expression-like characters can be used:</p> <ul> <li>Wildcard character (*) can be used to match anything. Wildcard can be only used at the end of the filter.</li> <li>These filters are valid:<ul> <li>\"*\"</li> <li>\"some/key*\"</li> </ul> </li> <li>These filters are invalid:<ul> <li>\"some/*/key\"</li> <li>\"*/key\"</li> </ul> </li> <li>Negation character (!) can be used to avoid matching certain keys. Negation can be only used at the beginning of a filter.   Note that a Negation and Wildcard can be used together.</li> <li>These filters are valid:<ul> <li>\"!some/key\"</li> <li>\"!some/*\"</li> </ul> </li> <li>These filters are invalid:<ul> <li>\"key!\"</li> <li>\"some/!key\"</li> </ul> </li> </ul> <p>Setting label / annotation filters is optional. If unset, all labels / annotations will be copied as a default.</p>"},{"location":"storage/clone_api/#template-label-template-annotation-filters","title":"Template Label &amp; Template Annotation filters","text":"<p>Some network CNIs such as Kube-OVN or OVN-Kubernetes inject network information into the annotations of a VM. When cloning a VM from a target VM the cloned VM will use the same network. To avoid this you can use template labels and annotation filters.</p>"},{"location":"storage/clone_api/#newmacaddresses","title":"newMacAddresses","text":"<p>This field is used to explicitly replace MAC addresses for certain interfaces. The field is a string to string map; the keys represent interface names and the values represent the new MAC address for the clone target.</p> <p>This field is optional. By default, all mac addresses are stripped out. This suits situations when kube-mac-pool is deployed in the cluster which would automatically assign the target with a fresh valid MAC address.</p>"},{"location":"storage/clone_api/#newsmbiosserial","title":"newSMBiosSerial","text":"<p>This field is used to explicitly set an SMBios serial for the target.</p> <p>This field is optional. By default, the target would have an auto-generated serial that's based on the VM name.</p>"},{"location":"storage/clone_api/#creating-a-virtualmachineclone-object","title":"Creating a VirtualMachineClone object","text":"<p>After the clone manifest is ready, we can create it: <pre><code>kubectl create -f clone.yaml\n</code></pre></p> <p>To wait for a clone to complete, execute: <pre><code>kubectl wait vmclone testclone --for condition=Ready\n</code></pre></p> <p>You can check the clone's phase in the clone's status. It can be one of the following:</p> <ul> <li> <p>SnapshotInProgress</p> </li> <li> <p>CreatingTargetVM</p> </li> <li> <p>RestoreInProgress</p> </li> <li> <p>Succeeded</p> </li> <li> <p>Failed</p> </li> <li> <p>Unknown</p> </li> </ul> <p>After the clone is finished, the target can be inspected: <pre><code>kubectl get vm vm-clone-target -o yaml\n</code></pre></p>"},{"location":"storage/clone_api/#future-roadmap","title":"Future roadmap","text":"<p>The clone API is in an early alpha version and may change dramatically. There are many improvements and features that are expected to be added, the most significant goals are:</p> <ul> <li>Add more supported source types like <code>VirtualMachineInstace</code> in the future.</li> <li>Add a cross-namespace clone support. This needs to be supported for snapshots / restores first.</li> </ul>"},{"location":"storage/clone_api/#using-clones-as-a-golden-vm-image","title":"Using clones as a \"golden VM image\"","text":"<p>One of the great things that could be accomplished with the clone API when the source is of kind <code>VirtualMachineSnapshot</code> is to create \"golden VM images\" (a.k.a. Templates / Bookmark VMs / etc). In other words, the following workflow would be available:</p> <p>Create a golden image</p> <ul> <li> <p>Create a VM</p> </li> <li> <p>Prepare a \"golden VM\" environment</p> </li> <li> <p>This can mean different things in different contexts. For example, write files, install applications, apply configurations,     or anything else.</p> </li> <li> <p>Snapshot the VM</p> </li> <li> <p>Delete the VM</p> </li> </ul> <p>Then, this \"golden image\" can be duplicated as many times as needed. To instantiate a VM from the snapshot:</p> <ul> <li>Create a Clone object where the source would point to the previously taken snapshot</li> <li>Create as many VMs you need</li> </ul> <p>This feature is still under discussions and may be implemented differently then explained here.</p>"},{"location":"storage/containerized_data_importer/","title":"Containerized Data Importer","text":"<p>The Containerized Data Importer (CDI) project provides facilities for enabling Persistent Volume Claims (PVCs) to be used as disks for KubeVirt VMs by way of DataVolumes. The three main CDI use cases are:</p> <ul> <li>Import a disk image from a web server or container registry to a DataVolume</li> <li>Clone an existing PVC to a DataVolume</li> <li>Upload a local disk image to a DataVolume</li> </ul> <p>This document deals with the third use case. So you should have CDI installed in your cluster, a VM disk that you'd like to upload, and virtctl in your path.</p>"},{"location":"storage/containerized_data_importer/#install-cdi","title":"Install CDI","text":"<p>Install the latest CDI release here</p> <pre><code>export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)\nexport VERSION=$(echo ${TAG##*/})\nkubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml\nkubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml\n</code></pre>"},{"location":"storage/containerized_data_importer/#expose-cdi-uploadproxy-service","title":"Expose cdi-uploadproxy service","text":"<p>The <code>cdi-uploadproxy</code> service must be accessible from outside the cluster. Here are some ways to do that:</p> <ul> <li> <p>NodePort     Service</p> </li> <li> <p>Ingress</p> </li> <li> <p>Route</p> </li> <li> <p>kubectl     port-forward     (not recommended for production clusters)</p> </li> </ul> <p>Look here for example manifests.</p>"},{"location":"storage/containerized_data_importer/#supported-image-formats","title":"Supported image formats","text":"<p>CDI supports the <code>raw</code> and <code>qcow2</code> image formats which are supported by qemu. See the qemu documentation for more details.  Bootable ISO images can also be used and are treated like <code>raw</code> images.  Images may be compressed with either the <code>gz</code> or <code>xz</code> format.</p> <p>The example in this document uses this CirrOS image</p>"},{"location":"storage/containerized_data_importer/#virtctl-image-upload","title":"virtctl image-upload","text":"<p>virtctl has an image-upload command with the following options:</p> <pre><code>virtctl image-upload --help\nUpload a VM image to a DataVolume/PersistentVolumeClaim.\n\nUsage:\n  virtctl image-upload [flags]\n\nExamples:\n  # Upload a local disk image to a newly created DataVolume:\n  virtctl image-upload dv dv-name --size=10Gi --image-path=/images/fedora30.qcow2\n\n  # Upload a local disk image to an existing DataVolume\n  virtctl image-upload dv dv-name --no-create --image-path=/images/fedora30.qcow2\n\n  # Upload a local disk image to an existing PersistentVolumeClaim\n  virtctl image-upload pvc pvc-name --image-path=/images/fedora30.qcow2\n\n  # Upload to a DataVolume with explicit URL to CDI Upload Proxy\n  virtctl image-upload dv dv-name --uploadproxy-url=https://cdi-uploadproxy.mycluster.com --image-path=/images/fedora30.qcow2\n\nFlags:\n      --access-mode string       The access mode for the PVC. (default \"ReadWriteOnce\")\n      --block-volume             Create a PVC with VolumeMode=Block (default Filesystem).\n  -h, --help                     help for image-upload\n      --image-path string        Path to the local VM image.\n      --insecure                 Allow insecure server connections when using HTTPS.\n      --no-create                Don't attempt to create a new DataVolume/PVC.\n      --pvc-name string          DEPRECATED - The destination DataVolume/PVC name.\n      --pvc-size string          DEPRECATED - The size of the PVC to create (ex. 10Gi, 500Mi).\n      --size string              The size of the DataVolume to create (ex. 10Gi, 500Mi).\n      --storage-class string     The storage class for the PVC.\n      --uploadproxy-url string   The URL of the cdi-upload proxy service.\n      --wait-secs uint           Seconds to wait for upload pod to start. (default 60)\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre> <p><code>virtctl image-upload</code> works by creating a DataVolume of the requested size, sending an <code>UploadTokenRequest</code> to the <code>cdi-apiserver</code>, and uploading the file to the <code>cdi-uploadproxy</code>.</p> <pre><code>virtctl image-upload dv cirros-vm-disk --size=500Mi --image-path=/home/mhenriks/images/cirros-0.4.0-x86_64-disk.img --uploadproxy-url=&lt;url to upload proxy service&gt;\n</code></pre>"},{"location":"storage/containerized_data_importer/#addressing-certificate-issues-when-uploading-images","title":"Addressing Certificate Issues when Uploading Images","text":"<p>Issues with the certificates can be circumvented by using the <code>--insecure</code> flag to prevent the virtctl command from verifying the remote host. It is better to resolve certificate issues that prevent uploading images using the <code>virtctl image-upload</code> command and not use the <code>--insecure</code> flag.</p> <p>The following are some common issues with certificates and some easy ways to fix them.</p>"},{"location":"storage/containerized_data_importer/#does-not-contain-any-ip-sans","title":"Does not contain any IP SANs","text":"<p>This issue happens when trying to upload images using an IP address instead of a resolvable name. For example, trying to upload to the IP address 192.168.39.32 at port 31001 would produce the following error.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://192.168.39.32:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://192.168.39.32:31001\n\n 0 B / 193.89 MiB [-------------------------------------------------------]   0.00% 0s\n\nPost https://192.168.39.32:31001/v1beta1/upload: x509: cannot validate certificate for 192.168.39.32 because it doesn't contain any IP SANs\n</code></pre> <p>It is easily fixed by adding an entry it your local name resolution service. This could be a DNS server or the local hosts file. The URL used to upload the proxy should be changed to reflect the resolvable name.</p> <p>The <code>Subject</code> and the <code>Subject Alternative Name</code> in the certificate contain valid names that can be used for resolution. Only one of these names needs to be resolvable. Use the <code>openssl</code> command to view the names of the cdi-uploadproxy service.</p> <pre><code>echo | openssl s_client -showcerts -connect 192.168.39.32:31001 2&gt;/dev/null \\\n     | openssl x509 -inform pem -noout -text \\\n     | sed -n -e '/Subject.*CN/p' -e '/Subject Alternative/{N;p}'\n\n    Subject: CN = cdi-uploadproxy\n        X509v3 Subject Alternative Name: \n            DNS:cdi-uploadproxy, DNS:cdi-uploadproxy.cdi, DNS:cdi-uploadproxy.cdi.svc\n</code></pre> <p>Adding the following entry to the /etc/hosts file, if it provides name resolution, should fix this issue. Any service that provides name resolution for the system could be used.</p> <pre><code>echo \"192.168.39.32  cdi-uploadproxy\" &gt;&gt; /etc/hosts\n</code></pre> <p>The upload should now work.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 193.89 MiB / 193.89 MiB [=============================================] 100.00% 1m38s\n\nUploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress\nProcessing completed successfully\nUploading Fedora-Cloud-Base-33-1.2.x86_64.raw.xz completed successfully\n</code></pre>"},{"location":"storage/containerized_data_importer/#certificate-signed-by-unknown-authority","title":"Certificate Signed by Unknown Authority","text":"<p>This happens because the cdi-uploadproxy certificate is self signed and the system does not trust the cdi-uploadproxy as a Certificate Authority.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 0 B / 193.89 MiB [-------------------------------------------------------]   0.00% 0s\n\nPost https://cdi-uploadproxy:31001/v1beta1/upload: x509: certificate signed by unknown authority\n</code></pre> <p>This can be fixed by adding the certificate to the systems trust store. Download the cdi-uploadproxy-server-cert.</p> <pre><code>kubectl get secret -n cdi cdi-uploadproxy-server-cert \\\n  -o jsonpath=\"{.data['tls\\.crt']}\" \\\n  | base64 -d &gt; cdi-uploadproxy-server-cert.crt\n</code></pre> <p>Add this certificate to the systems trust store. On Fedora, this can be done as follows.</p> <pre><code>sudo cp cdi-uploadproxy-server-cert.crt /etc/pki/ca-trust/source/anchors\n\nsudo update-ca-trust\n</code></pre> <p>The upload should now work.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 193.89 MiB / 193.89 MiB [=============================================] 100.00% 1m36s\n\nUploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress\nProcessing completed successfully\nUploading Fedora-Cloud-Base-33-1.2.x86_64.raw.xz completed successfully\n</code></pre>"},{"location":"storage/containerized_data_importer/#setting-the-url-of-the-cdi-upload-proxy-service","title":"Setting the URL of the cdi-upload Proxy Service","text":"<p>Setting the URL for the cdi-upload proxy service allows the <code>virtctl image-upload</code> command to upload the images without specifying the <code>--uploadproxy-url</code> flag. Permanently setting the URL is done by patching the CDI configuration.</p> <p>The following will set the default upload proxy to use port 31001 of cdi-uploadproxy. An IP address could also be used instead of the dns name.</p> <p>See the section Addressing Certificate Issues when Uploading for why cdi-uploadproxy was chosen and issues that can be encountered when using an IP address.</p> <pre><code>kubectl patch cdi cdi \\\n  --type merge \\\n  --patch '{\"spec\":{\"config\":{\"uploadProxyURLOverride\":\"https://cdi-uploadproxy:31001\"}}}'\n</code></pre>"},{"location":"storage/containerized_data_importer/#create-a-virtualmachineinstance","title":"Create a VirtualMachineInstance","text":"<p>To create a <code>VirtualMachineInstance</code> from a DataVolume, you can execute the following:</p> <pre><code>cat &lt;&lt;EOF | kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: cirros-vm\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: dvdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: dvdisk\n    dataVolume:\n      name: cirros-vm-disk\nstatus: {}\nEOF\n</code></pre>"},{"location":"storage/containerized_data_importer/#connect-to-virtualmachineinstance-console","title":"Connect to VirtualMachineInstance console","text":"<p>Use <code>virtctl</code> to connect to the newly create <code>VirtualMachineInstance</code>.</p> <pre><code>virtctl console cirros-vm\n</code></pre>"},{"location":"storage/disks_and_volumes/","title":"Filesystems, Disks and Volumes","text":"<p>Making persistent storage in the cluster (volumes) accessible to VMs consists of three parts. First, volumes are specified in <code>spec.volumes</code>. Second, disks are added to the VM by specifying them in <code>spec.domain.devices.disks</code>. Finally, a reference to the specified volume is added to the disk specification by name.</p>"},{"location":"storage/disks_and_volumes/#disks","title":"Disks","text":"<p>Like all other vmi devices a <code>spec.domain.devices.disks</code> element has a mandatory <code>name</code>, and furthermore, the disk's <code>name</code> must reference the <code>name</code> of a volume inside <code>spec.volumes</code>.</p> <p>A disk can be made accessible via four different types:</p> <ul> <li> <p>lun</p> </li> <li> <p>disk</p> </li> <li> <p>cdrom</p> </li> <li> <p>fileystems</p> </li> </ul> <p>All possible configuration options are available in the Disk API Reference.</p> <p>All types allow you to specify the <code>bus</code> attribute. The <code>bus</code> attribute determines how the disk will be presented to the guest operating system.</p>"},{"location":"storage/disks_and_volumes/#lun","title":"lun","text":"<p>A <code>lun</code> disk will expose the volume as a LUN device to the VM. This allows the VM to execute arbitrary iSCSI command passthrough.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>lun</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-lun\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a lun device\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#persistent-reservation","title":"persistent reservation","text":"<p>It is possible to reserve a LUN through the the SCSI Persistent Reserve commands. In order to issue privileged SCSI ioctls, the VM requires activation of the persistent resevation flag:</p> <pre><code>devices:\n  disks:\n  - name: mypvcdisk\n    lun:\n      reservation: true\n</code></pre> <p>This feature is enabled by the feature gate <code>PersistentReservation</code>:</p> <pre><code>configuration:\n  developerConfiguration:\n    featureGates:\n    -  PersistentReservation\n</code></pre> <p>Note: The persistent reservation feature enables an additional privileged component to be deployed together with virt-handler. Because this feature allows for sensitive security procedures, it is disabled by default and requires cluster administrator configuration.</p>"},{"location":"storage/disks_and_volumes/#disk","title":"disk","text":"<p>A <code>disk</code> disk will expose the volume as an ordinary disk to the VM.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>disk</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a disk\n        disk: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>You can set the disk <code>bus</code> type, overriding the defaults, which in turn depends on the chipset the VM is configured to use:</p> <pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a disk\n        disk:\n          # This makes it exposed as /dev/vda, being the only and thus first\n          # disk attached to the VM\n          bus: virtio\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#cdrom","title":"cdrom","text":"<p>A <code>cdrom</code> disk will expose the volume as a cdrom drive to the VM. It is read-only by default.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>cdrom</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-cdrom\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a cdrom\n        cdrom:\n          # This makes the cdrom writeable\n          readOnly: false\n          # This makes the cdrom be exposed as SATA device\n          bus: sata\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#filesystems","title":"filesystems","text":"<p>A <code>filesystem</code> device will expose the volume as a filesystem to the VM. <code>filesystems</code> rely on <code>virtiofs</code> to make visible external filesystems to <code>KubeVirt</code> VMs.  Further information about <code>virtiofs</code> can be found at the Official Virtiofs Site.</p> <p>Compared with <code>disk</code>, <code>filesystems</code> allow changes in the source to be dynamically reflected in the volumes inside the VM. For instance, if a given <code>configMap</code> is shared with <code>filesystems</code> any change made on it will be reflected in the VMs. However, it is important to note that <code>filesystems</code> do not allow live migration.</p> <p>Additionally, <code>filesystem</code> devices must be mounted inside the VM. This can be done through cloudInitNoCloud or manually connecting to the VM shell and targeting the same command. The main challenge is to understand how the device tag used to identify the new filesystem and mount it with the  <code>mount -t virtiofs [device tag] [path]</code> command. For that purpose, the tag is assigned to the filesystem in the VM spec <code>spec.domain.devices.filesystems.name</code>. For instance, if in a given VM spec is <code>spec.domain.devices.filesystems.name: foo</code>, the required command inside the VM to mount the filesystem in the <code>/tmp/foo</code> path will be <code>mount -t virtiofs foo /tmp/foo</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-filesystems\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: foo\n          virtiofs: {}\n      disks:\n        - name: containerdisk\n          disk:\n            bus: virtio\n        - name: cloudinitdisk\n          disk:\n            bus: virtio\n    volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk \n      - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: fedora\n              user: fedora\n              bootcmd:\n                - \"sudo mkdir /tmp/foo\"\n                - \"sudo mount -t virtiofs foo /tmp/foo\"\n      - persistentVolumeClaim:\n          claimName: mypvc\n        name: foo\n</code></pre> <p>Note: As stated, <code>filesystems</code> rely on <code>virtiofs</code>. Moreover, <code>virtiofs</code> requires kernel linux support to work in  the VM. To check if the linux image of the VM has the required support, you can address the following command: <code>modprobe virtiofs</code>. If the command output is <code>modprobe: FATAL: Module virtiofs not found</code>, the linux image of the VM does not support virtiofs. Also, you can check if the kernel version is up to 5.4 in any linux distribution or up to 4.18 in centos/rhel.  To check this, you can target the following command: <code>uname -r</code>.</p> <p>Refer to section Sharing Directories with VMs for usage examples of <code>filesystems</code>.</p>"},{"location":"storage/disks_and_volumes/#error-policy","title":"error policy","text":"<p>The error policy controls how the hypervisor should behave when an IO error occurs on a disk read or write. The default behaviour is to stop the guest and a Kubernetes event is generated. However, it is possible to change the value to either:</p> <ul> <li><code>report</code>: the error is reported in the guest</li> <li><code>ignore</code>: the error is ignored, but the read/write failure goes undetected</li> <li><code>enospace</code>: error when there isn't enough space on the disk</li> </ul> <p>The error policy can be specified per disk or lun.</p> <p>Example: <pre><code>spec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n        errorPolicy: \"report\"\n      - lun:\n          bus: scsi\n        name: scsi-disk\n        errorPolicy: \"report\"\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#volumes","title":"Volumes","text":"<p>Supported volume sources are</p> <ul> <li> <p>cloudInitNoCloud</p> </li> <li> <p>cloudInitConfigDrive</p> </li> <li> <p>persistentVolumeClaim</p> </li> <li> <p>dataVolume</p> </li> <li> <p>ephemeral</p> </li> <li> <p>containerDisk</p> </li> <li> <p>emptyDisk</p> </li> <li> <p>hostDisk</p> </li> <li> <p>configMap</p> </li> <li> <p>secret</p> </li> <li> <p>serviceAccount</p> </li> <li> <p>downwardMetrics</p> </li> </ul> <p>All possible configuration options are available in the Volume API Reference.</p>"},{"location":"storage/disks_and_volumes/#cloudinitnocloud","title":"cloudInitNoCloud","text":"<p>Allows attaching <code>cloudInitNoCloud</code> data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.</p> <p>A simple example which attaches a <code>Secret</code> as a cloud-init <code>disk</code> datasource may look like this:</p> <pre><code>metadata:\n  name: testvmi-cloudinitnocloud\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mybootdisk\n        lun: {}\n      - name: mynoclouddisk\n        disk: {}\n  volumes:\n    - name: mybootdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n    - name: mynoclouddisk\n      cloudInitNoCloud:\n        secretRef:\n          name: testsecret\n</code></pre>"},{"location":"storage/disks_and_volumes/#cloudinitconfigdrive","title":"cloudInitConfigDrive","text":"<p>Allows attaching <code>cloudInitConfigDrive</code> data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.</p> <p>A simple example which attaches a <code>Secret</code> as a cloud-init <code>disk</code> datasource may look like this:</p> <pre><code>metadata:\n  name: testvmi-cloudinitconfigdrive\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mybootdisk\n        lun: {}\n      - name: myconfigdrivedisk\n        disk: {}\n  volumes:\n    - name: mybootdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n    - name: myconfigdrivedisk\n      cloudInitConfigDrive:\n        secretRef:\n          name: testsecret\n</code></pre> <p>The <code>cloudInitConfigDrive</code> can also be used to configure VMs with Ignition. You just need to replace the cloud-init data by the Ignition data.</p>"},{"location":"storage/disks_and_volumes/#persistentvolumeclaim","title":"persistentVolumeClaim","text":"<p>Allows connecting a <code>PersistentVolumeClaim</code> to a VM disk.</p> <p>Use a PersistentVolumeClaim when the VirtualMachineInstance's disk needs to persist after the VM terminates. This allows for the VM's data to remain persistent between restarts.</p> <p>A <code>PersistentVolume</code> can be in \"filesystem\" or \"block\" mode:</p> <ul> <li> <p>Filesystem: For KubeVirt to be able to consume the disk present on a     PersistentVolume's filesystem, the disk must be named <code>disk.img</code> and     be placed in the root path of the filesystem. Currently the disk is     also required to be in raw format. &gt; Important: The     <code>disk.img</code> image file needs to be owned by the user-id <code>107</code> in     order to avoid permission issues.</p> <p>Note: If the <code>disk.img</code> image file has not been created manually before starting a VM then it will be created automatically with the <code>PersistentVolumeClaim</code> size. Since not every storage provisioner provides volumes with the exact usable amount of space as requested (e.g. due to filesystem overhead), KubeVirt tolerates up to 10% less available space. This can be configured with the <code>developerConfiguration.pvcTolerateLessSpaceUpToPercent</code> value in the KubeVirt CR (<code>kubectl edit kubevirt kubevirt -n kubevirt</code>).</p> </li> <li> <p>Block: Use a block volume for consuming raw block devices. Note: you     need to enable the <code>BlockVolume</code> feature gate.</p> </li> </ul> <p>A simple example which attaches a <code>PersistentVolumeClaim</code> as a <code>disk</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-pvc\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#thick-and-thin-volume-provisioning","title":"Thick and thin volume provisioning","text":"<p>Sparsification can make a disk thin-provisioned, in other words it allows to convert the freed space within the disk image into free space back on the host. The fstrim utility can be used on a mounted filesystem to discard the blocks not used by the filesystem. In order to be able to sparsify a disk inside the guest, the disk needs to be configured in the libvirt xml with the option <code>discard=unmap</code>. In KubeVirt, every disk is passed as default with this option enabled. It is possible to check if the trim configuration is supported in the guest by running<code>lsblk -D</code>, and check the discard options supported on every disk.</p> <p>Example: <pre><code>$ lsblk -D\nNAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO\nloop0         0        4K       4G         0\nloop1         0       64K       4M         0\nsr0           0        0B       0B         0\nrbd0          0       64K       4M         0\nvda         512      512B       2G         0\n\u2514\u2500vda1        0      512B       2G         0\n</code></pre></p> <p>However, in certain cases like preallocaton or when the disk is thick provisioned, the option needs to be disabled. The disk's PVC has to be marked with an annotation that contains <code>/storage.preallocation</code> or <code>/storage.thick-provisioned</code>, and set to true. If the volume is preprovisioned using CDI and the preallocation is enabled, then the PVC is automatically annotated with: <code>cdi.kubevirt.io/storage.preallocation: true</code> and the discard passthrough option is disabled.</p> <p>Example of a PVC definition with the annotation to disable discard passthrough: <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: pvc\n  annotations:\n    user.custom.annotation/storage.thick-provisioned: \"true\"\nspec:\n  storageClassName: local\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Filesystem\n  resources:\n    requests:\n      storage: 1Gi\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#disk-expansion","title":"disk expansion","text":"<p>For some storage methods, Kubernetes may support expanding storage in-use (allowVolumeExpansion feature). KubeVirt can respond to it by making the additional storage available for the virtual machines. This feature is currently off by default, and requires enabling a feature gate. To enable it, add the ExpandDisks feature gate in the kubevirt object:</p> <pre><code>spec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n      - ExpandDisks\n</code></pre> <p>Enabling this feature does two things: - Notify the virtual machine about size changes - If the disk is a Filesystem PVC, the matching file is expanded   to the remaining size (while reserving some space for file system overhead).</p>"},{"location":"storage/disks_and_volumes/#statically-provisioned-block-pvcs","title":"Statically provisioned block PVCs","text":"<p>To use an externally managed local block device from a host ( e.g. /dev/sdb , zvol, LVM, etc... ) in a VM directly, you would need a provisioner that supports block devices, such as OpenEBS LocalPV.</p> <p>Alternatively, local volumes can be provisioned by hand. I.e. the following PVC:</p> <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: myblock\nspec:\n  storageClassName: local-device\n  volumeMode: Block\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 100Gi\n</code></pre> <p>can claim a PersistentVolume pre-created by a cluster admin like so:</p> <pre><code>apiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n  name: local-device\nprovisioner: kubernetes.io/no-provisioner\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: myblock\nspec:\n  volumeMode: Block\n  storageClassName: local-device\n  nodeAffinity:\n    required:\n      nodeSelectorTerms:\n      - matchExpressions:\n        - key: kubernetes.io/hostname\n          operator: In\n          values:\n          - my-node\n  accessModes:\n    - ReadWriteOnce\n  capacity:\n    storage: 100Gi\n  local:\n    path: /dev/sdb\n</code></pre>"},{"location":"storage/disks_and_volumes/#datavolume","title":"dataVolume","text":"<p>DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine's launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user.</p>"},{"location":"storage/disks_and_volumes/#datavolume-vm-behavior","title":"DataVolume VM Behavior","text":"<p>DataVolumes can be defined in the VM spec directly by adding the DataVolumes to the <code>dataVolumeTemplates</code> list. Below is an example.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-alpine-datavolume\n  name: vm-alpine-datavolume\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-alpine-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 64M\n      volumes:\n      - dataVolume:\n          name: alpine-dv\n        name: datavolumedisk1\n  dataVolumeTemplates:\n  - metadata:\n      name: alpine-dv\n    spec:\n      storage:\n        resources:\n          requests:\n            storage: 2Gi\n      source:\n        http:\n          url: http://cdi-http-import-server.kubevirt/images/alpine.iso\n</code></pre> <p>You can see the DataVolume defined in the dataVolumeTemplates section has two parts. The source and pvc</p> <p>The source part declares that there is a disk image living on an http server that we want to use as a volume for this VM. The pvc part declares the spec that should be used to create the PVC that hosts the source data.</p> <p>When this VM manifest is posted to the cluster, as part of the launch flow a PVC will be created using the spec provided and the source data will be automatically imported into that PVC before the VM starts. When the VM is deleted, the storage provisioned by the DataVolume will automatically be deleted as well.</p>"},{"location":"storage/disks_and_volumes/#datavolume-vmi-behavior","title":"DataVolume VMI Behavior","text":"<p>For a VMI object, DataVolumes can be referenced as a volume source for the VMI. When this is done, it is expected that the referenced DataVolume exists in the cluster. The VMI will consume the DataVolume, but the DataVolume's life-cycle will not be tied to the VMI.</p> <p>Below is an example of a DataVolume being referenced by a VMI. It is expected that the DataVolume alpine-datavolume was created prior to posting the VMI manifest to the cluster. It is okay to post the VMI manifest to the cluster while the DataVolume is still having data imported. KubeVirt knows not to start the VMI until all referenced DataVolumes have finished their clone and import phases.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-alpine-datavolume\n  name: vmi-alpine-datavolume\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: disk1\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: disk1\n    dataVolume:\n      name: alpine-datavolume\n</code></pre>"},{"location":"storage/disks_and_volumes/#enabling-datavolume-support","title":"Enabling DataVolume support.","text":"<p>A DataVolume is a custom resource provided by the Containerized Data Importer (CDI) project. KubeVirt integrates with CDI in order to provide users a workflow for dynamically creating PVCs and importing data into those PVCs.</p> <p>In order to take advantage of the DataVolume volume source on a VM or VMI, CDI must be installed.</p> <p>Installing CDI</p> <p>Go to the CDI release page</p> <p>Pick the latest stable release and post the corresponding cdi-controller-deployment.yaml manifest to your cluster.</p>"},{"location":"storage/disks_and_volumes/#ephemeral","title":"ephemeral","text":"<p>An ephemeral volume is a local COW (copy on write) image that uses a network volume as a read-only backing store. With an ephemeral volume, the network backing store is never mutated. Instead all writes are stored on the ephemeral image which exists on local storage. KubeVirt dynamically generates the ephemeral images associated with a VM when the VM starts, and discards the ephemeral images when the VM stops.</p> <p>Ephemeral volumes are useful in any scenario where disk persistence is not desired. The COW image is discarded when VM reaches a final state (e.g., succeeded, failed).</p> <p>Currently, only <code>PersistentVolumeClaim</code> may be used as a backing store of the ephemeral volume.</p> <p>Up-to-date information on supported backing stores can be found in the KubeVirt API.</p> <pre><code>metadata:\n  name: testvmi-ephemeral-pvc\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      ephemeral:\n        persistentVolumeClaim:\n          claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#containerdisk","title":"containerDisk","text":"<p>containerDisk was originally registryDisk, please update your code when needed.</p> <p>The <code>containerDisk</code> feature provides the ability to store and distribute VM disks in the container image registry. <code>containerDisks</code> can be assigned to VMs in the disks section of the VirtualMachineInstance spec.</p> <p>No network shared storage devices are utilized by <code>containerDisks</code>. The disks are pulled from the container registry and reside on the local node hosting the VMs that consume the disks.</p>"},{"location":"storage/disks_and_volumes/#when-to-use-a-containerdisk","title":"When to use a containerDisk","text":"<p><code>containerDisks</code> are ephemeral storage devices that can be assigned to any number of active VirtualMachineInstances. This makes them an ideal tool for users who want to replicate a large number of VM workloads that do not require persistent data. <code>containerDisks</code> are commonly used in conjunction with VirtualMachineInstanceReplicaSets.</p>"},{"location":"storage/disks_and_volumes/#when-not-to-use-a-containerdisk","title":"When Not to use a containerDisk","text":"<p><code>containerDisks</code> are not a good solution for any workload that requires persistent root disks across VM restarts.</p>"},{"location":"storage/disks_and_volumes/#containerdisk-workflow-example","title":"containerDisk Workflow Example","text":"<p>Users can inject a VirtualMachineInstance disk into a container image in a way that is consumable by the KubeVirt runtime. Disks must be placed into the <code>/disk</code> directory inside the container. Raw and qcow2 formats are supported. Qcow2 is recommended in order to reduce the container image's size. <code>containerdisks</code> can and should be based on <code>scratch</code>. No content except the image is required.</p> <p>Note: Prior to kubevirt 0.20, the containerDisk image needed to have kubevirt/container-disk-v1alpha as base image.</p> <p>Note: The containerDisk needs to be readable for the user with the UID 107 (qemu).</p> <p>Example: Inject a local VirtualMachineInstance disk into a container image.</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD --chown=107:107 fedora25.qcow2 /disk/\nEND\n\ndocker build -t vmidisks/fedora25:latest .\n</code></pre> <p>Example: Inject a remote VirtualMachineInstance disk into a container image.</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD --chown=107:107 https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 /disk/\nEND\n</code></pre> <p>Example: Upload the ContainerDisk container image to a registry.</p> <pre><code>docker push vmidisks/fedora25:latest\n</code></pre> <p>Example: Attach the ContainerDisk as an ephemeral disk to a VM.</p> <pre><code>metadata:\n  name: testvmi-containerdisk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk: {}\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: vmidisks/fedora25:latest\n</code></pre> <p>Note that a <code>containerDisk</code> is file-based and therefore cannot be attached as a <code>lun</code> device to the VM.</p>"},{"location":"storage/disks_and_volumes/#custom-disk-image-path","title":"Custom disk image path","text":"<p>ContainerDisk also allows to store disk images in any folder, when required. The process is the same as previous. The main difference is, that in custom location, kubevirt does not scan for any image. It is your responsibility to provide full path for the disk image. Providing image <code>path</code> is optional. When no <code>path</code> is provided, kubevirt searches for disk images in default location: <code>/disk</code>.</p> <p>Example: Build container disk image:</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD fedora25.qcow2 /custom-disk-path/fedora25.qcow2\nEND\n\ndocker build -t vmidisks/fedora25:latest .\ndocker push vmidisks/fedora25:latest\n</code></pre> <p>Create VMI with container disk pointing to the custom location:</p> <pre><code>metadata:\n  name: testvmi-containerdisk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk: {}\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: vmidisks/fedora25:latest\n        path: /custom-disk-path/fedora25.qcow2\n</code></pre>"},{"location":"storage/disks_and_volumes/#emptydisk","title":"emptyDisk","text":"<p>An <code>emptyDisk</code> works similar to an <code>emptyDir</code> in Kubernetes. An extra sparse <code>qcow2</code> disk will be allocated and it will live as long as the VM. Thus it will survive guest side VM reboots, but not a VM re-creation. The disk <code>capacity</code> needs to be specified.</p> <p>Example: Boot cirros with an extra <code>emptyDisk</code> with a size of <code>2GiB</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: emptydisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: emptydisk\n      emptyDisk:\n        capacity: 2Gi\n</code></pre>"},{"location":"storage/disks_and_volumes/#when-to-use-an-emptydisk","title":"When to use an emptyDisk","text":"<p>Ephemeral VMs very often come with read-only root images and limited tmpfs space. In many cases this is not enough to install application dependencies and provide enough disk space for the application data. While this data is not critical and thus can be lost, it is still needed for the application to function properly during its lifetime. This is where an <code>emptyDisk</code> can be useful. An emptyDisk is often used and mounted somewhere in <code>/var/lib</code> or <code>/var/run</code>.</p>"},{"location":"storage/disks_and_volumes/#hostdisk","title":"hostDisk","text":"<p>A <code>hostDisk</code> volume type provides the ability to create or use a disk image located somewhere on a node. It works similar to a <code>hostPath</code> in Kubernetes and provides two usage types:</p> <ul> <li> <p><code>DiskOrCreate</code> if a disk image does not exist at a given location     then create one</p> </li> <li> <p><code>Disk</code> a disk image must exist at a given location</p> </li> </ul> <p>Note: you need to enable the HostDisk feature gate.</p> <p>Example: Create a 1Gi disk image located at /data/disk.img and attach it to a VM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-host-disk\n  name: vmi-host-disk\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: host-disk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - hostDisk:\n      capacity: 1Gi\n      path: /data/disk.img\n      type: DiskOrCreate\n    name: host-disk\nstatus: {}\n</code></pre> <p>Note: This does not always work as expected. Instead you may want to consider creating a PersistentVolume</p>"},{"location":"storage/disks_and_volumes/#configmap","title":"configMap","text":"<p>A <code>configMap</code> is a reference to a ConfigMap in Kubernetes.  A <code>configMap</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk","title":"As a disk","text":"<p>By using disk, an extra <code>iso</code> disk will be allocated which has to be mounted on a VM. To mount the <code>configMap</code> users can use <code>cloudInit</code> and the disk's serial number. The <code>name</code> needs to be set for a reference to the created kubernetes <code>ConfigMap</code>.</p> <p>Note: Currently, ConfigMap update is not propagate into the VMI. If a ConfigMap is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where ConfigMap keys are projected.</p> <p>Example: Attach the <code>configMap</code> to a VM and use <code>cloudInit</code> to mount the <code>iso</code> disk:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      - disk:\n        name: app-config-disk\n        # set serial\n        serial: CVLY623300HK240D\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          # mount the ConfigMap\n          - \"sudo mkdir /mnt/app-config\"\n          - \"sudo mount /dev/$(lsblk --nodeps -no name,serial | grep CVLY623300HK240D | cut -f1 -d' ') /mnt/app-config\"\n    name: cloudinitdisk\n  - configMap:\n      name: app-config\n    name: app-config-disk\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#as-a-filesystem","title":"As a filesystem","text":"<p>By using filesystem, <code>configMaps</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>configMaps</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>configMaps</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>configMap</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: config-fs\n          virtiofs: {}\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        chpasswd:\n          expire: false\n        password: fedora\n        user: fedora\n        bootcmd:\n          # mount the ConfigMap\n          - \"sudo mkdir /mnt/app-config\"\n          - \"sudo mount -t virtiofs config-fs /mnt/app-config\"\n    name: cloudinitdisk      \n  - configMap:\n      name: app-config\n    name: config-fs\n</code></pre>"},{"location":"storage/disks_and_volumes/#secret","title":"secret","text":"<p>A <code>secret</code> is a reference to a Secret in Kubernetes. A <code>secret</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk_1","title":"As a disk","text":"<p>By using disk, an extra <code>iso</code> disk will be allocated which has to be mounted on a VM. To mount the <code>secret</code> users can use <code>cloudInit</code> and the disks serial number. The <code>secretName</code> needs to be set for a reference to the created kubernetes <code>Secret</code>.</p> <p>Note: Currently, Secret update propagation is not supported. If a Secret is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where Secret keys are projected.</p> <p>Example: Attach the <code>secret</code> to a VM and use <code>cloudInit</code> to mount the <code>iso</code> disk:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      - disk:\n        name: app-secret-disk\n        # set serial\n        serial: D23YZ9W6WA5DJ487\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          # mount the Secret\n          - \"sudo mkdir /mnt/app-secret\"\n          - \"sudo mount /dev/$(lsblk --nodeps -no name,serial | grep D23YZ9W6WA5DJ487 | cut -f1 -d' ') /mnt/app-secret\"\n    name: cloudinitdisk\n  - secret:\n      secretName: app-secret\n    name: app-secret-disk\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#as-a-filesystem_1","title":"As a filesystem","text":"<p>By using filesystem, <code>secrets</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>secrets</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>secrets</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>secret</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: app-secret-fs\n          virtiofs: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          chpasswd:\n            expire: false\n          password: fedora\n          user: fedora\n          bootcmd:\n            # mount the Secret\n            - \"sudo mkdir /mnt/app-secret\"\n            - \"sudo mount -t virtiofs app-secret-fs /mnt/app-secret\"\n      name: cloudinitdisk\n    - secret:\n        secretName: app-secret\n      name: app-secret-fs\n</code></pre>"},{"location":"storage/disks_and_volumes/#serviceaccount","title":"serviceAccount","text":"<p>A <code>serviceAccount</code> volume references a Kubernetes <code>ServiceAccount</code>. A <code>serviceAccount</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk_2","title":"As a disk","text":"<p>By using disk, a new <code>iso</code> disk will be allocated with the content of the service account (<code>namespace</code>, <code>token</code> and <code>ca.crt</code>), which needs to be mounted in the VM. For automatic mounting, see the <code>configMap</code> and <code>secret</code> examples above.</p> <p>Note: Currently, ServiceAccount update propagation is not supported. If a ServiceAccount is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n        name: containerdisk\n      - disk:\n        name: serviceaccountdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: serviceaccountdisk\n    serviceAccount:\n      serviceAccountName: default\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#as-a-filesystem_2","title":"As a filesystem","text":"<p>By using filesystem, <code>serviceAccounts</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>serviceAccounts</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>serviceAccounts</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>serviceAccount</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: serviceaccount-fs\n          virtiofs: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          chpasswd:\n            expire: false\n          password: fedora\n          user: fedora\n          bootcmd:\n            # mount the ConfigMap\n            - \"sudo mkdir /mnt/serviceaccount\"\n            - \"sudo mount -t virtiofs serviceaccount-fs /mnt/serviceaccount\"\n      name: cloudinitdisk\n    - name: serviceaccount-fs\n      serviceAccount:\n        serviceAccountName: default\n</code></pre>"},{"location":"storage/disks_and_volumes/#downwardmetrics","title":"downwardMetrics","text":"<p><code>downwardMetrics</code> expose a limited set of VM and host metrics to the guest. The format is compatible with vhostmd.</p> <p>Getting a limited set of host and VM metrics is in some cases required to allow third-parties diagnosing performance issues on their appliances. One prominent example is SAP HANA.</p> <p>In order to expose <code>downwardMetrics</code> to VMs, the methods <code>disk</code> and <code>virtio-serial port</code> are supported.</p> <p>Note: The DownwardMetrics feature gate must be enabled to use the metrics. Available starting with KubeVirt v0.42.0.</p>"},{"location":"storage/disks_and_volumes/#disk_1","title":"Disk","text":"<p>A volume is created, and it is exposed to the guest as a raw block volume.  KubeVirt will update it periodically (by default, every 5 seconds).</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: metrics\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - name: metrics\n    downwardMetrics: {}\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#virtio-serial-port","title":"Virtio-serial port","text":"<p>This method uses a virtio-serial port to expose the metrics data to the VM. KubeVirt creates a port named <code>/dev/virtio-ports/org.github.vhostmd.1</code> inside the VM, in which the Virtio Transport protocol is supported. <code>downwardMetrics</code> can be retrieved from this port. See vhostmd documentation under <code>Virtio Transport</code> for further information.</p> <p>To expose the metrics using a virtio-serial port, a <code>downwardMetrics</code> device must be added (i.e., <code>spec.domain.devices.downwardMetrics: {}</code>).</p> <p>Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      downwardMetrics: {}\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n</code></pre>"},{"location":"storage/disks_and_volumes/#accessing-metrics-data","title":"Accessing Metrics Data","text":"<p>To access the DownwardMetrics shared with a disk or a virtio-serial port, the <code>vm-dump-metrics</code> tool can be used:</p> <pre><code>$ sudo dnf install -y vm-dump-metrics\n$ sudo vm-dump-metrics\n&lt;metrics&gt;\n  &lt;metric type=\"string\" context=\"host\"&gt;\n    &lt;name&gt;HostName&lt;/name&gt;\n    &lt;value&gt;node01&lt;/value&gt;\n[...]\n  &lt;metric type=\"int64\" context=\"host\" unit=\"s\"&gt;\n    &lt;name&gt;Time&lt;/name&gt;\n    &lt;value&gt;1619008605&lt;/value&gt;\n  &lt;/metric&gt;\n  &lt;metric type=\"string\" context=\"host\"&gt;\n    &lt;name&gt;VirtualizationVendor&lt;/name&gt;\n    &lt;value&gt;kubevirt.io&lt;/value&gt;\n  &lt;/metric&gt;\n&lt;/metrics&gt;\n</code></pre> <p><code>vm-dump-metrics</code> is useful as a standalone tool to verify the serial port is working and to inspect the metrics.  However, applications that consume metrics will usually connect to the virtio-serial port themselves.</p> <p>Note: The tool <code>vm-dump-metrics</code> provides the option <code>--virtio</code> in case the virtio-serial port is used. Please, refer to <code>vm-dump-metrics --help</code> for further information.</p>"},{"location":"storage/disks_and_volumes/#high-performance-features","title":"High Performance Features","text":""},{"location":"storage/disks_and_volumes/#iothreads","title":"IOThreads","text":"<p>Libvirt has the ability to use IOThreads for dedicated disk access (for supported devices). These are dedicated event loop threads that perform block I/O requests and improve scalability on SMP systems. KubeVirt exposes this libvirt feature through the <code>ioThreadsPolicy</code> setting. Additionally, each <code>Disk</code> device exposes a <code>dedicatedIOThread</code> setting. This is a boolean that indicates the specified disk should be allocated an exclusive IOThread that will never be shared with other disks.</p> <p>Currently valid policies are <code>shared</code> and <code>auto</code>. If <code>ioThreadsPolicy</code> is omitted entirely, use of IOThreads will be disabled. However, if any disk requests a dedicated IOThread, <code>ioThreadsPolicy</code> will be enabled and default to <code>shared</code>.</p>"},{"location":"storage/disks_and_volumes/#shared","title":"Shared","text":"<p>An <code>ioThreadsPolicy</code> of <code>shared</code> indicates that KubeVirt should use one thread that will be shared by all disk devices. This policy stems from the fact that large numbers of IOThreads is generally not useful as additional context switching is incurred for each thread.</p> <p>Disks with <code>dedicatedIOThread</code> set to <code>true</code> will not use the shared thread, but will instead be allocated an exclusive thread. This is generally useful if a specific Disk is expected to have heavy I/O traffic, e.g. a database spindle.</p>"},{"location":"storage/disks_and_volumes/#auto","title":"Auto","text":"<p><code>auto</code> IOThreads indicates that KubeVirt should use a pool of IOThreads and allocate disks to IOThreads in a round-robin fashion. The pool size is generally limited to twice the number of VCPU's allocated to the VM. This essentially attempts to dedicate disks to separate IOThreads, but only up to a reasonable limit. This would come in to play for systems with a large number of disks and a smaller number of CPU's for instance.</p> <p>As a caveat to the size of the IOThread pool, disks with <code>dedicatedIOThread</code> will always be guaranteed their own thread. This effectively diminishes the upper limit of the number of threads allocated to the rest of the disks. For example, a VM with 2 CPUs would normally use 4 IOThreads for all disks. However if one disk had <code>dedicatedIOThread</code> set to true, then KubeVirt would only use 3 IOThreads for the shared pool.</p> <p>There is always guaranteed to be at least one thread for disks that will use the shared IOThreads pool. Thus if a sufficiently large number of disks have dedicated IOThreads assigned, <code>auto</code> and <code>shared</code> policies would essentially result in the same layout.</p>"},{"location":"storage/disks_and_volumes/#iothreads-with-dedicated-pinned-cpus","title":"IOThreads with Dedicated (pinned) CPUs","text":"<p>When guest's vCPUs are pinned to a host's physical CPUs, it is also best to pin the IOThreads to specific CPUs to prevent these from floating between the CPUs. KubeVirt will automatically calculate and pin each IOThread to a CPU or a set of CPUs, depending on the ration between them. In case there are more IOThreads than CPUs, each IOThread will be pinned to a CPU, in a round-robin fashion. Otherwise, when there are fewer IOThreads than CPU, each IOThread will be pinned to a set of CPUs.</p>"},{"location":"storage/disks_and_volumes/#iothreads-with-qemu-emulator-thread-and-dedicated-pinned-cpus","title":"IOThreads with QEMU Emulator thread and Dedicated (pinned) CPUs","text":"<p>To further improve the vCPUs latency, KubeVirt can allocate an additional dedicated physical CPU<sup>1</sup>, exclusively for the emulator thread, to which it will be pinned. This will effectively \"isolate\" the emulator thread from the vCPUs of the VMI. When <code>ioThreadsPolicy</code> is set to <code>auto</code> IOThreads will also be \"isolated\" from the vCPUs and placed on the same physical CPU as the QEMU emulator thread.</p>"},{"location":"storage/disks_and_volumes/#examples","title":"Examples","text":""},{"location":"storage/disks_and_volumes/#shared-iothreads","title":"Shared IOThreads","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-shared\n  name: vmi-shared\nspec:\n  domain:\n    ioThreadsPolicy: shared\n    cpu:\n      cores: 2\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: vmi-shared_disk\n      - disk:\n          bus: virtio\n        name: emptydisk\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk2\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk3\n      - disk:\n          bus: virtio\n        name: emptydisk4\n      - disk:\n          bus: virtio\n        name: emptydisk5\n      - disk:\n          bus: virtio\n        name: emptydisk6\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: vmi-shared_disk\n    persistentVolumeClaim:\n      claimName: vmi-shared_pvc\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk2\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk3\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk4\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk5\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk6\n</code></pre> <p>In this example, emptydisk and emptydisk2 both request a dedicated IOThread. vmi-shared_disk, and emptydisk 3 through 6 will all shared one IOThread.</p> <pre><code>mypvc:        1\nemptydisk:    2\nemptydisk2:   3\nemptydisk3:   1\nemptydisk4:   1\nemptydisk5:   1\nemptydisk6:   1\n</code></pre>"},{"location":"storage/disks_and_volumes/#auto-iothreads","title":"Auto IOThreads","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-shared\n  name: vmi-shared\nspec:\n  domain:\n    ioThreadsPolicy: auto\n    cpu:\n      cores: 2\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: mydisk\n      - disk:\n          bus: virtio\n        name: emptydisk\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk2\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk3\n      - disk:\n          bus: virtio\n        name: emptydisk4\n      - disk:\n          bus: virtio\n        name: emptydisk5\n      - disk:\n          bus: virtio\n        name: emptydisk6\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: mydisk\n    persistentVolumeClaim:\n      claimName: mypvc\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk2\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk3\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk4\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk5\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk6\n</code></pre> <p>This VM is identical to the first, except it requests auto IOThreads. <code>emptydisk</code> and <code>emptydisk2</code> will still be allocated individual IOThreads, but the rest of the disks will be split across 2 separate iothreads (twice the number of CPU cores is 4).</p> <p>Disks will be assigned to IOThreads like this:</p> <pre><code>mypvc:        1\nemptydisk:    3\nemptydisk2:   4\nemptydisk3:   2\nemptydisk4:   1\nemptydisk5:   2\nemptydisk6:   1\n</code></pre>"},{"location":"storage/disks_and_volumes/#virtio-block-multi-queue","title":"Virtio Block Multi-Queue","text":"<p>Block Multi-Queue is a framework for the Linux block layer that maps Device I/O queries to multiple queues. This splits I/O processing up across multiple threads, and therefor multiple CPUs. libvirt recommends that the number of queues used should match the number of CPUs allocated for optimal performance.</p> <p>This feature is enabled by the <code>BlockMultiQueue</code> setting under <code>Devices</code>:</p> <pre><code>spec:\n  domain:\n    devices:\n      blockMultiQueue: true\n      disks:\n      - disk:\n          bus: virtio\n        name: mydisk\n</code></pre> <p>Note: Due to the way KubeVirt implements CPU allocation, blockMultiQueue can only be used if a specific CPU allocation is requested. If a specific number of CPUs hasn't been allocated to a VirtualMachine, KubeVirt will use all CPU's on the node on a best effort basis. In that case the amount of CPU allocation to a VM at the host level could change over time. If blockMultiQueue were to request a number of queues to match all the CPUs on a node, that could lead to over-allocation scenarios. To avoid this, KubeVirt enforces that a specific slice of CPU resources is requested in order to take advantage of this feature.</p>"},{"location":"storage/disks_and_volumes/#example","title":"Example","text":"<pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n        cpu: 4\n    devices:\n      blockMultiQueue: true\n      disks:\n      - name: mypvcdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>This example will enable Block Multi-Queue for the disk <code>mypvcdisk</code> and allocate 4 queues (to match the number of CPUs requested).</p>"},{"location":"storage/disks_and_volumes/#disk-device-cache","title":"Disk device cache","text":"<p>KubeVirt supports <code>none</code>, <code>writeback</code>, and <code>writethrough</code> KVM/QEMU cache modes.</p> <ul> <li> <p><code>none</code> I/O from the guest is not cached on the host. Use this option     for guests with large I/O requirements. This option is generally the     best choice.</p> </li> <li> <p><code>writeback</code> I/O from the guest is cached on the host and written through     to the physical media when the guest OS issues a flush.</p> </li> <li> <p><code>writethrough</code> I/O from the guest is cached on the host but must be written     through to the physical medium before the write operation completes.</p> </li> </ul> <p>Important: <code>none</code> cache mode is set as default if the file system supports direct I/O, otherwise, <code>writethrough</code> is used.</p> <p>Note: It is possible to force a specific cache mode, although if <code>none</code> mode has been chosen and the file system does not support direct I/O then started VMI will return an error.</p> <p>Example: force <code>writethrough</code> cache mode</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-pvc\n  name: vmi-pvc\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: pvcdisk\n        cache: writethrough\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: pvcdisk\n    persistentVolumeClaim:\n      claimName: disk-alpine\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#disk-sharing","title":"Disk sharing","text":"<p>Shareable disks allow multiple VMs to share the same underlying storage. In order to use this feature, special care is required because this could lead to data corruption and the loss of important data. Shareable disks demand either data synchronization at the application level or the use of clustered filesystems. These advanced configurations are not within the scope of this documentation and are use-case specific.</p> <p>If the <code>shareable</code> option is set, it indicates to libvirt/QEMU that the disk is going to be accessed by multiple VMs and not to create a lock for the writes.</p> <p>In this example, we use Rook Ceph in order to dynamically provisioning the PVC. <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: block-pvc\nspec:\n  accessModes:\n    - ReadWriteMany\n  volumeMode: Block\n  resources:\n    requests:\n      storage: 1Gi\n  storageClassName: rook-ceph-block\n</code></pre> <pre><code>$ kubectl get pvc\nNAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE\nblock-pvc   Bound    pvc-0a161bb2-57c7-4d97-be96-0a20ff0222e2   1Gi        RWO            rook-ceph-block   51s\n</code></pre> Then, we can declare 2 VMs and set the <code>shareable</code> option to true for the shared disk. <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-block-1\n  name: vm-block-1\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-block-1\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          - disk:\n              bus: virtio\n            shareable: true\n            name: block-disk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 2G\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\n      - name: block-disk\n        persistentVolumeClaim:\n          claimName: block-pvc\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-block-2\n  name: vm-block-2\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-block-2\n    spec:\n      affinity:\n        podAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n          - labelSelector:\n              matchExpressions:\n              - key: kubevirt.io/vm\n                operator: In\n                values:\n                - vm-block-1\n            topologyKey: \"kubernetes.io/hostname\"\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          - disk:\n              bus: virtio\n            shareable: true\n            name: block-disk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 2G\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\n      - name: block-disk\n        persistentVolumeClaim:\n          claimName: block-pvc                                        \n</code></pre> We can now attempt to write a string from the first guest and then read the string from the second guest to test that the sharing is working. <pre><code>$ virtctl console vm-block-1\n$ printf \"Test awesome shareable disks\" | sudo dd  of=/dev/vdc bs=1 count=150 conv=notrunc\n28+0 records in\n28+0 records out\n28 bytes copied, 0.0264182 s, 1.1 kB/s\n# Log into the second guest\n$ virtctl console vm-block-2\n$ sudo dd  if=/dev/vdc bs=1 count=150 conv=notrunc\nTest awesome shareable disks150+0 records in\n150+0 records out\n150 bytes copied, 0.136753 s, 1.1 kB/s\n</code></pre></p> <p>If you are using local devices or RWO PVCs, setting the affinity on the VMs that share the storage guarantees they will be scheduled on the same node. In the example, we set the affinity on the second VM using the label used on the first VM. If you are using shared storage with RWX PVCs, then the affinity rule is not necessary as the storage can be attached simultaneously on multiple nodes.</p>"},{"location":"storage/disks_and_volumes/#sharing-directories-with-vms","title":"Sharing Directories with VMs","text":"<p><code>Virtiofs</code> allows to make visible external filesystems to <code>KubeVirt</code> VMs. <code>Virtiofs</code> is a shared file system that lets VMs access a directory tree on the host. Further details can be found at Official Virtiofs Site.</p>"},{"location":"storage/disks_and_volumes/#non-privileged-and-privileged-sharing-modes","title":"Non-Privileged and Privileged Sharing Modes","text":"<p>KubeVirt supports two PVC sharing modes: non-privileged and privileged.</p> <p>The non-privileged mode is enabled by default. This mode has the advantage of not requiring any administrative privileges for creating the VM. However, it has some limitations:</p> <ul> <li>The virtiofsd daemon (the daemon in charge of sharing the PVC with the VM) will run with the QEMU UID/GID (107),   and cannot switch between different UIDs/GIDs.   Therefore, it will only have access to directories and files that UID/GID 107 has permission to.   Additionally, when creating new files they will always be created with QEMU's UID/GID regardless of the UID/GID of the   process within the guest.</li> <li>Extended attributes are not supported. </li> </ul> <p>To switch to the privileged mode, the feature gate ExperimentalVirtiofsSupport has to be enabled. Take into account that this mode requires privileges to run rootful containers. </p>"},{"location":"storage/disks_and_volumes/#sharing-persistent-volume-claims","title":"Sharing Persistent Volume Claims","text":""},{"location":"storage/disks_and_volumes/#cluster-configuration","title":"Cluster Configuration","text":"<p>We need to create a new VM definition including the <code>spec.devices.disk.filesystems.virtiofs</code> and a PVC. Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-fs\nspec:\n  domain:\n    devices:\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n        - disk:\n            bus: virtio\n          name: cloudinitdisk\n      filesystems:\n        - name: virtiofs-disk\n          virtiofs: {}\n    resources:\n      requests:\n        memory: 1024Mi\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          password: fedora\n          chpasswd: { expire: False }\n      name: cloudinitdisk\n    - name: virtiofs-disk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-vm","title":"Configuration Inside the VM","text":"<p>The following configuration can be done in using startup script. See cloudInitNoCloud section for  more details.  However, we can do it manually by logging in to the VM and mounting it.  Here are examples of how to mount it in a linux and windows VMs:</p> <ul> <li>Linux Example</li> </ul> <pre><code>$ sudo mkdir -p /mnt/disks/virtio\n$ sudo mount -t virtiofs virtiofs-disk /mnt/disks/virtio\n</code></pre> <ul> <li>Windows Example</li> </ul> <p>See this guide for details on startup steps needed for Windows VMs.</p>"},{"location":"storage/disks_and_volumes/#sharing-node-directories","title":"Sharing Node Directories","text":"<p>It is allowed using hostpaths. The following configuration example is shown for illustrative purposes. However, the PVCs method is preferred since using hostpath is generally discouraged for security reasons.</p>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-node","title":"Configuration Inside the Node","text":"<p>To share the directory with the VMs, we need to log in to the node, create the shared directory (if it does not already exist), and set the proper SELinux context label <code>container_file_t</code> to the shared directory. In this example we are going to share a new directory <code>/mnt/data</code> (if the desired directory is an existing one, you can skip the <code>mkdir</code> command):</p> <pre><code>$ mkdir /tmp/data\n$ sudo chcon -t container_file_t /tmp/data\n</code></pre> <p>Note: If you are attempting to share an existing directory, you must first check the SELinux context label with the command <code>ls -Z &lt;directory&gt;</code>. In the case that the label is not present or is not <code>container_file_t</code> you need to label  it with the <code>chcon</code>  command.</p>"},{"location":"storage/disks_and_volumes/#cluster-configuration_1","title":"Cluster Configuration","text":"<p>We need a <code>StorageClass</code> which uses the provider <code>no-provisioner</code>:</p> <pre><code>apiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n   name: no-provisioner-storage-class\nprovisioner: kubernetes.io/no-provisioner\nreclaimPolicy: Delete\nvolumeBindingMode: WaitForFirstConsumer\n</code></pre> <p>To make the shared directory available for VMs, we need to create a PV and a PVC that could be consumed by the VMs:</p> <pre><code>kind: PersistentVolume\napiVersion: v1\nmetadata:\n  name: hostpath\nspec:\n  capacity:\n    storage: 10Gi\n  accessModes:\n    - ReadWriteMany\n  hostPath:\n    path: \"/tmp/data\"\n  storageClassName: \"no-provisioner-storage-class\"\n  nodeAffinity:\n    required:\n      nodeSelectorTerms:\n      - matchExpressions:\n        - key: kubernetes.io/hostname\n          operator: In\n          values:\n          - node01\n---  \napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: hostpath-claim\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: \"no-provisioner-storage-class\"\n  resources:\n    requests:\n      storage: 10Gi\n</code></pre> <p>Note: Change the <code>node01</code> value for the node name where you want the shared directory will be located.</p> <p>The VM definitions have to request the PVC <code>hostpath-claim</code> and attach it as a virtiofs filesystem:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: hostpath-vm\n  name: hostpath\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: hostpath\n        kubevirt.io/vm: hostpath\n    spec:\n      domain:\n        cpu:\n          cores: 1\n          sockets: 1\n          threads: 1\n        devices:\n          filesystems:\n            - name: vm-hostpath\n              virtiofs: {}\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          interfaces:\n            - name: default\n              masquerade: {}\n          rng: {}\n        resources:\n          requests:\n            memory: 1Gi\n      networks:\n        - name: default\n          pod: {}\n      terminationGracePeriodSeconds: 180\n      volumes:\n        - containerDisk:\n            image: quay.io/containerdisks/fedora:latest\n          name: containerdisk\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: password\n              user: fedora\n          name: cloudinitdisk\n        - name: vm-hostpath\n          persistentVolumeClaim:\n            claimName: hostpath-claim\n</code></pre>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-vm_1","title":"Configuration Inside the VM","text":"<p>We need to log in to the VM and mount the shared directory:</p> <pre><code>$ sudo mount -t virtiofs vm-hostpath /mnt\n</code></pre>"},{"location":"storage/disks_and_volumes/#update-volumes-strategy","title":"Update volumes strategy","text":"<p>The <code>updateVolumesStrategy</code> field is used to specify the strategy for updating the volumes of a running VM. The following strategies are supported:   * <code>Replacement</code>: the update volumes will be replaced upon the VM restart.   * <code>Migration</code>: the update of the volumes will trigger a storage migration of     the old volumes to the new ones. More details about volume migration can be     found in the volume migration documentation.</p> <p>The update volume migration depends on the feature gate <code>VolumesUpdateStrategy</code> which depends on the VMLiveUpdateFeatures feature gate and configuration.</p> <p>KubeVirt CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n        - VolumesUpdateStrategy\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre></p>"},{"location":"storage/export_api/","title":"Export API","text":"<p>It can be desirable to export a Virtual Machine and its related disks out of a cluster so you can import that Virtual Machine into another system or cluster. The Virtual Machine disks are the most prominent things you will want to export. The export API makes it possible to declaratively export Virtual Machine disks. It is also possible to export individual PVCs and their contents, for instance when you have created a memory dump from a VM or are using virtio-fs to have a Virtual Machine populate a PVC.</p> <p>In order not to overload the kubernetes API server the data is transferred through a dedicated export proxy server. The proxy server can then be exposed to the outside world through a service associated with an Ingress/Route or NodePort. As an alternative, the <code>port-forward</code> flag can be used with the virtctl integration to bypass the need of an Ingress/Route.</p>"},{"location":"storage/export_api/#export-feature-gate","title":"Export Feature Gate","text":"<p>VMExport support must be enabled in the feature gates to be available. The feature gates field in the KubeVirt CR must be expanded by adding the <code>VMExport</code> to it.</p>"},{"location":"storage/export_api/#export-token","title":"Export token","text":"<p>In order to securely export a Virtual Machine Disk, you must create a token that is used to authorize users accessing the export endpoint. This token must be in the same namespace as the Virtual Machine. The contents of the secret can be passed as a token header or parameter to the export URL. The name of the header or argument is <code>x-kubevirt-export-token</code> with a value that matches the content of the secret. The secret can be named any valid secret in the namespace. We recommend you generate an alpha numeric token of at least 12 characters. The data key should be <code>token</code>. For example:</p> <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: example-token\nstringData:\n  token: 1234567890ab\n</code></pre>"},{"location":"storage/export_api/#export-virtual-machine-volumes","title":"Export Virtual Machine volumes","text":"<p>After you have created the token you can now create a VMExport CR that identifies the Virtual Machine you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\n</code></pre> <p>The following volumes present in the VM will be exported:</p> <ul> <li>PersistentVolumeClaims</li> <li>DataVolumes</li> <li>MemoryDump</li> </ul> <p>All other volume types are not exported. To avoid the export of inconsistent data, a Virtual Machine can only be exported while it is powered off. Any active VM exports will be terminated if the Virtual Machine is started. To export data from a running Virtual Machine you must first create a Virtual Machine Snapshot (see below).</p> <p>If the VM contains multiple volumes that can be exported, each volume will get its own URL links. If the VM contains no volumes that can be exported, the VMExport will go into a <code>Skipped</code> phase, and no export server is started.</p>"},{"location":"storage/export_api/#export-virtual-machine-snapshot-volumes","title":"Export Virtual Machine Snapshot volumes","text":"<p>You can create a VMExport CR that identifies the Virtual Machine Snapshot you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"snapshot.kubevirt.io\"\n    kind: VirtualMachineSnapshot\n    name: example-vmsnapshot\n</code></pre> <p>When you create a VMExport based on a Virtual Machine Snapshot, the controller will attempt to create PVCs from the volume snapshots contained in Virtual Machine Snapshot. Once all the PVCs are ready, the export server will start and you can begin the export. If the Virtual Machine Snapshot contains multiple volumes that can be exported, each volume will get its own URL links. If the Virtual Machine snapshot contains no volumes that can be exported, the VMExport will go into a <code>skipped</code> phase, and no export server is started.</p>"},{"location":"storage/export_api/#export-persistent-volume-claim","title":"Export Persistent Volume Claim","text":"<p>You can create a VMExport CR that identifies the Persistent Volume Claim (PVC) you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n</code></pre> <p>In this example the PVC name is <code>example-pvc</code>. Note the PVC doesn't need to contain a Virtual Machine Disk, it can contain any content, but the main use case is exporting Virtual Machine Disks. After you post this yaml to the cluster, a new export server is created in the same namespace as the PVC. If the source PVC is in use by another pod (such as the virt-launcher pod) then the export will remain pending until the PVC is no longer in use. If the exporter server is active and another pod starts using the PVC, the exporter server will be terminated until the PVC is not in use anymore.</p>"},{"location":"storage/export_api/#export-status-links","title":"Export status links","text":"<p>The VirtualMachineExport CR will contain a status with internal and external links to the export service. The internal links are only valid inside the cluster, and the external links are valid for external access through an Ingress or Route. The <code>cert</code> field will contain the CA that signed the certificate of the export server for internal links, or the CA that signed the Route or Ingress.</p>"},{"location":"storage/export_api/#kubevirt-content-type","title":"KubeVirt content-type","text":"<p>The following is an example of exporting a PVC that contains a KubeVirt disk image. The controller determines if the PVC contains a kubevirt disk by checking if there is a special annotation on the PVC, or if there is a DataVolume ownerReference on the PVC, or if the PVC has a volumeMode of block.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img\n        - format: gzip\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img\n        - format: gzip\n          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#archive-content-type","title":"Archive content-type","text":"<p>Archive content-type is automatically selected if we are unable to determine the PVC contains a KubeVirt disk. The archive will contain all the files that are in the PVC.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: dir\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example/dir\n        - format: tar.gz\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example/disk.tar.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: dir\n          url: https://virt-export-example-export.example.svc/volumes/example/dir\n        - format: tar.gz\n          url: https://virt-export-example-export.example.svc/volumes/example/disk.tar.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#manifests","title":"Manifests","text":"<p>The VirtualMachine manifests can be retrieved by accessing the <code>manifests</code> in the VirtualMachineExport status. The <code>all</code> type will return the VirtualMachine manifest, any DataVolumes, and a configMap that contains the public CA certificate of the Ingress/Route of the external URL, or the CA of the export server of the internal URL. The <code>auth-header-secret</code> will be a secret that contains a Containerized Data Importer (CDI) compatible header. This header contains a text version of the export token.</p> <p>Both internal and external links will contain a <code>manifests</code> field. If there are no external links, then there will not be any external manifests either. The virtualMachine <code>manifests</code> field is only available if the source is a <code>VirtualMachine</code> or <code>VirtualMachineSnapshot</code>. Exporting a <code>PersistentVolumeClaim</code> will not generate a Virtual Machine manifest.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      ...\n      manifests:\n      - type: all\n        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/all\n      - type: auth-header-secret\n        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret\n    internal:\n      ...\n      manifests:\n      - type: all\n        url: https://virt-export-export-pvc.default.svc/internal/manifests/all\n      - type: auth-header-secret\n        url: https://virt-export-export-pvc.default.svc/internal/manifests/secret\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#format-types","title":"Format types","text":"<p>There are 4 format types that are possible:</p> <ul> <li>Raw. The unaltered raw KubeVirt disk image.</li> <li>Gzip. The raw KubeVirt disk image but gzipped to help with transferring efficiency.</li> <li>Dir. A directory listing, allowing you to find the files contained in the PVC.</li> <li>Tar.gz The contents of the PVC tarred and gzipped in a single file.</li> </ul> <p>Raw and Gzip will be selected if the PVC is determined to be a KubeVirt disk. KubeVirt disks contain a single disk.img file (or are a block device). Dir will return a list of the files in the PVC, to download a specific file you can replace <code>/dir</code> in the URL with the path and file name. For instance if the PVC contains the file <code>/example/data.txt</code> you can replace <code>/dir</code> with <code>/example/data.txt</code> to download just data.txt file. Or you can use the tar.gz URL to get all the contents of the PVC in a tar file.</p>"},{"location":"storage/export_api/#internal-link-certificates","title":"Internal link certificates","text":"<p>The export server certificate is valid for 7 days after which it is rotated by deleting the export server pod and associated secret and generating a new one. If for whatever reason the export server pod dies, the associated secret is also automatically deleted and a new pod and secret are generated. The VirtualMachineExport object status will be automatically updated to reflect the new certificate.</p>"},{"location":"storage/export_api/#external-link-certificates","title":"External link certificates","text":"<p>The external link certificates are associated with the Ingress/Route that points to the service created by the KubeVirt operator. The CA that signed the Ingress/Route will part of the certificates.</p>"},{"location":"storage/export_api/#ttl-time-to-live-for-an-export","title":"TTL (Time to live) for an Export","text":"<p>For various reasons (security being one), users should be able to specify a TTL for the VMExport objects that limits the lifetime of an export. This is done via the <code>ttlDuration</code> field which accepts a k8s duration, which defaults to 2 hours when not specified. <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n    name: example-export\nspec:\n    source:\n        apiGroup: \"kubevirt.io\"\n        kind: VirtualMachine\n        name: example-vm\n    tokenSecretRef: example-token\n    ttlDuration: 1h\n</code></pre></p>"},{"location":"storage/export_api/#virtctl-integration-vmexport","title":"virtctl integration: vmexport","text":"<p>The virtctl <code>vmexport</code> command allows users to interact with the export API in an easy-to-use way.</p> <p><code>vmexport</code> uses two mandatory arguments:</p> <ul> <li>The vmexport functions (create|delete|download).</li> <li>The VirtualMachineExport name.</li> </ul> <p>These three functions are:</p>"},{"location":"storage/export_api/#create","title":"Create","text":"<pre><code># Creates a VMExport object according to the specified flag.\n\n# The flag should either be:\n\n# --pvc, to specify the name of the pvc to export.\n# --snapshot, to specify the name of the VM snapshot to export.\n# --vm, to specify the name of the Virtual Machine to export.\n\n$ virtctl vmexport create name [flags]\n</code></pre>"},{"location":"storage/export_api/#delete","title":"Delete","text":"<pre><code># Deletes the specified VMExport object.\n\n$ virtctl vmexport delete name\n</code></pre>"},{"location":"storage/export_api/#download","title":"Download","text":"<pre><code># Downloads a volume from the defined VMExport object.\n\n# The main available flags are:\n\n# --output, mandatory flag to specify the output file.\n# --volume, optional flag to specify the name of the downloadable volume.\n# --vm|--snapshot|--pvc, if specified, are used to create the VMExport object assuming it doesn't exist. The name of the object to export has to be specified.\n# --format, optional flag to specify wether to download the file in compressed (default) or raw format.\n# --port-forward, optional flag to easily download the volume without the need of an ingress or route. Also, the local port can be optionally specified with the --local-port flag.\n\n$ virtctl vmexport download name [flags]\n</code></pre> <p>By default, the volume will be downloaded in compressed format. Users can specify the desired format (gzip or raw) by using the <code>format</code> flag, as shown below:</p> <pre><code># Downloads a volume from the defined VMExport object and, if necessary, decompresses it.\n$ virtctl vmexport download name --format=raw [flags]\n</code></pre>"},{"location":"storage/export_api/#ttl-time-to-live","title":"TTL (Time to live)","text":"<p>TTL can also be added when creating a VMExport via virtctl <pre><code>$ virtctl vmexport create name --ttl=1h\n</code></pre></p> <p>For more information about usage and examples:</p> <pre><code>$ virtctl vmexport --help\n\nExport a VM volume.\n\nUsage:\n  virtctl vmexport [flags]\n\nExamples:\n  # Create a VirtualMachineExport to export a volume from a virtual machine:\n    virtctl vmexport create vm1-export --vm=vm1\n\n    # Create a VirtualMachineExport to export a volume from a virtual machine snapshot\n    virtctl vmexport create snap1-export --snapshot=snap1\n\n    # Create a VirtualMachineExport to export a volume from a PVC\n    virtctl vmexport create pvc1-export --pvc=pvc1\n\n    # Delete a VirtualMachineExport resource\n    virtctl vmexport delete snap1-export\n\n    # Download a volume from an already existing VirtualMachineExport (--volume is optional when only one volume is available)\n    virtctl vmexport download vm1-export --volume=volume1 --output=disk.img.gz\n\n    # Create a VirtualMachineExport and download the requested volume from it\n    virtctl vmexport download vm1-export --vm=vm1 --volume=volume1 --output=disk.img.gz\n\nFlags:\n  -h, --help              help for vmexport\n      --insecure          When used with the 'download' option, specifies that the http request should be insecure.\n      --keep-vme          When used with the 'download' option, specifies that the vmexport object should not be deleted after the download finishes.\n      --output string     Specifies the output path of the volume to be downloaded.\n      --pvc string        Sets PersistentVolumeClaim as vmexport kind and specifies the PVC name.\n      --snapshot string   Sets VirtualMachineSnapshot as vmexport kind and specifies the snapshot name.\n      --vm string         Sets VirtualMachine as vmexport kind and specifies the vm name.\n      --volume string     Specifies the volume to be downloaded.\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre>"},{"location":"storage/export_api/#use-cases","title":"Use cases","text":""},{"location":"storage/export_api/#clone-vm-from-one-cluster-to-another-cluster","title":"Clone VM from one cluster to another cluster","text":"<p>If you want to transfer KubeVirt disk images from a source cluster to another target cluster, you can use the VMExport in the source to expose the disks and use Containerized Data Importer (CDI) in the target cluster to import the image into the target cluster. Let's assume we have an Ingress or Route in the source cluster that exposes the export proxy with the following example domain <code>virt-exportproxy-example.example.com</code> and we have a Virtual Machine in the source cluster with one disk, which looks like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-example-datavolume\n  name: example-vm\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: example-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        resources:\n          requests:\n            storage: 20Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://quay.io/containerdisks/centos-stream:9\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-example-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 2Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: example-dv\n        name: datavolumedisk1\n</code></pre> <p>This is a VM that has a DataVolume (DV) <code>example-dv</code> that is populated from a container disk and we want to export that disk to the target cluster. To export this VM we have to create a token that we can use in the target cluster to get access to the export, or we can let the export controller generate one for us. For example</p> <p><pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: example-token\nstringData:\n  token: 1234567890ab\n</code></pre> The value of the token is <code>1234567890ab</code> hardly a secure token, but it is an example. We can now create a VMExport that looks like this:</p> <p><pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token #optional, if omitted the export controller will generate a token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\n</code></pre> If the VM is not running the status of the VMExport object will get updated once the export-server pod is running to look something like this:</p> <p><pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\nstatus:\n  conditions:\n  - lastProbeTime: null\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img\n        - format: gzip\n          url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img\n        - format: gzip\n          url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre> Note in this example we are in the <code>example</code> namespace in the source cluster, which is why the internal links domain ends with <code>.example.svc</code>. The external links are what will be visible to outside of the source cluster, so we can use that for when we import into the target cluster.</p> <p>Now we are ready to import this disk into the target cluster. In order for CDI to import, we will need to provide appropriate yaml that contains the following: - CA cert (as config map) - The token needed to access the disk images in a CDI compatible format - The VM yaml - DataVolume yaml (optional if not part of the VM definition)</p> <p>virtctl provides an additional argument to the download command called <code>--manifest</code> that will retrieve the appropriate information from the export server, and either save it to a file with the <code>--output</code> argument or write to standard out. By default this output will not contain the header secret as it contains the token in plaintext. To get the header secret you specify the <code>--include-secret</code> argument. The default output format is <code>yaml</code> but it is possible to get <code>json</code> output as well.</p> <p>Assuming there is a running VirtualMachineExport called <code>example-export</code> and the same namespace exists in the target cluster. The name of the kubeconfig of the target cluster is named <code>kubeconfig-target</code>, to clone the vm into the target cluster run the following commands:</p> <pre><code>$ virtctl vmexport download example-export --manifest --include-secret --output=import.yaml\n$ kubectl apply -f import.yaml --kubeconfig=kubeconfig-target\n</code></pre> <p>The first command generates the yaml and writes it to <code>import.yaml</code>. The second command applies the generated yaml to the target cluster. It is possible to combine the two commands writing to standard <code>out</code> with the first command, and piping it into the second command. Use this option if the export token should not be written to a file anywhere. This will create the VM in the target cluster, and provides CDI in the target cluster with everything required to import the disk images.</p> <p>After the import completes you should be able to start the VM in the target cluster.</p>"},{"location":"storage/export_api/#download-a-vm-volume-locally-using-virtctl-vmexport","title":"Download a VM volume locally using virtctl vmexport","text":"<p>Several steps from the previous section can be simplified considerably by using the <code>vmexport</code> command.</p> <p>Again, let's assume we have an Ingress or Route in our cluster that exposes the export proxy, and that we have a Virtual Machine in the cluster with one disk like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-example-datavolume\n  name: example-vm\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: example-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        resources:\n          requests:\n            storage: 20Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://quay.io/containerdisks/centos-stream:9\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-example-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 2Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: example-dv\n        name: datavolumedisk1\n</code></pre> <p>Once we meet these requirements, the process of downloading the volume locally can be accomplished by different means:</p>"},{"location":"storage/export_api/#performing-each-step-separately","title":"Performing each step separately","text":"<p>We can download the volume by performing every single step in a different command. We start by creating the export object:</p> <pre><code># We use an arbitrary name for the VMExport object, but specify our VM name in the flag.\n\n$ virtctl vmexport create vmexportname --vm=example-vm\n</code></pre> <p>Then, we download the volume in the specified output:</p> <pre><code># Since our virtual machine only has one volume, there's no need to specify the volume name with the --volume flag.\n\n# After the download, the VMExport object is deleted by default, so we are using the optional --keep-vme flag to delete it manually.\n\n$ virtctl vmexport download vmexportname --output=/tmp/disk.img --keep-vme\n</code></pre> <p>Lastly, we delete the VMExport object:</p> <pre><code>$ virtctl vmexport delete vmexportname\n</code></pre>"},{"location":"storage/export_api/#performing-one-single-step","title":"Performing one single step","text":"<p>All the previous steps can be simplified in one, single command:</p> <pre><code># Since we are using a create flag (--vm) with download, the command creates the object assuming the VMExport doesn't exist.\n\n# Also, since we are not using --keep-vme, the VMExport object is deleted after the download.\n\n$ virtctl vmexport download vmexportname --vm=example-vm --output=/tmp/disk.img\n</code></pre> <p>After the download finishes, we can find our disk in <code>/tmp/disk.img</code>.</p>"},{"location":"storage/guestfs/","title":"Usage of libguestfs-tools and virtctl guestfs","text":"<p>Libguestfs tools are a set of utilities for accessing and modifying VM disk images. The command <code>virtctl guestfs</code> helps to deploy an interactive container with the libguestfs-tools and the PVC attached to it. This command is particularly useful if the users need to modify, inspect or debug VM disks on a PVC. <pre><code>$ virtctl guestfs -h\nCreate a pod with libguestfs-tools, mount the pvc and attach a shell to it. The pvc is mounted under the /disks directory inside the pod for filesystem-based pvcs, or as /dev/vda for block-based pvcs\n\nUsage:\n  virtctl guestfs [flags]\n\nExamples:\n  # Create a pod with libguestfs-tools, mount the pvc and attach a shell to it:\n  virtctl guestfs &lt;pvc-name&gt;\n\nFlags:\n  -h, --help                 help for guestfs\n      --image string         libguestfs-tools container image\n      --kvm                  Use kvm for the libguestfs-tools container (default true)\n      --pull-policy string   pull policy for the libguestfs image (default \"IfNotPresent\")\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre></p> <p>By default <code>virtctl guestfs</code> sets up <code>kvm</code> for the interactive container. This considerably speeds up the execution of the libguestfs-tools since they use QEMU. If the cluster doesn't have any kvm supporting nodes, the user must disable kvm by setting the option <code>--kvm=false</code>. If not set, the libguestfs-tools pod will remain pending because it cannot be scheduled on any node.</p> <p>The command automatically uses the image exposed by KubeVirt under the http endpoint <code>/apis/subresources.kubevirt.io/&lt;kubevirt-version&gt;/guestfs</code>, but it can be configured to use a custom image by using the option <code>--image</code>. Users can also overwrite the pull policy of the image by setting the option <code>pull-policy</code>.</p> <p>The command checks if a PVC is used by another pod in which case it will fail. However, once libguestfs-tools has started, the setup doesn't prevent a new pod starting and using the same PVC. The user needs to verify that there are no active virtctl guestfs pods before starting the VM which accesses the same PVC.</p> <p>Currently, <code>virtctl guestfs</code> supports only a single PVC. Future versions might support multiple PVCs attached to the interactive pod.</p>"},{"location":"storage/guestfs/#examples-and-use-cases","title":"Examples and use-cases","text":"<p>Generally, the user can take advantage of the <code>virtctl guestfs</code> command for all typical usage of libguestfs-tools. It is strongly recommended to consult the official documentation. This command simply aims to help in configuring the correct containerized environment in the Kubernetes cluster where KubeVirt is installed.</p> <p>For all the examples, the user has to start the interactive container by referencing the PVC in the <code>virtctl guestfs</code> command. This will deploy the interactive pod and attach the stdin and stdout. </p> <p>Example:</p> <p><pre><code>$ virtctl guestfs pvc-test\nUse image: registry:5000/kubevirt/libguestfs-tools@sha256:6644792751b2ba9442e06475a809448b37d02d1937dbd15ad8da4d424b5c87dd \nThe PVC has been mounted at /disk \nWaiting for container libguestfs still in pending, reason: ContainerCreating, message:  \nWaiting for container libguestfs still in pending, reason: ContainerCreating, message:  \nbash-5.0#\n</code></pre> Once the libguestfs-tools pod has been deployed, the user can access the disk and execute the desired commands. Later, once the user has completed the operations on the disk, simply <code>exit</code> the container and the pod be will automatically terminated.</p> <ol> <li>Inspect the disk filesystem to retrive the version of the OS on the disk: <pre><code>bash-5.0# virt-cat -a disk.img /etc/os-release \nNAME=Fedora\nVERSION=\"34 (Cloud Edition)\"\nID=fedora\nVERSION_ID=34\nVERSION_CODENAME=\"\"\nPLATFORM_ID=\"platform:f34\"\nPRETTY_NAME=\"Fedora 34 (Cloud Edition)\"\nANSI_COLOR=\"0;38;2;60;110;180\"\nLOGO=fedora-logo-icon\nCPE_NAME=\"cpe:/o:fedoraproject:fedora:34\"\nHOME_URL=\"https://fedoraproject.org/\"\nDOCUMENTATION_URL=\"https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/\"\nSUPPORT_URL=\"https://fedoraproject.org/wiki/Communicating_and_getting_help\"\nBUG_REPORT_URL=\"https://bugzilla.redhat.com/\"\nREDHAT_BUGZILLA_PRODUCT=\"Fedora\"\nREDHAT_BUGZILLA_PRODUCT_VERSION=34\nREDHAT_SUPPORT_PRODUCT=\"Fedora\"\nREDHAT_SUPPORT_PRODUCT_VERSION=34\nPRIVACY_POLICY_URL=\"https://fedoraproject.org/wiki/Legal:PrivacyPolicy\"\nVARIANT=\"Cloud Edition\"\nVARIANT_ID=cloud\n</code></pre></li> <li>Add users (for example after the disk has been imported using CDI) <pre><code>bash-5.0# virt-customize -a disk.img --run-command 'useradd -m test-user -s /bin/bash' --password  'test-user:password:test-password'\n[   0.0] Examining the guest ...\n[   4.1] Setting a random seed\n[   4.2] Setting the machine ID in /etc/machine-id\n[   4.2] Running: useradd -m test-user -s /bin/bash\n[   4.3] Setting passwords\n[   5.3] Finishing off\n</code></pre></li> <li> <p>Run virt-rescue and repair a broken partition or initrd (for example by running dracut) <pre><code>bash-5.0# virt-rescue -a disk.img\n[...]\nThe virt-rescue escape key is \u2018^]\u2019.  Type \u2018^] h\u2019 for help.\n\n------------------------------------------------------------\n\nWelcome to virt-rescue, the libguestfs rescue shell.\n\nNote: The contents of / (root) are the rescue appliance.\nYou have to mount the guest\u2019s partitions under /sysroot\nbefore you can examine them.\n&gt;&lt;rescue&gt; fdisk -l\nDisk /dev/sda: 6 GiB, 6442450944 bytes, 12582912 sectors\nDisk model: QEMU HARDDISK   \nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\nDisklabel type: gpt\nDisk identifier: F8DC0844-9194-4B34-B432-13FA4B70F278\n\nDevice       Start      End  Sectors Size Type\n/dev/sda1     2048     4095     2048   1M BIOS boot\n/dev/sda2     4096  2101247  2097152   1G Linux filesystem\n/dev/sda3  2101248 12580863 10479616   5G Linux filesystem\n\n\nDisk /dev/sdb: 4 GiB, 4294967296 bytes, 8388608 sectors\nDisk model: QEMU HARDDISK   \nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\n&gt;&lt;rescue&gt; mount /dev/sda3 sysroot/\n&gt;&lt;rescue&gt; mount /dev/sda2 sysroot/boot\n&gt;&lt;rescue&gt; chroot sysroot/\n&gt;&lt;rescue&gt; ls boot/\nSystem.map-5.11.12-300.fc34.x86_64\nconfig-5.11.12-300.fc34.x86_64\nefi\ngrub2\ninitramfs-0-rescue-8afb5b540fab48728e48e4196a3a48ee.img\ninitramfs-5.11.12-300.fc34.x86_64.img\nloader\nvmlinuz-0-rescue-8afb5b540fab48728e48e4196a3a48ee\n&gt;&lt;rescue&gt; dracut -f boot/initramfs-5.11.12-300.fc34.x86_64.img 5.11.12-300.fc34.x86_64\n[...]\n&gt;&lt;rescue&gt; exit # &lt;- exit from chroot\n&gt;&lt;rescue&gt; umount sysroot/boot\n&gt;&lt;rescue&gt; umount sysroot\n&gt;&lt;rescue&gt; exit\n</code></pre></p> </li> <li> <p>Install an OS from scratch <pre><code>bash-5.0# virt-builder centos-8.2 -o disk.img --root-password password:password-test\n[   1.5] Downloading: http://builder.libguestfs.org/centos-8.2.xz\n######################################################################## 100.0%#=#=#                                                    ######################################################################## 100.0%\n[  58.3] Planning how to build this image\n[  58.3] Uncompressing\n[  65.7] Opening the new disk\n[  70.8] Setting a random seed\n[  70.8] Setting passwords\n[  72.0] Finishing off\n                   Output file: disk.img\n                   Output size: 6.0G\n                 Output format: raw\n            Total usable space: 5.3G\n                    Free space: 4.0G (74%)\n</code></pre></p> </li> <li>Identify the partition and filesystem on the disk <pre><code>bash-5.0# virt-filesystems -a disk.img --partitions --filesystem --long\nName       Type        VFS   Label  MBR  Size        Parent\n/dev/sda2  filesystem  ext4  -      -    1023303680  -\n/dev/sda4  filesystem  xfs   -      -    4710203392  -\n/dev/sda1  partition   -     -      -    1048576     /dev/sda\n/dev/sda2  partition   -     -      -    1073741824  /dev/sda\n/dev/sda3  partition   -     -      -    644874240   /dev/sda\n/dev/sda4  partition   -     -      -    4720689152  /dev/sda\n</code></pre></li> </ol>"},{"location":"storage/guestfs/#limitations","title":"Limitations","text":"<p>Currently, it is not possible to resize the xfs filesystem.</p>"},{"location":"storage/hotplug_volumes/","title":"Hotplug Volumes","text":"<p>KubeVirt now supports hotplugging volumes into a running Virtual Machine Instance (VMI). The volume must be either a block volume or contain a disk image. When a VM that has hotplugged volumes is rebooted, the hotplugged volumes will be attached to the restarted VM. If the volumes are persisted they will become part of the VM spec, and will not be considered hotplugged. If they are not persisted, the volumes will be reattached as hotplugged volumes</p>"},{"location":"storage/hotplug_volumes/#enabling-hotplug-volume-support","title":"Enabling hotplug volume support","text":"<p>Hotplug volume support must be enabled in the feature gates to be supported. The feature gates field in the KubeVirt CR must be expanded by adding the <code>HotplugVolumes</code> to it.</p>"},{"location":"storage/hotplug_volumes/#virtctl-support","title":"Virtctl support","text":"<p>In order to hotplug a volume, you must first prepare a volume. This can be done by using a DataVolume (DV). In the example we will use a blank DV in order to add some extra storage to a running VMI</p> <p><pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: example-volume-hotplug\nspec:\n  source:\n    blank: {}\n  storage:\n    resources:\n      requests:\n        storage: 5Gi\n</code></pre> In this example we are using <code>ReadWriteOnce</code> accessMode, and the default FileSystem volume mode. Volume hotplugging supports all combinations of block volume mode and <code>ReadWriteMany</code>/<code>ReadWriteOnce</code>/<code>ReadOnlyMany</code> accessModes, if your storage supports the combination.</p>"},{"location":"storage/hotplug_volumes/#addvolume","title":"Addvolume","text":"<p>Now lets assume we have started a VMI like the Fedora VMI in examples and the name of the VMI is 'vmi-fedora'. We can add the above blank volume to this running VMI by using the 'addvolume' command  available with virtctl</p> <pre><code>$ virtctl addvolume vmi-fedora --volume-name=example-volume-hotplug\n</code></pre> <p>This will hotplug the volume into the running VMI, and set the serial of the disk to the volume name. In this example it is set to example-hotplug-volume.</p>"},{"location":"storage/hotplug_volumes/#why-virtio-scsi","title":"Why virtio-scsi","text":"<p>The bus of hotplug disk is specified as a <code>scsi</code> disk. Why is it not specified as <code>virtio</code> instead, like regular disks? The reason is a limitation of <code>virtio</code> disks that each disk uses a pcie slot in the virtual machine and there is a maximum of 32 slots. This means there is a low limit on the maximum number of disks you can hotplug especially given that other things will also need pcie slots. Another issue is these slots need to be reserved ahead of time. So if the number of hotplugged disks is not known ahead of time, it is impossible to properly reserve the required number of slots. To work around this issue, each VM has a virtio-scsi controller, which allows the use of a <code>scsi</code> bus for hotplugged disks. This controller allows for hotplugging of over 4 million disks. <code>virtio-scsi</code> is very close in performance to <code>virtio</code></p>"},{"location":"storage/hotplug_volumes/#serial","title":"Serial","text":"<p>You can change the serial of the disk by specifying the --serial parameter, for example: <pre><code>$ virtctl addvolume vmi-fedora --volume-name=example-volume-hotplug --serial=1234567890\n</code></pre></p> <p>The serial will be used in the guest so you can identify the disk inside the guest by the serial. For instance in Fedora the disk by id will contain the serial. <pre><code>$ virtctl console vmi-fedora\n\nFedora 32 (Cloud Edition)\nKernel 5.6.6-300.fc32.x86_64 on an x86_64 (ttyS0)\n\nSSH host key: SHA256:c8ik1A9F4E7AxVrd6eE3vMNOcMcp6qBxsf8K30oC/C8 (ECDSA)\nSSH host key: SHA256:fOAKptNAH2NWGo2XhkaEtFHvOMfypv2t6KIPANev090 (ED25519)\neth0: 10.244.196.144 fe80::d8b7:51ff:fec4:7099\nvmi-fedora login:fedora\nPassword:fedora\n[fedora@vmi-fedora ~]$ ls /dev/disk/by-id\nscsi-0QEMU_QEMU_HARDDISK_1234567890\n[fedora@vmi-fedora ~]$ \n</code></pre> As you can see the serial is part of the disk name, so you can uniquely identify it.</p> <p>The format and length of serials are specified according to the libvirt documentation: <pre><code>    If present, this specify serial number of virtual hard drive. For example, it may look like &lt;serial&gt;WD-WMAP9A966149&lt;/serial&gt;. Not supported for scsi-block devices, that is those using disk type 'block' using device 'lun' on bus 'scsi'. Since 0.7.1\n\n    Note that depending on hypervisor and device type the serial number may be truncated silently. IDE/SATA devices are commonly limited to 20 characters. SCSI devices depending on hypervisor version are limited to 20, 36 or 247 characters.\n\n    Hypervisors may also start rejecting overly long serials instead of truncating them in the future so it's advised to avoid the implicit truncation by testing the desired serial length range with the desired device and hypervisor combination.\n</code></pre></p>"},{"location":"storage/hotplug_volumes/#supported-disk-types","title":"Supported Disk types","text":"<p>Kubevirt supports hotplugging disk devices of type disk and lun. As with other volumes, using type <code>disk</code> will expose the hotplugged volume as a regular disk, while using <code>lun</code> allows additional functionalities like the execution of iSCSI commands.</p> <p>You can specify the desired type by using the --disk-type parameter, for example:</p> <pre><code># Allowed values are lun and disk. If no option is specified, we use disk by default.\n$ virtctl addvolume vmi-fedora --volume-name=example-lun-hotplug --disk-type=lun\n</code></pre>"},{"location":"storage/hotplug_volumes/#retain-hotplugged-volumes-after-restart","title":"Retain hotplugged volumes after restart","text":"<p>In many cases it is desirable to keep hotplugged volumes after a VM restart. It may also be desirable to be able to unplug these volumes after the restart. The <code>persist</code> option makes it impossible to unplug the disks after a restart. If you don't specify <code>persist</code> the default behaviour is to retain hotplugged volumes as hotplugged volumes after a VM restart. This makes the <code>persist</code> flag mostly obsolete unless you want to make a volume permanent on restart.</p>"},{"location":"storage/hotplug_volumes/#persist","title":"Persist","text":"<p>In some cases you want a hotplugged volume to become part of the standard disks after a restart of the VM. For instance if you added some permanent storage to the VM. We also assume that the running VMI has a matching VM that defines it specification. You can call the addvolume command with the --persist flag. This will update the VM domain disks section in addition to updating the VMI domain disks. This means that when you restart the VM, the disk is already defined in the VM, and thus in the new VMI.</p> <pre><code>$ virtctl addvolume vm-fedora --volume-name=example-volume-hotplug --persist\n</code></pre> <p>In the VM spec this will now show as a new disk <pre><code>spec:\ndomain:\n    devices:\n        disks:\n        - disk:\n            bus: virtio\n            name: containerdisk\n        - disk:\n            bus: virtio\n            name: cloudinitdisk\n        - disk:\n            bus: scsi\n            name: example-volume-hotplug\n    machine:\n      type: \"\"\n</code></pre></p>"},{"location":"storage/hotplug_volumes/#removevolume","title":"Removevolume","text":"<p>In addition to hotplug plugging the volume, you can also unplug it by using the 'removevolume' command available with virtctl <pre><code>$ virtctl removevolume vmi-fedora --volume-name=example-volume-hotplug\n</code></pre></p> <p>NOTE You can only unplug volumes that were dynamically added with addvolume, or using the API.</p>"},{"location":"storage/hotplug_volumes/#volumestatus","title":"VolumeStatus","text":"<p>VMI objects have a new <code>status.VolumeStatus</code> field. This is an array containing each disk, hotplugged or not. For example, after hotplugging the volume in the addvolume example, the VMI status will contain this: <pre><code>volumeStatus:\n- name: cloudinitdisk\n  target: vdb\n- name: containerdisk\n  target: vda\n- hotplugVolume:\n    attachPodName: hp-volume-7fmz4\n    attachPodUID: 62a7f6bf-474c-4e25-8db5-1db9725f0ed2\n  message: Successfully attach hotplugged volume volume-hotplug to VM\n  name: example-volume-hotplug\n  phase: Ready\n  reason: VolumeReady\n  target: sda\n</code></pre> Vda is the container disk that contains the Fedora OS, vdb is the cloudinit disk. As you can see those just contain the name and target used when assigning them to the VM. The target is the value passed to QEMU when specifying the disks. The value is unique for the VM and does NOT represent the naming inside the guest. For instance for a Windows Guest OS the target has no meaning. The same will be true for hotplugged volumes. The target is just a unique identifier meant for QEMU, inside the guest the disk can be assigned a different name.</p> <p>The hotplugVolume has some extra information that regular volume statuses do not have. The attachPodName is the name of the pod that was used to attach the volume to the node the VMI is running on. If this pod is deleted it will also stop the VMI as we cannot guarantee the volume will remain attached to the node. The other fields are similar to conditions and indicate the status of the hot plug process. Once a Volume is ready it can be used by the VM.</p>"},{"location":"storage/hotplug_volumes/#live-migration","title":"Live Migration","text":"<p>Currently Live Migration is enabled for any VMI that has volumes hotplugged into it. </p> <p>NOTE However there is a known issue that the migration may fail for VMIs with hotplugged block volumes if the target node uses CPU manager with static policy and <code>runc</code> prior to version <code>v1.0.0</code>.</p>"},{"location":"storage/snapshot_restore_api/","title":"Snapshot Restore API","text":"<p>The <code>snapshot.kubevirt.io</code> API Group defines resources for snapshotting and restoring KubeVirt <code>VirtualMachines</code></p>"},{"location":"storage/snapshot_restore_api/#prerequesites","title":"Prerequesites","text":""},{"location":"storage/snapshot_restore_api/#volumesnapshotclass","title":"VolumeSnapshotClass","text":"<p>KubeVirt leverages the <code>VolumeSnapshot</code> functionality of Kubernetes CSI drivers for capturing persistent <code>VirtualMachine</code> state.  So, you should make sure that your <code>VirtualMachine</code> uses <code>DataVolumes</code> or <code>PersistentVolumeClaims</code> backed by a <code>StorageClass</code> that supports <code>VolumeSnapshots</code> and a <code>VolumeSnapshotClass</code> is properly configured for that <code>StorageClass</code>.</p> <p>KubeVirt looks for Kubernetes Volume Snapshot related APIs/resources in the <code>v1</code> version. To make sure that KubeVirt's snapshot controller is able to snapshot the VirtualMachine and referenced volumes as expected, Kubernetes Volume Snapshot APIs must be served from <code>v1</code> version.</p> <p>To list <code>VolumeSnapshotClasses</code>:</p> <pre><code>kubectl get volumesnapshotclass\n</code></pre> <p>Make sure the <code>provisioner</code> property of your <code>StorageClass</code> matches the <code>driver</code> property of the <code>VolumeSnapshotClass</code></p> <p>Even if you have no <code>VolumeSnapshotClasses</code> in your cluster, <code>VirtualMachineSnapshots</code> are not totally useless.  They will still backup your <code>VirtualMachine</code> configuration.</p>"},{"location":"storage/snapshot_restore_api/#snapshot-feature-gate","title":"Snapshot Feature Gate","text":"<p>Snapshot/Restore support must be enabled in the feature gates to be supported. The feature gates field in the KubeVirt CR must be expanded by adding the <code>Snapshot</code> to it.</p>"},{"location":"storage/snapshot_restore_api/#snapshot-a-virtualmachine","title":"Snapshot a VirtualMachine","text":"<p>Snapshotting a virtualMachine is supported for online and offline vms.</p> <p>When snapshotting a running vm the controller will check for qemu guest agent in the vm. If the agent exists it will freeze the vm filesystems before taking the snapshot and unfreeze after the snapshot. It is recommended to take online snapshots with the guest agent for a better snapshot, if not present a best effort snapshot will be taken.</p> <p>Note To check if your vm has a qemu-guest-agent check for 'AgentConnected' in the vm status.</p> <p>There will be an indication in the vmSnapshot status if the snapshot was taken online and with or without guest agent participation.</p> <p>Note Online snapshot with hotplugged disks is supported, only persistent hotplugged disks will be included in the snapshot.</p> <p>To snapshot a <code>VirtualMachine</code> named <code>larry</code>, apply the following yaml.</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineSnapshot\nmetadata:\n  name: snap-larry\nspec:\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n</code></pre> <p>To wait for a snapshot to complete, execute:</p> <pre><code>kubectl wait vmsnapshot snap-larry --for condition=Ready\n</code></pre> <p>You can check the vmSnapshot phase in the vmSnapshot status. It can be one of the following:</p> <ul> <li>InProgress</li> <li>Succeeded</li> <li>Failed.</li> </ul> <p>The vmSnapshot has a default deadline of 5 minutes. If the vmSnapshot has not succeessfully completed before the deadline, it will be marked as Failed. The VM will be unfrozen and the created snapshot content will be cleaned up if necessary. The vmSnapshot object will remain in Failed state until deleted by the user. To change the default deadline add 'FailureDeadline' to the VirtualMachineSnapshot spec with a new value. The allowed format is a duration string which is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as \"300ms\", \"-1.5h\" or \"2h45m\"</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineSnapshot\nmetadata:\n  name: snap-larry\nspec:\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n  failureDeadline: 1m\n</code></pre> <p>In order to set an infinite deadline you can set it to 0 (not recommended).</p>"},{"location":"storage/snapshot_restore_api/#restoring-a-virtualmachine","title":"Restoring a VirtualMachine","text":"<p>To restore the <code>VirtualMachine</code> <code>larry</code> from <code>VirtualMachineSnapshot</code> <code>snap-larry</code>, Stop the VM, wait for it to be stopped and then apply the following yaml.</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineRestore\nmetadata:\n  name: restore-larry\nspec:\n  target:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n  virtualMachineSnapshotName: snap-larry\n</code></pre> <p>To wait for a restore to complete, execute:</p> <pre><code>kubectl wait vmrestore restore-larry --for condition=Ready\n</code></pre>"},{"location":"storage/snapshot_restore_api/#cleanup","title":"Cleanup","text":"<p>Keep <code>VirtualMachineSnapshots</code> (and their corresponding <code>VirtualMachineSnapshotContents</code>) around as long as you may want to restore from them again.</p> <p>Feel free to delete <code>restore-larry</code> as it is not needed once the restore is complete.</p>"},{"location":"storage/volume_migration/","title":"Migration update volume strategy and volume migration","text":"<p>Storage migration is possible while the VM is running by using the update volume strategy. Storage migration can be useful in the cases where the users need to change the underlying storage, for example, if the storage class has been deprecated, or there is a new more performant driver available.</p> <p>This feature doesn't handle the volume creation or cover migration between storage classes, but rather implements a basic API which can be used by overlaying tools to perform more advanced migration planning.</p> <p>If <code>Migration</code> is specified as <code>updateVolumesStrategy</code>, KubeVirt will try to migrate the storage from the old volume set to the new one when the VirtualMachine spec is updated. The migration considers the changed volumes present into a single update. A single update may contain modifications to more than one volume, but sequential changes to the volume set will be handled as separate migrations.</p> <p>Updates are declarative and GitOps compatible. For example, a new version of the VM specification with the new volume set and the migration volume update strategy can be directly applied using <code>kubectl apply</code> or interactively editing the VM with <code>kubectl edit</code></p> <p>Example: Original VM with a datavolume and datavolume template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-dv\n  name: vm-dv\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: src-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        volumeMode: Filesystem\n        resources:\n          requests:\n            storage: 2Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://registry:5000/kubevirt/alpine-container-disk-demo:devel\n  runStrategy: \"Always\"\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-dv\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n          interfaces:\n          - masquerade: {}\n            name: default\n        resources:\n          requests:\n            memory: 128Mi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: src-dv\n        name: datavolumedisk1\n      networks:\n      - name: default\n        pod: {}\n</code></pre></p> <p>The datavolume <code>src-dv</code> is migrated to the pvc <code>dest-pvc</code>: <pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: dst-dv\nspec:\n  source:\n      blank: {}\n  storage:\n    accessModes:\n      - ReadWriteMany\n    volumeMode: Block\n    resources:\n      requests:\n        storage: 5Gi\n    storageClassName: rook-ceph\n</code></pre></p> <p>by updating the VM: <pre><code> apiVersion: kubevirt.io/v1\n kind: VirtualMachine\n     kubevirt.io/vm: vm-dv\n   name: vm-dv\n spec:\n+  updateVolumesStrategy: Migration\n   dataVolumeTemplates:\n   - metadata:\n-      name: src-pvc\n+      name: dst-dv\n\n       volumes:\n       - dataVolume:\n-          name: src-pvc\n+          name: dst-dv\n         name: datavolumedisk1\n</code></pre></p> <p>The destination volume may be of a different type or size than the source. It is possible to migrate from and to a block volume as well as a filesystem volume. The destination volume should be equal to or larger than the source volume. However, the additional difference in the size of the destination volume is not instantly visible within the VM and must be manually resized because the guest is unaware of the migration.</p> <p>The volume migration depends on the <code>VolumeMigration</code> and <code>VolumesUpdateStrategy</code> feature gates and the <code>LiveMigrate</code> workloadUpdateStrategy. To fully enable the feature, add the following to the KubeVirt CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n        - VolumesUpdateStrategy\n        - VolumeMigration\n    vmRolloutStrategy: LiveUpdate\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre></p> <p>The volume migration progress can be monitored by watching the corresponding VirtualMachineInstanceMigration object using the label <code>kubevirt.io/volume-update-in-progress: &lt;vm-name&gt;</code>. Example: <pre><code>$ kubectl get virtualmachineinstancemigrations -l kubevirt.io/volume-update-in-progress: vmi` --watch=true\nNAME                           PHASE       VMI\nkubevirt-workload-update-abcd  Running     vmi\n</code></pre></p>"},{"location":"storage/volume_migration/#datavolume-template","title":"Datavolume template","text":"<p>Updating a datavolume that is referenced by a datavolume template requires special caution. The volumes section must include a reference to the name of the datavolume template. This means that the datavolume templates must either be entirely deleted or updated as well.</p> <p>Example of updating the datavolume for the original VM in the first example: <pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: dst-dv\nspec:\n  source:\n      blank: {}\n  storage:\n    accessModes:\n      - ReadWriteMany\n    volumeMode: Block\n    resources:\n      requests:\n        storage: 10Gi\n    storageClassName: rook-ceph-block\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-dv\n  name: vm-dv\nspec:\n  updateVolumesStrategy: Migration\n  dataVolumeTemplates:\n  - metadata:\n      name: dst-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        volumeMode: Filesystem\n        resources:\n          requests:\n            storage: 2Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://registry:5000/kubevirt/alpine-container-disk-demo:devel\n  runStrategy: \"Always\"\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-dv\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n          interfaces:\n          - masquerade: {}\n            name: default\n        resources:\n          requests:\n            memory: 128Mi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: dst-dv\n        name: datavolumedisk1\n      networks:\n      - name: default\n        pod: {}\n</code></pre></p>"},{"location":"storage/volume_migration/#limitations","title":"Limitations","text":"<p>Only certain types of disks and volumes are supported to be migrated. For an invalid type of volume the RestartRequired condition is set and volumes will be replaced upon VM restart. Currently, the volume migration is supported between PersistentVolumeClaims and Datavolumes. Additionally, volume migration is forbidden if the disk is: * shareable, since it cannot guarantee the data consistency with multiple writers * hotpluggable, this case isn't currently supported * filesystem, since virtiofs doesn't currently support live-migration * lun, originally the disk might support SCSI protocol but the destination PVC class does not. This case isn't currently supported.</p> <p>Currently, KubeVirt only enables live migration between separate nodes. Volume migration relies on live migration; hence, live migrating storage on the same node is also not possible. Volume migration is possible between local storage, like between 2 PVCs with RWO access mode, but they need to be located on two different host.</p>"},{"location":"user_workloads/accessing_virtual_machines/","title":"Accessing Virtual Machines","text":""},{"location":"user_workloads/accessing_virtual_machines/#graphical-and-serial-console-access","title":"Graphical and Serial Console Access","text":"<p>Once a virtual machine is started you are able to connect to the consoles it exposes. Usually there are two types of consoles:</p> <ul> <li>Serial Console</li> <li>Graphical Console (VNC)</li> </ul> <p>Note: You need to have <code>virtctl</code> installed to gain access to the VirtualMachineInstance.</p>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-serial-console","title":"Accessing the Serial Console","text":"<p>The serial console of a virtual machine can be accessed by using the <code>console</code> command:</p> <pre><code>virtctl console testvm\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-graphical-console-vnc","title":"Accessing the Graphical Console (VNC)","text":"<p>To access the graphical console of a virtual machine the VNC protocol is typically used. This requires <code>remote-viewer</code> to be installed. Once the tool is installed, you can access the graphical console using:</p> <pre><code>virtctl vnc testvm\n</code></pre> <p>If you only want to open a vnc-proxy without executing the <code>remote-viewer</code> command, it can be accomplished with:</p> <pre><code>virtctl vnc --proxy-only testvm\n</code></pre> <p>This would print the port number on your machine where you can manually connect using any VNC viewer.</p>"},{"location":"user_workloads/accessing_virtual_machines/#debugging-console-access","title":"Debugging console access","text":"<p>If the connection fails, you can use the <code>-v</code> flag to get more verbose output from both <code>virtctl</code> and the <code>remote-viewer</code> tool to troubleshoot the problem.</p> <pre><code>virtctl vnc testvm -v 4\n</code></pre> <p>Note: If you are using virtctl via SSH on a remote machine, you need to forward the X session to your machine. Look up the -X and -Y flags of <code>ssh</code> if you are not familiar with that. As an alternative you can proxy the API server port with SSH to your machine (either direct or in combination with <code>kubectl proxy</code>).</p>"},{"location":"user_workloads/accessing_virtual_machines/#ssh-access","title":"SSH Access","text":"<p>A common operational pattern used when managing virtual machines is to inject SSH public keys into the virtual machines at boot. This allows automation tools (like Ansible) to provision the virtual machine. It also gives operators a way of gaining secure and passwordless access to a virtual machine.</p> <p>KubeVirt provides multiple ways to inject SSH public keys into a virtual machine.</p> <p>In general, these methods fall into two categories:  - Static key injection, which places keys on the virtual machine the first time it is booted.  - Dynamic key injection, which allows keys to be dynamically updated both at boot and during runtime.</p> <p>Once a SSH public key is injected into the virtual machine, it can be accessed via <code>virtctl</code>.</p>"},{"location":"user_workloads/accessing_virtual_machines/#static-ssh-public-key-injection-via-cloud-init","title":"Static SSH public key injection via cloud-init","text":"<p>Users creating virtual machines can provide startup scripts to their virtual machines, allowing multiple customization operations.</p> <p>One option for injecting public SSH keys into a VM is via cloud-init startup script. However, there are more flexible options available.</p> <p>The virtual machine's access credential API allows statically injecting SSH public keys at startup time independently of the cloud-init user data by placing the SSH public key into a Kubernetes <code>Secret</code>. This allows keeping the application data in the cloud-init user data separate from the credentials used to access the virtual machine.</p> <p>A Kubernetes <code>Secret</code> can be created from an SSH public key like this:</p> <pre><code># Place SSH public key into a Secret\nkubectl create secret generic my-pub-key --from-file=key1=id_rsa.pub\n</code></pre> <p>The <code>Secret</code> containing the public key is then assigned to a virtual machine using the access credentials API with the <code>noCloud</code> propagation method.</p> <p>KubeVirt injects the SSH public key into the virtual machine by using the generated cloud-init metadata instead of the user data. This separates the application user data and user credentials.</p> <p>Note: The cloud-init <code>userData</code> is not touched.</p> <pre><code># Create a VM referencing the Secret using propagation method noCloud\nkubectl create -f - &lt;&lt;EOF\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      terminationGracePeriodSeconds: 0\n      accessCredentials:\n      - sshPublicKey:\n          source:\n            secret:\n              secretName: my-pub-key\n          propagationMethod:\n            noCloud: {}\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\nEOF\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#dynamic-ssh-public-key-injection-via-qemu-guest-agent","title":"Dynamic SSH public key injection via qemu-guest-agent","text":"<p>KubeVirt allows the dynamic injection of SSH public keys into a VirtualMachine with the access credentials API.</p> <p>Utilizing the <code>qemuGuestAgent</code> propagation method, configured Secrets are attached to a VirtualMachine when the VM is started.  This allows for dynamic injection of SSH public keys at runtime by updating the attached Secrets.</p> <p>Please note that new Secrets cannot be attached to a running VM: You must restart the VM to attach the new Secret.</p> <p>Note: This requires the qemu-guest-agent to be installed within the guest.</p> <p>Note: When using qemuGuestAgent propagation, the <code>/home/$USER/.ssh/authorized_keys</code> file will be owned by the guest agent. Changes to the file not made by the guest agent will be lost.</p> <p>Note: More information about the motivation behind the access credentials API can be found in the pull request description that introduced the API.</p> <p>In the example below the <code>Secret</code> containing the SSH public key is attached to the virtual machine via the access credentials API with the <code>qemuGuestAgent</code> propagation method. This allows updating the contents of the <code>Secret</code> at any time, which will result in the changes getting applied to the running virtual machine immediately. The <code>Secret</code> may also contain multiple SSH public keys.</p> <pre><code># Place SSH public key into a secret\nkubectl create secret generic my-pub-key --from-file=key1=id_rsa.pub\n</code></pre> <p>Now reference this secret in the <code>VirtualMachine</code> spec with the access credentials API using <code>qemuGuestAgent</code> propagation.</p> <pre><code># Create a VM referencing the Secret using propagation method qemuGuestAgent\nkubectl create -f - &lt;&lt;EOF\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      terminationGracePeriodSeconds: 0\n      accessCredentials:\n      - sshPublicKey:\n          source:\n            secret:\n              secretName: my-pub-key\n          propagationMethod:\n            qemuGuestAgent:\n              users:\n              - fedora\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n            # Disable SELinux for now, so qemu-guest-agent can write the authorized_keys file\n            # The selinux-policy is too restrictive currently, see open bugs:\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=1917024\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=2028762\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=2057310\n            bootcmd:\n              - setenforce 0\n        name: cloudinitdisk\nEOF\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-vmi-using-virtctl","title":"Accessing the VMI using virtctl","text":"<p>The user can create a websocket backed network tunnel to a port inside the instance by using the <code>virtualmachineinstances/portforward</code> subresource of the <code>VirtualMachineInstance</code>.</p> <p>One use-case for this subresource is to forward SSH traffic into the <code>VirtualMachineInstance</code> either from the CLI or a web-UI.</p> <p>To connect to a <code>VirtualMachineInstance</code> from your local machine, <code>virtctl</code> provides a lightweight SSH client with the <code>ssh</code> command, that uses port forwarding. Refer to the command's help for more details.</p> <pre><code>virtctl ssh\n</code></pre> <p>To transfer files from or to a <code>VirtualMachineInstance</code> <code>virtctl</code> also provides a lightweight SCP client with the <code>scp</code> command. Its usage is similar to the <code>ssh</code> command. Refer to the command's help for more details.</p> <pre><code>virtctl scp\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#using-virtctl-as-proxy","title":"Using virtctl as proxy","text":"<p>If you prefer to use your local OpenSSH client, there are two ways of doing that in combination with virtctl.</p> <p>Note: Most of this applies to the <code>virtctl scp</code> command too.</p> <ol> <li>The <code>virtctl ssh</code> command has a <code>--local-ssh</code> option. With this option    <code>virtctl</code> wraps the local OpenSSH client transparently to the user. The    executed SSH command can be viewed by increasing the verbosity (<code>-v 3</code>).</li> </ol> <pre><code>virtctl ssh --local-ssh -v 3 testvm\n</code></pre> <ol> <li>The <code>virtctl port-forward</code> command provides an option to tunnel a single    port to your local stdout/stdin. This allows the command to be used in    combination with the OpenSSH client's <code>ProxyCommand</code> option.</li> </ol> <pre><code>ssh -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' fedora@testvm.mynamespace\n</code></pre> <p>To provide easier access to arbitrary virtual machines you can add the following lines to your SSH <code>config</code>:</p> <pre><code>Host vmi/*\n   ProxyCommand virtctl port-forward --stdio=true %h %p\nHost vm/*\n   ProxyCommand virtctl port-forward --stdio=true %h %p\n</code></pre> <p>This allows you to simply call <code>ssh user@vmi/testvmi.mynamespace</code> and your SSH config and virtctl will do the rest. Using this method it becomes easy to set up different identities for different namespaces inside your SSH <code>config</code>.</p> <p>This feature can also be used with Ansible to automate configuration of virtual machines running on KubeVirt. You can put the snippet above into its own file (e.g. <code>~/.ssh/virtctl-proxy-config</code>) and add the following lines to your <code>.ansible.cfg</code>:</p> <pre><code>[ssh_connection]\nssh_args = -F ~/.ssh/virtctl-proxy-config\n</code></pre> <p>Note that all port forwarding traffic will be sent over the Kubernetes control plane. A high amount of connections and traffic can increase pressure on the API server. If you regularly need a high amount of connections and traffic consider using a dedicated Kubernetes <code>Service</code> instead.</p>"},{"location":"user_workloads/accessing_virtual_machines/#example","title":"Example","text":"<ol> <li> <p>Create virtual machine and inject SSH public key as explained above</p> </li> <li> <p>SSH into virtual machine</p> </li> </ol> <pre><code># Add --local-ssh to transparently use local OpenSSH client\nvirtctl ssh -i id_rsa fedora@testvm\n</code></pre> <p>or</p> <pre><code>ssh -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' -i id_rsa fedora@vmi/testvm.mynamespace\n</code></pre> <ol> <li>SCP file to the virtual machine</li> </ol> <pre><code># Add --local-ssh to transparently use local OpenSSH client\nvirtctl scp -i id_rsa testfile fedora@testvm:/tmp\n</code></pre> <p>or</p> <pre><code>scp -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' -i id_rsa testfile fedora@testvm.mynamespace:/tmp\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#rbac-permissions-for-consolevncssh-access","title":"RBAC permissions for Console/VNC/SSH access","text":""},{"location":"user_workloads/accessing_virtual_machines/#using-default-rbac-cluster-roles","title":"Using default RBAC cluster roles","text":"<p>Every KubeVirt installation starting with version v0.5.1 ships a set of default RBAC cluster roles that can be used to grant users access to VirtualMachineInstances.</p> <p>The <code>kubevirt.io:admin</code> and <code>kubevirt.io:edit</code> cluster roles have console, VNC and SSH respectively port-forwarding access permissions built into them. By binding either of these roles to a user, they will have the ability to use virtctl to access the console, VNC and SSH.</p>"},{"location":"user_workloads/accessing_virtual_machines/#using-custom-rbac-cluster-role","title":"Using custom RBAC cluster role","text":"<p>The default KubeVirt cluster roles grant access to more than just the console, VNC and port-forwarding. The <code>ClusterRole</code> below demonstrates how to craft a custom role, that only allows access to the console, VNC and port-forwarding.</p> <pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n  name: allow-console-vnc-port-forward-access\nrules:\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/console\n      - virtualmachineinstances/vnc\n    verbs:\n      - get\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/portforward\n    verbs:\n      - update\n</code></pre> <p>When bound with a <code>ClusterRoleBinding</code> the <code>ClusterRole</code> above grants access to virtual machines across all namespaces.</p> <p>In order to reduce the scope to a single namespace, bind this <code>ClusterRole</code> using a <code>RoleBinding</code> that targets a single namespace.</p>"},{"location":"user_workloads/basic_use/","title":"Basic use","text":"<p>Using KubeVirt should be fairly natural if you are used to working with Kubernetes.</p> <p>The primary way of using KubeVirt is by working with the KubeVirt kinds in the Kubernetes API:</p> <pre><code>$ kubectl create -f vmi.yaml\n$ kubectl wait --for=condition=Ready vmis/my-vmi\n$ kubectl get vmis\n$ kubectl delete vmis testvmi\n</code></pre> <p>The following pages describe how to use and discover the API, manage, and access virtual machines.</p>"},{"location":"user_workloads/basic_use/#user-interface","title":"User Interface","text":"<p>KubeVirt does not come with a UI, it is only extending the Kubernetes API with virtualization functionality.</p>"},{"location":"user_workloads/boot_from_external_source/","title":"Booting From External Source","text":"<p>When installing a new guest virtual machine OS, it is often useful to boot directly from a kernel and initrd stored in the host physical machine OS, allowing command line arguments to be passed directly to the installer.</p> <p>Booting from an external source is supported in Kubevirt starting from version v0.42.0-rc.0. This enables the capability to define a Virtual Machine that will use a custom kernel / initrd binary, with possible custom arguments, during its boot process.</p> <p>The binaries are provided though a container image. The container is pulled from the container registry and resides on the local node hosting the VMs.</p>"},{"location":"user_workloads/boot_from_external_source/#use-cases","title":"Use cases","text":"<p>Some use cases for this may be: - For a kernel developer it may be very convenient to launch VMs that are defined to boot from the latest kernel binary that is often being changed. - Initrd can be set with files that need to reside on-memory during all the VM's life-cycle.</p>"},{"location":"user_workloads/boot_from_external_source/#workflow","title":"Workflow","text":"<p>Defining an external boot source can be done in the following way: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: ext-kernel-boot-vm\nspec:\n  runStrategy: Manual\n  template:\n    spec:\n      domain:\n        devices: {}\n        firmware:\n          kernelBoot:\n            container:\n              image: vmi_ext_boot/kernel_initrd_binaries_container:latest\n              initrdPath: /boot/initramfs-virt\n              kernelPath: /boot/vmlinuz-virt\n              imagePullPolicy: Always\n              imagePullSecret: IfNotPresent\n            kernelArgs: console=ttyS0\n        resources:\n          requests:\n            memory: 1Gi\n</code></pre></p> <p>Notes:</p> <ul> <li> <p><code>initrdPath</code> and <code>kernelPath</code> define the path for the binaries inside the container.</p> </li> <li> <p>Kernel and Initrd binaries must be owned by <code>qemu</code> user &amp; group.</p> </li> <li> <p>To change ownership: <code>chown qemu:qemu &lt;binary&gt;</code> when <code>&lt;binary&gt;</code> is the binary file.</p> </li> <li> <p><code>kernelArgs</code> can only be provided if a kernel binary is provided (i.e. <code>kernelPath</code> not defined). These arguments will be passed to the default kernel the VM boots from.</p> </li> <li> <p><code>imagePullSecret</code> and <code>imagePullPolicy</code> are optional</p> </li> <li> <p>if <code>imagePullPolicy</code> is <code>Always</code> and the container image is updated then the VM will be booted   into the new kernel when VM restarts</p> </li> </ul>"},{"location":"user_workloads/component_monitoring/","title":"Component monitoring","text":"<p>All KubeVirt system-components expose Prometheus metrics at their <code>/metrics</code> REST endpoint.</p> <p>You can consult the complete and up-to-date metric list at kubevirt/monitoring.</p>"},{"location":"user_workloads/component_monitoring/#custom-service-discovery","title":"Custom Service Discovery","text":"<p>Prometheus supports service discovery based on Pods and Endpoints out of the box. Both can be used to discover KubeVirt services.</p> <p>All Pods which expose metrics are labeled with <code>prometheus.kubevirt.io</code> and contain a port-definition which is called <code>metrics</code>. In the KubeVirt release-manifests, the default <code>metrics</code> port is <code>8443</code>.</p> <p>The above labels and port informations are collected by a <code>Service</code> called <code>kubevirt-prometheus-metrics</code>. Kubernetes automatically creates a corresponding <code>Endpoint</code> with an equal name:</p> <pre><code>$ kubectl get endpoints -n kubevirt kubevirt-prometheus-metrics -o yaml\napiVersion: v1\nkind: Endpoints\nmetadata:\n  labels:\n    kubevirt.io: \"\"\n    prometheus.kubevirt.io: \"\"\n  name: kubevirt-prometheus-metrics\n  namespace: kubevirt\nsubsets:\n- addresses:\n  - ip: 10.244.0.5\n    nodeName: node01\n    targetRef:\n      kind: Pod\n      name: virt-handler-cjzg6\n      namespace: kubevirt\n      resourceVersion: \"4891\"\n      uid: c67331f9-bfcf-11e8-bc54-525500d15501\n  - ip: 10.244.0.6\n  [...]\n  ports:\n  - name: metrics\n    port: 8443\n    protocol: TCP\n</code></pre> <p>By watching this endpoint for added and removed IPs to <code>subsets.addresses</code> and appending the <code>metrics</code> port from <code>subsets.ports</code>, it is possible to always get a complete list of ready-to-be-scraped Prometheus targets.</p>"},{"location":"user_workloads/component_monitoring/#integrating-with-the-prometheus-operator","title":"Integrating with the prometheus-operator","text":"<p>The prometheus-operator can make use of the <code>kubevirt-prometheus-metrics</code> service to automatically create the appropriate Prometheus config.</p> <p>KubeVirt's <code>virt-operator</code> checks if the <code>ServiceMonitor</code> custom resource exists when creating an install strategy for deployment. KubeVirt will automatically create a <code>ServiceMonitor</code> resource in the <code>monitorNamespace</code>, as well as an appropriate role and rolebinding in KubeVirt's namespace.</p> <p>Three settings are exposed in the <code>KubeVirt</code> custom resource to direct KubeVirt to create these resources correctly:</p> <ul> <li> <p><code>monitorNamespace</code>: The namespace that prometheus-operator runs in.     Defaults to <code>openshift-monitoring</code>.</p> </li> <li> <p><code>monitorAccount</code>: The serviceAccount that prometheus-operator runs     with. Defaults to <code>prometheus-k8s</code>.</p> </li> <li> <p><code>serviceMonitorNamespace</code>: The namespace that the serviceMonitor runs in.     Defaults to be <code>monitorNamespace</code> </p> </li> </ul> <p>Please note that if you decide to set <code>serviceMonitorNamespace</code> than this  namespace must be included in <code>serviceMonitorNamespaceSelector</code> field of  Prometheus spec.</p> <p>If the prometheus-operator for a given deployment uses these defaults, then these values can be omitted.</p> <p>An example of the KubeVirt resource depicting these default values:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\nspec:\n  monitorNamespace: openshift-monitoring\n  monitorAccount: prometheus-k8s\n</code></pre>"},{"location":"user_workloads/component_monitoring/#integrating-with-the-okd-cluster-monitoring-operator","title":"Integrating with the OKD cluster-monitoring-operator","text":"<p>After the cluster-monitoring-operator is up and running, KubeVirt will detect the existence of the <code>ServiceMonitor</code> resource. Because the definition contains the <code>openshift.io/cluster-monitoring</code> label, it will automatically be picked up by the cluster monitor.</p>"},{"location":"user_workloads/component_monitoring/#metrics-about-virtual-machines","title":"Metrics about Virtual Machines","text":"<p>The endpoints report metrics related to the runtime behaviour of the Virtual Machines. All the relevant metrics are prefixed with <code>kubevirt_vmi</code>.</p> <p>The metrics have labels that allow to connect to the VMI objects they refer to. At minimum, the labels will expose <code>node</code>, <code>name</code> and <code>namespace</code> of the related VMI object.</p> <p>For example, reported metrics could look like</p> <pre><code>kubevirt_vmi_memory_resident_bytes{domain=\"default_vm-test-01\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\"} 2.5595904e+07\nkubevirt_vmi_network_traffic_bytes_total{domain=\"default_vm-test-01\",interface=\"vnet0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",type=\"rx\"} 8431\nkubevirt_vmi_network_traffic_bytes_total{domain=\"default_vm-test-01\",interface=\"vnet0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",type=\"tx\"} 1835\nkubevirt_vmi_vcpu_seconds_total{domain=\"default_vm-test-01\",id=\"0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",state=\"1\"} 19\n</code></pre> <p>Please note the <code>domain</code> label in the above example. This label is deprecated and it will be removed in a future release. You should identify the VMI using the <code>node</code>, <code>namespace</code>, <code>name</code> labels instead.</p>"},{"location":"user_workloads/component_monitoring/#important-queries","title":"Important Queries","text":""},{"location":"user_workloads/component_monitoring/#detecting-connection-issues-for-the-rest-client","title":"Detecting connection issues for the REST client","text":"<p>Use the following query to get a counter for all REST call which indicate connection issues:</p> <pre><code>rest_client_requests_total{code=\"&lt;error&gt;\"}\n</code></pre> <p>If this counter is continuously increasing, it is an indicator that the corresponding KubeVirt component has general issues to connect to the apiserver</p>"},{"location":"user_workloads/creating_vms/","title":"Creating VirtualMachines","text":"<p>The virtctl sub command <code>create vm</code> allows easy creation of VirtualMachine manifests from the command line. It leverages instance types and preferences and inference by default (see Specifying or inferring instance types and preferences) and provides several flags to control details of the created virtual machine.</p> <p>For example there are flags to specify the name or run strategy of a virtual machine or flags to add volumes to a virtual machine. Instance types and preferences can either be specified directly or it is possible to let KubeVirt infer those from the volume used to boot the virtual machine.</p> <p>For a full set of flags and their description use the following command:</p> <pre><code>virtctl create vm -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#creating-virtualmachines-on-a-cluster","title":"Creating VirtualMachines on a cluster","text":"<p>The output of virtctl <code>create vm</code> can be piped into <code>kubectl</code> to directly create a VirtualMachine on a cluster, e.g.:</p> <pre><code># Create a VM with name my-vm on the cluster\nvirtctl create vm --name my-vm | kubectl create -f -\nvirtualmachine.kubevirt.io/my-vm created\n</code></pre>"},{"location":"user_workloads/creating_vms/#creating-instance-types","title":"Creating Instance Types","text":"<p>The virtctl subcommand <code>create instancetype</code> allows easy creation of an instance type manifest from the command line. The command also provides several flags that can be used to create your desired manifest.</p> <p>There are two required flags that need to be specified: the number of vCPUs and the amount of memory to be requested. Additionally, there are several optional flags that can be used, such as specifying a list of GPUs for passthrough, choosing the desired IOThreadsPolicy, or simply providing the name of our instance type.</p> <p>By default, the command creates the cluster-wide resource. If the user wants to create the namespaced version, they need to provide the namespaced flag. The namespace name can be specified by using the namespace flag.</p> <p>For a complete list of flags and their descriptions, use the following command:</p> <pre><code>virtctl create instancetype -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples","title":"Examples","text":"<p>Create a manifest for a VirtualMachineClusterInstancetype with the required --cpu and --memory flags <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi\n</code></pre></p> <p>Create a manifest for a VirtualMachineInstancetype with a specified namespace <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi --namespace my-namespace\n</code></pre></p> <p>Create a manifest for a VirtualMachineInstancetype without a specified namespace name <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi --namespaced\n</code></pre></p>"},{"location":"user_workloads/creating_vms/#creating-preferences","title":"Creating Preferences","text":"<p>The virtctl subcommand <code>create preference</code> allows easy creation of a preference manifest from the command line. This command serves as a starting point to create the basic structure of a manifest, as it does not allow specifying all of the options that are supported in preferences.</p> <p>The current set of flags allows us, for example, to specify the preferred CPU topology, machine type or a storage class.</p> <p>By default, the command creates the cluster-wide resource. If the user wants to create the namespaced version, they need to provide the namespaced flag. The namespace name can be specified by using the namespace flag.</p> <p>For a complete list of flags and their descriptions, use the following command:</p> <pre><code>virtctl create preference -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples_1","title":"Examples","text":"<p>Create a manifest for a VirtualMachineClusterPreference with a preferred cpu topology <pre><code>virtctl create preference --cpu-topology preferSockets\n</code></pre></p> <p>Create a manifest for a VirtualMachinePreference with a specified namespace <pre><code>virtctl create preference --namespace my-namespace\n</code></pre></p> <p>Create a manifest for a VirtualMachinePreference with the preferred storage class <pre><code>virtctl create preference --namespaced --volume-storage-class my-storage\n</code></pre></p>"},{"location":"user_workloads/creating_vms/#specifying-or-inferring-instance-types-and-preferences","title":"Specifying or inferring instance types and preferences","text":"<p>Instance types and preference can be specified with the appropriate flags, e.g.:</p> <pre><code>virtctl create vm --instancetype my-instancetype --preference my-preference\n</code></pre> <p>The type of the instance type or preference (namespaced or cluster scope) can be controlled by prefixing the instance type or preference name with the corresponding CRD name, e.g.:</p> <pre><code># Using a cluster scoped instancetype and a namespaced preference\nvirtctl create vm \\\n  --instancetype virtualmachineclusterinstancetype/my-instancetype \\\n  --preference virtualmachinepreference/my-preference\n</code></pre> <p>If a prefix was not supplied the cluster scoped resources will be used by default.</p> <p>To explicitly infer instance types and/or preferences from the volume used to boot the virtual machine add the following flags:</p> <pre><code>virtctl create vm --infer-instancetype --infer-preference\n</code></pre> <p>The implicit default is to always try inferring an instancetype and preference from the boot volume. This feature makes use of the <code>IgnoreInferFromVolumeFailure</code> policy, which suppresses failures on inference of instancetypes and preferences. If one of the above switches was provided explicitly, then the <code>RejectInferFromVolumeFailure</code> policy is used instead. This way users are made aware of potential issues during the virtual machine creation.</p>"},{"location":"user_workloads/creating_vms/#boot-order-of-added-volumes","title":"Boot order of added volumes","text":"<p>Please note that volumes of different kinds currently have the following fixed boot order regardless of the order their flags were specified on the command line:</p> <ol> <li>ContainerDisk</li> <li>DataSource</li> <li>Cloned PVC</li> <li>Directly used PVC</li> </ol> <p>If multiple volumes of the same kind were specified their order is determined by the order in which their flags were specified.</p>"},{"location":"user_workloads/creating_vms/#specifying-cloud-init-user-data","title":"Specifying cloud-init user data","text":"<p>To pass cloud-init user data to virtctl it needs to be encoded into a base64 string. Here is an example how to do it:</p> <pre><code># Put your cloud-init user data into a file.\n# This will add an authorized key to the default user.\n# To get the default username read the documentation for the cloud image\n$ cat cloud-init.txt\n#cloud-config\nssh_authorized_keys:\n  - ssh-rsa AAAA...\n\n# Base64 encode the contents of the file without line wraps and store it in a variable\n$ CLOUD_INIT_USERDATA=$(base64 -w 0 cloud-init.txt)\n\n# Show the contents of the variable\n$ echo $CLOUD_INIT_USERDATA I2Nsb3VkLWNvbmZpZwpzc2hfYXV0aG9yaXplZF9rZXlzOgogIC0gc3NoLXJzYSBBQUFBLi4uCg==\n</code></pre> <p>You can now use this variable as an argument to the <code>--cloud-init-user-data</code> flag:</p> <pre><code>virtctl create vm --cloud-init-user-data $CLOUD_INIT_USERDATA\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples_2","title":"Examples","text":"<p>Create a manifest for a VirtualMachine with a random name:</p> <pre><code>virtctl create vm\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified name and RunStrategy Always</p> <pre><code>virtctl create vm --name=my-vm --run-strategy=Always\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineClusterInstancetype</p> <pre><code>virtctl create vm --instancetype=my-instancetype\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineInstancetype (namespaced)</p> <pre><code>virtctl create vm --instancetype=virtualmachineinstancetype/my-instancetype\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineClusterPreference</p> <pre><code>virtctl create vm --preference=my-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachinePreference (namespaced)</p> <pre><code>virtctl create vm --preference=virtualmachinepreference/my-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with an ephemeral containerdisk volume</p> <pre><code>virtctl create vm --volume-containerdisk=src:my.registry/my-image:my-tag\n</code></pre> <p>Create a manifest for a VirtualMachine with a cloned DataSource in namespace and specified size</p> <pre><code>virtctl create vm --volume-datasource=src:my-ns/my-ds,size:50Gi\n</code></pre> <p>Create a manifest for a VirtualMachine with a cloned DataSource and inferred instancetype and preference</p> <pre><code>virtctl create vm --volume-datasource=src:my-annotated-ds --infer-instancetype --infer-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and cloned PVC</p> <pre><code>virtctl create vm --volume-clone-pvc=my-ns/my-pvc\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and directly used PVC</p> <pre><code>virtctl create vm --volume-pvc=my-pvc\n</code></pre> <p>Create a manifest for a VirtualMachine with a clone DataSource and a blank volume</p> <pre><code>virtctl create vm --volume-datasource=src:my-ns/my-ds --volume-blank=size:50Gi\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and cloned DataSource</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-datasource=src:my-ds\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and two cloned DataSources (flag can be provided multiple times)</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-datasource=src:my-ds1 --volume-datasource=src:my-ds2\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and directly used PVC</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-pvc=my-pvc\n</code></pre>"},{"location":"user_workloads/deploy_common_instancetypes/","title":"Deploy common-instancetypes","text":"<p>The <code>kubevirt/common-instancetypes</code> provide a set of instancetypes and preferences to help create KubeVirt <code>VirtualMachines</code>.</p> <p>Beginning with the 1.1 release of KubeVirt, cluster wide resources can be deployed directly through KubeVirt, without another operator. This allows deployment of a set of default instancetypes and preferences along side KubeVirt.</p>"},{"location":"user_workloads/deploy_common_instancetypes/#enable-automatic-deployment-of-common-instancetypes","title":"Enable automatic deployment of common-instancetypes","text":"<p>To enable the deployment of cluster-wide common-instancetypes through the KubeVirt <code>virt-operator</code>, the <code>CommonInstancetypesDeploymentGate</code> feature gate needs to be enabled.</p> <p>See Activating feature gates on how to enable it.</p>"},{"location":"user_workloads/deploy_common_instancetypes/#deploy-common-instancetypes-manually","title":"Deploy common-instancetypes manually","text":"<p>For customization purposes or to install namespaced resources, common-instancetypes can also be deployed by hand.</p> <p>To install all resources provided by the <code>kubevirt/common-instancetypes</code> project without further customizations, simply apply with <code>kustomize</code> enabled (-k flag):</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git\n</code></pre> <p>Alternatively, targets for each of the available custom resource types (e.g. namespaced instancetypes) are available.</p> <p>For example, to deploy <code>VirtualMachineInstancetypes</code> run the following command:</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git/VirtualMachineInstancetypes\n</code></pre>"},{"location":"user_workloads/guest_agent_information/","title":"Guest Agent information","text":"<p>Guest Agent (GA) is an optional component that can run inside of Virtual Machines. The GA provides plenty of additional runtime information about the running operating system (OS). More technical detail about available GA commands is available here.</p>"},{"location":"user_workloads/guest_agent_information/#guest-agent-info-in-virtual-machine-status","title":"Guest Agent info in Virtual Machine status","text":"<p>GA presence in the Virtual Machine is signaled with a condition in the VirtualMachineInstance status. The condition tells that the GA is connected and can be used.</p> <p>GA condition on VirtualMachineInstance</p> <pre><code>status:\n  conditions:\n  - lastProbeTime: \"2020-02-28T10:22:59Z\"\n    lastTransitionTime: null\n    status: \"True\"\n    type: AgentConnected\n</code></pre> <p>When the GA is connected, additional OS information is shown in the status. This information comprises:</p> <ul> <li>guest info, which contains OS runtime data</li> <li>interfaces info, which shows QEMU interfaces merged with GA interfaces info.</li> </ul> <p>Below is the example of the information shown in the VirtualMachineInstance status.</p> <p>GA info with merged into status</p> <pre><code>status:\n  guestOSInfo:\n    id: fedora\n    kernelRelease: 4.18.16-300.fc29.x86_64\n    kernelVersion: '#1 SMP Sat Oct 20 23:24:08 UTC 2018'\n    name: Fedora\n    prettyName: Fedora 29 (Cloud Edition)\n    version: \"29\"\n    versionId: \"29\"\n  interfaces:\n  - infoSource: domain, guest-agent\n    interfaceName: eth0\n    ipAddress: 10.244.0.23/24\n    ipAddresses:\n    - 10.244.0.23/24\n    - fe80::858:aff:fef4:17/64\n    mac: 0a:58:0a:f4:00:17\n    name: default\n</code></pre> <p>When the Guest Agent is not present in the Virtual Machine, the Guest Agent information is not shown. No error is reported because the Guest Agent is an optional component.</p> <p>The infoSource field indicates where the info is gathered from. Valid values:</p> <ul> <li>domain: the info is based on the domain spec</li> <li>guest-agent: the info is based on Guest Agent report</li> <li>domain, guest-agent: the info is based on both the domain spec and the Guest Agent report</li> </ul>"},{"location":"user_workloads/guest_agent_information/#guest-agent-info-available-through-the-api","title":"Guest Agent info available through the API","text":"<p>The data shown in the VirtualMachineInstance status are a subset of the information available. The rest of the data is available via the REST API exposed in the Kubernetes <code>kube-api</code> server.</p> <p>There are three new subresources added to the VirtualMachineInstance object:</p> <pre><code>- guestosinfo\n- userlist\n- filesystemlist\n</code></pre> <p>The whole GA data is returned via <code>guestosinfo</code> subresource available behind the API endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/guestosinfo\n</code></pre> <p>GuestOSInfo sample data:</p> <pre><code>{\n    \"fsInfo\": {\n        \"disks\": [\n            {\n                \"diskName\": \"vda1\",\n                \"fileSystemType\": \"ext4\",\n                \"mountPoint\": \"/\",\n                \"totalBytes\": 0,\n                \"usedBytes\": 0\n            }\n        ]\n    },\n    \"guestAgentVersion\": \"2.11.2\",\n    \"hostname\": \"testvmi6m5krnhdlggc9mxfsrnhlxqckgv5kqrwcwpgr5mdpv76grrk\",\n    \"metadata\": {\n        \"creationTimestamp\": null\n    },\n    \"os\": {\n        \"id\": \"fedora\",\n        \"kernelRelease\": \"4.18.16-300.fc29.x86_64\",\n        \"kernelVersion\": \"#1 SMP Sat Oct 20 23:24:08 UTC 2018\",\n        \"machine\": \"x86_64\",\n        \"name\": \"Fedora\",\n        \"prettyName\": \"Fedora 29 (Cloud Edition)\",\n        \"version\": \"29 (Cloud Edition)\",\n        \"versionId\": \"29\"\n    },\n    \"timezone\": \"UTC, 0\"\n}\n</code></pre> <p>Items FSInfo and UserList are capped to the max capacity of 10 items, as a precaution for VMs with thousands of users.</p> <p>Full list of Filesystems is available through the subresource <code>filesystemlist</code> which is available as endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/filesystemlist\n</code></pre> <p>Filesystem sample data:</p> <pre><code>{\n    \"items\": [\n        {\n            \"diskName\": \"vda1\",\n            \"fileSystemType\": \"ext4\",\n            \"mountPoint\": \"/\",\n            \"totalBytes\": 3927900160,\n            \"usedBytes\": 1029201920\n        }\n    ],\n    \"metadata\": {}\n}\n</code></pre> <p>Full list of the Users is available through the subresource <code>userlist</code> which is available as endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/userlist\n</code></pre> <p>Userlist sample data:</p> <pre><code>{\n    \"items\": [\n        {\n            \"loginTime\": 1580467675.876078,\n            \"userName\": \"fedora\"\n        }\n    ],\n    \"metadata\": {}\n}\n</code></pre> <p>User LoginTime is in fractional seconds since epoch time. It is left for the consumer to convert to the desired format.</p>"},{"location":"user_workloads/guest_operating_system_information/","title":"Guest Operating System Information","text":"<p>Guest operating system identity for the VirtualMachineInstance will be provided by the label <code>kubevirt.io/os</code> :</p> <pre><code>metadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win2k12r2\n</code></pre> <p>The <code>kubevirt.io/os</code> label is based on the short OS identifier from libosinfo database. The following Short IDs are currently supported:</p> Short ID Name Version Family ID <p>win2k12r2</p> <p>Microsoft Windows Server 2012 R2</p> <p>6.3</p> <p>winnt</p> <p>https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2012-r2</p>"},{"location":"user_workloads/guest_operating_system_information/#use-with-presets","title":"Use with presets","text":"<p>A VirtualMachineInstancePreset representing an operating system with a <code>kubevirt.io/os</code> label could be applied on any given VirtualMachineInstance that have and match the <code>kubevirt.io/os</code> label.</p> <p>Default presets for the OS identifiers above are included in the current release.</p>"},{"location":"user_workloads/guest_operating_system_information/#windows-server-2012r2-virtualmachineinstancepreset-example","title":"Windows Server 2012R2 <code>VirtualMachineInstancePreset</code> Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: windows-server-2012r2\n  selector:\n    matchLabels:\n      kubevirt.io/os: win2k12r2\nspec:\n  domain:\n    cpu:\n      cores: 2\n    resources:\n      requests:\n        memory: 2G\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    clock:\n      utc: {}\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/os: win2k12r2\n  name: windows2012r2\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    firmware:\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n    devices:\n      disks:\n      - name: server2012r2\n        disk:\n          dev: vda\n  volumes:\n    - name: server2012r2\n      persistentVolumeClaim:\n        claimName: my-windows-image\n\nOnce the `VirtualMachineInstancePreset` is applied to the\n`VirtualMachineInstance`, the resulting resource would look like this:\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/windows-server-2012r2: kubevirt.io/v1\n  labels:\n    kubevirt.io/os: win2k12r2\n  name: windows2012r2\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    cpu:\n      cores: 2\n    resources:\n      requests:\n        memory: 2G\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    clock:\n      utc: {}\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n    firmware:\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n    devices:\n      disks:\n      - name: server2012r2\n        disk:\n          dev: vda\n  volumes:\n    - name: server2012r2\n      persistentVolumeClaim:\n        claimName: my-windows-image\n</code></pre> <p>For more information see VirtualMachineInstancePresets</p>"},{"location":"user_workloads/guest_operating_system_information/#hyperv-optimizations","title":"HyperV optimizations","text":"<p>KubeVirt supports quite a lot of so-called \"HyperV enlightenments\", which are optimizations for Windows Guests. Some of these optimization may require an up to date host kernel support to work properly, or to deliver the maximum performance gains.</p> <p>KubeVirt can perform extra checks on the hosts before to run Hyper-V enabled VMs, to make sure the host has no known issues with Hyper-V support, properly expose all the required features and thus we can expect optimal performance. These checks are disabled by default for backward compatibility and because they depend on the node-feature-discovery and on extra configuration.</p> <p>To enable strict host checking, the user may expand the <code>featureGates</code> field in the KubeVirt CR by adding the <code>HypervStrictCheck</code> to it.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"HypervStrictCheck\"\n</code></pre> <p>Alternatively, users can edit an existing kubevirt CR:</p> <p><code>kubectl edit kubevirt kubevirt -n kubevirt</code></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"HypervStrictCheck\"\n        - \"CPUManager\"\n</code></pre>"},{"location":"user_workloads/hook-sidecar/","title":"Hook Sidecar Container","text":""},{"location":"user_workloads/hook-sidecar/#introduction","title":"Introduction","text":"<p>In KubeVirt, a Hook Sidecar container is a sidecar container (a secondary container that runs along with the main application container within the same Pod) used to apply customizations before the Virtual Machine is initialized. This ability is provided since configurable elements in the VMI specification do not cover all of the libvirt domain XML elements.</p> <p>The sidecar containers communicate with the main container over a socket with a gRPC protocol. There are two main sidecar hooks:</p> <ol> <li><code>onDefineDomain</code>: This hook helps to customize libvirt's XML and return the new XML over gRPC for the VM creation.</li> <li><code>preCloudInitIso</code>: This hook helps to customize the cloud-init configuration. It operates on and returns JSON    formatted cloud-init data.</li> </ol>"},{"location":"user_workloads/hook-sidecar/#enabling-sidecar-feature-gate","title":"Enabling <code>Sidecar</code> feature gate","text":"<p><code>Sidecar</code> feature gate can be enabled by following the steps mentioned in Activating feature gates.</p> <p>In case of a development cluster created using kubevirtci, follow the steps mentioned in the  developer doc to enable  the feature gate.</p>"},{"location":"user_workloads/hook-sidecar/#sidecar-shim-container-image","title":"Sidecar-shim container image","text":"<p>To run a VM with custom modifications, the sidecar-shim-image  takes care of implementing the communication with the main container.</p> <p>The image contains the <code>sidecar-shim</code> binary built using <code>sidecar_shim.go</code> which should be kept as the entrypoint of the container. This binary will search in <code>$PATH</code> for binaries named after the hook names (e.g <code>onDefineDomain</code> and <code>preCloudInitIso</code>) and run them. Users must provide the necessary arguments as command line options (flags).</p> <p>In the case of <code>onDefineDomain</code>, the arguments will be the VMI information as JSON string, (e.g <code>--vmi vmiJSON</code>) and the current domain XML (e.g <code>--domain domainXML</code>). It outputs the modified domain XML on the standard output.</p> <p>In the case of <code>preCloudInitIso</code>, the arguments will be the VMI information as JSON string, (e.g <code>--vmi vmiJSON</code>) and the CloudInitData (e.g <code>--cloud-init cloudInitJSON</code>). It outputs the modified CloudInitData (as JSON) on the standard ouput.</p> <p>Shell or python scripts can be used as alternatives to the binary, by making them available at the expected location  (<code>/usr/bin/onDefineDomain</code> or <code>/usr/bin/preCloudInitIso</code> depending upon the hook).</p> <p>A prebuilt image named <code>sidecar-shim</code> capable of running Shell or Python scripts is shipped as part of KubeVirt releases.</p>"},{"location":"user_workloads/hook-sidecar/#go-python-shell-pick-any-one","title":"Go, Python, Shell - pick any one","text":"<p>Although a binary doesn't strictly need to be generated from Go code, and a script doesn't strictly need to be one among Shell or Python, for the purpose of this guide, we will use those as examples.</p>"},{"location":"user_workloads/hook-sidecar/#go-binary","title":"Go binary","text":"<p>Example Go code modifiying the SMBIOS system information can be found in the KubeVirt repo. Binary generated from this code, when available under <code>/usr/bin/ondefinedomain</code> in the sidecar-shim-image, is run right before VMI creation and the baseboard manufacturer value is modified to reflect what's provided in the <code>smbios.vm.kubevirt.io/baseBoardManufacturer</code> annotation in VMI spec.</p>"},{"location":"user_workloads/hook-sidecar/#shell-or-python-script","title":"Shell or Python script","text":"<p>If you pefer writing a shell or python script instead of a Go program, create a Kubernetes ConfigMap and use annotations to make sure the script is run before the VMI creation. The flow would be as below:</p> <ol> <li>Create a ConfigMap containing the shell or python script you want to run</li> <li>Create a VMI containing the annotation <code>hooks.kubevirt.io/hookSidecars</code> and mention the ConfigMap information in it.</li> <li>In this case a predefined image can be used to handle the communication with the main container.</li> </ol>"},{"location":"user_workloads/hook-sidecar/#configmap-with-shell-script","title":"ConfigMap with shell script","text":"<pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;baseBoard&gt;&lt;/baseBoard&gt;|&lt;baseBoard&gt;&lt;entry name='manufacturer'&gt;Radical Edward&lt;/entry&gt;&lt;/baseBoard&gt;|\" $tempFile\n    cat $tempFile\n</code></pre>"},{"location":"user_workloads/hook-sidecar/#configmap-with-python-script","title":"ConfigMap with python script","text":"<pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/usr/bin/env python3\n\n    import xml.etree.ElementTree as ET\n    import sys\n\n    def main(s):\n        # write to a temporary file\n        f = open(\"/tmp/orig.xml\", \"w\")\n        f.write(s)\n        f.close()\n\n        # parse xml from file\n        xml = ET.parse(\"/tmp/orig.xml\")\n        # get the root element\n        root = xml.getroot()\n        # find the baseBoard element\n        baseBoard = root.find(\"sysinfo\").find(\"baseBoard\")\n\n        # prepare new element to be inserted into the xml definition\n        element = ET.Element(\"entry\", {\"name\": \"manufacturer\"})\n        element.text = \"Radical Edward\"\n        # insert the element\n        baseBoard.insert(0, element)\n\n        # write to a new file\n        xml.write(\"/tmp/new.xml\")\n        # print file contents to stdout\n        f = open(\"/tmp/new.xml\")\n        print(f.read())\n        f.close()\n\n    if __name__ == \"__main__\":\n        main(sys.argv[4])\n</code></pre> <p>After creating one of the above ConfigMap, create the VMI using the manifest in this example. Of importance here is the ConfigMap information stored in the annotations:</p> <pre><code>annotations:\n  hooks.kubevirt.io/hookSidecars: &gt;\n    [\n        {\n            \"args\": [\"--version\", \"v1alpha2\"],\n            \"configMap\": {\"name\": \"my-config-map\", \"key\": \"my_script.sh\", \"hookPath\": \"/usr/bin/onDefineDomain\"}\n        }\n    ]\n</code></pre> <p>The <code>name</code> field indicates the name of the ConfigMap on the cluster which contains the script you want to execute. The <code>key</code> field indicates the key in the ConfigMap which contains the script to be executed. Finally, <code>hookPath</code> indicates the path where you want the script to be mounted. It could be either of <code>/usr/bin/onDefineDomain</code> or <code>/usr/bin/preCloudInitIso</code> depending upon the hook you want to execute. An optional value can be specified with the <code>\"image\"</code> key if a custom image is needed, if omitted the default Sidecar-shim image built together with the other KubeVirt images will be used. The default Sidecar-shim image, if not override with a custom value, will also be updated as other images as for Updating KubeVirt Workloads.</p>"},{"location":"user_workloads/hook-sidecar/#verify-everything-works","title":"Verify everything works","text":"<p>Whether you used the Go binary or a Shell/Python script from the above examples, you would be able to see the newly  created VMI have the modified baseboard manufacturer information. After creating the VMI, verify that it is in the  <code>Running</code> state, and connect to its console and see if the desired changes to baseboard manufacturer get reflected:</p> <pre><code># Once the VM is ready, connect to its display and login using name and password \"fedora\"\ncluster/virtctl.sh vnc vmi-with-sidecar-hook-configmap\n\n# Check whether the base board manufacturer value was successfully overwritten\nsudo dmidecode -s baseboard-manufacturer\n</code></pre>"},{"location":"user_workloads/instancetypes/","title":"Instance types and preferences","text":"<p>FEATURE STATE: </p> <ul> <li><code>instancetype.kubevirt.io/v1alpha1</code> (Experimental) as of the <code>v0.56.0</code> KubeVirt release</li> <li><code>instancetype.kubevirt.io/v1alpha2</code> (Experimental) as of the <code>v0.58.0</code> KubeVirt release</li> <li><code>instancetype.kubevirt.io/v1beta1</code> as of the <code>v1.0.0</code> KubeVirt release</li> </ul> <p>See the Version History section for more details.</p>"},{"location":"user_workloads/instancetypes/#introduction","title":"Introduction","text":"<p>KubeVirt's <code>VirtualMachine</code> API contains many advanced options for tuning the performance of a VM that goes beyond what typical users need to be aware of. Users have previously been unable to simply define the storage/network they want assigned to their VM and then declare in broad terms what quality of resources and kind of performance characteristics they need for their VM.</p> <p>Instance types and preferences provide a way to define a set of resource, performance and other runtime characteristics, allowing users to reuse these definitions across multiple <code>VirtualMachines</code>.</p>"},{"location":"user_workloads/instancetypes/#virtualmachineinstancetype","title":"VirtualMachineInstancetype","text":"<pre><code>---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachineInstancetype\nmetadata:\n  name: example-instancetype\nspec:\n  cpu:\n    guest: 1\n  memory:\n    guest: 128Mi\n</code></pre> <p>KubeVirt provides two <code>CRDs</code> for instance types, a cluster wide <code>VirtualMachineClusterInstancetype</code> and a namespaced <code>VirtualMachineInstancetype</code>. These <code>CRDs</code> encapsulate the following resource related characteristics of a <code>VirtualMachine</code> through a shared <code>VirtualMachineInstancetypeSpec</code>:</p> <ul> <li><code>CPU</code> : Required number of vCPUs presented to the guest</li> <li><code>Memory</code> : Required amount of memory presented to the guest</li> <li><code>GPUs</code> : Optional list of vGPUs to passthrough</li> <li><code>HostDevices</code> : Optional list of <code>HostDevices</code> to passthrough</li> <li><code>IOThreadsPolicy</code> : Optional <code>IOThreadsPolicy</code> to be used</li> <li><code>LaunchSecurity</code>: Optional <code>LaunchSecurity</code> to be used</li> </ul> <p>Anything provided within an instance type cannot be overridden within the <code>VirtualMachine</code>. For example, as <code>CPU</code> and <code>Memory</code> are both required attributes of an instance type, if a user makes any requests for <code>CPU</code> or <code>Memory</code> resources within the underlying <code>VirtualMachine</code>, the instance type will conflict and the request will be rejected during creation.</p>"},{"location":"user_workloads/instancetypes/#virtualmachinepreference","title":"VirtualMachinePreference","text":"<pre><code>---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: example-preference\nspec:\n  devices:\n    preferredDiskBus: virtio\n    preferredInterfaceModel: virtio\n</code></pre> <p>KubeVirt also provides two further preference based <code>CRDs</code>, again a cluster wide <code>VirtualMachineClusterPreference</code> and namespaced <code>VirtualMachinePreference</code>. These <code>CRDs</code>encapsulate the preferred value of any remaining attributes of a <code>VirtualMachine</code> required to run a given workload, again this is through a shared <code>VirtualMachinePreferenceSpec</code>.</p> <p>Unlike instance types, preferences only represent the preferred values and as such, they can be overridden by values in the <code>VirtualMachine</code> provided by the user.</p> <p>In the example shown below, a user has provided a <code>VirtualMachine</code> with a disk bus already defined within a <code>DiskTarget</code> and has also selected a set of preferences with <code>DevicePreference</code> and <code>preferredDiskBus</code> , so the user's original choice within the <code>VirtualMachine</code> and <code>DiskTarget</code> are used:</p> <pre><code>$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: example-preference-disk-virtio\nspec:\n  devices:\n    preferredDiskBus: virtio\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: example-preference-user-override\nspec:\n  preference:\n    kind: VirtualMachinePreference\n    name: example-preference-disk-virtio\n  running: false\n  template:\n    spec:\n      domain:\n        memory:\n          guest: 128Mi\n        devices:\n          disks:\n          - disk:\n              bus: sata\n            name: containerdisk\n          - disk: {}\n            name: cloudinitdisk\n        resources: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |\n            #!/bin/sh\n\n            echo 'printed from cloud-init userdata'\n        name: cloudinitdisk\nEOF\nvirtualmachinepreference.instancetype.kubevirt.io/example-preference-disk-virtio created\nvirtualmachine.kubevirt.io/example-preference-user-override configured\n\n\n$ virtctl start example-preference-user-override\nVM example-preference-user-override was scheduled to start\n\n# We can see the original request from the user within the VirtualMachine lists `containerdisk` with a `SATA` bus\n$ kubectl get vms/example-preference-user-override -o json | jq .spec.template.spec.domain.devices.disks\n[\n  {\n    \"disk\": {\n      \"bus\": \"sata\"\n    },\n    \"name\": \"containerdisk\"\n  },\n  {\n    \"disk\": {},\n    \"name\": \"cloudinitdisk\"\n  }\n]\n\n# This is still the case in the VirtualMachineInstance with the remaining disk using the `preferredDiskBus` from the preference of `virtio`\n$ kubectl get vmis/example-preference-user-override -o json | jq .spec.domain.devices.disks\n[\n  {\n    \"disk\": {\n      \"bus\": \"sata\"\n    },\n    \"name\": \"containerdisk\"\n  },\n  {\n    \"disk\": {\n      \"bus\": \"virtio\"\n    },\n    \"name\": \"cloudinitdisk\"\n  }\n]\n</code></pre>"},{"location":"user_workloads/instancetypes/#virtualmachine","title":"VirtualMachine","text":"<pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: example-vm\nspec:\n  instancetype:\n    kind: VirtualMachineInstancetype\n    name: example-instancetype\n  preference:\n    kind: VirtualMachinePreference\n    name: example-preference\n</code></pre> <p>The previous instance type and preference CRDs are matched to a given <code>VirtualMachine</code> through the use of a matcher. Each matcher consists of the following:</p> <ul> <li><code>Name</code> (string): Name of the resource being referenced</li> <li><code>Kind</code> (string):  Optional, defaults to the cluster wide CRD kinds of <code>VirtualMachineClusterInstancetype</code> or <code>VirtualMachineClusterPreference</code> if not provided</li> <li><code>RevisionName</code> (string) : Optional, name of a <code>ControllerRevision</code> containing a copy of the <code>VirtualMachineInstancetypeSpec</code> or <code>VirtualMachinePreferenceSpec</code> taken when the <code>VirtualMachine</code> is first created. See the Versioning section below for more details on how and why this is captured.</li> <li><code>InferFromVolume</code> (string): Optional, see the Inferring defaults from a Volume section below for more details.</li> </ul>"},{"location":"user_workloads/instancetypes/#creating-instancetypes-preferences-and-virtualmachines","title":"Creating InstanceTypes, Preferences and VirtualMachines","text":"<p>It is possible to streamline the creation of instance types, preferences, and virtual machines with the usage of the virtctl command-line tool. To read more about it, please see the Creating VirtualMachines.</p>"},{"location":"user_workloads/instancetypes/#versioning","title":"Versioning","text":"<p>Versioning of these resources is required to ensure the eventual <code>VirtualMachineInstance</code> created when starting a <code>VirtualMachine</code> does not change between restarts if any referenced instance type or set of preferences are updated during the lifetime of the <code>VirtualMachine</code>.</p> <p>This is currently achieved by using <code>ControllerRevision</code> to retain a copy of the <code>VirtualMachineInstancetype</code> or <code>VirtualMachinePreference</code> at the time the <code>VirtualMachine</code> is created. A reference to these <code>ControllerRevisions</code> are then retained in the <code>InstancetypeMatcher</code> and <code>PreferenceMatcher</code> within the <code>VirtualMachine</code> for future use.</p> <pre><code>$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall created\nvirtualmachine.kubevirt.io/vm-cirros-csmall created\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\"\n}\n\n$ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .\n{\n  \"apiVersion\": \"apps/v1\",\n  \"data\": {\n    \"apiVersion\": \"instancetype.kubevirt.io/v1beta1\",\n    \"kind\": \"VirtualMachineInstancetype\",\n    \"metadata\": {\n      \"creationTimestamp\": \"2022-09-30T12:20:19Z\",\n      \"generation\": 1,\n      \"name\": \"csmall\",\n      \"namespace\": \"default\",\n      \"resourceVersion\": \"10303\",\n      \"uid\": \"72c3a35b-6e18-487d-bebf-f73c7d4f4a40\"\n    },\n    \"spec\": {\n      \"cpu\": {\n        \"guest\": 1\n      },\n      \"memory\": {\n        \"guest\": \"128Mi\"\n      }\n    }\n  },\n  \"kind\": \"ControllerRevision\",\n  \"metadata\": {\n    \"creationTimestamp\": \"2022-09-30T12:20:19Z\",\n    \"name\": \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\",\n    \"namespace\": \"default\",\n    \"ownerReferences\": [\n      {\n        \"apiVersion\": \"kubevirt.io/v1\",\n        \"blockOwnerDeletion\": true,\n        \"controller\": true,\n        \"kind\": \"VirtualMachine\",\n        \"name\": \"vm-cirros-csmall\",\n        \"uid\": \"5216527a-1d31-4637-ad3a-b640cb9949a2\"\n      }\n    ],\n    \"resourceVersion\": \"10307\",\n    \"uid\": \"a7bc784b-4cea-45d7-8432-15418e1dd7d3\"\n  },\n  \"revision\": 0\n}\n\n\n$ kubectl delete vm/vm-cirros-csmall\nvirtualmachine.kubevirt.io \"vm-cirros-csmall\" deleted\n\n$ kubectl get controllerrevision/controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\nError from server (NotFound): controllerrevisions.apps \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\" not found\n</code></pre> <p>Users can opt in to moving to a newer generation of an instance type or preference by removing the referenced <code>revisionName</code> from the appropriate matcher within the <code>VirtualMachine</code> object. This will result in fresh <code>ControllerRevisions</code> being captured and used.</p> <p>The following example creates a <code>VirtualMachine</code> using an initial version of the csmall instance type before increasing the number of vCPUs provided by the instance type:</p> <pre><code>$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall created\nvirtualmachine.kubevirt.io/vm-cirros-csmall created\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-1\"\n}\n\n$ virtctl start vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to start\n\n$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu\n{\n  \"cores\": 1,\n  \"model\": \"host-model\",\n  \"sockets\": 1,\n  \"threads\": 1\n}\n\n$ kubectl patch VirtualMachineInstancetype/csmall --type merge -p '{\"spec\":{\"cpu\":{\"guest\":2}}}'\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall patched\n</code></pre> <p>In order for this change to be picked up within the <code>VirtualMachine</code>, we need to stop the running <code>VirtualMachine</code> and clear the <code>revisionName</code> referenced by the <code>InstancetypeMatcher</code>:</p> <pre><code>$ virtctl stop vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to stop\n\n$ kubectl patch vm/vm-cirros-csmall --type merge -p '{\"spec\":{\"instancetype\":{\"revisionName\":\"\"}}}'\nvirtualmachine.kubevirt.io/vm-cirros-csmall patched\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-2\"\n}\n</code></pre> <p>As you can see above, the <code>InstancetypeMatcher</code> now references a new <code>ControllerRevision</code> containing generation 2 of the instance type. We can now start the <code>VirtualMachine</code> again and see the new number of vCPUs being used by the <code>VirtualMachineInstance</code>:</p> <pre><code>$ virtctl start vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to start\n\n$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu\n{\n  \"cores\": 1,\n  \"model\": \"host-model\",\n  \"sockets\": 2,\n  \"threads\": 1\n}\n</code></pre>"},{"location":"user_workloads/instancetypes/#inferfromvolume","title":"inferFromVolume","text":"<p>The <code>inferFromVolume</code> attribute of both the <code>InstancetypeMatcher</code> and <code>PreferenceMatcher</code> allows a user to request that defaults are inferred from a volume. When requested, KubeVirt will look for the following labels on the underlying <code>PVC</code>, <code>DataSource</code> or <code>DataVolume</code> to determine the default name and kind:</p> <ul> <li><code>instancetype.kubevirt.io/default-instancetype</code></li> <li><code>instancetype.kubevirt.io/default-instancetype-kind</code> (optional, defaults to <code>VirtualMachineClusterInstancetype</code>)</li> <li><code>instancetype.kubevirt.io/default-preference</code></li> <li><code>instancetype.kubevirt.io/default-preference-kind</code> (optional, defaults to <code>VirtualMachineClusterPreference</code>)</li> </ul> <p>These values are then written into the appropriate matcher by the mutation webhook and used during validation before the <code>VirtualMachine</code> is formally accepted.</p> <p>The validation can be controlled by the value provided to <code>inferFromVolumeFailurePolicy</code> in either the <code>InstancetypeMatcher</code> or <code>PreferenceMatcher</code> of a <code>VirtualMachine</code>.</p> <p>The default value of <code>Reject</code> will cause the request to be rejected on failure to find the referenced <code>Volume</code> or labels on an underlying resource.</p> <p>If <code>Ignore</code> was provided, the respective <code>InstancetypeMatcher</code> or <code>PreferenceMatcher</code> will be cleared on a failure instead.</p> <p>Example with implicit default value of <code>Reject</code>:</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git\n[..]\n$ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img\n[..]\n$ kubectl label pvc/cirros-pvc \\\n  instancetype.kubevirt.io/default-instancetype=server.tiny \\\n  instancetype.kubevirt.io/default-preference=cirros\n[..]\n$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataSource\nmetadata:\n  name: cirros-datasource\nspec:\n  source:\n    pvc:\n      name: cirros-pvc\n      namespace: default\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: cirros\nspec:\n  instancetype:\n    inferFromVolume: cirros-volume\n  preference:\n    inferFromVolume: cirros-volume\n  running: false\n  dataVolumeTemplates:\n    - metadata:\n        name: cirros-datavolume\n      spec:\n        storage:\n          resources:\n            requests:\n              storage: 1Gi\n          storageClassName: local\n        sourceRef:\n          kind: DataSource\n          name: cirros-datasource\n          namespace: default\n  template:\n    spec:\n      domain:\n        devices: {}\n      volumes:\n        - dataVolume:\n            name: cirros-datavolume\n          name: cirros-volume\nEOF\n[..]\nkubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'\n{\n  \"kind\": \"virtualmachineclusterinstancetype\",\n  \"name\": \"server.tiny\",\n  \"revisionName\": \"cirros-server.tiny-76454433-3d82-43df-a7e5-586e48c71f68-1\"\n}\n{\n  \"kind\": \"virtualmachineclusterpreference\",\n  \"name\": \"cirros\",\n  \"revisionName\": \"cirros-cirros-85823ddc-9e8c-4d23-a94c-143571b5489c-1\"\n}\n</code></pre> <p>Example with explicit value of <code>Ignore</code>:</p> <pre><code>$ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img\n$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataSource\nmetadata:\n  name: cirros-datasource\nspec:\n  source:\n    pvc:\n      name: cirros-pvc\n      namespace: default\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: cirros\nspec:\n  instancetype:\n    inferFromVolume: cirros-volume\n    inferFromVolumeFailurePolicy: Ignore\n  preference:\n    inferFromVolume: cirros-volume\n    inferFromVolumeFailurePolicy: Ignore\n  running: false\n  dataVolumeTemplates:\n    - metadata:\n        name: cirros-datavolume\n      spec:\n        storage:\n          accessModes:\n            - ReadWriteOnce\n          resources:\n            requests:\n              storage: 1Gi\n          storageClassName: local\n        sourceRef:\n          kind: DataSource\n          name: cirros-datasource\n          namespace: default\n  template:\n    spec:\n      domain:\n        devices: {}\n      volumes:\n        - dataVolume:\n            name: cirros-datavolume\n          name: cirros-volume\nEOF\n[..]\nkubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'\nnull\nnull\n</code></pre>"},{"location":"user_workloads/instancetypes/#common-instancetypes","title":"common-instancetypes","text":"<p>The <code>kubevirt/common-instancetypes</code> provide a set of instancetypes and preferences to help create KubeVirt <code>VirtualMachines</code>.</p> <p>See Deploy common-instancetypes on how to deploy them.</p>"},{"location":"user_workloads/instancetypes/#examples","title":"Examples","text":"<p>Various examples are available within the <code>kubevirt</code> repo under <code>/examples</code>. The following uses an example <code>VirtualMachine</code> provided by the <code>containerdisk/fedora</code> repo and replaces much of the <code>DomainSpec</code> with the equivalent instance type and preferences:</p> <pre><code>$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachineInstancetype\nmetadata:\n  name: cmedium\nspec:\n  cpu:\n    guest: 1\n  memory:\n    guest: 1Gi\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: fedora\nspec:\n  devices:\n    preferredDiskBus: virtio\n    preferredInterfaceModel: virtio\n    preferredRng: {}\n  features:\n    preferredAcpi: {}\n    preferredSmm: {}\n  firmware:\n    preferredUseEfi: true\n    preferredUseSecureBoot: true    \n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  creationTimestamp: null\n  name: fedora\nspec:\n  instancetype:\n    name: cmedium\n    kind: virtualMachineInstancetype\n  preference:\n    name: fedora\n    kind: virtualMachinePreference\n  runStrategy: Always\n  template:\n    metadata:\n      creationTimestamp: null\n    spec:\n      domain:\n        devices: {}\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            ssh_authorized_keys:\n              - ssh-rsa AAAA...\n        name: cloudinit\nEOF\n</code></pre>"},{"location":"user_workloads/instancetypes/#version-history","title":"Version History","text":""},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1alpha1-experimental","title":"<code>instancetype.kubevirt.io/v1alpha1</code> (Experimental)","text":"<ul> <li>Initial development version.</li> </ul>"},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1alpha2-experimental","title":"<code>instancetype.kubevirt.io/v1alpha2</code> (Experimental)","text":"<ul> <li> <p>This version captured complete <code>VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}</code> objects within the created <code>ControllerRevisions</code></p> </li> <li> <p>This version is backwardly compatible with <code>instancetype.kubevirt.io/v1alpha1</code>.</p> </li> </ul>"},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1beta1","title":"<code>instancetype.kubevirt.io/v1beta1</code>","text":"<ul> <li>The following instance type attribute has been added:</li> <li> <p><code>Spec.Memory.OvercommitPercent</code></p> </li> <li> <p>The following preference attributes have been added:</p> </li> <li><code>Spec.CPU.PreferredCPUFeatures</code></li> <li><code>Spec.Devices.PreferredInterfaceMasquerade</code></li> <li><code>Spec.PreferredSubdomain</code></li> <li><code>Spec.PreferredTerminationGracePeriodSeconds</code></li> <li> <p><code>Spec.Requirements</code></p> </li> <li> <p>This version is backwardly compatible with <code>instancetype.kubevirt.io/v1alpha1</code> and <code>instancetype.kubevirt.io/v1alpha2</code> objects, no modifications are required to existing  <code>VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}</code> or <code>ControllerRevisions</code>.</p> </li> <li> <p>As with the migration to <code>kubevirt.io/v1</code> it is recommend previous users of <code>instancetype.kubevirt.io/v1alpha1</code> or <code>instancetype.kubevirt.io/v1alpha2</code> use <code>kube-storage-version-migrator</code> to upgrade any stored objects to <code>instancetype.kubevirt.io/v1beta1</code>.</p> </li> </ul>"},{"location":"user_workloads/lifecycle/","title":"Lifecycle","text":"<p>Every <code>VirtualMachineInstance</code> represents a single virtual machine instance. In general, the management of VirtualMachineInstances is kept similar to how <code>Pods</code> are managed: Every VM that is defined in the cluster is expected to be running, just like Pods. Deleting a VirtualMachineInstance is equivalent to shutting it down, this is also equivalent to how Pods behave.</p>"},{"location":"user_workloads/lifecycle/#launching-a-virtual-machine","title":"Launching a virtual machine","text":"<p>In order to start a VirtualMachineInstance, you just need to create a <code>VirtualMachineInstance</code> object using <code>kubectl</code>:</p> <pre><code>$ kubectl create -f vmi.yaml\n</code></pre>"},{"location":"user_workloads/lifecycle/#listing-virtual-machines","title":"Listing virtual machines","text":"<p>VirtualMachineInstances can be listed by querying for VirtualMachineInstance objects:</p> <pre><code>$ kubectl get vmis\n</code></pre>"},{"location":"user_workloads/lifecycle/#retrieving-a-virtual-machine-instance-definition","title":"Retrieving a virtual machine instance definition","text":"<p>A single VirtualMachineInstance definition can be retrieved by getting the specific VirtualMachineInstance object:</p> <pre><code>$ kubectl get vmis testvmi\n</code></pre>"},{"location":"user_workloads/lifecycle/#stopping-a-virtual-machine-instance","title":"Stopping a virtual machine instance","text":"<p>To stop the VirtualMachineInstance, you just need to delete the corresponding <code>VirtualMachineInstance</code> object using <code>kubectl</code>.</p> <pre><code>$ kubectl delete -f vmi.yaml\n# OR\n$ kubectl delete vmis testvmi\n</code></pre> <p>Note: Stopping a VirtualMachineInstance implies that it will be deleted from the cluster. You will not be able to start this VirtualMachineInstance object again.</p>"},{"location":"user_workloads/lifecycle/#starting-and-stopping-a-virtual-machine","title":"Starting and stopping a virtual machine","text":"<p>Virtual machines, in contrast to VirtualMachineInstances, have a running state. Thus on VM you can define if it should be running, or not. VirtualMachineInstances are, if they are defined in the cluster, always running and consuming resources.</p> <p><code>virtctl</code> is used in order to start and stop a VirtualMachine:</p> <pre><code>$ virtctl start my-vm\n$ virtctl stop my-vm\n</code></pre> <p>Note: You can force stop a VM (which is like pulling the power cord, with all its implications like data inconsistencies or [in the worst case] data loss) by</p> <pre><code>$ virtctl stop my-vm --grace-period 0 --force\n</code></pre>"},{"location":"user_workloads/lifecycle/#pausing-and-unpausing-a-virtual-machine","title":"Pausing and unpausing a virtual machine","text":"<p>Note: Pausing in this context refers to libvirt's <code>virDomainSuspend</code> command: \"The process is frozen without further access to CPU resources and I/O but the memory used by the domain at the hypervisor level will stay allocated\"</p> <p>To pause a virtual machine, you need the <code>virtctl</code> command line tool. Its <code>pause</code> command works on either <code>VirtualMachine</code> s or <code>VirtualMachinesInstance</code> s:</p> <pre><code>$ virtctl pause vm testvm\n# OR\n$ virtctl pause vmi testvm\n</code></pre> <p>Paused VMIs have a <code>Paused</code> condition in their status:</p> <pre><code>$ kubectl get vmi testvm -o=jsonpath='{.status.conditions[?(@.type==\"Paused\")].message}'\nVMI was paused by user\n</code></pre> <p>Unpausing works similar to pausing:</p> <pre><code>$ virtctl unpause vm testvm\n# OR\n$ virtctl unpause vmi testvm\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/","title":"Liveness and Readiness Probes","text":"<p>It is possible to configure Liveness and Readiness Probes in a similar fashion like it is possible to configure Liveness and Readiness Probes on Containers.</p> <p>Liveness Probes will effectively stop the VirtualMachineInstance if they fail, which will allow higher level controllers, like VirtualMachine or VirtualMachineInstanceReplicaSet to spawn new instances, which will hopefully be responsive again.</p> <p>Readiness Probes are an indicator for Services and Endpoints if the VirtualMachineInstance is ready to receive traffic from Services. If Readiness Probes fail, the VirtualMachineInstance will be removed from the Endpoints which back services until the probe recovers.</p> <p>Watchdogs focus on ensuring that an Operating System is still responsive. They complement the probes which are more workload centric. Watchdogs require kernel support from the guest and additional tooling like the commonly used <code>watchdog</code> binary.</p> <p>Exec probes are Liveness or Readiness probes specifically intended for VMs. These probes run a command inside the VM and determine the VM ready/live state based on its success. For running commands inside the VMs, the qemu-guest-agent package is used. A command supplied to an exec probe will be wrapped by <code>virt-probe</code> in the  operator and forwarded to the guest.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-a-http-liveness-probe","title":"Define a HTTP Liveness Probe","text":"<p>The following VirtualMachineInstance configures a HTTP Liveness Probe via <code>spec.livenessProbe.httpGet</code>, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  livenessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    httpGet:\n      port: 1500\n    timeoutSeconds: 10\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-a-tcp-liveness-probe","title":"Define a TCP Liveness Probe","text":"<p>The following VirtualMachineInstance configures a TCP Liveness Probe via <code>spec.livenessProbe.tcpSocket</code>, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  livenessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    tcpSocket:\n      port: 1500\n    timeoutSeconds: 10\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-readiness-probes","title":"Define Readiness Probes","text":"<p>Readiness Probes are configured in a similar way like liveness probes. Instead of <code>spec.livenessProbe</code>, <code>spec.readinessProbe</code> needs to be filled:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  readinessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    timeoutSeconds: 10\n    failureThreshold: 3\n    successThreshold: 3\n    httpGet:\n      port: 1500\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre> <p>Note that in the case of Readiness Probes, it is also possible to set a <code>failureThreshold</code> and a <code>successThreashold</code> to only flip between ready and non-ready state if the probe succeeded or failed multiple times.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#dual-stack-considerations","title":"Dual-stack considerations","text":"<p>Some context is needed to understand the limitations imposed by a dual-stack network configuration on readiness - or liveness - probes. Users must be fully aware that a dual-stack configuration is currently only available when using a masquerade binding type. Furthermore, it must be recalled that accessing a VM using masquerade binding type is performed via the pod IP address; in dual-stack mode, both IPv4 and IPv6 addresses can be used to reach the VM.</p> <p>Dual-stack networking configurations have a limitation when using HTTP / TCP probes - you cannot probe the VMI by its IPv6 address. The reason for this is the <code>host</code> field for both the HTTP and TCP probe actions default to the pod's IP address, which is currently always the IPv4 address.</p> <p>Since the pod's IP address is not known before creating the VMI, it is not possible to pre-provision the probe's host field.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#defining-a-watchdog","title":"Defining a Watchdog","text":"<p>A watchdog is a more VM centric approach where the responsiveness of the Operating System is focused on. One can configure the <code>i6300esb</code> watchdog device:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-with-watchdog\n  name: vmi-with-watchdog\nspec:\n  domain:\n    devices:\n      watchdog:\n        name: mywatchdog\n        i6300esb:\n          action: \"poweroff\"\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"busybox\"]\n    name: cloudinitdisk\n</code></pre> <p>The example above configures it with the <code>poweroff</code> action. It defines what will happen if the OS can't respond anymore. Other possible actions are <code>reset</code> and <code>shutdown</code>. The VM in this example will have the device exposed as <code>/dev/watchdog</code>. This device can then be used by the <code>watchdog</code> binary. For example, if root executes this command inside the VM:</p> <pre><code>sudo busybox watchdog -t 2000ms -T 4000ms /dev/watchdog\n</code></pre> <p>the watchdog will send a heartbeat every two seconds to <code>/dev/watchdog</code> and after four seconds without a heartbeat the defined action will be executed. In this case a hard <code>poweroff</code>.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#defining-guest-agent-ping-probes","title":"Defining Guest-Agent Ping Probes","text":"<p>Guest-Agent probes are based on qemu-guest-agent <code>guest-ping</code>.  This will ping the guest and return an error if the guest is not up and running.  To easily define this on VM spec, specify <code>guestAgentPing: {}</code> in VM's  <code>spec.template.spec.readinessProbe</code>.  <code>virt-controller</code> will translate this  into a corresponding command wrapped by <code>virt-probe</code>.</p> <p>Note: You can only define one of the type of probe, i.e. guest-agent exec  or ping probes.</p> <p>Important: If the qemu-guest-agent is not installed and enabled inside the VM, the probe will fail.  Many images don't enable the agent by default so make sure you either run one that does or enable it. </p> <p>Make sure to provide enough delay and failureThreshold for the VM and the agent to be online.</p> <p>In the following example the Fedora image does have qemu-guest-agent available by default. Nevertheless, in case qemu-guest-agent is not installed, it will be installed and enabled via cloud-init as shown in the example below.  Also, cloud-init assigns the proper SELinux context, i.e. virt_qemu_ga_exec_t, to the <code>/tmp/healthy.txt</code> file.  Otherwise, SELinux will deny the attempts to open the <code>/tmp/healthy.txt</code> file causing the probe to fail.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-guest-probe-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-guest-probe\n        kubevirt.io/vm: vmi-guest-probe\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  readinessProbe:\n    exec:\n      command: [\"cat\", \"/tmp/healthy.txt\"]\n    failureThreshold: 10\n    initialDelaySeconds: 20\n    periodSeconds: 10\n    timeoutSeconds: 5\n  terminationGracePeriodSeconds: 180\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        packages:\n          qemu-guest-agent\n        runcmd:\n          - [\"touch\", \"/tmp/healthy.txt\"]\n          - [\"sudo\", \"chcon\", \"-t\", \"virt_qemu_ga_exec_t\", \"/tmp/healthy.txt\"]\n          - [\"sudo\", \"systemctl\", \"enable\", \"--now\", \"qemu-guest-agent\"]\n    name: cloudinitdisk\n</code></pre> <p>Note that, in the above example if SELinux is not installed in your container disk image, the command <code>chcon</code> should be removed from the VM manifest shown below. Otherwise, the <code>chcon</code>  command will fail.</p> <p>The <code>.status.ready</code> field will switch to <code>true</code> indicating that probes are returning successfully:</p> <pre><code>kubectl wait vmis/vmi-guest-probe --for=condition=Ready --timeout=5m\n</code></pre> <p>Additionally, the following command can be used inside the VM to watch the incoming qemu-ga commands:</p> <pre><code>journalctl _COMM=qemu-ga --follow \n</code></pre>"},{"location":"user_workloads/pool/","title":"VirtualMachinePool","text":"<p>A VirtualMachinePool tries to ensure that a specified number of VirtualMachine replicas and their respective VirtualMachineInstances are in the ready state at any time. In other words, a VirtualMachinePool makes sure that a VirtualMachine or a set of VirtualMachines is always up and ready. </p> <p>No state is kept and no guarantees are made about the maximum number of VirtualMachineInstance replicas running at any time. For example, the VirtualMachinePool may decide to create new replicas if possibly still running VMs are entering an unknown state.</p>"},{"location":"user_workloads/pool/#using-virtualmachinepool","title":"Using VirtualMachinePool","text":"<p>The VirtualMachinePool allows us to specify a VirtualMachineTemplate in <code>spec.virtualMachineTemplate</code>. It consists of <code>ObjectMetadata</code> in <code>spec.virtualMachineTemplate.metadata</code>, and a <code>VirtualMachineSpec</code> in <code>spec.virtualMachineTemplate.spec</code>. The specification of the virtual machine is equal to the specification of the virtual machine in the <code>VirtualMachine</code> workload.</p> <p><code>spec.replicas</code> can be used to specify how many replicas are wanted. If unspecified, the default value is 1. This value can be updated anytime. The controller will react to the changes.</p> <p><code>spec.selector</code> is used by the controller to keep track of managed virtual machines. The selector specified there must be able to match the virtual machine labels as specified in <code>spec.virtualMachineTemplate.metadata.labels</code>. If the selector does not match these labels, or they are empty, the controller will simply do nothing except log an error. The user is responsible for avoiding the creation of other virtual machines or VirtualMachinePools which may conflict with the selector and the template labels.</p>"},{"location":"user_workloads/pool/#creating-a-virtualmachinepool","title":"Creating a VirtualMachinePool","text":"<p>VirtualMachinePool is part of the Kubevirt API <code>pool.kubevirt.io/v1alpha1</code>.</p> <p>The example below shows how to create a simple <code>VirtualMachinePool</code>:</p>"},{"location":"user_workloads/pool/#example","title":"Example","text":"<pre><code>apiVersion: pool.kubevirt.io/v1alpha1\nkind: VirtualMachinePool\nmetadata:\n  name: vm-pool-cirros\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      kubevirt.io/vmpool: vm-pool-cirros\n  virtualMachineTemplate:\n    metadata:\n      creationTimestamp: null\n      labels:\n        kubevirt.io/vmpool: vm-pool-cirros\n    spec:\n      running: true\n      template:\n        metadata:\n          creationTimestamp: null\n          labels:\n            kubevirt.io/vmpool: vm-pool-cirros\n        spec:\n          domain:\n            devices:\n              disks:\n              - disk:\n                  bus: virtio\n                name: containerdisk\n            resources:\n              requests:\n                memory: 128Mi\n          terminationGracePeriodSeconds: 0\n          volumes:\n          - containerDisk:\n              image: kubevirt/cirros-container-disk-demo:latest\n            name: containerdisk \n</code></pre> <p>Saving this manifest into <code>vm-pool-cirros.yaml</code> and submitting it to Kubernetes will create three virtual machines based on the template.</p> <pre><code>$ kubectl create -f vm-pool-cirros.yaml\nvirtualmachinepool.pool.kubevirt.io/vm-pool-cirros created\n$ kubectl describe vmpool vm-pool-cirros\nName:         vm-pool-cirros\nNamespace:    default\nLabels:       &lt;none&gt;\nAnnotations:  &lt;none&gt;\nAPI Version:  pool.kubevirt.io/v1alpha1\nKind:         VirtualMachinePool\nMetadata:\n  Creation Timestamp:  2023-02-09T18:30:08Z\n  Generation:          1\n    Manager:      kubectl-create\n    Operation:    Update\n    Time:         2023-02-09T18:30:08Z\n    API Version:  pool.kubevirt.io/v1alpha1\n    Fields Type:  FieldsV1\n    fieldsV1:\n      f:status:\n        .:\n        f:labelSelector:\n        f:readyReplicas:\n        f:replicas:\n    Manager:         virt-controller\n    Operation:       Update\n    Subresource:     status\n    Time:            2023-02-09T18:30:44Z\n  Resource Version:  6606\n  UID:               ba51daf4-f99f-433c-89e5-93f39bc9989d\nSpec:\n  Replicas:  3\n  Selector:\n    Match Labels:\n      kubevirt.io/vmpool:  vm-pool-cirros\n  Virtual Machine Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        kubevirt.io/vmpool:  vm-pool-cirros\n    Spec:\n      Running:  true\n      Template:\n        Metadata:\n          Creation Timestamp:  &lt;nil&gt;\n          Labels:\n            kubevirt.io/vmpool:  vm-pool-cirros\n        Spec:\n          Domain:\n            Devices:\n              Disks:\n                Disk:\n                  Bus:  virtio\n                Name:   containerdisk\n            Resources:\n              Requests:\n                Memory:                      128Mi\n          Termination Grace Period Seconds:  0\n          Volumes:\n            Container Disk:\n              Image:  kubevirt/cirros-container-disk-demo:latest\n            Name:     containerdisk\nStatus:\n  Label Selector:  kubevirt.io/vmpool=vm-pool-cirros\n  Ready Replicas:  2\n  Replicas:        3\nEvents:\n  Type    Reason            Age   From                           Message\n  ----    ------            ----  ----                           -------\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-0\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-2\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-1\n</code></pre> <p><code>Replicas</code> is <code>3</code> and <code>Ready Replicas</code> is <code>2</code>. This means that at the moment when showing the status, three Virtual Machines were already created, but only two are running and ready.</p>"},{"location":"user_workloads/pool/#scaling-via-the-scale-subresource","title":"Scaling via the Scale Subresource","text":"<p>Note: This requires KubeVirt 0.59 or newer.</p> <p>The <code>VirtualMachinePool</code> supports the <code>scale</code> subresource. As a consequence it is possible to scale it via <code>kubectl</code>:</p> <pre><code>$ kubectl scale vmpool vm-pool-cirros --replicas 5\n</code></pre>"},{"location":"user_workloads/pool/#removing-a-virtualmachine-from-virtualmachinepool","title":"Removing a VirtualMachine from VirtualMachinePool","text":"<p>It is also possible to remove a <code>VirtualMachine</code> from its <code>VirtualMachinePool</code>.</p> <p>In this scenario, the <code>ownerReferences</code> needs to be removed from the <code>VirtualMachine</code>. This can be achieved either by using <code>kubectl edit</code> or <code>kubectl patch</code>. Using <code>kubectl patch</code> it would look like:</p> <pre><code>kubectl patch vm vm-pool-cirros-0 --type merge --patch '{\"metadata\":{\"ownerReferences\":null}}'\n</code></pre> <p>Note: You may want to update your VirtualMachine labels as well to avoid impact on selectors.</p>"},{"location":"user_workloads/pool/#using-the-horizontal-pod-autoscaler","title":"Using the Horizontal Pod Autoscaler","text":"<p>Note: This requires KubeVirt 0.59 or newer.</p> <p>The HorizontalPodAutoscaler (HPA) can be used with a <code>VirtualMachinePool</code>. Simply reference it in the spec of the autoscaler:</p> <pre><code>apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n  creationTimestamp: null\n  name: vm-pool-cirros\nspec:\n  maxReplicas: 10\n  minReplicas: 3\n  scaleTargetRef:\n    apiVersion: pool.kubevirt.io/v1alpha1\n    kind: VirtualMachinePool\n    name: vm-pool-cirros\n  targetCPUUtilizationPercentage: 50\n</code></pre> <p>or use <code>kubectl autoscale</code> to define the HPA via the commandline:</p> <pre><code>$ kubectl autoscale vmpool vm-pool-cirros --min=3 --max=10 --cpu-percent=50\n</code></pre>"},{"location":"user_workloads/pool/#exposing-a-virtualmachinepool-as-a-service","title":"Exposing a VirtualMachinePool as a Service","text":"<p>A VirtualMachinePool may be exposed as a service. When this is done, one of the VirtualMachine replicas will be picked for the actual delivery of the service.</p> <p>For example, exposing SSH port (22) as a ClusterIP service:</p> <p><pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vm-pool-cirros-ssh\nspec:\n  type: ClusterIP\n  selector:\n    kubevirt.io/vmpool: vm-pool-cirros\n  ports:\n    - protocol: TCP\n      port: 2222\n      targetPort: 22\n</code></pre> Saving this manifest into <code>vm-pool-cirros-ssh.yaml</code> and submitting it to Kubernetes will create the <code>ClusterIP</code> service listening on port 2222 and forwarding to port 22.</p> <p>See Service Objects for more details.</p>"},{"location":"user_workloads/pool/#using-persistent-storage","title":"Using Persistent Storage","text":"<p>Note: DataVolumes are part of CDI</p> <p>Usage of a <code>DataVolumeTemplates</code> within a <code>spec.virtualMachineTemplate.spec</code> will result in the creation of unique persistent storage for each VM within a VMPool. The <code>DataVolumeTemplate</code> name will have the VM's sequential postfix appended to it when the VM is created from the <code>spec.virtualMachineTemplate.spec.dataVolumeTemplates</code>. This makes each VM a completely unique stateful workload.</p>"},{"location":"user_workloads/pool/#using-unique-cloudinit-and-configmap-volumes-with-virtualmachinepools","title":"Using Unique CloudInit and ConfigMap Volumes with VirtualMachinePools","text":"<p>By default, any secrets or configMaps references in a <code>spec.virtualMachineTemplate.spec.template</code> Volume section will be used directly as is, without any modification to the naming. This means if you specify a secret in a <code>CloudInitNoCloud</code> volume, that every VM instance spawned from the VirtualMachinePool with this volume will get the exact same secret used for their cloud-init user data.</p> <p>This default behavior can be modified by setting the <code>AppendPostfixToSecretReferences</code> and <code>AppendPostfixToConfigMapReferences</code> booleans to true on the VMPool spec. When these booleans are enabled, references to secret and configMap names will have the VM's sequential postfix appended to the secret and configmap name. This allows someone to pre-generate unique per VM <code>secret</code> and <code>configMap</code> data for a VirtualMachinePool ahead of time in a way that will be predictably assigned to VMs within the VirtualMachinePool.</p>"},{"location":"user_workloads/presets/","title":"Presets","text":"<p>FEATURE STATE: </p> <ul> <li><code>VirtualMachineInstancePresets</code> are deprecated as of the <code>v0.57.0</code> release and will be removed in a future release. </li> <li>Users should instead look to use Instancetypes and preferences as a replacement.</li> </ul> <p><code>VirtualMachineInstancePresets</code> are an extension to general <code>VirtualMachineInstance</code> configuration behaving much like <code>PodPresets</code> from Kubernetes. When a <code>VirtualMachineInstance</code> is created, any applicable <code>VirtualMachineInstancePresets</code> will be applied to the existing spec for the <code>VirtualMachineInstance</code>. This allows for re-use of common settings that should apply to multiple <code>VirtualMachineInstances</code>.</p>"},{"location":"user_workloads/presets/#create-a-virtualmachineinstancepreset","title":"Create a VirtualMachineInstancePreset","text":"<p>You can describe a <code>VirtualMachineInstancePreset</code> in a YAML file. For example, the <code>vmi-preset.yaml</code> file below describes a <code>VirtualMachineInstancePreset</code> that requests a <code>VirtualMachineInstance</code> be created with a resource request for 64M of RAM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: small-qemu\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/size: small\n  domain:\n    resources:\n      requests:\n        memory: 64M\n</code></pre> <ul> <li>Create a <code>VirtualMachineInstancePreset</code> based on that YAML file:</li> </ul> <pre><code>    kubectl create -f vmipreset.yaml\n</code></pre>"},{"location":"user_workloads/presets/#required-fields","title":"Required Fields","text":"<p>As with most Kubernetes resources, a <code>VirtualMachineInstancePreset</code> requires <code>apiVersion</code>, <code>kind</code> and <code>metadata</code> fields.</p> <p>Additionally <code>VirtualMachineInstancePresets</code> also need a <code>spec</code> section. While not technically required to satisfy syntax, it is strongly recommended to include a <code>Selector</code> in the <code>spec</code> section, otherwise a <code>VirtualMachineInstancePreset</code> will match all <code>VirtualMachineInstances</code> in a namespace.</p>"},{"location":"user_workloads/presets/#virtualmachine-selector","title":"VirtualMachine Selector","text":"<p>KubeVirt uses Kubernetes <code>Labels</code> and <code>Selectors</code> to determine which <code>VirtualMachineInstancePresets</code> apply to a given <code>VirtualMachineInstance</code>, similarly to how <code>PodPresets</code> work in Kubernetes. If a setting from a <code>VirtualMachineInstancePreset</code> is applied to a <code>VirtualMachineInstance</code>, the <code>VirtualMachineInstance</code> will be marked with an Annotation upon completion.</p> <p>Any domain structure can be listed in the <code>spec</code> of a <code>VirtualMachineInstancePreset</code>, e.g. Clock, Features, Memory, CPU, or Devices such as network interfaces. All elements of the <code>spec</code> section of a <code>VirtualMachineInstancePreset</code> will be applied to the <code>VirtualMachineInstance</code>.</p> <p>Once a <code>VirtualMachineInstancePreset</code> is successfully applied to a <code>VirtualMachineInstance</code>, the <code>VirtualMachineInstance</code> will be marked with an annotation to indicate that it was applied. If a conflict occurs while a <code>VirtualMachineInstancePreset</code> is being applied, that portion of the <code>VirtualMachineInstancePreset</code> will be skipped.</p> <p>Any valid <code>Label</code> can be matched against, but it is suggested that a general rule of thumb is to use os/shortname, e.g. <code>kubevirt.io/os: rhel7</code>.</p>"},{"location":"user_workloads/presets/#updating-a-virtualmachineinstancepreset","title":"Updating a VirtualMachineInstancePreset","text":"<p>If a <code>VirtualMachineInstancePreset</code> is modified, changes will not be applied to existing <code>VirtualMachineInstances</code>. This applies to both the <code>Selector</code> indicating which <code>VirtualMachineInstances</code> should be matched, and also the <code>Domain</code> section which lists the settings that should be applied to a <code>VirtualMachine</code>.</p>"},{"location":"user_workloads/presets/#overrides","title":"Overrides","text":"<p><code>VirtualMachineInstancePresets</code> use a similar conflict resolution strategy to Kubernetes <code>PodPresets</code>. If a portion of the domain spec is present in both a <code>VirtualMachineInstance</code> and a <code>VirtualMachineInstancePreset</code> and both resources have the identical information, then creation of the <code>VirtualMachineInstance</code> will continue normally. If however there is a difference between the resources, an Event will be created indicating which <code>DomainSpec</code> element of which <code>VirtualMachineInstancePreset</code> was overridden. For example: If both the <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code> define a <code>CPU</code>, but use a different number of <code>Cores</code>, KubeVirt will note the difference.</p> <p>If any settings from the <code>VirtualMachineInstancePreset</code> were successfully applied, the <code>VirtualMachineInstance</code> will be annotated.</p> <p>In the event that there is a difference between the <code>Domains</code> of a <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code>, KubeVirt will create an <code>Event</code>. <code>kubectl get events</code> can be used to show all <code>Events</code>. For example:</p> <pre><code>    $ kubectl get events\n    ....\n    Events:\n      FirstSeen                         LastSeen                        Count From                              SubobjectPath                Reason    Message\n      2m          2m           1         myvmi.1515bbb8d397f258                       VirtualMachineInstance                                     Warning   Conflict                  virtualmachineinstance-preset-controller   Unable to apply VirtualMachineInstancePreset 'example-preset': spec.cpu: &amp;{6} != &amp;{4}\n</code></pre>"},{"location":"user_workloads/presets/#usage","title":"Usage","text":"<p><code>VirtualMachineInstancePresets</code> are namespaced resources, so should be created in the same namespace as the <code>VirtualMachineInstances</code> that will use them:</p> <p><code>kubectl create -f &lt;preset&gt;.yaml [--namespace &lt;namespace&gt;]</code></p> <p>KubeVirt will determine which <code>VirtualMachineInstancePresets</code> apply to a Particular <code>VirtualMachineInstance</code> by matching <code>Labels</code>. For example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: example-preset\n  selector:\n    matchLabels:\n      kubevirt.io/os: win10\n  ...\n</code></pre> <p>would match any <code>VirtualMachineInstance</code> in the same namespace with a <code>Label</code> of <code>flavor: foo</code>. For example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win10\n  ...\n</code></pre>"},{"location":"user_workloads/presets/#conflicts","title":"Conflicts","text":"<p>When multiple <code>VirtualMachineInstancePresets</code> match a particular <code>VirtualMachineInstance</code>, if they specify the same settings within a Domain, those settings must match. If two <code>VirtualMachineInstancePresets</code> have conflicting settings (e.g. for the number of CPU cores requested), an error will occur, and the <code>VirtualMachineInstance</code> will enter the <code>Failed</code> state, and a <code>Warning</code> event will be emitted explaining which settings of which <code>VirtualMachineInstancePresets</code> were problematic.</p>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances","title":"Matching Multiple <code>VirtualMachineInstances</code>","text":"<p>The main use case for <code>VirtualMachineInstancePresets</code> is to create re-usable settings that can be applied across various machines. Multiple methods are available to match the labels of a <code>VirtualMachineInstance</code> using selectors.</p> <ul> <li> <p>matchLabels: Each <code>VirtualMachineInstance</code> can use a specific label     shared by all</p> <p>instances. * matchExpressions: Logical operators for sets can be used to match multiple</p> <p>labels.</p> </li> </ul> <p>Using matchLabels, the label used in the <code>VirtualMachineInstancePreset</code> must match one of the labels of the <code>VirtualMachineInstance</code>:</p> <pre><code>selector:\n  matchLabels:\n    kubevirt.io/memory: large\n</code></pre> <p>would match</p> <pre><code>metadata:\n  labels:\n    kubevirt.io/memory: large\n    kubevirt.io/os: win10\n</code></pre> <p>or</p> <pre><code>metadata:\n  labels:\n    kubevirt.io/memory: large\n    kubevirt.io/os: fedora27\n</code></pre> <p>Using matchExpressions allows for matching multiple labels of <code>VirtualMachineInstances</code> without needing to explicity list a label.</p> <pre><code>selector:\n  matchExpressions:\n    - {key: kubevirt.io/os, operator: In, values: [fedora27, fedora26]}\n</code></pre> <p>would match both: <pre><code>metadata:\n  labels:\n    kubevirt.io/os: fedora26\n\nmetadata:\n  labels:\n    kubevirt.io/os: fedora27\n</code></pre></p> <p>The Kubernetes documentation has a detailed explanation. Examples are provided below.</p>"},{"location":"user_workloads/presets/#exclusions","title":"Exclusions","text":"<p>Since <code>VirtualMachineInstancePresets</code> use <code>Selectors</code> that indicate which <code>VirtualMachineInstances</code> their settings should apply to, there needs to exist a mechanism by which <code>VirtualMachineInstances</code> can opt out of <code>VirtualMachineInstancePresets</code> altogether. This is done using an annotation:</p> <pre><code>kind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  annotations:\n    virtualmachineinstancepresets.admission.kubevirt.io/exclude: \"true\"\n  ...\n</code></pre>"},{"location":"user_workloads/presets/#examples","title":"Examples","text":""},{"location":"user_workloads/presets/#simple-virtualmachineinstancepreset-example","title":"Simple <code>VirtualMachineInstancePreset</code> Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nversion: v1\nmetadata:\n  name: example-preset\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/os: win10\n  domain:\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win10\nspec:\n  domain:\n    firmware:\n      uuid: c8f99fc8-20f5-46c4-85e5-2b841c547cef\n</code></pre> <p>Once the <code>VirtualMachineInstancePreset</code> is applied to the <code>VirtualMachineInstance</code>, the resulting resource would look like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/example-preset: kubevirt.io/v1\n  labels:\n    kubevirt.io/os: win10\n    kubevirt.io/nodeName: master\n  name: myvmi\n  namespace: default\nspec:\n  domain:\n    devices: {}\n    features:\n      acpi:\n        enabled: true\n      apic:\n        enabled: true\n      hyperv:\n        relaxed:\n          enabled: true\n        spinlocks:\n          enabled: true\n          spinlocks: 8191\n        vapic:\n          enabled: true\n    firmware:\n      uuid: c8f99fc8-20f5-46c4-85e5-2b841c547cef\n    machine:\n      type: q35\n    resources:\n      requests:\n        memory: 8Mi\n</code></pre>"},{"location":"user_workloads/presets/#conflict-example","title":"Conflict Example","text":"<p>This is an example of a merge conflict. In this case both the <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code> request different number of CPU's.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nversion: v1\nmetadata:\n  name: example-preset\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/flavor: default-features\n  domain:\n    cpu:\n      cores: 4\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/flavor: default-features\nspec:\n  domain:\n    cpu:\n      cores: 6\n</code></pre> <p>In this case the <code>VirtualMachineInstance</code> Spec will remain unmodified. Use <code>kubectl get events</code> to show events.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n  generation: 0\n  labels:\n    kubevirt.io/flavor: default-features\n  name: myvmi\n  namespace: default\nspec:\n  domain:\n    cpu:\n      cores: 6\n    devices: {}\n    machine:\n      type: \"\"\n    resources: {}\nstatus: {}\n</code></pre> <p>Calling <code>kubectl get events</code> would have a line like:</p> <pre><code>2m 2m 1 myvmi.1515bbb8d397f258 VirtualMachineInstance Warning Conflict virtualmachineinstance-preset-controller Unable to apply VirtualMachineInstancePreset example-preset: spec.cpu: &amp;{6} != &amp;{4}\n</code></pre>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances-using-matchlabels","title":"Matching Multiple VirtualMachineInstances Using MatchLabels","text":"<p>These <code>VirtualMachineInstances</code> have multiple labels, one that is unique and one that is shared.</p> <p>Note: This example breaks from the convention of using os-shortname as a <code>Label</code> for demonstration purposes.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: twelve-cores\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/cpu: dodecacore\n  domain:\n    cpu:\n      cores: 12\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: windows-10\n  labels:\n    kubevirt.io/os: win10\n    kubevirt.io/cpu: dodecacore\nspec:\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: windows-7\n  labels:\n    kubevirt.io/os: win7\n    kubevirt.io/cpu: dodecacore\nspec:\n  terminationGracePeriodSeconds: 0\n</code></pre> <p>Adding this <code>VirtualMachineInstancePreset</code> and these <code>VirtualMachineInstances</code> will result in:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/twelve-cores: kubevirt.io/v1\n  labels:\n    kubevirt.io/cpu: dodecacore\n    kubevirt.io/os: win10\n  name: windows-10\nspec:\n  domain:\n    cpu:\n      cores: 12\n    devices: {}\n    resources:\n      requests:\n        memory: 4Gi\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/twelve-cores: kubevirt.io/v1\n  labels:\n    kubevirt.io/cpu: dodecacore\n    kubevirt.io/os: win7\n  name: windows-7\nspec:\n  domain:\n    cpu:\n      cores: 12\n    devices: {}\n    resources:\n      requests:\n        memory: 4Gi\n  terminationGracePeriodSeconds: 0\n</code></pre>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances-using-matchexpressions","title":"Matching Multiple VirtualMachineInstances Using MatchExpressions","text":"<p>This <code>VirtualMachineInstancePreset</code> has a matchExpression that will match two labels: <code>kubevirt.io/os: win10</code> and <code>kubevirt.io/os: win7</code>.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: windows-vmis\nspec:\n  selector:\n    matchExpressions:\n      - {key: kubevirt.io/os, operator: In, values: [win10, win7]}\n  domain:\n    resources:\n      requests:\n        memory: 128M\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: smallvmi\n  labels:\n    kubevirt.io/os: win10\nspec:\n  terminationGracePeriodSeconds: 60\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: largevmi\n  labels:\n    kubevirt.io/os: win7\nspec:\n  terminationGracePeriodSeconds: 120\n</code></pre> <p>Applying the preset to both VM's will result in:</p> <pre><code>apiVersion: v1\nitems:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachineInstance\n  metadata:\n    annotations:\n      presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n      virtualmachineinstancepreset.kubevirt.io/windows-vmis: kubevirt.io/v1\n    labels:\n      kubevirt.io/os: win7\n    name: largevmi\n  spec:\n    domain:\n      resources:\n        requests:\n          memory: 128M\n    terminationGracePeriodSeconds: 120\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachineInstance\n  metadata:\n    annotations:\n      presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n      virtualmachineinstancepreset.kubevirt.io/windows-vmis: kubevirt.io/v1\n    labels:\n      kubevirt.io/os: win10\n    name: smallvmi\n  spec:\n    domain:\n      resources:\n        requests:\n          memory: 128M\n    terminationGracePeriodSeconds: 60\n</code></pre>"},{"location":"user_workloads/replicaset/","title":"VirtualMachineInstanceReplicaSet","text":"<p>A VirtualMachineInstanceReplicaSet tries to ensures that a specified number of VirtualMachineInstance replicas are running at any time. In other words, a VirtualMachineInstanceReplicaSet makes sure that a VirtualMachineInstance or a homogeneous set of VirtualMachineInstances is always up and ready. It is very similar to a Kubernetes ReplicaSet.</p> <p>No state is kept and no guarantees about the maximum number of VirtualMachineInstance replicas which are up are given. For example, the VirtualMachineInstanceReplicaSet may decide to create new replicas if possibly still running VMs are entering an unknown state.</p>"},{"location":"user_workloads/replicaset/#using-virtualmachineinstancereplicaset","title":"Using VirtualMachineInstanceReplicaSet","text":"<p>The VirtualMachineInstanceReplicaSet allows us to specify a VirtualMachineInstanceTemplate in <code>spec.template</code>. It consists of <code>ObjectMetadata</code> in <code>spec.template.metadata</code>, and a <code>VirtualMachineInstanceSpec</code> in <code>spec.template.spec</code>. The specification of the virtual machine is equal to the specification of the virtual machine in the <code>VirtualMachineInstance</code> workload.</p> <p><code>spec.replicas</code> can be used to specify how many replicas are wanted. If unspecified, the default value is 1. This value can be updated anytime. The controller will react to the changes.</p> <p><code>spec.selector</code> is used by the controller to keep track of managed virtual machines. The selector specified there must be able to match the virtual machine labels as specified in <code>spec.template.metadata.labels</code>. If the selector does not match these labels, or they are empty, the controller will simply do nothing except from logging an error. The user is responsible for not creating other virtual machines or VirtualMachineInstanceReplicaSets which conflict with the selector and the template labels.</p>"},{"location":"user_workloads/replicaset/#exposing-a-virtualmachineinstancereplicaset-as-a-service","title":"Exposing a VirtualMachineInstanceReplicaSet as a Service","text":"<p>A VirtualMachineInstanceReplicaSet could be exposed as a service. When this is done, one of the VirtualMachineInstances replicas will be picked for the actual delivery of the service.</p> <p>For example, exposing SSH port (22) as a ClusterIP service using virtctl on a VirtualMachineInstanceReplicaSet:</p> <pre><code>$ virtctl expose vmirs vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>All service exposure options that apply to a VirtualMachineInstance apply to a VirtualMachineInstanceReplicaSet. See Exposing VirtualMachineInstance for more details.</p>"},{"location":"user_workloads/replicaset/#when-to-use-a-virtualmachineinstancereplicaset","title":"When to use a VirtualMachineInstanceReplicaSet","text":"<p>Note: The base assumption is that referenced disks are read-only or that the VMIs are writing internally to a tmpfs. The most obvious volume sources for VirtualMachineInstanceReplicaSets which KubeVirt supports are referenced below. If other types are used data corruption is possible.</p> <p>Using VirtualMachineInstanceReplicaSet is the right choice when one wants many identical VMs and does not care about maintaining any disk state after the VMs are terminated.</p> <p>Volume types which work well in combination with a VirtualMachineInstanceReplicaSet are:</p> <ul> <li>cloudInitNoCloud</li> <li>ephemeral</li> <li>containerDisk</li> <li>emptyDisk</li> <li>configMap</li> <li>secret</li> <li>any other type, if the VMI writes internally to a tmpfs</li> </ul>"},{"location":"user_workloads/replicaset/#fast-starting-ephemeral-virtual-machines","title":"Fast starting ephemeral Virtual Machines","text":"<p>This use-case involves small and fast booting VMs with little provisioning performed during initialization.</p> <p>In this scenario, migrations are not important. Redistributing VM workloads between Nodes can be achieved simply by deleting managed VirtualMachineInstances which are running on an overloaded Node. The <code>eviction</code> of such a VirtualMachineInstance can happen by directly deleting the VirtualMachineInstance instance (KubeVirt aware workload redistribution) or by deleting the corresponding Pod where the Virtual Machine runs in (Only Kubernetes aware workload redistribution).</p>"},{"location":"user_workloads/replicaset/#slow-starting-ephemeral-virtual-machines","title":"Slow starting ephemeral Virtual Machines","text":"<p>In this use-case one has big and slow booting VMs, and complex or resource intensive provisioning is done during boot. More specifically, the timespan between the creation of a new VM and it entering the ready state is long.</p> <p>In this scenario, one still does not care about the state, but since re-provisioning VMs is expensive, migrations are important. Workload redistribution between Nodes can be achieved by migrating VirtualMachineInstances to different Nodes. A workload redistributor needs to be aware of KubeVirt and create migrations, instead of <code>evicting</code> VirtualMachineInstances by deletion.</p> <p>Note: The simplest form of having a migratable ephemeral VirtualMachineInstance, will be to use local storage based on <code>ContainerDisks</code> in combination with a file based backing store. However, migratable backing store support has not officially landed yet in KubeVirt and is untested.</p>"},{"location":"user_workloads/replicaset/#example","title":"Example","text":"<p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceReplicaSet\nmetadata:\n  name: testreplicaset\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      myvmi: myvmi\n  template:\n    metadata:\n      name: test\n      labels:\n        myvmi: myvmi\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n            name: containerdisk\n        resources:\n          requests:\n            memory: 64M\n      volumes:\n      - name: containerdisk\n        containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n</code></pre> Saving this manifest into <code>testreplicaset.yaml</code> and submitting it to Kubernetes will create three virtual machines based on the template.</p> <pre><code>$ kubectl create -f testreplicaset.yaml\nvirtualmachineinstancereplicaset \"testreplicaset\" created\n$ kubectl describe vmirs testreplicaset\nName:         testreplicaset\nNamespace:    default\nLabels:       &lt;none&gt;\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachineInstanceReplicaSet\nMetadata:\n  Cluster Name:\n  Creation Timestamp:  2018-01-03T12:42:30Z\n  Generation:          0\n  Resource Version:    6380\n  Self Link:           /apis/kubevirt.io/v1/namespaces/default/virtualmachineinstancereplicasets/testreplicaset\n  UID:                 903a9ea0-f083-11e7-9094-525400ee45b0\nSpec:\n  Replicas:  3\n  Selector:\n    Match Labels:\n      Myvmi:  myvmi\n  Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        Myvmi:  myvmi\n      Name:    test\n    Spec:\n      Domain:\n        Devices:\n          Disks:\n            Disk:\n            Name:         containerdisk\n            Volume Name:  containerdisk\n        Resources:\n          Requests:\n            Memory:  64M\n      Volumes:\n        Name:  containerdisk\n        Container Disk:\n          Image:  kubevirt/cirros-container-disk-demo:latest\nStatus:\n  Conditions:      &lt;nil&gt;\n  Ready Replicas:  2\n  Replicas:        3\nEvents:\n  Type    Reason            Age   From                                 Message\n  ----    ------            ----  ----                                 -------\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: testh8998\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: testf474w\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: test5lvkd\n</code></pre> <p><code>Replicas</code> is <code>3</code> and <code>Ready Replicas</code> is <code>2</code>. This means that at the moment when showing the status, three Virtual Machines were already created, but only two are running and ready.</p>"},{"location":"user_workloads/replicaset/#scaling-via-the-scale-subresource","title":"Scaling via the Scale Subresource","text":"<p>Note: This requires the <code>CustomResourceSubresources</code> feature gate to be enabled for clusters prior to 1.11.</p> <p>The <code>VirtualMachineInstanceReplicaSet</code> supports the <code>scale</code> subresource. As a consequence it is possible to scale it via <code>kubectl</code>:</p> <pre><code>$ kubectl scale vmirs myvmirs --replicas 5\n</code></pre>"},{"location":"user_workloads/replicaset/#using-the-horizontal-pod-autoscaler","title":"Using the Horizontal Pod Autoscaler","text":"<p>Note: This requires at cluster newer or equal to 1.11.</p> <p>The HorizontalPodAutoscaler (HPA) can be used with a <code>VirtualMachineInstanceReplicaSet</code>. Simply reference it in the spec of the autoscaler:</p> <pre><code>apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: myhpa\nspec:\n  scaleTargetRef:\n    kind: VirtualMachineInstanceReplicaSet\n    name: vmi-replicaset-cirros\n    apiVersion: kubevirt.io/v1\n  minReplicas: 3\n  maxReplicas: 10\n  targetCPUUtilizationPercentage: 50\n</code></pre> <p>or use <code>kubectl autoscale</code> to define the HPA via the commandline:</p> <pre><code>$ kubectl autoscale vmirs vmi-replicaset-cirros --min=3 --max=10\n</code></pre>"},{"location":"user_workloads/startup_scripts/","title":"Startup Scripts","text":"<p>KubeVirt supports the ability to assign a startup script to a VirtualMachineInstance instance which is executed automatically when the VM initializes.</p> <p>These scripts are commonly used to automate injection of users and SSH keys into VMs in order to provide remote access to the machine. For example, a startup script can be used to inject credentials into a VM that allows an Ansible job running on a remote host to access and provision the VM.</p> <p>Startup scripts are not limited to any specific use case though. They can be used to run any arbitrary script in a VM on boot.</p>"},{"location":"user_workloads/startup_scripts/#cloud-init","title":"Cloud-init","text":"<p>cloud-init is a widely adopted project used for early initialization of a VM. Used by cloud providers such as AWS and GCP, cloud-init has established itself as the defacto method of providing startup scripts to VMs.</p> <p>Cloud-init documentation can be found here: Cloud-init Documentation.</p> <p>KubeVirt supports cloud-init's NoCloud  and ConfigDrive datasources which involve injecting startup scripts into a VM instance through the use of an ephemeral disk. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.</p>"},{"location":"user_workloads/startup_scripts/#ignition","title":"Ignition","text":"<p>Ignition is an alternative to cloud-init which allows for configuring the VM disk on first boot. You can find the Ignition documentation here.  You can also find a comparison between cloud-init and Ignition here.</p> <p>Ignition can be used with Kubevirt by using the <code>cloudInitConfigDrive</code> volume.</p>"},{"location":"user_workloads/startup_scripts/#sysprep","title":"Sysprep","text":"<p>Sysprep is an automation tool for Windows that automates Windows installation, setup, and custom software provisioning.</p> <p>The general flow is:</p> <ol> <li> <p>Seal the vm image with the Sysprep tool, for example by running:</p> <pre><code>%WINDIR%\\system32\\sysprep\\sysprep.exe /generalize /shutdown /oobe /mode:vm\n</code></pre> <p>Note</p> <p>We need to make sure the base vm does not restart, which can be done by setting the vm run strategy as <code>RerunOnFailure</code>.</p> <p>VM runStrategy:</p> <pre><code>spec:\n  runStrategy: RerunOnFailure\n</code></pre> <p>More information can be found here:</p> <ul> <li>Sysprep Process Overview</li> <li>Sysprep (Generalize) a Windows installation</li> </ul> <p>Note</p> <p>It is important that there is no answer file detected when the Sysprep Tool is triggered, because Windows Setup searches for answer files at the beginning of each configuration pass and caches it. If that happens, when the OS will start - it will just use the cached answer file, ignoring the one we provide through the Sysprep API. More information can be found here.</p> </li> <li> <p>Providing an Answer file named <code>autounattend.xml</code> in an attached media. The answer file can be provided in a ConfigMap or a Secret with the key <code>autounattend.xml</code></p> <p>The configuration file can be generated with Windows SIM or it can be specified manually according to the information found here:</p> <ul> <li>Answer files (unattend.xml)</li> <li>Answer File Reference</li> <li>Answer File Components Reference</li> </ul> <p>Note</p> <p>There are also many easy to find online tools available for creating an answer file.</p> </li> </ol>"},{"location":"user_workloads/startup_scripts/#cloud-init-examples","title":"Cloud-init Examples","text":""},{"location":"user_workloads/startup_scripts/#user-data","title":"User Data","text":"<p>KubeVirt supports the cloud-init NoCloud and ConfigDrive data sources which involve injecting startup scripts through the use of a disk attached to the VM.</p> <p>In order to assign a custom userdata script to a VirtualMachineInstance using this method, users must define a disk and a volume for the NoCloud or ConfigDrive datasource in the VirtualMachineInstance's spec.</p>"},{"location":"user_workloads/startup_scripts/#data-sources","title":"Data Sources","text":"<p>Under most circumstances users should stick to the NoCloud data source as it is the simplest cloud-init data source. Only if NoCloud is not supported by the cloud-init implementation (e.g. coreos-cloudinit) users should switch the data source to ConfigDrive.</p> <p>Switching the cloud-init data source to ConfigDrive is as easy as changing the volume type in the VirtualMachineInstance's spec from <code>cloudInitNoCloud</code> to <code>cloudInitConfigDrive</code>.</p> <p>NoCloud data source:</p> <pre><code>volumes:\n  - name: cloudinitvolume\n    cloudInitNoCloud:\n      userData: \"#cloud-config\"\n</code></pre> <p>ConfigDrive data source:</p> <pre><code>volumes:\n  - name: cloudinitvolume\n    cloudInitConfigDrive:\n      userData: \"#cloud-config\"\n</code></pre> <p>When using the ConfigDrive datasource, the <code>networkData</code> part has to be in the OpenStack Metadata Service Network format: <pre><code>  spec:\n    domain:\n      interfaces: \n        - name: secondary-net\n          bridge: {}\n          macAddress: '02:26:19:00:00:30'\n          model: virtio\n    networks: \n        - multus:\n            networkName: my-ns/my-net\n          name: secondary-net\n    volumes:\n      - name: cloudinitvolume\n        cloudInitConfigDrive:\n          networkData: |\n            {\"links\":[{\"id\":\"enp2s0\",\"type\":\"phy\",\"ethernet_mac_address\":\"02:26:19:00:00:30\"}],\"networks\":[{\"id\":\"NAD1\",\"type\":\"ipv4\",\"link\":\"enp2s0\",\"ip_address\":\"10.184.0.244\",\"netmask\":\"255.255.240.0\",\"routes\":[{\"network\":\"0.0.0.0\",\"netmask\":\"0.0.0.0\",\"gateway\":\"23.253.157.1\"}],\"network_id\":\"\"}],\"services\":[]}\n          userData: \"#cloud-config\"\n</code></pre></p> <p>Note The MAC address of the secondary interface should be predefined and identical in  the network interface and the cloud-init networkData. </p> <p>See the examples below for more complete cloud-init examples.</p>"},{"location":"user_workloads/startup_scripts/#cloud-init-user-data-as-clear-text","title":"Cloud-init user-data as clear text","text":"<p>In the example below, a SSH key is stored in the cloudInitNoCloud Volume's userData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <pre><code># Create a VM manifest with the startup script\n# a cloudInitNoCloud volume's userData field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userData: |\n          #cloud-config\n          ssh_authorized_keys:\n            - ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com\n\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-user-data-as-base64-string","title":"Cloud-init user-data as base64 string","text":"<p>In the example below, a simple bash script is base64 encoded and stored in the cloudInitNoCloud Volume's userDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <p>Users also have the option of storing the startup script in a Kubernetes Secret and referencing the Secret in the VM's spec. Examples further down in the document illustrate how that is done.</p> <pre><code># Create a simple startup script\n\ncat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\necho \"Hi from startup script!\"\nEND\n\n# Create a VM manifest with the startup script base64 encoded into\n# a cloudInitNoCloud volume's userDataBase64 field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script.sh | base64 -w0)\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-userdata-as-k8s-secret","title":"Cloud-init UserData as k8s Secret","text":"<p>Users who wish to not store the cloud-init userdata directly in the VirtualMachineInstance spec have the option to store the userdata into a Kubernetes Secret and reference that Secret in the spec.</p> <p>Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init userdata.</p> <p>Below is an example of how to create a Kubernetes Secret containing a startup script and reference that Secret in the VM's spec.</p> <pre><code># Create a simple startup script\n\ncat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\necho \"Hi from startup script!\"\nEND\n\n# Store the startup script in a Kubernetes Secret\nkubectl create secret generic my-vmi-secret --from-file=userdata=startup-script.sh\n\n# Create a VM manifest and reference the Secret's name in the cloudInitNoCloud\n# Volume's secretRef field\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        secretRef:\n          name: my-vmi-secret\nEND\n\n# Post the VM\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#injecting-ssh-keys-with-cloud-inits-cloud-config","title":"Injecting SSH keys with Cloud-init's Cloud-config","text":"<p>In the examples so far, the cloud-init userdata script has been a bash script. Cloud-init has it's own configuration that can handle some common tasks such as user creation and SSH key injection.</p> <p>More cloud-config examples can be found here: Cloud-init Examples</p> <p>Below is an example of using cloud-config to inject an SSH key for the default user (fedora in this case) of a Fedora Atomic disk image.</p> <pre><code># Create the cloud-init cloud-config userdata.\ncat &lt;&lt; END &gt; startup-script\n#cloud-config\npassword: atomic\nchpasswd: { expire: False }\nssh_pwauth: False\nssh_authorized_keys:\n    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain\nEND\n\n# Create the VM spec\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: sshvmi\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          dev: vda\n      - name: cloudinitdisk\n        disk:\n          dev: vdb\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/fedora-atomic-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script | base64 -w0)\nEND\n\n# Post the VirtualMachineInstance spec to KubeVirt.\nkubectl create -f my-vmi.yaml\n\n# Connect to VM with passwordless SSH key\nssh -i &lt;insert private key here&gt; fedora@&lt;insert ip here&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#inject-ssh-key-using-a-custom-shell-script","title":"Inject SSH key using a Custom Shell Script","text":"<p>Depending on the boot image in use, users may have a mixed experience using cloud-init's cloud-config to create users and inject SSH keys.</p> <p>Below is an example of creating a user and injecting SSH keys for that user using a script instead of cloud-config.</p> <pre><code>cat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\nexport NEW_USER=\"foo\"\nexport SSH_PUB_KEY=\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J $NEW_USER@localhost.localdomain\"\n\nsudo adduser -U -m $NEW_USER\necho \"$NEW_USER:atomic\" | chpasswd\nsudo mkdir /home/$NEW_USER/.ssh\nsudo echo \"$SSH_PUB_KEY\" &gt; /home/$NEW_USER/.ssh/authorized_keys\nsudo chown -R ${NEW_USER}: /home/$NEW_USER/.ssh\nEND\n\n# Create the VM spec\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: sshvmi\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          dev: vda\n      - name: cloudinitdisk\n        disk:\n          dev: vdb\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/fedora-atomic-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script.sh | base64 -w0)\nEND\n\n# Post the VirtualMachineInstance spec to KubeVirt.\nkubectl create -f my-vmi.yaml\n\n# Connect to VM with passwordless SSH key\nssh -i &lt;insert private key here&gt; foo@&lt;insert ip here&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#network-config","title":"Network Config","text":"<p>A cloud-init network version 1 configuration can be set to configure the network at boot.</p> <p>Cloud-init user-data must be set for cloud-init to parse network-config even if it is just the user-data config header:</p> <pre><code>#cloud-config\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-clear-text","title":"Cloud-init network-config as clear text","text":"<p>In the example below, a simple cloud-init network-config is stored in the cloudInitNoCloud Volume's networkData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <pre><code># Create a VM manifest with the network-config in\n# a cloudInitNoCloud volume's networkData field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkData: |\n          network:\n            version: 1\n            config:\n            - type: physical\n            name: eth0\n            subnets:\n              - type: dhcp\n\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-base64-string","title":"Cloud-init network-config as base64 string","text":"<p>In the example below, a simple network-config is base64 encoded and stored in the cloudInitNoCloud Volume's networkDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <p>Users also have the option of storing the network-config in a Kubernetes Secret and referencing the Secret in the VM's spec. Examples further down in the document illustrate how that is done.</p> <pre><code># Create a simple network-config\n\ncat &lt;&lt; END &gt; network-config\nnetwork:\n  version: 1\n  config:\n  - type: physical\n  name: eth0\n  subnets:\n    - type: dhcp\nEND\n\n# Create a VM manifest with the networkData base64 encoded into\n# a cloudInitNoCloud volume's networkDataBase64 field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkDataBase64: $(cat network-config | base64 -w0)\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-k8s-secret","title":"Cloud-init network-config as k8s Secret","text":"<p>Users who wish to not store the cloud-init network-config directly in the VirtualMachineInstance spec have the option to store the network-config into a Kubernetes Secret and reference that Secret in the spec.</p> <p>Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init network-config.</p> <p>Below is an example of how to create a Kubernetes Secret containing a network-config and reference that Secret in the VM's spec.</p> <pre><code># Create a simple network-config\n\ncat &lt;&lt; END &gt; network-config\nnetwork:\n  version: 1\n  config:\n  - type: physical\n  name: eth0\n  subnets:\n    - type: dhcp\nEND\n\n# Store the network-config in a Kubernetes Secret\nkubectl create secret generic my-vmi-secret --from-file=networkdata=network-config\n\n# Create a VM manifest and reference the Secret's name in the cloudInitNoCloud\n# Volume's secretRef field\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkDataSecretRef:\n          name: my-vmi-secret\nEND\n\n# Post the VM\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#debugging","title":"Debugging","text":"<p>Depending on the operating system distribution in use, cloud-init output is often printed to the console output on boot up. When developing userdata scripts, users can connect to the VM's console during boot up to debug.</p> <p>Example of connecting to console using virtctl:</p> <pre><code>virtctl console &lt;name of vmi&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#device-role-tagging","title":"Device Role Tagging","text":"<p>KubeVirt provides a mechanism for users to tag devices such as Network Interfaces with a specific role. The tag will be matched to the hardware address of the device and this mapping exposed to the guest OS via cloud-init.</p> <p>This additional metadata will help the guest OS users with multiple networks interfaces to identify the devices that may have a specific role, such as a network device dedicated to a specific service or a disk intended to be used by a specific application (database, webcache, etc.)</p> <p>This functionality already exists in platforms such as OpenStack. KubeVirt will provide the data in a similar format, known to users and services like cloud-init.</p> <p>For example: <pre><code>kind: VirtualMachineInstance\nspec:\n  domain:\n    devices:\n      interfaces:\n      - masquerade: {}\n        name: default\n      - bridge: {}\n        name: ptp\n        tag: ptp\n      - name: sriov-net\n        sriov: {}\n        tag: nfvfunc\n  networks:\n  - name: default\n    pod: {}\n  - multus:\n      networkName: ptp-conf\n    name: ptp\n  - multus:\n      networkName: sriov/sriov-network\n    name: sriov-net\n\nThe metadata will be available in the guests config drive `openstack/latest/meta_data.json`\n\n{\n  \"devices\": [\n    {\n        \"type\": \"nic\",\n        \"bus\": \"pci\",\n        \"address\": \"0000:00:02.0\",\n        \"mac\": \"01:22:22:42:22:21\",\n        \"tags\": [\"ptp\"]\n    },\n    {\n        \"type\": \"nic\",\n        \"bus\": \"pci\",\n        \"address\": \"0000:81:10.1\",\n        \"mac\": \"01:22:22:42:22:22\",\n        \"tags\": [\"nfvfunc\"]\n    },\n  ]\n}\n</code></pre></p>"},{"location":"user_workloads/startup_scripts/#ignition-examples","title":"Ignition Examples","text":"<p>Ignition data can be passed into a <code>cloudInitConfigDrive</code> source using  either clear text, a base64 string or a k8s Secret.</p> <p>Some examples of Ignition configurations can be found  in the examples  given by the Ignition documentation.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-clear-text","title":"Ignition as clear text","text":"<p>Here is a complete example of a Kubevirt VM using Ignition to add an ssh key to the  <code>coreos</code> user at first boot : </p> <pre><code>apiVersion: kubevirt.io/v1alpha3\nkind: VirtualMachine\nmetadata:\n  name: ign-demo\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/size: small\n        kubevirt.io/domain: ign-demo\n    spec:\n      domain:\n        devices:\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          interfaces:\n            - name: default\n              masquerade: {}\n        resources:\n          requests:\n            memory: 2G\n      networks:\n        - name: default\n          pod: {}\n      volumes:\n        - name: containerdisk\n          containerDisk:\n            image: quay.io/containerdisks/rhcos:4.9\n        - name: cloudinitdisk\n          cloudInitConfigDrive:\n            userData: |\n              {\n                \"ignition\": {\n                  \"config\": {},\n                  \"proxy\": {},\n                  \"security\": {},\n                  \"timeouts\": {},\n                  \"version\": \"3.2.0\"\n                },\n                \"passwd\": {\n                  \"users\": [\n                    {\n                      \"name\": \"coreos\",\n                      \"sshAuthorizedKeys\": [\n                        \"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPL3axFGHI3db9iJWkPXVbYzD7OaWTtHuqmxLvj+DztB user@example\"\n                      ]\n                    }\n                  ]\n                },\n                \"storage\": {},\n                \"systemd\": {}\n              }\n</code></pre> <p>See that the Ignition config is simply passed to the <code>userData</code> annotation of the  <code>cloudInitConfigDrive</code> volume.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-base64","title":"Ignition as base64","text":"<p>You can also pass the Ignition config as a base64 string by using the <code>userDatabase64</code> annotation : </p> <pre><code>...\ncloudInitConfigDrive:\n  userDataBase64: eyJpZ25pdGlvbiI6eyJjb25maWciOnt9LCJwcm94eSI6e30sInNlY3VyaXR5Ijp7fSwidGltZW91dHMiOnt9LCJ2ZXJzaW9uIjoiMy4yLjAifSwicGFzc3dkIjp7InVzZXJzIjpbeyJuYW1lIjoiY29yZW9zIiwic3NoQXV0aG9yaXplZEtleXMiOlsic3NoLWVkMjU1MTlBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJUEwzYXhGR0hJM2RiOWlKV2tQWFZiWXpEN09hV1R0SHVxbXhMdmorRHp0QiB1c2VyQGV4YW1wbGUiXX1dfSwic3RvcmFnZSI6e30sInN5c3RlbWQiOnt9fQ==\n</code></pre> <p>You can obtain the base64 string by doing <code>cat ignition.json | base64 -w0</code> in your terminal.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-k8s-secret","title":"Ignition as k8s Secret","text":"<p>If you do not want to store the Ignition config into the VM configuration, you can use a k8s Secret.</p> <p>First, create the secret with the ignition data in it :</p> <pre><code>kubectl create secret generic my-ign-secret --from-file=ignition=ignition.json\n</code></pre> <p>Then specify this secret into your VM configuration :</p> <pre><code>...\ncloudInitConfigDrive:\n  secretRef:\n    name: my-ign-secret\n</code></pre>"},{"location":"user_workloads/startup_scripts/#sysprep-examples","title":"Sysprep Examples","text":""},{"location":"user_workloads/startup_scripts/#sysprep-in-a-configmap","title":"Sysprep in a ConfigMap","text":"<p>The answer file can be provided in a ConfigMap:</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: sysprep-config\ndata:\n  autounattend.xml: |\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n    ...\n    &lt;/unattend&gt;\n</code></pre> <p>And attached to the VM like so:</p> <pre><code>kind: VirtualMachine\nmetadata:\n  name: windows-with-sysprep\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: windows-with-sysprep\n    spec:\n      domain:\n        cpu:\n          cores: 3\n        devices:\n          disks:\n          - bootOrder: 1\n            disk:\n              bus: virtio\n            name: harddrive\n          - name: sysprep\n            cdrom:\n              bus: sata\n        machine:\n          type: q35\n        resources:\n          requests:\n            memory: 6G\n      volumes:\n      - name: harddrive\n        persistentVolumeClaim:\n          claimName: windows_pvc\n      - name: sysprep\n        sysprep:\n          configMap:\n            name: sysprep-config\n</code></pre>"},{"location":"user_workloads/startup_scripts/#sysprep-in-a-secret","title":"Sysprep in a Secret","text":"<p>The answer file can be provided in a Secret:</p> <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: sysprep-config\nstringData:\ndata:\n  autounattend.xml: |\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n    ...\n    &lt;/unattend&gt;\n</code></pre> <p>And attached to the VM like so:</p> <pre><code>kind: VirtualMachine\nmetadata:\n  name: windows-with-sysprep\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: windows-with-sysprep\n    spec:\n      domain:\n        cpu:\n          cores: 3\n        devices:\n          disks:\n          - bootOrder: 1\n            disk:\n              bus: virtio\n            name: harddrive\n          - name: sysprep\n            cdrom:\n              bus: sata\n        machine:\n          type: q35\n        resources:\n          requests:\n            memory: 6G\n      volumes:\n      - name: harddrive\n        persistentVolumeClaim:\n          claimName: windows_pvc\n      - name: sysprep\n        sysprep:\n          secret:\n            name: sysprep-secret\n</code></pre>"},{"location":"user_workloads/startup_scripts/#base-sysprep-vm","title":"Base Sysprep VM","text":"<p>In the example below, a configMap with <code>autounattend.xml</code> file is used to modify the Windows iso image which is downloaded from Microsoft and creates a base installed Windows machine with virtio drivers installed and all the commands executed in <code>post-install.ps1</code> For the below manifests to work it needs to have <code>win10-iso</code> DataVolume.</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: win10-template-configmap\ndata:\n  autounattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n      &lt;settings pass=\"windowsPE\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-International-Core-WinPE\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;SetupUILanguage&gt;\n            &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;/SetupUILanguage&gt;\n          &lt;InputLocale&gt;0409:00000409&lt;/InputLocale&gt;\n          &lt;SystemLocale&gt;en-US&lt;/SystemLocale&gt;\n          &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;UILanguageFallback&gt;en-US&lt;/UILanguageFallback&gt;\n          &lt;UserLocale&gt;en-US&lt;/UserLocale&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-PnpCustomizationsWinPE\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;DriverPaths&gt;\n            &lt;PathAndCredentials wcm:keyValue=\"4b29ba63\" wcm:action=\"add\"&gt;\n              &lt;Path&gt;E:\\amd64\\2k19&lt;/Path&gt;\n            &lt;/PathAndCredentials&gt;\n            &lt;PathAndCredentials wcm:keyValue=\"25fe51ea\" wcm:action=\"add\"&gt;\n              &lt;Path&gt;E:\\NetKVM\\2k19\\amd64&lt;/Path&gt;\n            &lt;/PathAndCredentials&gt;\n          &lt;/DriverPaths&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;DiskConfiguration&gt;\n            &lt;Disk wcm:action=\"add\"&gt;\n              &lt;CreatePartitions&gt;\n                &lt;CreatePartition wcm:action=\"add\"&gt;\n                  &lt;Order&gt;1&lt;/Order&gt;\n                  &lt;Type&gt;Primary&lt;/Type&gt;\n                  &lt;Size&gt;100&lt;/Size&gt;\n                &lt;/CreatePartition&gt;\n                &lt;CreatePartition wcm:action=\"add\"&gt;\n                  &lt;Extend&gt;true&lt;/Extend&gt;\n                  &lt;Order&gt;2&lt;/Order&gt;\n                  &lt;Type&gt;Primary&lt;/Type&gt;\n                &lt;/CreatePartition&gt;\n              &lt;/CreatePartitions&gt;\n              &lt;ModifyPartitions&gt;\n                &lt;ModifyPartition wcm:action=\"add\"&gt;\n                  &lt;Format&gt;NTFS&lt;/Format&gt;\n                  &lt;Label&gt;System Reserved&lt;/Label&gt;\n                  &lt;Order&gt;1&lt;/Order&gt;\n                  &lt;PartitionID&gt;1&lt;/PartitionID&gt;\n                  &lt;TypeID&gt;0x27&lt;/TypeID&gt;\n                &lt;/ModifyPartition&gt;\n                &lt;ModifyPartition wcm:action=\"add\"&gt;\n                  &lt;Format&gt;NTFS&lt;/Format&gt;\n                  &lt;Label&gt;OS&lt;/Label&gt;\n                  &lt;Letter&gt;C&lt;/Letter&gt;\n                  &lt;Order&gt;2&lt;/Order&gt;\n                  &lt;PartitionID&gt;2&lt;/PartitionID&gt;\n                &lt;/ModifyPartition&gt;\n              &lt;/ModifyPartitions&gt;\n              &lt;DiskID&gt;0&lt;/DiskID&gt;\n              &lt;WillWipeDisk&gt;true&lt;/WillWipeDisk&gt;\n            &lt;/Disk&gt;\n          &lt;/DiskConfiguration&gt;\n          &lt;ImageInstall&gt;\n            &lt;OSImage&gt;\n              &lt;InstallFrom&gt;\n                &lt;MetaData wcm:action=\"add\"&gt;\n                  &lt;Key&gt;/Image/Description&lt;/Key&gt;\n                  &lt;Value&gt;Windows 10 Pro&lt;/Value&gt;\n                &lt;/MetaData&gt;\n              &lt;/InstallFrom&gt;\n              &lt;InstallTo&gt;\n                &lt;DiskID&gt;0&lt;/DiskID&gt;\n                &lt;PartitionID&gt;2&lt;/PartitionID&gt;\n              &lt;/InstallTo&gt;\n            &lt;/OSImage&gt;\n          &lt;/ImageInstall&gt;\n          &lt;UserData&gt;\n            &lt;AcceptEula&gt;true&lt;/AcceptEula&gt;\n            &lt;FullName/&gt;\n            &lt;Organization/&gt;\n            &lt;ProductKey&gt;\n              &lt;Key/&gt;\n            &lt;/ProductKey&gt;\n          &lt;/UserData&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"offlineServicing\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-LUA-Settings\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;EnableLUA&gt;false&lt;/EnableLUA&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"specialize\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-International-Core\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;InputLocale&gt;0409:00000409&lt;/InputLocale&gt;\n          &lt;SystemLocale&gt;en-US&lt;/SystemLocale&gt;\n          &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;UILanguageFallback&gt;en-US&lt;/UILanguageFallback&gt;\n          &lt;UserLocale&gt;en-US&lt;/UserLocale&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Security-SPP-UX\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;SkipAutoActivation&gt;true&lt;/SkipAutoActivation&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-SQMApi\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;CEIPEnabled&gt;0&lt;/CEIPEnabled&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"oobeSystem\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Shell-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;OOBE&gt;\n            &lt;HideEULAPage&gt;true&lt;/HideEULAPage&gt;\n            &lt;HideOEMRegistrationScreen&gt;true&lt;/HideOEMRegistrationScreen&gt;\n            &lt;HideOnlineAccountScreens&gt;true&lt;/HideOnlineAccountScreens&gt;\n            &lt;HideWirelessSetupInOOBE&gt;true&lt;/HideWirelessSetupInOOBE&gt;\n            &lt;NetworkLocation&gt;Work&lt;/NetworkLocation&gt;\n            &lt;SkipUserOOBE&gt;true&lt;/SkipUserOOBE&gt;\n            &lt;SkipMachineOOBE&gt;true&lt;/SkipMachineOOBE&gt;\n            &lt;ProtectYourPC&gt;3&lt;/ProtectYourPC&gt;\n          &lt;/OOBE&gt;\n          &lt;AutoLogon&gt;\n            &lt;Password&gt;\n              &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/Password&gt;\n            &lt;Enabled&gt;true&lt;/Enabled&gt;\n            &lt;Username&gt;Administrator&lt;/Username&gt;\n          &lt;/AutoLogon&gt;\n          &lt;UserAccounts&gt;\n            &lt;AdministratorPassword&gt;\n              &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/AdministratorPassword&gt;\n          &lt;/UserAccounts&gt;\n          &lt;RegisteredOrganization/&gt;\n          &lt;RegisteredOwner/&gt;\n          &lt;TimeZone&gt;Eastern Standard Time&lt;/TimeZone&gt;\n          &lt;FirstLogonCommands&gt;\n            &lt;SynchronousCommand wcm:action=\"add\"&gt;\n              &lt;CommandLine&gt;powershell -ExecutionPolicy Bypass -NoExit -NoProfile f:\\post-install.ps1&lt;/CommandLine&gt;\n              &lt;RequiresUserInput&gt;false&lt;/RequiresUserInput&gt;\n              &lt;Order&gt;1&lt;/Order&gt;\n              &lt;Description&gt;Post Installation Script&lt;/Description&gt;\n            &lt;/SynchronousCommand&gt;\n          &lt;/FirstLogonCommands&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n    &lt;/unattend&gt;\n\n\n  post-install.ps1: |-\n    # Remove AutoLogin\n    # https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/microsoft-windows-shell-setup-autologon-logoncount#logoncount-known-issue\n    reg add \"HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon\" /v AutoAdminLogon /t REG_SZ /d 0 /f\n\n    # install Qemu Tools (Drivers)\n    Start-Process msiexec -Wait -ArgumentList '/i e:\\virtio-win-gt-x64.msi /qn /passive /norestart'\n\n    # install Guest Agent\n    Start-Process msiexec -Wait -ArgumentList '/i e:\\guest-agent\\qemu-ga-x86_64.msi /qn /passive /norestart'\n\n    # Rename cached unattend.xml to avoid it is picked up by sysprep\n    mv C:\\Windows\\Panther\\unattend.xml C:\\Windows\\Panther\\unattend.install.xml\n\n    # Eject CD, to avoid that the autounattend.xml on the CD is picked up by sysprep\n    (new-object -COM Shell.Application).NameSpace(17).ParseName('F:').InvokeVerb('Eject')\n\n    # Run Sysprep and Shutdown\n    C:\\Windows\\System32\\Sysprep\\sysprep.exe /generalize /oobe /shutdown /mode:vm\n\n---\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  annotations:\n    name.os.template.kubevirt.io/win10: Microsoft Windows 10\n    vm.kubevirt.io/validations: |\n      [\n        {\n          \"name\": \"minimal-required-memory\",\n          \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n          \"rule\": \"integer\",\n          \"message\": \"This VM requires more memory.\",\n          \"min\": 2147483648\n        }, {\n          \"name\": \"windows-virtio-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"virto disk bus type has better performance, install virtio drivers in VM and change bus type\",\n          \"values\": [\"virtio\"],\n          \"justWarning\": true\n        }, {\n          \"name\": \"windows-disk-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"disk bus has to be either virtio or sata or scsi\",\n          \"values\": [\"virtio\", \"sata\", \"scsi\"]\n        }, {\n          \"name\": \"windows-cd-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].cdrom.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].cdrom.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"cd bus has to be sata\",\n          \"values\": [\"sata\"]\n        }\n      ]\n  name: win10-template\n  namespace: default\n  labels:\n    app: win10-template\n    flavor.template.kubevirt.io/medium: 'true'\n    os.template.kubevirt.io/win10: 'true'\n    vm.kubevirt.io/template: windows10-desktop-medium\n    vm.kubevirt.io/template.namespace: openshift\n    vm.kubevirt.io/template.revision: '1'\n    vm.kubevirt.io/template.version: v0.14.0\n    workload.template.kubevirt.io/desktop: 'true'\nspec:\n  runStrategy: RerunOnFailure\n  dataVolumeTemplates:\n    - metadata:\n        name: win10-template-windows-iso\n      spec:\n        storage: {}\n        source:\n          pvc:\n            name: windows10-iso\n            namespace: default\n    - metadata:\n        name: win10-template\n      spec:\n        storage:\n          resources:\n            requests:\n              storage: 25Gi\n          volumeMode: Filesystem\n        source:\n          blank: {}\n  template:\n    metadata:\n      annotations:\n        vm.kubevirt.io/flavor: medium\n        vm.kubevirt.io/os: windows10\n        vm.kubevirt.io/workload: desktop\n      labels:\n        flavor.template.kubevirt.io/medium: 'true'\n        kubevirt.io/domain: win10-template\n        kubevirt.io/size: medium\n        os.template.kubevirt.io/win10: 'true'\n        vm.kubevirt.io/name: win10-template\n        workload.template.kubevirt.io/desktop: 'true'\n    spec:\n      domain:\n        clock:\n          timer:\n            hpet:\n              present: false\n            hyperv: {}\n            pit:\n              tickPolicy: delay\n            rtc:\n              tickPolicy: catchup\n          utc: {}\n        cpu:\n          cores: 1\n          sockets: 1\n          threads: 1\n        devices:\n          disks:\n            - bootOrder: 1\n              disk:\n                bus: virtio\n              name: win10-template\n            - bootOrder: 2\n              cdrom:\n                bus: sata\n              name: windows-iso\n            - cdrom:\n                bus: sata\n              name: windows-guest-tools\n            - name: sysprep\n              cdrom:\n                bus: sata\n          inputs:\n            - bus: usb\n              name: tablet\n              type: tablet\n          interfaces:\n            - masquerade: {}\n              model: virtio\n              name: default\n        features:\n          acpi: {}\n          apic: {}\n          hyperv:\n            reenlightenment: {}\n            ipi: {}\n            synic: {}\n            synictimer:\n              direct: {}\n            spinlocks:\n              spinlocks: 8191\n            reset: {}\n            relaxed: {}\n            vpindex: {}\n            runtime: {}\n            tlbflush: {}\n            frequencies: {}\n            vapic: {}\n        machine:\n          type: pc-q35-rhel8.4.0\n        resources:\n          requests:\n            memory: 4Gi\n      hostname: win10-template\n      networks:\n        - name: default\n          pod: {}\n      volumes:\n        - dataVolume:\n            name: win10-iso\n          name: windows-iso\n        - dataVolume:\n            name: win10-template-windows-iso\n          name: win10-template\n        - containerDisk:\n            image: quay.io/kubevirt/virtio-container-disk\n          name: windows-guest-tools\n        - name: sysprep\n          sysprep:\n            configMap:\n              name: win10-template-configmap\n</code></pre>"},{"location":"user_workloads/startup_scripts/#launching-a-vm-from-template","title":"Launching a VM from template","text":"<p>From the above example after the sysprep command is executed in the <code>post-install.ps1</code> and the vm is in shutdown state, A new VM can be launched from the base <code>win10-template</code> with additional changes mentioned from the below <code>unattend.xml</code> in <code>sysprep-config</code>. The new VM can take upto 5 minutes to be in running state since Windows goes through oobe setup in the background with the customizations specified in the below <code>unattend.xml</code> file.</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: sysprep-config\ndata:\n  autounattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;!-- responsible for installing windows, ignored on sysprepped images --&gt;\n  unattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n      &lt;settings pass=\"oobeSystem\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Shell-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;OOBE&gt;\n            &lt;HideEULAPage&gt;true&lt;/HideEULAPage&gt;\n            &lt;HideOEMRegistrationScreen&gt;true&lt;/HideOEMRegistrationScreen&gt;\n            &lt;HideOnlineAccountScreens&gt;true&lt;/HideOnlineAccountScreens&gt;\n            &lt;HideWirelessSetupInOOBE&gt;true&lt;/HideWirelessSetupInOOBE&gt;\n            &lt;NetworkLocation&gt;Work&lt;/NetworkLocation&gt;\n            &lt;SkipUserOOBE&gt;true&lt;/SkipUserOOBE&gt;\n            &lt;SkipMachineOOBE&gt;true&lt;/SkipMachineOOBE&gt;\n            &lt;ProtectYourPC&gt;3&lt;/ProtectYourPC&gt;\n          &lt;/OOBE&gt;\n          &lt;AutoLogon&gt;\n            &lt;Password&gt;\n            &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/Password&gt;\n            &lt;Enabled&gt;true&lt;/Enabled&gt;\n        &lt;Username&gt;Administrator&lt;/Username&gt;\n    &lt;/AutoLogon&gt;\n    &lt;UserAccounts&gt;\n         &lt;AdministratorPassword&gt;\n                &lt;Value&gt;123456&lt;/Value&gt;\n                &lt;PlainText&gt;true&lt;/PlainText&gt;\n        &lt;/AdministratorPassword&gt;\n          &lt;/UserAccounts&gt;\n          &lt;RegisteredOrganization&gt;Kuebvirt&lt;/RegisteredOrganization&gt;\n          &lt;RegisteredOwner&gt;Kubevirt&lt;/RegisteredOwner&gt;\n          &lt;TimeZone&gt;Eastern Standard Time&lt;/TimeZone&gt;\n                &lt;FirstLogonCommands&gt;\n                    &lt;SynchronousCommand wcm:action=\"add\"&gt;\n                    &lt;CommandLine&gt;powershell -ExecutionPolicy Bypass -NoExit -WindowStyle Hidden -NoProfile d:\\customize.ps1&lt;/CommandLine&gt;\n                        &lt;RequiresUserInput&gt;false&lt;/RequiresUserInput&gt;\n                        &lt;Order&gt;1&lt;/Order&gt;\n                &lt;Description&gt;Customize Script&lt;/Description&gt;\n            &lt;/SynchronousCommand&gt;\n                &lt;/FirstLogonCommands&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n    &lt;/unattend&gt;\n  customize.ps1: |-\n    # Enable RDP\n    Set-ItemProperty -Path 'HKLM:\\System\\CurrentControlSet\\Control\\Terminal Server' -name \"fDenyTSConnections\" -value 0\n    Enable-NetFirewallRule -DisplayGroup \"Remote Desktop\"\n\n\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse\n    # Install the OpenSSH Server\n    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0\n    # Start the sshd service\n    Start-Service sshd\n\n    Set-Service -Name sshd -StartupType 'Automatic'\n\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration\n    # use powershell as default shell for ssh\n    New-ItemProperty -Path \"HKLM:\\SOFTWARE\\OpenSSH\" -Name DefaultShell -Value \"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\" -PropertyType String -Force\n\n\n    # Add ssh authorized_key for administrator\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement\n    $MyDir = $MyInvocation.MyCommand.Path | Split-Path -Parent\n    $PublicKey = Get-Content -Path $MyDir\\id_rsa.pub\n    $authrized_keys_path = $env:ProgramData + \"\\ssh\\administrators_authorized_keys\" \n    Add-Content -Path $authrized_keys_path -Value $PublicKey\n    icacls.exe $authrized_keys_path /inheritance:r /grant \"Administrators:F\" /grant \"SYSTEM:F\"\n\n\n    # install application via exe file installer from url\n    function Install-Exe {\n      $dlurl = $args[0]\n      $installerPath = Join-Path $env:TEMP (Split-Path $dlurl -Leaf)\n      Invoke-WebRequest -UseBasicParsing $dlurl -OutFile $installerPath\n      Start-Process -FilePath $installerPath -Args \"/S\" -Verb RunAs -Wait\n      Remove-Item $installerPath\n\n    }\n\n    # Wait for networking before running a task at startup\n    do {\n      $ping = test-connection -comp kubevirt.io -count 1 -Quiet\n    } until ($ping)\n\n    # Installing the Latest Notepad++ with PowerShell\n    $BaseUri = \"https://notepad-plus-plus.org\"\n    $BasePage = Invoke-WebRequest -Uri $BaseUri -UseBasicParsing\n    $ChildPath = $BasePage.Links | Where-Object { $_.outerHTML -like '*Current Version*' } | Select-Object -ExpandProperty href\n    $DownloadPageUri = $BaseUri + $ChildPath\n    $DownloadPage = Invoke-WebRequest -Uri $DownloadPageUri -UseBasicParsing\n    $DownloadUrl = $DownloadPage.Links | Where-Object { $_.outerHTML -like '*npp.*.Installer.x64.exe\"*' } | Select-Object -ExpandProperty href\n    Install-Exe $DownloadUrl\n  id_rsa.pub: |-\n    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain\n</code></pre>"},{"location":"user_workloads/templates/","title":"Templates","text":"<p>Note</p> <p>By deploying KubeVirt on top of OpenShift the user can benefit from the OpenShift Template functionality.</p>"},{"location":"user_workloads/templates/#virtual-machine-templates","title":"Virtual machine templates","text":""},{"location":"user_workloads/templates/#what-is-a-virtual-machine-template","title":"What is a virtual machine template?","text":"<p>The KubeVirt projects provides a set of templates to create VMs to handle common usage scenarios. These templates provide a combination of some key factors that could be further customized and processed to have a Virtual Machine object. The key factors which define a template are</p> <ul> <li> <p>Workload Most Virtual Machine should be server or desktop to have maximum     flexibility; the highperformance workload trades some of this     flexibility to provide better performances.</p> </li> <li> <p>Guest Operating System (OS) This allow to ensure that the emulated     hardware is compatible with the guest OS. Furthermore, it allows to     maximize the stability of the VM, and allows performance     optimizations.</p> </li> <li> <p>Size (flavor) Defines the amount of resources (CPU, memory) to     allocate to the VM.</p> </li> </ul> <p>More documentation is available in the common templates subproject</p>"},{"location":"user_workloads/templates/#accessing-the-virtual-machine-templates","title":"Accessing the virtual machine templates","text":"<p>If you installed KubeVirt using a supported method you should find the common templates preinstalled in the cluster. Should you want to upgrade the templates, or install them from scratch, you can use one of the supported releases</p> <p>To install the templates:</p> <pre><code>    $ export VERSION=$(curl -s https://api.github.com/repos/kubevirt/common-templates/releases | grep tag_name | grep -v -- '-rc' | head -1 | awk -F': ' '{print $2}' | sed 's/,//' | xargs)\n    $ oc create -f https://github.com/kubevirt/common-templates/releases/download/$VERSION/common-templates-$VERSION.yaml\n</code></pre>"},{"location":"user_workloads/templates/#editable-fields","title":"Editable fields","text":"<p>You can edit the fields of the templates which define the amount of resources which the VMs will receive.</p> <p>Each template can list a different set of fields that are to be considered editable. The fields are used as hints for the user interface, and also for other components in the cluster.</p> <p>The editable fields are taken from annotations in the template. Here is a snippet presenting a couple of most commonly found editable fields:</p> <pre><code>    metadata:\n      annotations:\n        template.kubevirt.io/editable: |\n          /objects[0].spec.template.spec.domain.cpu.sockets\n          /objects[0].spec.template.spec.domain.cpu.cores\n          /objects[0].spec.template.spec.domain.cpu.threads\n          /objects[0].spec.template.spec.domain.resources.requests.memory\n</code></pre> <p>Each entry in the editable field list must be a jsonpath. The jsonpath root is the objects: element of the template. The actually editable field is the last entry (the \"leaf\") of the path. For example, the following minimal snippet highlights the fields which you can edit:</p> <pre><code>    objects:\n      spec:\n        template:\n          spec:\n            domain:\n              cpu:\n                sockets:\n                  VALUE # this is editable\n                cores:\n                  VALUE # this is editable\n                threads:\n                  VALUE # this is editable\n              resources:\n                requests:\n                  memory:\n                    VALUE # this is editable\n</code></pre>"},{"location":"user_workloads/templates/#relationship-between-templates-and-vms","title":"Relationship between templates and VMs","text":"<p>Once processed the templates produce VM objects to be used in the cluster. The VMs produced from templates will have a <code>vm.kubevirt.io/template</code> label, whose value will be the name of the parent template, for example <code>fedora-desktop-medium</code>:</p> <pre><code>      metadata:\n        labels:\n          vm.kubevirt.io/template: fedora-desktop-medium\n</code></pre> <p>In addition, these VMs can include an optional label <code>vm.kubevirt.io/template-namespace</code>, whose value will be the namespace of the parent template, for example:</p> <pre><code>      metadata:\n        labels:\n          vm.kubevirt.io/template-namespace: openshift\n</code></pre> <p>If this label is not defined, the template is expected to belong to the same namespace as the VM.</p> <p>This make it possible to query for all the VMs built from any template.</p> <p>Example:</p> <pre><code>    oc process -o yaml -f dist/templates/rhel8-server-tiny.yaml NAME=rheltinyvm SRC_PVC_NAME=rhel SRC_PVC_NAMESPACE=kubevirt\n</code></pre> <p>And the output:</p> <pre><code>apiVersion: v1\nitems:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachine\n  metadata:\n    annotations:\n      vm.kubevirt.io/flavor: tiny\n      vm.kubevirt.io/os: rhel8\n      vm.kubevirt.io/validations: |\n        [\n          {\n            \"name\": \"minimal-required-memory\",\n            \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n            \"rule\": \"integer\",\n            \"message\": \"This VM requires more memory.\",\n            \"min\": 1610612736\n          }\n        ]\n      vm.kubevirt.io/workload: server\n    labels:\n      app: rheltinyvm\n      vm.kubevirt.io/template: rhel8-server-tiny\n      vm.kubevirt.io/template.revision: \"45\"\n      vm.kubevirt.io/template.version: 0.11.3\n    name: rheltinyvm\n  spec:\n    dataVolumeTemplates:\n    - apiVersion: cdi.kubevirt.io/v1beta1\n      kind: DataVolume\n      metadata:\n        name: rheltinyvm\n      spec:\n        storage:\n          accessModes:\n          - ReadWriteMany\n        source:\n          pvc:\n            name: rhel\n            namespace: kubevirt\n    running: false\n    template:\n      metadata:\n        labels:\n          kubevirt.io/domain: rheltinyvm\n          kubevirt.io/size: tiny\n      spec:\n        domain:\n          cpu:\n            cores: 1\n            sockets: 1\n            threads: 1\n          devices:\n            disks:\n            - disk:\n                bus: virtio\n              name: rheltinyvm\n            - disk:\n                bus: virtio\n              name: cloudinitdisk\n            interfaces:\n            - masquerade: {}\n              name: default\n            networkInterfaceMultiqueue: true\n            rng: {}\n          resources:\n            requests:\n              memory: 1.5Gi\n        networks:\n        - name: default\n          pod: {}\n        terminationGracePeriodSeconds: 180\n        volumes:\n        - dataVolume:\n            name: rheltinyvm\n          name: rheltinyvm\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              user: cloud-user\n              password: lymp-fda4-m1cv\n              chpasswd: { expire: False }\n          name: cloudinitdisk\nkind: List\nmetadata: {}\n</code></pre> <p>You can add the VM from the template to the cluster in one go</p> <pre><code>    oc process rhel8-server-tiny NAME=rheltinyvm SRC_PVC_NAME=rhel SRC_PVC_NAMESPACE=kubevirt | oc apply -f -\n</code></pre> <p>Please note that after the generation step VM and template objects have no relationship with each other besides the aforementioned label.  Changes in templates do not automatically affect VMs or vice versa.</p>"},{"location":"user_workloads/templates/#common-template-customization","title":"common template customization","text":"<p>The templates provided by the kubevirt project provide a set of conventions and annotations that augment the basic feature of the openshift templates. You can customize your kubevirt-provided templates editing these annotations, or you can add them to your existing templates to make them consumable by the kubevirt services.</p> <p>Here's a description of the kubevirt annotations. Unless otherwise specified, the following keys are meant to be top-level entries of the template metadata, like</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: windows-10\n  annotations:\n    openshift.io/display-name: \"Generic demo template\"\n</code></pre> <p>All the following annotations are prefixed with <code>defaults.template.kubevirt.io</code>, which is omitted below for brevity. So the actual annotations you should use will look like</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: windows-10\n  annotations:\n    defaults.template.kubevirt.io/disk: default-disk\n    defaults.template.kubevirt.io/volume: default-volume\n    defaults.template.kubevirt.io/nic: default-nic\n    defaults.template.kubevirt.io/network: default-network\n</code></pre> <p>Unless otherwise specified, all annotations are meant to be safe defaults, both for performance and compatibility, and hints for the CNV-aware UI and tooling.</p>"},{"location":"user_workloads/templates/#disk","title":"disk","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/disk: rhel-disk\n</code></pre>"},{"location":"user_workloads/templates/#nic","title":"nic","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Windows\n  annotations:\n    defaults.template.kubevirt.io/nic: my-nic\n</code></pre>"},{"location":"user_workloads/templates/#volume","title":"volume","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/volume: custom-volume\n</code></pre>"},{"location":"user_workloads/templates/#network","title":"network","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/network: fast-net\n</code></pre>"},{"location":"user_workloads/templates/#references","title":"references","text":"<p>The default values for network, nic, volume, disk are meant to be the name of a section later in the document that the UI will find and consume to find the default values for the corresponding types. For example, considering the annotation <code>defaults.template.kubevirt.io/disk: my-disk</code>: we assume that later in the document it exists an element called <code>my-disk</code> that the UI can use to find the data it needs. The names actually don't matter as long as they are legal for kubernetes and consistent with the content of the document.</p>"},{"location":"user_workloads/templates/#complete-example","title":"complete example","text":"<p><code>demo-template.yaml</code></p> <pre><code>apiversion: v1\nitems:\n- apiversion: kubevirt.io/v1\n  kind: virtualmachine\n  metadata:\n    labels:\n      vm.kubevirt.io/template: rhel7-generic-tiny\n    name: rheltinyvm\n    osinfoname: rhel7.0\n    defaults.template.kubevirt.io/disk: rhel-default-disk\n    defaults.template.kubevirt.io/nic: rhel-default-net\n  spec:\n    running: false\n    template:\n      spec:\n        domain:\n          cpu:\n            sockets: 1\n            cores: 1\n            threads: 1\n          devices:\n            rng: {}\n          resources:\n            requests:\n              memory: 1g\n        terminationgraceperiodseconds: 0\n        volumes:\n        - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n          name: rhel-default-disk\n        networks:\n        - genie:\n          networkName: flannel\n          name: rhel-default-net\nkind: list\nmetadata: {}\n</code></pre> <p>once processed becomes: <code>demo-vm.yaml</code></p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    vm.kubevirt.io/template: rhel7-generic-tiny\n  name: rheltinyvm\n  osinfoname: rhel7.0\nspec:\n  running: false\n  template:\n    spec:\n      domain:\n        cpu:\n          sockets: 1\n          cores: 1\n          threads: 1\n        resources:\n          requests:\n            memory: 1g\n        devices:\n          rng: {}\n          disks:\n          - disk:\n            name: rhel-default-disk\n        interfaces:\n        - bridge: {}\n          name: rhel-default-nic\n      terminationgraceperiodseconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n        name: containerdisk\n      networks:\n      - genie:\n          networkName: flannel\n        name: rhel-default-nic\n</code></pre>"},{"location":"user_workloads/templates/#virtual-machine-creation","title":"Virtual machine creation","text":""},{"location":"user_workloads/templates/#overview","title":"Overview","text":"<p>The KubeVirt projects provides a set of templates to create VMs to handle common usage scenarios. These templates provide a combination of some key factors that could be further customized and processed to have a Virtual Machine object.</p> <p>The key factors which define a template are - Workload Most Virtual Machine should be server or desktop to have maximum flexibility; the highperformance workload trades some of this flexibility to provide better performances. - Guest Operating System (OS) This allow to ensure that the emulated hardware is compatible with the guest OS. Furthermore, it allows to maximize the stability of the VM, and allows performance optimizations. - Size (flavor) Defines the amount of resources (CPU, memory) to allocate to the VM.</p>"},{"location":"user_workloads/templates/#openshift-console","title":"Openshift Console","text":"<p>VMs can be created through OpenShift Cluster Console UI . This UI supports creation VM using templates and templates features - flavors and workload profiles. To create VM from template, choose WorkLoads in the left panel &gt;&gt; choose Virtualization &gt;&gt; press to the \"Create Virtual Machine\" blue button &gt;&gt; choose \"Create from wizard\". Next, you have to see \"Create Virtual Machine\" window</p>"},{"location":"user_workloads/templates/#common-templates","title":"Common-templates","text":"<p>There is the common-templates subproject. It provides official prepared and useful templates. You can also create templates by hand. You can find an example below, in the \"Example template\" section.</p>"},{"location":"user_workloads/templates/#example-template","title":"Example template","text":"<p>In order to create a virtual machine via OpenShift CLI, you need to provide a template defining the corresponding object and its metadata.</p> <p>NOTE Only <code>VirtualMachine</code> object is currently supported.</p> <p>Here is an example template that defines an instance of the <code>VirtualMachine</code> object:</p> <pre><code>apiVersion: template.openshift.io/v1\nkind: Template\nmetadata:\n  name: fedora-desktop-large\n  annotations:\n    openshift.io/display-name: \"Fedora 32+ VM\"\n    description: &gt;-\n      Template for Fedora 32 VM or newer.\n      A PVC with the Fedora disk image must be available.\n      Recommended disk image:\n      https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.qcow2\n    tags: \"hidden,kubevirt,virtualmachine,fedora\"\n    iconClass: \"icon-fedora\"\n    openshift.io/provider-display-name: \"KubeVirt\"\n    openshift.io/documentation-url: \"https://github.com/kubevirt/common-templates\"\n    openshift.io/support-url: \"https://github.com/kubevirt/common-templates/issues\"\n    template.openshift.io/bindable: \"false\"\n    template.kubevirt.io/version: v1alpha1\n    defaults.template.kubevirt.io/disk: rootdisk\n    template.kubevirt.io/editable: |\n      /objects[0].spec.template.spec.domain.cpu.sockets\n      /objects[0].spec.template.spec.domain.cpu.cores\n      /objects[0].spec.template.spec.domain.cpu.threads\n      /objects[0].spec.template.spec.domain.resources.requests.memory\n      /objects[0].spec.template.spec.domain.devices.disks\n      /objects[0].spec.template.spec.volumes\n      /objects[0].spec.template.spec.networks\n    name.os.template.kubevirt.io/fedora32: Fedora 32 or higher\n    name.os.template.kubevirt.io/fedora33: Fedora 32 or higher\n    name.os.template.kubevirt.io/silverblue32: Fedora 32 or higher\n    name.os.template.kubevirt.io/silverblue33: Fedora 32 or higher\n  labels:\n    os.template.kubevirt.io/fedora32: \"true\"\n    os.template.kubevirt.io/fedora33: \"true\"\n    os.template.kubevirt.io/silverblue32: \"true\"\n    os.template.kubevirt.io/silverblue33: \"true\"\n    workload.template.kubevirt.io/desktop: \"true\"\n    flavor.template.kubevirt.io/large: \"true\"\n    template.kubevirt.io/type: \"base\"\n    template.kubevirt.io/version: \"0.11.3\"\nobjects:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachine\n  metadata:\n    name: ${NAME}\n    labels:\n      vm.kubevirt.io/template: fedora-desktop-large\n      vm.kubevirt.io/template.version: \"0.11.3\"\n      vm.kubevirt.io/template.revision: \"45\"\n      app: ${NAME}\n    annotations:\n      vm.kubevirt.io/os: \"fedora\"\n      vm.kubevirt.io/workload: \"desktop\"\n      vm.kubevirt.io/flavor: \"large\"\n      vm.kubevirt.io/validations: |\n        [\n          {\n            \"name\": \"minimal-required-memory\",\n            \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n            \"rule\": \"integer\",\n            \"message\": \"This VM requires more memory.\",\n            \"min\": 1073741824\n          }\n        ]\n  spec:\n    dataVolumeTemplates:\n    - apiVersion: cdi.kubevirt.io/v1beta1\n      kind: DataVolume\n      metadata:\n        name: ${NAME}\n      spec:\n        storage:\n          accessModes:\n            - ReadWriteMany\n        source:\n          pvc:\n            name: ${SRC_PVC_NAME}\n            namespace: ${SRC_PVC_NAMESPACE}\n    running: false\n    template:\n      metadata:\n        labels:\n          kubevirt.io/domain: ${NAME}\n          kubevirt.io/size: large\n      spec:\n        domain:\n          cpu:\n            sockets: 2\n            cores: 1\n            threads: 1\n          resources:\n            requests:\n              memory: 8Gi\n          devices:\n            rng: {}\n            networkInterfaceMultiqueue: true\n            inputs:\n              - type: tablet\n                bus: virtio\n                name: tablet\n            disks:\n            - disk:\n                bus: virtio\n              name: ${NAME}\n            - disk:\n                bus: virtio\n              name: cloudinitdisk\n            interfaces:\n            - masquerade: {}\n              name: default\n        terminationGracePeriodSeconds: 180\n        networks:\n        - name: default\n          pod: {}\n        volumes:\n        - dataVolume:\n            name: ${NAME}\n          name: ${NAME}\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              user: fedora\n              password: ${CLOUD_USER_PASSWORD}\n              chpasswd: { expire: False }\n          name: cloudinitdisk\nparameters:\n- description: VM name\n  from: 'fedora-[a-z0-9]{16}'\n  generate: expression\n  name: NAME\n- name: SRC_PVC_NAME\n  description: Name of the PVC to clone\n  value: 'fedora'\n- name: SRC_PVC_NAMESPACE\n  description: Namespace of the source PVC\n  value: kubevirt-os-images\n- description: Randomized password for the cloud-init user fedora\n  from: '[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}'\n  generate: expression\n  name: CLOUD_USER_PASSWORD\n</code></pre> <p>Note that the template above defines free parameters (<code>NAME</code>, <code>SRC_PVC_NAME</code>, <code>SRC_PVC_NAMESPACE</code>, <code>CLOUD_USER_PASSWORD</code>) and the <code>NAME</code> parameter does not have specified default value.</p> <p>An OpenShift template has to be converted into the JSON file via <code>oc process</code> command, that also allows you to set the template parameters.</p> <p>A complete example can be found in the KubeVirt repository.</p> <p>!&gt; You need to be logged in by <code>oc login</code> command.</p> <pre><code>$ oc process -f cluster/vmi-template-fedora.yaml\\\n    -p NAME=testvmi \\\n    -p SRC_PVC_NAME=fedora \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n{\n    \"kind\": \"List\",\n    \"apiVersion\": \"v1\",\n    \"metadata\": {},\n    \"items\": [\n        {\n</code></pre> <p>The JSON file is usually applied directly by piping the processed output to <code>oc create</code> command.</p> <pre><code>$ oc process -f cluster/examples/vm-template-fedora.yaml \\\n    -p NAME=testvm \\\n    -p SRC_PVC_NAME=fedora \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n    | oc create -f -\nvirtualmachine.kubevirt.io/testvm created\n</code></pre> <p>The command above results in creating a Kubernetes object according to the specification given by the template \\(in this example it is an instance of the VirtualMachine object\\).</p> <p>It's possible to get list of available parameters using the following command:</p> <pre><code>$ oc process -f dist/templates/fedora-desktop-large.yaml --parameters\nNAME                  DESCRIPTION                                          GENERATOR           VALUE\nNAME                  VM name                                              expression          fedora-[a-z0-9]{16}\nSRC_PVC_NAME          Name of the PVC to clone                                                 fedora\nSRC_PVC_NAMESPACE     Namespace of the source PVC                                              kubevirt-os-images\nCLOUD_USER_PASSWORD   Randomized password for the cloud-init user fedora   expression          [a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}\n</code></pre>"},{"location":"user_workloads/templates/#starting-virtual-machine-from-the-created-object","title":"Starting virtual machine from the created object","text":"<p>The created object is now a regular VirtualMachine object and from now it can be controlled by accessing Kubernetes API resources. The preferred way how to do this from within the OpenShift environment is to use <code>oc patch</code> command.</p> <pre><code>$ oc patch virtualmachine testvm --type merge -p '{\"spec\":{\"running\":true}}'\nvirtualmachine.kubevirt.io/testvm patched\n</code></pre> <p>Do not forget about virtctl tool. Using it in the real cases instead of using kubernetes API can be more convenient. Example:</p> <pre><code>$ virtctl start testvm\nVM testvm was scheduled to start\n</code></pre> <p>As soon as VM starts, Kubernetes creates new type of object - VirtualMachineInstance. It has similar name to VirtualMachine. Example (not full output, it's too big):</p> <pre><code>$ kubectl describe vm testvm\nname:         testvm\nNamespace:    myproject\nLabels:       kubevirt-vm=vm-testvm\n              kubevirt.io/os=fedora33\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachine\n</code></pre>"},{"location":"user_workloads/templates/#cloud-init-script-and-parameters","title":"Cloud-init script and parameters","text":"<p>Kubevirt VM templates, just like kubevirt VM/VMI yaml configs, supports cloud-init scripts</p>"},{"location":"user_workloads/templates/#hack-use-pre-downloaded-image","title":"Hack - use pre-downloaded image","text":"<p>Kubevirt VM templates, just like kubevirt VM/VMI yaml configs, can use pre-downloaded VM image, which can be a useful feature especially in the debug/development/testing cases. No special parameters required in the VM template or VM/VMI yaml config. The main idea is to create Kubernetes PersistentVolume and PersistentVolumeClaim corresponding to existing image in the file system. Example:</p> <pre><code>---\nkind: PersistentVolume\napiVersion: v1\nmetadata:\n  name: mypv\n  labels:\n    type: local\nspec:\n  storageClassName: manual\n  capacity:\n    storage: 10G\n  accessModes:\n    - ReadWriteOnce\n  hostPath:\n    path: \"/mnt/sda1/images/testvm\"\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: mypvc\nspec:\n  storageClassName: manual\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 10G\n</code></pre>"},{"location":"user_workloads/templates/#using-datavolumes","title":"Using DataVolumes","text":"<p>Kubevirt VM templates are using dataVolumeTemplates. Before using dataVolumes, CDI has to be installed in cluster. After that, source Datavolume can be created.</p> <pre><code>---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: fedora-datavolume-original\n  namespace: kubevirt\nspec:\n  source:\n    registry:\n      url: \"image_url\"\n  storage:\n    resources:\n      requests:\n        storage: 30Gi\n</code></pre> <p>After import is completed, VM can be created: <pre><code>$ oc process -f cluster/examples/vm-template-fedora.yaml \\\n    -p NAME=testvmi \\\n    -p SRC_PVC_NAME=fedora-datavolume-original \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n    | oc create -f -\nvirtualmachine.kubevirt.io/testvm created\n</code></pre></p>"},{"location":"user_workloads/templates/#additional-information","title":"Additional information","text":"<p>You can follow Virtual Machine Lifecycle Guide for further reference.</p>"},{"location":"user_workloads/virtctl_client_tool/","title":"Download and Install the virtctl Command Line Interface","text":""},{"location":"user_workloads/virtctl_client_tool/#download-the-virtctl-client-tool","title":"Download the <code>virtctl</code> client tool","text":"<p>Basic VirtualMachineInstance operations can be performed with the stock <code>kubectl</code> utility. However, the <code>virtctl</code> binary utility is required to use advanced features such as:</p> <ul> <li>Serial and graphical console access</li> </ul> <p>It also provides convenience commands for:</p> <ul> <li> <p>Starting and stopping VirtualMachineInstances</p> </li> <li> <p>Live migrating VirtualMachineInstances and canceling live migrations</p> </li> <li> <p>Uploading virtual machine disk images</p> </li> </ul> <p>There are two ways to get it:</p> <ul> <li> <p>the most recent version of the tool can be retrieved from the     official release     page</p> </li> <li> <p>it can be installed as a <code>kubectl</code> plugin using     krew</p> </li> </ul> <p>Example:</p> <pre><code>export VERSION=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)\nwget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64\n</code></pre>"},{"location":"user_workloads/virtctl_client_tool/#install-virtctl-with-krew","title":"Install <code>virtctl</code> with <code>krew</code>","text":"<p>It is required to install <code>krew</code> plugin manager beforehand. If <code>krew</code> is installed, <code>virtctl</code> can be installed via <code>krew</code>:</p> <pre><code>$ kubectl krew install virt\n</code></pre> <p>Then <code>virtctl</code> can be used as a kubectl plugin. For a list of available commands run:</p> <pre><code>$ kubectl virt help\n</code></pre> <p>Every occurrence throughout this guide of</p> <pre><code>$ ./virtctl &lt;command&gt;...\n</code></pre> <p>should then be read as</p> <pre><code>$ kubectl virt &lt;command&gt;...\n</code></pre>"},{"location":"user_workloads/virtual_machine_instances/","title":"Virtual Machines Instances","text":"<p>The <code>VirtualMachineInstance</code> type conceptionally has two parts:</p> <ul> <li> <p>Information for making scheduling decisions</p> </li> <li> <p>Information about the virtual machine API</p> </li> </ul> <p>Every <code>VirtualMachineInstance</code> object represents a single running virtual machine instance.</p>"},{"location":"user_workloads/virtual_machine_instances/#api-overview","title":"API Overview","text":"<p>With the installation of KubeVirt, new types are added to the Kubernetes API to manage Virtual Machines.</p> <p>You can interact with the new resources (via <code>kubectl</code>) as you would with any other API resource.</p>"},{"location":"user_workloads/virtual_machine_instances/#virtualmachineinstance-api","title":"VirtualMachineInstance API","text":"<p>Note: A full API reference is available at https://kubevirt.io/api-reference/.</p> <p>Here is an example of a VirtualMachineInstance object:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: emptydisk\n        disk:\n          bus: virtio\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: emptydisk\n    emptyDisk:\n      capacity: \"2Gi\"\n  - name: cloudinitdisk\n    cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n</code></pre> <p>This example uses a fedora cloud image in combination with cloud-init and an ephemeral empty disk with a capacity of <code>2Gi</code>. For the sake of simplicity, the volume sources in this example are ephemeral and don't require a provisioner in your cluster.</p>"},{"location":"user_workloads/virtual_machine_instances/#additional-information","title":"Additional Information","text":"<ul> <li> <p>Using instancetypes and preferences with a VirtualMachine:     Instancetypes and preferences</p> </li> <li> <p>More information about persistent and ephemeral volumes:     Disks and Volumes</p> </li> <li> <p>How to access a VirtualMachineInstance via <code>console</code> or <code>vnc</code>:     Console Access</p> </li> <li> <p>How to customize VirtualMachineInstances with <code>cloud-init</code>:     Cloud Init</p> </li> </ul>"},{"location":"user_workloads/vm_rollout_strategies/","title":"VM Rollout Strategies","text":"<p>In KubeVirt, the VM rollout strategy defines how changes to a VM object affect a running guest. In other words, it defines when and how changes to a VM object get propagated to its corresponding VMI object.</p> <p>There are currently 2 rollout strategies: <code>LiveUpdate</code> and <code>Stage</code>. Only 1 can be specified and the default is <code>Stage</code>.</p>"},{"location":"user_workloads/vm_rollout_strategies/#feature-gate","title":"Feature Gate","text":"<p>As long as the <code>VMLiveUpdateFeatures</code> is not enabled, the VM Rollout Strategy is ignored and defaults to \"Stage\". The feature gate is set in the KubeVirt custom resource (CR) like that:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#liveupdate","title":"LiveUpdate","text":"<p>The <code>LiveUpdate</code> VM rollout strategy tries to propagate VM object changes to running VMIs as soon as possible. For example, changing the number of CPU sockets will trigger a CPU hotplug.</p> <p>Enable the <code>LiveUpdate</code> VM rollout strategy in the KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#stage","title":"Stage","text":"<p>The <code>Stage</code> VM rollout strategy stages every change made to the VM object until its next reboot.  </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"Stage\"\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#restartrequired-condition","title":"RestartRequired condition","text":"<p>Any change made to a VM object when the rollout strategy is <code>Stage</code> will trigger the <code>RestartRequired</code> VM condition. When the rollout strategy is <code>LiveUpdate</code>, only non-propagatable changes will trigger the condition.</p> <p>Once the <code>RestartRequired</code> condition is set on a VM object, no further changes can be propagated, even if the strategy is set to <code>LiveUpdate</code>. Changes will become effective on next reboot, and the condition will be removed.</p>"},{"location":"user_workloads/vm_rollout_strategies/#limitations","title":"Limitations","text":"<p>The current implementation has the following limitations:</p> <ul> <li>Once the <code>RestartRequired</code> condition is set, the only way to get rid of it is to restart the VM. In the future, we plan on implementing a way to get rid of it by reverting the VM template spec to its last non-RestartRequired state.</li> <li>Cluster defaults are excluded from this logic. It means that changing a cluster-wide setting that impacts VM specs will not be live-updated, regardless of the rollout strategy.</li> <li>The <code>RestartRequired</code> condition comes with a message stating what kind of change triggered the condition (CPU/memory/other). That message pertains only to the first change that triggered the condition. Additional changes that would usually trigger the condition will just get staged and no additional <code>RestartRequired</code> condition will be added.</li> </ul>"},{"location":"user_workloads/windows_virtio_drivers/","title":"Windows virtio drivers","text":"<p>Purpose of this document is to explain how to install virtio drivers for Microsoft Windows running in a fully virtualized guest.</p>"},{"location":"user_workloads/windows_virtio_drivers/#do-i-need-virtio-drivers","title":"Do I need virtio drivers?","text":"<p>Yes. Without the virtio drivers, you cannot use paravirtualized hardware properly. It would either not work, or will have a severe performance penalty.</p> <p>For more information about VirtIO and paravirtualization, see VirtIO and paravirtualization</p> <p>For more details on configuring your VirtIO driver please refer to Installing VirtIO driver on a new Windows virtual machine and Installing VirtIO driver on an existing Windows virtual machine.</p>"},{"location":"user_workloads/windows_virtio_drivers/#which-drivers-i-need-to-install","title":"Which drivers I need to install?","text":"<p>There are usually up to 8 possible devices that are required to run Windows smoothly in a virtualized environment. KubeVirt currently supports only:</p> <ul> <li> <p>viostor, the block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>viorng, the entropy source driver, applies to PCI Device in the     Other devices group.</p> </li> <li> <p>NetKVM, the network driver, applies to Ethernet Controller in     the Other devices group. Available only if a virtio NIC is     configured.</p> </li> </ul> <p>Other virtio drivers, that exists and might be supported in the future:</p> <ul> <li> <p>Balloon, the balloon driver, applies to PCI Device in the Other     devices group</p> </li> <li> <p>vioserial, the paravirtual serial driver, applies to PCI Simple     Communications Controller in the Other devices group.</p> </li> <li> <p>vioscsi, the SCSI block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>qemupciserial, the emulated PCI serial driver, applies to PCI Serial     Port in the Other devices group.</p> </li> <li> <p>qxl, the paravirtual video driver, applied to Microsoft Basic     Display Adapter in the Display adapters group.</p> </li> <li> <p>pvpanic, the paravirtual panic driver, applies to Unknown device in     the Other devices group.</p> </li> </ul> <p>Note</p> <p>Some drivers are required in the installation phase. When you are installing Windows onto the virtio block storage you have to provide an appropriate virtio driver. Namely, choose viostor driver for your version of Microsoft Windows, eg. does not install XP driver when you run Windows 10.</p> <p>Other drivers can be installed after the successful windows installation. Again, please install only drivers matching your Windows version.</p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-install-during-windows-install","title":"How to install during Windows install?","text":"<p>To install drivers before the Windows starts its install, make sure you have virtio-win package attached to your VirtualMachine as SATA CD-ROM. In the Windows installation, choose advanced install and load driver. Then please navigate to loaded Virtio CD-ROM and install one of viostor or vioscsi, depending on whichever you have set up.</p> <p>Step by step screenshots:</p> <p></p> <p></p> <p></p> <p></p> <p></p> <p></p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-install-after-windows-install","title":"How to install after Windows install?","text":"<p>After windows install, please go to Device Manager. There you should see undetected devices in \"available devices\" section. You can install virtio drivers one by one going through this list.</p> <p></p> <p></p> <p></p> <p></p> <p>For more details on how to choose a proper driver and how to install the driver, please refer to the Windows Guest Virtual Machines on Red Hat Enterprise Linux 7.</p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-obtain-virtio-drivers","title":"How to obtain virtio drivers?","text":"<p>The virtio Windows drivers are distributed in a form of containerDisk, which can be simply mounted to the VirtualMachine. The container image, containing the disk is located at: https://quay.io/repository/kubevirt/virtio-container-disk?tab=tags and the image be pulled as any other docker container:</p> <pre><code>docker pull quay.io/kubevirt/virtio-container-disk\n</code></pre> <p>However, pulling image manually is not required, it will be downloaded if not present by Kubernetes when deploying VirtualMachine.</p>"},{"location":"user_workloads/windows_virtio_drivers/#attaching-to-virtualmachine","title":"Attaching to VirtualMachine","text":"<p>KubeVirt distributes virtio drivers for Microsoft Windows in a form of container disk. The package contains the virtio drivers and QEMU guest agent. The disk was tested on Microsoft Windows Server 2012. Supported Windows version is XP and up.</p> <p>The package is intended to be used as CD-ROM attached to the virtual machine with Microsoft Windows. It can be used as SATA CDROM during install phase or to provide drivers in an existing Windows installation.</p> <p>Attaching the virtio-win package can be done simply by adding ContainerDisk to you VirtualMachine.</p> <pre><code>spec:\n  domain:\n    devices:\n      disks:\n        - name: virtiocontainerdisk\n          # Any other disk you want to use, must go before virtioContainerDisk.\n          # KubeVirt boots from disks in order ther are defined.\n          # Therefore virtioContainerDisk, must be after bootable disk.\n          # Other option is to choose boot order explicitly:\n          #  - https://kubevirt.io/api-reference/v0.13.2/definitions.html#_v1_disk\n          # NOTE: You either specify bootOrder explicitely or sort the items in\n          #       disks. You can not do both at the same time.\n          # bootOrder: 2\n          cdrom:\n            bus: sata\nvolumes:\n  - containerDisk:\n      image: quay.io/kubevirt/virtio-container-disk\n    name: virtiocontainerdisk\n</code></pre> <p>Once you are done installing virtio drivers, you can remove virtio container disk by simply removing the disk from yaml specification and restarting the VirtualMachine.</p>"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-,:!=\\[\\]\\(\\)\"/]+|\\.(?!\\d)","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome","text":"<p>The KubeVirt User Guide is divided into the following sections:</p> <ul> <li>Architecture: Technical and conceptual overview of KubeVirt components</li> <li>Quickstarts: A list of resources to help you learn KubeVirt basics</li> <li>Cluster Administration: Cluster-level administration concepts and tasks</li> <li>User Workloads: Creating, customizing, using, and monitoring virtual machines </li> <li>Compute: Resource allocation and optimization for the virtualization layer</li> <li>Network: Concepts and tasks for the networking and service layers </li> <li>Storage: Concepts and tasks for the storage layer, including importing and exporting.</li> <li>Release Notes: The release notes for all KubeVirt releases</li> <li>Contributing: How you can contribute to this guide or the KubeVirt project </li> <li>Virtualization Debugging: How to debug your KubeVirt cluster and virtual resources</li> </ul>"},{"location":"#try-it-out","title":"Try it out","text":"<ul> <li> <p>Kubevirt on Killercoda: https://killercoda.com/kubevirt</p> </li> <li> <p>Kubevirt on Minikube: https://kubevirt.io/quickstart_minikube/</p> </li> <li> <p>Kubevirt on Kind: https://kubevirt.io/quickstart_kind/</p> </li> <li> <p>Kubevirt on cloud providers: https://kubevirt.io/quickstart_cloud/</p> </li> </ul>"},{"location":"#kubevirt-labs","title":"KubeVirt Labs","text":"<ul> <li> <p>Use KubeVirt</p> </li> <li> <p>Experiment with Containerized Data Importer (CDI)</p> </li> <li> <p>Experiment with KubeVirt Upgrades</p> </li> <li> <p>Live Migration</p> </li> </ul>"},{"location":"#getting-help","title":"Getting help","text":"<ul> <li> <p>File a bug: https://github.com/kubevirt/kubevirt/issues</p> </li> <li> <p>Mailing list: https://groups.google.com/forum/#!forum/kubevirt-dev</p> </li> <li> <p>Slack: https://kubernetes.slack.com/messages/virtualization</p> </li> </ul>"},{"location":"#developer","title":"Developer","text":"<ul> <li> <p>Start contributing: Contributing</p> </li> <li> <p>API Reference: http://kubevirt.io/api-reference/</p> </li> </ul>"},{"location":"#privacy","title":"Privacy","text":"<ul> <li> <p>Check our privacy policy at: https://kubevirt.io/privacy/</p> </li> <li> <p>We do use https://netlify.com Open Source Plan for Rendering Pull   Requests to the documentation repository</p> </li> </ul>"},{"location":"architecture/","title":"Architecture","text":"<p>KubeVirt is built using a service oriented architecture and a choreography pattern.</p>"},{"location":"architecture/#stack","title":"Stack","text":"<pre><code>  +---------------------+\n  | KubeVirt            |\n~~+---------------------+~~\n  | Orchestration (K8s) |\n  +---------------------+\n  | Scheduling (K8s)    |\n  +---------------------+\n  | Container Runtime   |\n~~+---------------------+~~\n  | Operating System    |\n  +---------------------+\n  | Virtual(kvm)        |\n~~+---------------------+~~\n  | Physical            |\n  +---------------------+\n</code></pre> <p>Users requiring virtualization services are speaking to the Virtualization API (see below) which in turn is speaking to the Kubernetes cluster to schedule requested Virtual Machine Instances (VMIs). Scheduling, networking, and storage  are all delegated to Kubernetes, while KubeVirt provides the virtualization functionality.</p>"},{"location":"architecture/#additional-services","title":"Additional Services","text":"<p>KubeVirt provides additional functionality to your Kubernetes cluster, to perform virtual machine management</p> <p>If we recall how Kubernetes is handling Pods, then we remember that Pods are created by posting a Pod specification to the Kubernetes API Server. This specification is then transformed into an object inside the API Server, this object is of a specific type or kind - that is how it's called in the specification. A Pod is of the type <code>Pod</code>. Controllers within Kubernetes know how to handle these Pod objects. Thus once a new Pod object is seen, those controllers perform the necessary actions to bring the Pod alive, and to match the required state.</p> <p>This same mechanism is used by KubeVirt. Thus KubeVirt delivers three things to provide the new functionality:</p> <ol> <li>Additional types - so called Custom Resource Definition (CRD) - are added to the Kubernetes API</li> <li>Additional controllers for cluster wide logic associated with these new types</li> <li>Additional daemons for node specific logic associated with new types</li> </ol> <p>Once all three steps have been completed, you are able to</p> <ul> <li>create new objects of these new types in Kubernetes (VMIs in our   case)</li> <li>and the new controllers take care to get the VMIs scheduled on some host,</li> <li>and a daemon - the <code>virt-handler</code> - is taking care of a host - alongside the   <code>kubelet</code> - to launch the VMI and configure it until it matches the required   state.</li> </ul> <p>One final note; both controllers and daemons are running as Pods (or similar) on top of the Kubernetes cluster, and are not installed alongside it. The type is - as said before - even defined inside the Kubernetes API server. This allows users to speak to Kubernetes, but modify VMIs.</p> <p>The following diagram illustrates how the additional controllers and daemons communicate with Kubernetes and where the additional types are stored:</p> <p></p> <p>And a simplified version:</p> <p></p>"},{"location":"architecture/#application-layout","title":"Application Layout","text":"<ul> <li>Cluster</li> <li>KubeVirt Components<ul> <li>virt-controller</li> <li>virt-handler</li> <li>libvirtd</li> <li>\u2026</li> </ul> </li> <li>KubeVirt Managed Pods<ul> <li>VMI Foo</li> <li>VMI Bar</li> <li>\u2026</li> </ul> </li> <li>KubeVirt Custom Resources<ul> <li>VirtualMachine (VM) Foo     -&gt; VirtualMachineInstance (VMI) Foo</li> <li>VirtualMachineInstanceReplicaSet (VMIRS) Bar     -&gt; VirtualMachineInstance (VMI) Bar</li> </ul> </li> </ul> <p>VirtualMachineInstance (VMI) is the custom resource that represents the basic ephemeral building block of an instance. In a lot of cases this object won't be created directly by the user but by a high level resource. High level resources for VMI can be:</p> <ul> <li>VirtualMachine (VM) - StateFul VM that can be stopped and started while keeping the VM data and state.</li> <li>VirtualMachineInstanceReplicaSet (VMIRS) - Similar to pods ReplicaSet, a group of ephemeral VMIs with similar configuration defined in a template.</li> </ul>"},{"location":"architecture/#native-workloads","title":"Native Workloads","text":"<p>KubeVirt is deployed on top of a Kubernetes cluster. This means that you can continue to run your Kubernetes-native workloads next to the VMIs managed through KubeVirt.</p> <p>Furthermore: if you can run native workloads, and you have KubeVirt installed, you should be able to run VM-based workloads, too. For example, Application Operators should not require additional permissions to use cluster features for VMs, compared to using that feature with a plain Pod.</p> <p>Security-wise, installing and using KubeVirt must not grant users any permission they do not already have regarding native workloads. For example, a non-privileged Application Operator must never gain access to a privileged Pod by using a KubeVirt feature.</p>"},{"location":"architecture/#the-razor","title":"The Razor","text":"<p>We love virtual machines, think that they are very important and work hard to make them easy to use in Kubernetes. But even more than VMs, we love good design and modular, reusable components. Quite frequently, we face a dilemma: should we solve a problem in KubeVirt in a way that is best optimized for VMs, or should we take a longer path and introduce the solution to Pod-based workloads too?</p> <p>To decide these dilemmas we came up with the KubeVirt Razor: \"If something is useful for Pods, we should not implement it only for VMs\".</p> <p>For example, we debated how we should connect VMs to external network resources. The quickest way seems to introduce KubeVirt-specific code, attaching a VM to a host bridge. However, we chose the longer path of integrating with Multus and CNI and improving them.</p>"},{"location":"architecture/#virtualmachine","title":"VirtualMachine","text":"<p>A <code>VirtualMachine</code> provides additional management capabilities to a VirtualMachineInstance inside the cluster. That includes:</p> <ul> <li> <p>API stability</p> </li> <li> <p>Start/stop/restart capabilities on the controller level</p> </li> <li> <p>Offline configuration change with propagation on     VirtualMachineInstance recreation</p> </li> <li> <p>Ensure that the VirtualMachineInstance is running if it should be     running</p> </li> </ul> <p>It focuses on a 1:1 relationship between the controller instance and a virtual machine instance. In many ways it is very similar to a StatefulSet with <code>spec.replica</code> set to <code>1</code>.</p>"},{"location":"architecture/#how-to-use-a-virtualmachine","title":"How to use a VirtualMachine","text":"<p>A VirtualMachine will make sure that a VirtualMachineInstance object with an identical name will be present in the cluster, if <code>spec.running</code> is set to <code>true</code>. Further it will make sure that a VirtualMachineInstance will be removed from the cluster if <code>spec.running</code> is set to <code>false</code>.</p> <p>There exists a field <code>spec.runStrategy</code> which can also be used to control the state of the associated VirtualMachineInstance object. To avoid confusing and contradictory states, these fields are mutually exclusive.</p> <p>An extended explanation of <code>spec.runStrategy</code> vs <code>spec.running</code> can be found in Run Strategies</p>"},{"location":"architecture/#starting-and-stopping","title":"Starting and stopping","text":"<p>After creating a VirtualMachine it can be switched on or off like this:</p> <pre><code># Start the virtual machine:\nvirtctl start vm\n\n# Stop the virtual machine:\nvirtctl stop vm\n</code></pre> <p><code>kubectl</code> can be used too:</p> <pre><code># Start the virtual machine:\nkubectl patch virtualmachine vm --type merge -p \\\n    '{\"spec\":{\"running\":true}}'\n\n# Stop the virtual machine:\nkubectl patch virtualmachine vm --type merge -p \\\n    '{\"spec\":{\"running\":false}}'\n</code></pre> <p>Find more details about a VM's life-cycle in the relevant section</p>"},{"location":"architecture/#controller-status","title":"Controller status","text":"<p>Once a VirtualMachineInstance is created, its state will be tracked via <code>status.created</code> and <code>status.ready</code> fields of the VirtualMachine. If a VirtualMachineInstance exists in the cluster, <code>status.created</code> will equal <code>true</code>. If the VirtualMachineInstance is also ready, <code>status.ready</code> will equal <code>true</code> too.</p> <p>If a VirtualMachineInstance reaches a final state but the <code>spec.running</code> equals <code>true</code>, the VirtualMachine controller will set <code>status.ready</code> to <code>false</code> and re-create the VirtualMachineInstance.</p> <p>Additionally, the <code>status.printableStatus</code> field provides high-level summary information about the state of the VirtualMachine. This information is also displayed when listing VirtualMachines using the CLI:</p> <pre><code>$ kubectl get virtualmachines\nNAME     AGE   STATUS    VOLUME\nvm1      4m    Running\nvm2      11s   Stopped\n</code></pre> <p>Here's the list of states currently supported and their meanings. Note that states may be added/removed in future releases, so caution should be used if consumed by automated programs.</p> <ul> <li>Stopped: The virtual machine is currently stopped and isn't expected to start.</li> <li>Provisioning: Cluster resources associated with the virtual machine (e.g., DataVolumes) are being provisioned and prepared.</li> <li>Starting: The virtual machine is being prepared for running.</li> <li>Running: The virtual machine is running.</li> <li>Paused: The virtual machine is paused.</li> <li>Migrating: The virtual machine is in the process of being migrated to another host.</li> <li>Stopping: The virtual machine is in the process of being stopped.</li> <li>Terminating: The virtual machine is in the process of deletion, as well as its associated resources (VirtualMachineInstance, DataVolumes, \u2026).</li> <li>Unknown: The state of the virtual machine could not be obtained, typically due to an error in communicating with the host on which it's running.</li> </ul>"},{"location":"architecture/#restarting","title":"Restarting","text":"<p>A VirtualMachineInstance restart can be triggered by deleting the VirtualMachineInstance. This will also propagate configuration changes from the template in the VirtualMachine:</p> <pre><code># Restart the virtual machine (you delete the instance!):\nkubectl delete virtualmachineinstance vm\n</code></pre> <p>To restart a VirtualMachine named vm using virtctl:</p> <pre><code>$ virtctl restart vm\n</code></pre> <p>This would perform a normal restart for the VirtualMachineInstance and would reschedule the VirtualMachineInstance on a new virt-launcher Pod</p> <p>To force restart a VirtualMachine named vm using virtctl:</p> <pre><code>$ virtctl restart vm --force --grace-period=0\n</code></pre> <p>This would try to perform a normal restart, and would also delete the virt-launcher Pod of the VirtualMachineInstance with setting GracePeriodSeconds to the seconds passed in the command.</p> <p>Currently, only setting grace-period=0 is supported.</p> <p>Note</p> <p>Force restart can cause data corruption, and should be used in cases of kernel panic or VirtualMachine being unresponsive to normal restarts.</p>"},{"location":"architecture/#fencing-considerations","title":"Fencing considerations","text":"<p>A VirtualMachine will never restart or re-create a VirtualMachineInstance until the current instance of the VirtualMachineInstance is deleted from the cluster.</p>"},{"location":"architecture/#exposing-as-a-service","title":"Exposing as a Service","text":"<p>A VirtualMachine can be exposed as a service. The actual service will be available once the VirtualMachineInstance starts without additional interaction.</p> <p>For example, exposing SSH port (22) as a <code>ClusterIP</code> service using <code>virtctl</code> after the VirtualMachine was created, but before it started:</p> <pre><code>$ virtctl expose virtualmachine vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>All service exposure options that apply to a VirtualMachineInstance apply to a VirtualMachine.</p> <p>See Service Objects for more details.</p>"},{"location":"architecture/#when-to-use-a-virtualmachine","title":"When to use a VirtualMachine","text":""},{"location":"architecture/#when-api-stability-is-required-between-restarts","title":"When API stability is required between restarts","text":"<p>A <code>VirtualMachine</code> makes sure that VirtualMachineInstance API configurations are consistent between restarts. A classical example are licenses which are bound to the firmware UUID of a virtual machine. The <code>VirtualMachine</code> makes sure that the UUID will always stay the same without the user having to take care of it.</p> <p>One of the main benefits is that a user can still make use of defaulting logic, although a stable API is needed.</p>"},{"location":"architecture/#when-config-updates-should-be-picked-up-on-the-next-restart","title":"When config updates should be picked up on the next restart","text":"<p>If the VirtualMachineInstance configuration should be modifiable inside the cluster and these changes should be picked up on the next VirtualMachineInstance restart. This means that no hotplug is involved.</p>"},{"location":"architecture/#when-you-want-to-let-the-cluster-manage-your-individual-virtualmachineinstance","title":"When you want to let the cluster manage your individual VirtualMachineInstance","text":"<p>Kubernetes as a declarative system can help you to manage the VirtualMachineInstance. You tell it that you want this VirtualMachineInstance with your application running, the VirtualMachine will try to make sure it stays running.</p> <p>Note</p> <p>The current belief is that if it is defined that the VirtualMachineInstance should be running, it should be running. This is different from many classical virtualization platforms, where VMs stay down if they were switched off. Restart policies may be added if needed. Please provide your use-case if you need this!</p>"},{"location":"architecture/#example","title":"Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-cirros\n  name: vm-cirros\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-cirros\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 64M\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - name: containerdisk\n        containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n      - cloudInitNoCloud:\n          userDataBase64: IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK\n        name: cloudinitdisk\n</code></pre> <p>Saving this manifest into <code>vm.yaml</code> and submitting it to Kubernetes will create the controller instance:</p> <pre><code>$ kubectl create -f vm.yaml\nvirtualmachine \"vm-cirros\" created\n</code></pre> <p>Since <code>spec.running</code> is set to <code>false</code>, no vmi will be created:</p> <pre><code>$ kubectl get vmis\nNo resources found.\n</code></pre> <p>Let's start the VirtualMachine:</p> <pre><code>$ virtctl start vm vm-cirros\n</code></pre> <p>As expected, a VirtualMachineInstance called <code>vm-cirros</code> got created:</p> <pre><code>$ kubectl describe vm vm-cirros\nName:         vm-cirros\nNamespace:    default\nLabels:       kubevirt.io/vm=vm-cirros\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachine\nMetadata:\n  Cluster Name:\n  Creation Timestamp:  2018-04-30T09:25:08Z\n  Generation:          0\n  Resource Version:    6418\n  Self Link:           /apis/kubevirt.io/v1/namespaces/default/virtualmachines/vm-cirros\n  UID:                 60043358-4c58-11e8-8653-525500d15501\nSpec:\n  Running:  true\n  Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        Kubevirt . Io / Ovmi:  vm-cirros\n    Spec:\n      Domain:\n        Devices:\n          Disks:\n            Disk:\n              Bus:        virtio\n            Name:         containerdisk\n            Volume Name:  containerdisk\n            Disk:\n              Bus:        virtio\n            Name:         cloudinitdisk\n            Volume Name:  cloudinitdisk\n        Machine:\n          Type:\n        Resources:\n          Requests:\n            Memory:                      64M\n      Termination Grace Period Seconds:  0\n      Volumes:\n        Name:  containerdisk\n        Registry Disk:\n          Image:  kubevirt/cirros-registry-disk-demo:latest\n        Cloud Init No Cloud:\n          User Data Base 64:  IyEvYmluL3NoCgplY2hvICdwcmludGVkIGZyb20gY2xvdWQtaW5pdCB1c2VyZGF0YScK\n        Name:                 cloudinitdisk\nStatus:\n  Created:  true\n  Ready:    true\nEvents:\n  Type    Reason            Age   From                              Message\n  ----    ------            ----  ----                              -------\n  Normal  SuccessfulCreate  15s   virtualmachine-controller  Created virtual machine: vm-cirros\n</code></pre>"},{"location":"architecture/#kubectl-commandline-interactions","title":"Kubectl commandline interactions","text":"<p>Whenever you want to manipulate the VirtualMachine through the commandline you can use the kubectl command. The following are examples demonstrating how to do it.</p> <pre><code>    # Define a virtual machine:\n    kubectl create -f vm.yaml\n\n    # Start the virtual machine:\n    kubectl patch virtualmachine vm --type merge -p \\\n        '{\"spec\":{\"running\":true}}'\n\n    # Look at virtual machine status and associated events:\n    kubectl describe virtualmachine vm\n\n    # Look at the now created virtual machine instance status and associated events:\n    kubectl describe virtualmachineinstance vm\n\n    # Stop the virtual machine instance:\n    kubectl patch virtualmachine vm --type merge -p \\\n        '{\"spec\":{\"running\":false}}'\n\n    # Restart the virtual machine (you delete the instance!):\n    kubectl delete virtualmachineinstance vm\n\n    # Implicit cascade delete (first deletes the virtual machine and then the virtual machine instance)\n    kubectl delete virtualmachine vm\n\n    # Explicit cascade delete (first deletes the virtual machine and then the virtual machine instance)\n    kubectl delete virtualmachine vm --cascade=true\n\n    # Orphan delete (The running virtual machine is only detached, not deleted)\n    # Recreating the virtual machine would lead to the adoption of the virtual machine instance\n    kubectl delete virtualmachine vm --cascade=false\n</code></pre>"},{"location":"contributing/","title":"Contributing","text":"<p>Welcome!! And thank you for taking the first step to contributing to the KubeVirt project. On this page you should be able to find all the information required to get started on your contirbution journey, as well as information on how to become a community member and grow into roles of responsibility. </p> <p>If you think something might be missing from this page, please help us by raising a bug!</p>"},{"location":"contributing/#prerequisites","title":"Prerequisites","text":"<p>Reviewing the following will prepare you for contributing:</p> <ul> <li>If this is your first step in the world of open source, consider reading the CNCF's Start Contributing to Open Source page for an introduction to key concepts.</li> <li>You should be comfortable with git. Most contributions follow the GitHub workflow of fork, branch, commit, open pull request, review changes, and merge to work effectively in the KubeVirt community.  If you're new to git, git-scm.com has a nice set of tutorials.</li> <li>Familiarize yourself with the various repositories of the KubeVirt GitHub organization.</li> <li>Try the one of our quick start labs on killercoda, minikube, or kind.</li> <li>See the \"Other ways to contribute\" section below.</li> </ul> <p>For code contributors:</p> <ul> <li>You need to be familiar with writing code in golang.  See the golang tour to familiarize yourself.</li> <li>To contribute to the core of the project, read the Developer contribution page and the getting started page in the kubevirt/kubevirt repo.</li> <li>Alternatively, to contribute to its storage management add-on, check out the kubevirt/containerized-data-importer (CDI) repo, and their contribution page.</li> </ul>"},{"location":"contributing/#your-first-contribution","title":"Your first contribution","text":"<p>The following will help you decide where to start:</p> <ul> <li>Check a repository issues list and label <code>good-first-issue</code> for issues that make good entry points.</li> <li>Open a pull request using GitHub to documentation. The tutorials found here can be helpful https://lab.github.com/</li> <li>Review a pull request from other community members for accuracy and language.</li> </ul>"},{"location":"contributing/#important-community-resources","title":"Important community resources","text":"<p>You should familiarize yourself with the following documents, which are critical to being a member of the community:</p> <ul> <li>Code of Conduct: Everyone is expected to abide by the CoC to ensure an open and welcoming environment. Our CoC is based off the CNCF Code of conduct which also has a variety of translations. </li> <li>Our community membership policy: How to become a member and grow into roles of responsibility.</li> <li>Project governance: Project Maintainer responsibilities.</li> </ul>"},{"location":"contributing/#other-ways-to-contribute","title":"Other ways to contribute","text":"<ul> <li>Visit the KubeVirt community page, participate on Twitter or Slack, learn about local meetups and events.</li> <li>Visit the KubeVirt website repository and submit a blog post, case study or lab.</li> <li>Visit the KubeVirt user-guide repository and find feature documentation that could use an update.</li> </ul>"},{"location":"quickstarts/","title":"Quickstarts","text":""},{"location":"quickstarts/#quickstart-guides","title":"Quickstart Guides","text":"<p>Killercoda provides an interactive environment for exploring KubeVirt scenarios:</p> <ul> <li>KubeVirt on killercoda</li> </ul> <p>Guides for deploying KubeVirt with different Kubernetes tools:</p> <ul> <li> <p>KubeVirt on minikube</p> </li> <li> <p>KubeVirt on kind</p> </li> <li> <p>KubeVirt on cloud providers</p> </li> </ul>"},{"location":"release_notes/","title":"KubeVirt release notes","text":""},{"location":"release_notes/#v130","title":"v1.3.0","text":"<p>Release on: Wed Jul 17 2024</p> <p>KubeVirt v1.3 is built for Kubernetes v1.30 and additionally supported for the previous two versions. See the KubeVirt support matrix for more information.</p> <p>To see the list of fine folks who contributed to this release, see the KubeVirt release tag for v1.3.0.</p>"},{"location":"release_notes/#api-change","title":"API change","text":"<ul> <li>[PR #11156] [nunnatsa] Move some verification from the VMI create validation webhook to the CRD</li> <li>[PR #11500] [iholder101] Support HyperV Passthrough: automatically use all available HyperV features</li> <li>[PR #11641] [alicefr] Add kubevirt.io/testWorkloadUpdateMigrationAbortion annotation and a mechanism to abort workload updates</li> <li>[PR #11700] [alicefr] Add the updateVolumeStrategy field</li> <li>[PR #11729] [lyarwood] <code>spreadOptions</code> have been introduced to preferences in order to allow for finer grain control of the <code>preferSpread</code> <code>preferredCPUTopology</code>. This includes the ability to now spread vCPUs across guest visible sockets, cores and threads.</li> <li>[PR #10545] [lyarwood] <code>ControllerRevisions</code> containing instance types and preferences are now upgraded to their latest available version when the <code>VirtualMachine</code> owning them is resync'd by <code>virt-controller</code>.</li> <li>[PR #11955] [mhenriks] Introduce snapshot.kibevirt.io/v1beta1</li> <li>[PR #11956] [mhenriks] Introduce export.kibevirt.io/v1beta1</li> <li>[PR #11533] [alicefr] Implement volume migration and introduce the migration updateVolumesStrategy field</li> <li>[PR #10490] [jschintag] Add support for building and running kubevirt on s390x.</li> <li>[PR #11330] [jean-edouard] More information in the migration state of VMI / migration objects</li> <li>[PR #11773] [jean-edouard] Persistent TPM/UEFI will use the default storage class if none is specified in the CR.</li> </ul>"},{"location":"release_notes/#bug-fix","title":"Bug fix","text":"<ul> <li>[PR #12296] [orelmisan] Network binding plugins: Enable the ability to specify compute memory overhead</li> <li>[PR #12279] [fossidhelm] Fix: persistent tpm can be used with vmis containing dots in their name</li> <li>[PR #12226] [awels] Virt export route has an edge termination of redirect</li> <li>[PR #12249] [machadovilaca] Fix missing performance metrics for VMI resources</li> <li>[PR #12237] [vladikr] Only a single vgpu display option with ramfb will be configured per VMI.</li> <li>[PR #12064] [akalenyu] BugFix: Graceful deletion skipped for any delete call to the VM (not VMI) resource</li> <li>[PR #11996] [ShellyKa13] BugFix: Fix restore panic in case of volumesnapshot missing</li> <li>[PR #11973] [fossedihelm] Bug fix: Correctly reflect RestartRequired condition</li> <li>[PR #11922] [alromeros] Bugfix: Fix VM manifest rendering in export controller</li> <li>[PR #11367] [alromeros] Bugfix: Allow vmexport download redirections by printing logs into stderr</li> <li>[PR #11219] [alromeros] Bugfix: Improve handling of IOThreads with incompatible buses</li> <li>[PR #11372] [xpivarc] Bug-fix: Fix nil panic if VM update fails</li> <li>[PR #11267] [mhenriks] BugFix: Ensure DataVolumes created by virt-controller (DataVolumeTemplates) are recreated and owned by the VM in the case of DR and backup/restore.</li> <li>[PR #10900] [KarstenB] BugFix: Fixed incorrect APIVersion of APIResourceList</li> <li>[PR #11306] [fossedihelm] fix(ksm): set the <code>kubevirt.io/ksm-enabled</code> node label to true if the ksm is managed by KubeVirt, instead of reflect the actual ksm value.</li> <li>[PR #11264] [machadovilaca] Fix perfscale buckets error</li> <li>[PR #11058] [fossedihelm] fix(vmclone): delete vmclone resource when the target vm is deleted</li> <li>[PR #11265] [xpivarc] Bug fix: VM controller doesn't corrupt its cache anymore</li> <li>[PR #11205] [akalenyu] Fix migration breaking in case the VM has an rng device after hotplugging a block volume on cgroupsv2</li> <li>[PR #11051] [alromeros] Bugfix: Improve error reporting when fsfreeze fails</li> <li>[PR #12016] [acardace] fix starting VM with Manual RunStrategy</li> <li>[PR #11963] [acardace] Fix RerunOnFailure RunStrategy</li> <li>[PR #11718] [fossedihelm] Fix: SEV methods in client-go now satisfy the proxy server configuration, if provided</li> <li>[PR #12122] [kubevirt-bot] Fix VMPools when <code>LiveUpdate</code> as <code>vmRolloutStrategy</code> is used.</li> <li>[PR #12201] [kubevirt-bot] fix RerunOnFailure stuck in Provisioning</li> <li>[PR #12151] [kubevirt-bot] Bugfix: Implement retry mechanism in export server and vmexport</li> <li>[PR #12146] [kubevirt-bot] Memory Hotplug fixes and stabilization</li> <li>[PR #12185] [kubevirt-bot] VMs with a single socket and NetworkInterfaceMultiqueue enabled require a restart to hotplug additional CPU sockets.</li> <li>[PR #12171] [kubevirt-bot] <code>PreferredDiskDedicatedIoThread</code> is now only applied to <code>virtio</code> disk devices</li> </ul>"},{"location":"release_notes/#deprecation","title":"Deprecation","text":"<ul> <li>[PR #11701] [EdDev] The SLIRP core binding is deprecated and removed.</li> <li>[PR #11901] [EdDev] The 'macvtap' core network binding is discontinued and removed.</li> <li>[PR #11915] [ormergi] The 'passt' core network binding is discontinued and removed.</li> <li>[PR #11404] [avlitman] KubeVirtComponentExceedsRequestedCPU and KubeVirtComponentExceedsRequestedMemory alerts are deprecated; they do not indicate a genuine issue.</li> </ul>"},{"location":"release_notes/#sig-compute","title":"SIG-compute","text":"<ul> <li>[PR #11498] [acardace] Allow to hotplug memory for VMs with memory limits set</li> <li>[PR #11479] [vladikr] virtual machines instance will no longer be stuck in an irrecoverable state after an interrupted postcopy migration. Instead, these will fail and could be restarted again.</li> <li>[PR #11685] [fossedihelm] Updated go version of the client-go to 1.21</li> <li>[PR #11344] [aerosouund] Refactor device plugins to use a base plugin and define a common interface</li> <li>[PR #12025] [fossedihelm] Add descheduler compatibility</li> <li>[PR #12109] [acardace] Support Memory Hotplug with Hugepages</li> <li>[PR #11883] [orelmisan] Restart of a VM is required when the CPU socket count is reduced</li> <li>[PR #11655] [acardace] Allow to hotplug vcpus for VMs with CPU requests and/or limits set</li> <li>[PR #11455] [lyarwood] <code>LiveUpdates</code>  of VMs using instance types are now supported with the same caveats as when making changes to a vanilla VM.</li> <li>[PR #11681] [lyarwood] The <code>CommonInstancetypesDeployment</code> feature and gate are retrospectively moved to Beta from the 1.2.0 release.</li> <li>[PR #11648] [kubevirt-bot] Updated common-instancetypes bundles to v1.0.0</li> <li>[PR #12240] [kubevirt-bot] Updated common-instancetypes bundles to v1.0.1</li> </ul>"},{"location":"release_notes/#sig-storage","title":"SIG-storage","text":"<ul> <li>[PR #11095] [ShellyKa13] Expose volumesnapshot error in vmsnapshot object</li> <li>[PR #11312] [alromeros] Improve handling of export resources in virtctl vmexport</li> <li>[PR #11770] [alicefr] Fix the live updates for volumes and disks</li> <li>[PR #11957] [mhenriks] snapshot: Ignore unfreeze error if VMSnapshot deleting</li> </ul>"},{"location":"release_notes/#sig-network","title":"SIG-network","text":"<ul> <li>[PR #11653] [EdDev] Build the <code>passt</code>custom CNI binary statically, for the <code>passt</code> network binding plugin.</li> <li>[PR #11678] [Vicente-Cheng] Improve the handling of ordinal pod interface name for upgrade</li> <li>[PR #11618] [AlonaKaplan] Extend network binding plugin to support device-info DownwardAPI.</li> <li>[PR #11256] [andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 10.0.0 and QEMU 8.2.0.</li> <li>[PR #11788] [ormergi] The network-info annotation is now used for mapping between SR-IOV network and the underlying device PCI address</li> <li>[PR #11659] [iholder101] Require scheduling infra components onto control-plane nodes</li> <li>[PR #12079] [EdDev] Network hotplug feature is declared as Beta.</li> </ul>"},{"location":"release_notes/#sig-scale","title":"SIG-scale","text":"<ul> <li>[PR #11272] [dharmit] Make 'image' field in hook sidecar annotation optional.</li> <li>[PR #11676] [machadovilaca] Rename rest client metrics to include kubevirt prefix</li> <li>[PR #11387] [alaypatel07] add perf-scale benchmarks for release v1.2</li> </ul>"},{"location":"release_notes/#monitoring","title":"Monitoring","text":"<ul> <li>[PR #11307] [machadovilaca] Add e2e tests for metrics</li> <li>[PR #11294] [machadovilaca] Fix kubevirt_vm_created_total being broken down by virt-api pod</li> <li>[PR #11557] [avlitman] New memory statistics added named kubevirt_memory_delta_from_requested_bytes</li> <li>[PR #11283] [assafad] Collect VMI OS info from the Guest agent as <code>kubevirt_vmi_phase_count</code> metric labels</li> <li>[PR #11906] [machadovilaca] Create kubevirt_vmi_info metric</li> <li>[PR #11934] [assafad] Add kubevirt_vmi_last_connection_timestamp_seconds metric</li> <li>[PR #12000] [machadovilaca] Create kubevirt_vmi_launcher_memory_overhead_bytes metric</li> <li>[PR #11484] [jcanocan] Reduce the downwardMetrics server maximum number of request per second to 1.</li> </ul>"},{"location":"release_notes/#uncategorized","title":"Uncategorized","text":"<ul> <li>[PR #12132] [kubevirt-bot] Introduce validatingAdmissionPolicy to restrict node patches on virt-handler</li> <li>[PR #12009] [xpivarc] By enabling NodeRestriction feature gate, Kubevirt now authorize virt-handler's requests to modify VMs.</li> <li>[PR #12089] [jean-edouard] Less privileged virt-operator ClusterRole</li> <li>[PR #11969] [iholder101] Infra components control-plane nodes NoSchedule toleration</li> <li>[PR #11835] [talcoh2x] add Intel Gaudi to adopters.</li> <li>[PR #11790] [aburdenthehand] Re-adding Cloudflare to our ADOPTERS list</li> <li>[PR #11331] [anjuls] add cloudraft to adopters.</li> <li>[PR #11942] [ido106] Update virtctl to use v1beta1 endpoint for both regular and async image upload</li> <li>[PR #11908] [victortoso] sidecar-shim: allow stderr log from binary hooks</li> <li>[PR #11846] [victortoso] SMBios sidecar can be built out-of-tree</li> <li>[PR #11482] [brianmcarey] Build KubeVirt with go v1.22.2</li> <li>[PR #11470] [brianmcarey] Build KubeVirt with Go version 1.21.8</li> <li>[PR #12097] [fossedihelm] Bump k8s deps to 0.30.0</li> <li>[PR #11416] [dhiller] emission of k8s logs when using programmatic focus with <code>FIt</code></li> <li>[PR #11183] [dhiller] Extend OWNERS for sig-buildsystem</li> <li>[PR #11149] [0xFelix] virtctl: It is possible to import volumes from GCS when creating a VM now</li> <li>[PR #11146] [RamLavi] node-labeller: Remove obsolete functionalities</li> </ul>"},{"location":"release_notes/#v120","title":"v1.2.0","text":"<p>Released on: Tue Mar 05 2024</p> <p>KubeVirt v1.2 is built for Kubernetes v1.29 and additionally supported for the previous two versions. See the KubeVirt support matrix for more information.</p>"},{"location":"release_notes/#api-change_1","title":"API change","text":"<ul> <li>[PR #11064] [AlonaKaplan] Introduce a new API to mark a binding plugin as migratable.</li> <li>[PR #10970] [alromeros] Expose fs disk information via GuestOsInfo</li> <li>[PR #10905] [tiraboschi] Aggregate DVs conditions on VMI (and so VM)</li> <li>[PR #10872] [RamLavi] IsolateEmulatorThread: Add cluster-wide parity completion setting</li> <li>[PR #10846] [RamLavi] Change vm.status.PrintableStatus default value to \"Stopped\"</li> <li>[PR #10774] [victortoso] Windows offline activation with ACPI SLIC table</li> <li>[PR #10732] [AlonaKaplan] Extend kubvirt CR by adding domain attachment option to the network binding plugin API.</li> <li>[PR #10658] [matthewei] Support \"Clone API\" to filter VirtualMachine.spec.template.annotation and VirtualMachine.spec.template.label</li> </ul>"},{"location":"release_notes/#bug-fix_1","title":"Bug fix","text":"<ul> <li>[PR #11271] [kubevirt-bot] Bug fix: VM controller doesn't corrupt its cache anymore</li> <li>[PR #11242] [kubevirt-bot] Fix migration breaking in case the VM has an rng device after hotplugging a block volume on cgroupsv2</li> <li>[PR #11069] [ormergi] Bug fix: Packet drops during the initial phase of VM live migration https://issues.redhat.com/browse/CNV-28040</li> <li>[PR #11065] [fossedihelm] fix(vmclone): Generate VM patches from vmsnapshotcontent, instead of current VM</li> <li>[PR #11050] [fossedihelm] restrict default cluster role to authenticated only users</li> <li>[PR #11047] [jschintag] Fix potential crash when trying to list USB devices on host without any</li> <li>[PR #10963] [alromeros] Bugfix: Reject volume exports when no output is specified</li> <li>[PR #10916] [orelmisan] Fix the value of VMI <code>Status.GuestOSInfo.Version</code></li> <li>[PR #10888] [fossedihelm] [Bugfix] Clone VM with WaitForFirstConsumer binding mode PVC now works.</li> <li>[PR #10860] [akalenyu] BugFix: Double cloning with filter fails  isolateEmulatorThread feature (BZ#2228103).</li> <li>[PR #10845] [orelmisan] Reject VirtualMachineClone creation when target name is equal to source name</li> <li>[PR #10753] [victortoso] Fixes  permission when using USB host passthrough</li> <li>[PR #10747] [acardace] Fix KubeVirt for CRIO 1.28 by using checksums to verify containerdisks when migrating VMIs</li> <li>[PR #10699] [qinqon] virt-launcher: fix qemu non root log path</li> <li>[PR #10689] [akalenyu] BugFix: cgroupsv2 device allowlist is bound to virt-handler internal state/block disk device overwritten on hotplug</li> <li>[PR #10593] [RamLavi] Fixes SMT Alignment Error in virt-launcher pod by optimizing</li> </ul>"},{"location":"release_notes/#deprecation_1","title":"Deprecation","text":"<ul> <li>[PR #10924] [AlonaKaplan] Deprecate macvtap</li> </ul>"},{"location":"release_notes/#sig-compute_1","title":"SIG-compute","text":"<ul> <li>[PR #11054] [jean-edouard] New cluster-wide <code>vmRolloutStrategy</code> setting to define whether changes to VMs should either be always staged or live-updated when possible.</li> <li>[PR #11001] [fossedihelm] Allow <code>kubevirt.io:default</code> clusterRole to get,list kubevirts</li> <li>[PR #10961] [jcanocan] Reduced VM rescheduling time on node failure</li> <li>[PR #10918] [orelmisan] VMClone: Emit an event in case restore creation fails</li> <li>[PR #10898] [matthewei] vmi status's guestOsInfo adds <code>Machine</code></li> <li>[PR #10840] [acardace] Requests/Limits can now be configured when using CPU/Memory hotplug</li> <li>[PR #10839] [RamLavi] Change second emulator thread assign strategy to best-effort.</li> <li>[PR #10809] [orelmisan] Source virt-launcher: Log migration info by default</li> <li>[PR #10783] [RamLavi] Support multiple CPUs in Housekeeping cgroup</li> <li>[PR #10571] [tiraboschi] vmi memory footprint increase by 35M when guest serial console logging is turned on (default on).</li> </ul>"},{"location":"release_notes/#sig-storage_1","title":"SIG-storage","text":"<ul> <li>[PR #10657] [germag] Exposing Filesystem Persistent Volumes (PVs)  to the VM using unprivilege virtiofsd.</li> <li>[PR #10529] [alromeros] Allow LUN disks to be hotplugged</li> </ul>"},{"location":"release_notes/#sig-network_1","title":"SIG-network","text":"<ul> <li>[PR #10981] [AlonaKaplan] Report IP of interfaces using network binding plugin.</li> <li>[PR #10866] [AlonaKaplan] Raise an error in case passt feature gate or API are used.</li> <li>[PR #10800] [AlonaKaplan] Support macvtap as a binding plugin</li> <li>[PR #10425] [ormergi] Introduce network binding plugin for Passt networking, interfacing with Kubevirt new network binding plugin API.</li> </ul>"},{"location":"release_notes/#sig-infra","title":"SIG-infra","text":"<ul> <li>[PR #11025] [0xFelix] Allow unprivileged users read-only access to VirtualMachineCluster{Instancetypes,Preferences} by default.</li> <li>[PR #10922] [kubevirt-bot] Updated common-instancetypes bundles to v0.4.0</li> </ul>"},{"location":"release_notes/#monitoring_1","title":"Monitoring","text":"<ul> <li>[PR #10982] [machadovilaca] Refactor monitoring metrics</li> <li>[PR #10962] [machadovilaca] Update monitoring file structure</li> <li>[PR #10853] [machadovilaca] Refactor monitoring collectors</li> <li>[PR #10700] [machadovilaca] Refactor monitoring alerts</li> <li>[PR #10693] [machadovilaca] Remove MigrateVmiDiskTransferRateMetric</li> <li>[PR #10651] [machadovilaca] Refactor monitoring  recording-rules</li> <li>[PR #10570] [machadovilaca] Fix LowKVMNodesCount not firing</li> <li>[PR #10418] [machadovilaca] Add total VMs created metric</li> </ul>"},{"location":"release_notes/#uncategorized_1","title":"Uncategorized","text":"<ul> <li>[PR #11144] [0xFelix] virtctl: Specifying size when creating a VM and using --volume-import to clone a PVC or a VolumeSnapshot is optional now</li> <li>[PR #11122] [brianmcarey] Update runc dependency to v1.1.12</li> <li>[PR #11068] [brianmcarey] Update container base image to use current stable debian 12 base</li> <li>[PR #10914] [brianmcarey] KubeVirt is now built with go 1.21.5</li> <li>[PR #10879] [brianmcarey] Built with golang 1.20.12</li> <li>[PR #10863] [dhiller] Remove year from generated code copyright</li> <li>[PR #10787] [matthewei] virtctl support to add template label and annotation filters</li> <li>[PR #10720] [awels] Restored hotplug attachment pod request/limit to original value</li> <li>[PR #10637] [dharmit] Functional tests for sidecar hook with ConfigMap</li> <li>[PR #10615] [orelmisan] Remove leftover NonRoot feature gate</li> <li>[PR #10598] [alicefr] Add PVC option to the hook sidecars for supplying additional debugging tools</li> <li>[PR #10596] [mhenriks] Disable HTTP/2 to mitigate CVE-2023-44487</li> <li>[PR #10582] [orelmisan] Remove leftover NonRootExperimental feature gate</li> <li>[PR #10567] [awels] Attachment pod creation is now rate limited</li> <li>[PR #10526] [cfilleke]  Documents steps to build the KubeVirt builder container</li> <li>[PR #10479] [dharmit] Ability to run scripts through hook sidecardevice</li> <li>[PR #10244] [hshitomi] Added \u201cadm\u201d subcommand under \u201cvirtctl\u201d, and \u201clog-verbosity\" subcommand under \u201cadm\u201d. The log-verbosity command is: to show the log verbosity of one or more components, to set the log verbosity of one or more components, and to reset the log verbosity of all components (reset to the default verbosity (2)).</li> <li>[PR #10046] [victortoso] Add v1alpha3 for hooks and fix migration when using sidecars</li> </ul>"},{"location":"release_notes/#v110","title":"v1.1.0","text":"<p>Released on: Tue Nov 07 2023</p>"},{"location":"release_notes/#api-change_2","title":"API change","text":"<ul> <li>[#10568][ormergi] Network binding plugin API support CNIs, new integration point on virt-launcher pod creation.</li> <li>[#10309][lyarwood] cluster-wide <code>common-instancetypes</code> resources can now deployed by <code>virt-operator</code> using the <code>CommonInstancetypesDeploymentGate</code> feature gate.</li> <li>[#10463][0xFelix] VirtualMachines: Introduce InferFromVolumeFailurePolicy in Instancetype- and PreferenceMatchers</li> <li>[#10447][fossedihelm] Add a Feature Gate to KV CR to automatically set memory limits when a resource quota with memory limits is associated to the creation namespace</li> <li>[#10477][jean-edouard] Dynamic KSM enabling and configuration</li> <li>[#10110][tiraboschi] Stream guest serial console logs from a dedicated container</li> <li>[#10015][victortoso] Implements USB host passthrough in permittedHostDevices of KubeVirt CRD</li> <li>[#10184][acardace] Add memory hotplug feature</li> <li>[#10231][kvaps] Propogate public-keys to cloud-init NoCloud meta-data</li> <li>[#9673][germag] DownwardMetrics: Expose DownwardMetrics through virtio-serial channel.</li> <li>[#10086][vladikr] allow live updating VM affinity and node selector</li> <li>[#10272][ormergi] Introduce network binding plugin for Slirp networking, interfacing with Kubevirt new network binding plugin API.</li> <li>[#10284][AlonaKaplan] Introduce an API for network binding plugins. The feature is behind \"NetworkBindingPlugins\" gate.</li> <li>[#10101][acardace] Deprecate <code>spec.config.machineType</code> in KubeVirt CR.</li> <li>[#9878][jean-edouard] The EFI NVRAM can now be configured to persist across reboots</li> <li>[#9932][lyarwood] <code>ControllerRevisions</code> containing <code>instancetype.kubevirt.io</code> <code>CRDs</code> are now decorated with labels detailing specific metadata of the underlying stashed object</li> <li>[#10058][alicefr] Add field errorPolicy for disks</li> <li>[#10004][AlonaKaplan] Hoyplug/unplug interfaces should be done by updating the VM spec template. virtctl and REST API endpoints were removed.</li> <li>[#9896][ormergi] The VM controller now replicates spec interfaces MAC addresses to the corresponding interfaces in the VMI spec.</li> <li>[#7708][VirrageS] <code>nodeSelector</code> and <code>schedulerName</code> fields have been added to VirtualMachineInstancetype spec.</li> <li>[#7197][vasiliy-ul] Experimantal support of SEV attestation via the new API endpoints</li> <li>[#9737][AlonaKaplan] On hotunplug - remove bridge, tap and dummy interface from virt-launcher and the caches (file and volatile) from the node.</li> </ul>"},{"location":"release_notes/#bug-fixes","title":"Bug fixes:","text":"<ul> <li>[#10515][iholder101] Bug-fix: Stop copying VMI spec to VM during snapshots</li> <li>[#10393][iholder101] [Bugfix] [Clone API] Double-cloning is now working as expected.</li> <li>[#10391][awels] BugFix: VMExport now works in a namespace with quotas defined.</li> <li>[#10380][alromeros] Bugfix: Allow image-upload to recover from PendingPopulation phase</li> <li>[#10099][iholder101] Bugfix: target virt-launcher pod hangs when migration is cancelled.</li> <li>[#10165][awels] BugFix: deleting hotplug attachment pod will no longer detach volumes that were not removed.</li> <li>[#10067][iholder101] Bug fix: <code>virtctl create clone</code> marshalling and replacement of <code>kubectl</code> with <code>kubectl virt</code></li> <li>[#9935][xpivarc] Bug fix - correct logging in container disk</li> <li>[#9872][alromeros] Bugfix: Allow lun disks to be mapped to DataVolume sources</li> <li>[#10039][simonyangcj] fix guaranteed qos of virt-launcher pod broken when use virtiofs</li> <li>[#9861][rmohr] Fix the possibility of data corruption when requesting a force-restart via \"virtctl restart\"</li> </ul>"},{"location":"release_notes/#deprecation_2","title":"Deprecation","text":"<ul> <li>[#10486][assafad] Deprecation notice for the metrics listed in the PR. Please update your systems to use the new metrics names.</li> <li>[#9821][sradco] Deprecation notice for the metrics listed in the PR. Please update your systems to use the new metrics names.</li> </ul>"},{"location":"release_notes/#sig-compute_2","title":"SIG-compute","text":"<ul> <li>[#10566][fossedihelm] Add 100Mi of memory overhead for vmi with dedicatedCPU or that wants GuaranteedQos</li> <li>[#10496][fossedihelm] Automatically set cpu limits when a resource quota with cpu limits is associated to the creation namespace and the <code>AutoResourceLimits</code> FeatureGate is enabled</li> <li>[#10543][0xFelix] Clear VM guest memory when ignoring inference failures</li> <li>[#10320][victortoso] sidecar-shim implements PreCloudInitIso hook</li> <li>[#10253][rmohr] Stop trying to create unused directory /var/run/kubevirt-ephemeral-disk in virt-controller</li> <li>[#10050][victortoso] Updating the virt stack: QEMU 8.0.0, libvirt to 9.5.0, edk2 20230524, passt 20230818, libguestfs and guestfs-tools 1.50.1, virtiofsd 1.7.2</li> <li>[#9231][victortoso] Introduces sidecar-shim container image</li> <li>[#10254][rmohr] Don't mark the KubeVirt \"Available\" condition as false on up-to-date and ready but misscheduled virt-handler pods.</li> <li>[#10182][iholder101] Stop considering nodes without <code>kubevirt.io/schedulable</code> label when finding lowest TSC frequency on the cluster</li> <li>[#10056][jean-edouard] UEFI guests now use Bochs display instead of VGA emulation</li> <li>[#10106][acardace] Add boot-menu wait time when starting the VM as paused.</li> </ul>"},{"location":"release_notes/#sig-storage_2","title":"SIG-storage","text":"<ul> <li>[#10532][alromeros] Add --volume-mode flag in image-upload</li> <li>[#10020][akalenyu] Use auth API for DataVolumes, stop importing kubevirt.io/containerized-data-importer</li> <li>[#10400][alromeros] Add new vmexport flags to download raw images, either directly (--raw) or by decompressing (--decompress) them</li> <li>[#10148][alromeros] Add port-forward functionalities to vmexport</li> <li>[#10275][awels] Ensure new hotplug attachment pod is ready before deleting old attachment pod</li> <li>[#10118][akalenyu] Change exportserver default UID to succeed exporting CDI standalone PVCs (not attached to VM)</li> <li>[#9918][ShellyKa13] Fix for hotplug with WFFC SCI storage class which uses CDI populators</li> </ul>"},{"location":"release_notes/#sig-network_2","title":"SIG-network","text":"<ul> <li>[#10366][ormergi] Kubevirt now delegates Slirp networking configuration to Slirp network binding plugin.  In case you haven't registered Slirp network binding plugin image yet (i.e.: specify in Kubevirt config) the following default image would be used: <code>quay.io/kubevirt/network-slirp-binding:20230830_638c60fc8</code>. On next release (v1.2.0) no default image will be set and registering an image would be mandatory.</li> <li>[#10185][AlonaKaplan] Add support to migration based SRIOV hotplug.</li> <li>[#10116][ormergi] Existing detached interfaces with 'absent' state will be cleared from VMI spec.</li> <li>[#9958][AlonaKaplan] Disable network interface hotplug/unplug for VMIs. It will be supported for VMs only.</li> <li>[#10489][maiqueb] Remove the network-attachment-definition <code>list</code> and <code>watch</code> verbs from virt-controller's RBAC</li> </ul>"},{"location":"release_notes/#sig-infra_1","title":"SIG-infra","text":"<ul> <li>[#10438][lyarwood] A new <code>instancetype.kubevirt.io:view</code> <code>ClusterRole</code> has been introduced that can be bound to users via a <code>ClusterRoleBinding</code> to provide read only access to the cluster scoped <code>VirtualMachineCluster{Instancetype,Preference}</code> resources.</li> </ul>"},{"location":"release_notes/#sig-scale_1","title":"SIG-scale","text":"<ul> <li>[#9989][alaypatel07] Add perf scale benchmarks for VMIs</li> </ul>"},{"location":"release_notes/#uncategorized_2","title":"Uncategorized","text":"<ul> <li>[#9590][xuzhenglun] fix embed version info of virt-operator</li> <li>[#10044][machadovilaca] Add operator-observability package</li> <li>[#10450][0xFelix] virtctl: Enable inference in create vm subcommand by default</li> <li>[#10386][liuzhen21] KubeSphere added to the adopter's file!</li> <li>[#10167][0xFelix] virtctl: Apply namespace to created manifests</li> <li>[#10173][rmohr] Move coordination/lease RBAC permissions to Roles</li> <li>[#10138][machadovilaca] Change <code>kubevirt_vmi_*_usage_seconds</code> from Gauge to Counter</li> <li>[#10107][PiotrProkop] Expose <code>kubevirt_vmi_vcpu_delay_seconds_total</code> reporting amount of seconds VM spent in waiting in the queue instead of running.</li> <li>[#10070][machadovilaca] Remove affinities label from <code>kubevirt_vmi_cpu_affinity</code> and use sum as value</li> <li>[#9982][fabiand] Introduce a support lifecycle and Kubernetes target version.</li> <li>[#10001][machadovilaca] Fix <code>kubevirt_vmi_phase_count</code> not being created</li> <li>[#9840][dhiller] Increase probability for flake checker script to find flakes</li> <li>[#9988][enp0s3] Always deploy the outdated VMI workload alert</li> <li>[#9882][dhiller] Add some context for initial contributors about automated testing and draft pull requests.</li> <li>[#9552][phoracek] gRPC client now works correctly with non-Go gRPC servers</li> <li>[#9818][akrejcir] Added \"virtctl credentials\" commands to dynamically change SSH keys in a VM, and to set user's password.</li> <li>[#9073][machadovilaca] Fix incorrect KubevirtVmHighMemoryUsage description</li> </ul>"},{"location":"release_notes/#v100","title":"v1.0.0","text":"<p>Released on: Thu Jul 11 17:39:42 2023 +0000</p>"},{"location":"release_notes/#api-changes","title":"API changes","text":"<ul> <li>[PR #9572][fossedihelm] Enable freePageReporting for new non high performance vmi</li> <li>[PR #8156][jean-edouard] TPM VM device can now be set to persistent</li> <li>[PR #8575][iholder101] QEMU-level migration parallelism (a.k.a. multifd) + Upgrade QEMU to 7.2.0-11.el9</li> <li>[PR #9322][iholder101] Add guest-to-request memory headroom ratio.</li> <li>[PR #9422][awels] Ability to specify cpu/mem request limit for supporting containers (hotplug/container disk/virtiofs/side car)</li> <li>[PR #9177][alicefr] Adding SCSI persistent reservation</li> <li>[PR #9145][awels] Show VirtualMachine name in the VMExport status</li> <li>[PR #9491][orelmisan] API, AddInterfaceOptions: Rename NetworkName to NetworkAttachmentDefinitionName and InterfaceName to Name</li> <li>[PR #9442][EdDev] Remove the VMI Status interface <code>podConfigDone</code> field in favor of a new source option in <code>infoSource</code>.</li> <li>[PR #6852][maiqueb] Dev preview: Enables network interface hotplug for VMs / VMIs</li> <li>[PR #9193][qinqon] Add annotation for live migration and bridged pod interface</li> <li>[PR #9421][lyarwood] Requests to update the target <code>Name</code> of a <code>{Instancetype,Preference}Matcher</code> without also updating the <code>RevisionName</code> are now rejected.</li> </ul>"},{"location":"release_notes/#bug-fixes_1","title":"Bug fixes","text":"<ul> <li>[PR #9591][awels] BugFix: allow multiple NFS disks to be used/hotplugged</li> <li>[PR #9536][akalenyu] BugFix: virtualmachineclusterinstancetypes/preferences show up for get all -n  <li>[PR #9300][xpivarc] Bug fix: API and virtctl invoked migration is not rejected when the VM is paused</li> <li>[PR #9189][xpivarc] Bug fix: DNS integration continues to work after migration</li> <li>[PR #9241][akalenyu] BugFix: Guestfs image url not constructed correctly</li> <li>[PR #9260][ShellyKa13] Fix bug of possible re-trigger of memory dump</li> <li>[PR #9478][xpivarc] Bug fix: Fixes case when migration is not retried if the migration Pod gets denied.</li> <li>[PR #9330][qinqon] Skip label kubevirt.io/migrationTargetNodeName from virtctl expose service selector</li> <li>[PR #9603][qinqon] Adapt node-labeller.sh script to work at non kvm envs with emulation.</li>"},{"location":"release_notes/#deprecation_3","title":"Deprecation","text":"<ul> <li>[PR #9047][machadovilaca] Deprecate VM stuck in status alerts</li> </ul>"},{"location":"release_notes/#sig-compute_3","title":"SIG-compute","text":"<ul> <li>[PR #9640][jean-edouard] TSC-enabled VMs can now migrate to a node with a non-identical (but close-enough) frequency</li> <li>[PR #9629][0xFelix] virtctl: Allow to specify the boot order of volumes when creating VMs</li> <li>[PR #9435][rmohr] Ensure existence of all PVCs attached to the VMI before creating the VM target pod.</li> <li>[PR #9470][machadovilaca] Enable libvirt GetDomainStats on paused VMs</li> <li>[PR #9163][vladikr] fixes the requests/limits CPU number mismatch for VMs with isolatedEmulatorThread</li> <li>[PR #9250][vladikr] externally created mediated devices will not be deleted by virt-handler</li> </ul>"},{"location":"release_notes/#sig-storage_3","title":"SIG-storage","text":"<ul> <li>[PR #9376][ShellyKa13] Fix vmrestore with WFFC snapshotable storage class</li> <li>[PR #9392][awels] virtctl supports retrieving vm manifest for VM export</li> <li>[PR #9188][awels] Default RBAC for clone and export</li> <li>[PR #9133][ShellyKa13] Fix addvolume not rejecting adding existing volume source, fix removevolume allowing to remove non hotpluggable volume</li> </ul>"},{"location":"release_notes/#sig-network_3","title":"SIG-network","text":"<ul> <li>[PR #9399][maiqueb] Compute the interfaces to be hotplugged based on the current domain info, rather than on the interface status.</li> <li>[PR #9220][orelmisan] client-go: Added context to VirtualMachine's methods.</li> </ul>"},{"location":"release_notes/#sig-infra_2","title":"SIG-infra","text":"<ul> <li>[PR #9651][0xFelix] virtctl: Allow to specify memory of created VMs. Default to 512Mi if no instancetype was specified or is inferred.</li> <li>[PR #9169][lyarwood] The <code>dedicatedCPUPlacement</code> attribute is once again supported within the <code>VirtualMachineInstancetype</code> and <code>VirtualMachineClusterInstancetype</code> CRDs after a recent bugfix improved <code>VirtualMachine</code> validations, ensuring defaults are applied before any attempt to validate.</li> </ul>"},{"location":"release_notes/#uncategorized_3","title":"Uncategorized","text":"<ul> <li>[PR #9632][toelke] * Add Genesis Cloud to the adopters list</li> <li>[PR #9596][iholder101] Add \"virtctl create clone\" command</li> <li>[PR #9407][assafad] Use env <code>RUNBOOK_URL_TEMPLATE</code> for the runbooks URL template</li> <li>[PR #9327][jcanocan] DownwardMetrics: Swap KubeVirt build info with qemu version in VirtProductInfo field</li> <li>[PR #9367][machadovilaca] Add VM instancetype and preference label to vmi_phase_count metric</li> <li>[PR #8906][machadovilaca] Alert if there are no available nodes to run VMs</li> <li>[PR #9320][darfux] node-labeller: Check arch on the handler side</li> <li>[PR #9127][fossedihelm] Use ECDSA instead of RSA for key generation</li> <li>[PR #9228][rumans] Bump virtiofs container limit</li> <li>[PR #9159][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 9.0.0 and QEMU 7.2.0.</li> <li>[PR #8989][rthallisey] Integrate multi-architecture container manifests into the bazel make recipes</li> <li>[PR #8937][fossedihelm] Added foreground finalizer to  virtual machine</li> </ul>"},{"location":"release_notes/#v0590","title":"v0.59.0","text":"<p>Released on: Wed Mar 1 16:49:27 2023 +0000</p> <ul> <li>[PR #9311][kubevirt-bot] fixes the requests/limits CPU number mismatch for VMs with isolatedEmulatorThread</li> <li>[PR #9276][fossedihelm] Added foreground finalizer to  virtual machine</li> <li>[PR #9295][kubevirt-bot] Fix bug of possible re-trigger of memory dump</li> <li>[PR #9270][kubevirt-bot] BugFix: Guestfs image url not constructed correctly</li> <li>[PR #9234][kubevirt-bot] The <code>dedicatedCPUPlacement</code> attribute is once again supported within the <code>VirtualMachineInstancetype</code> and <code>VirtualMachineClusterInstancetype</code> CRDs after a recent bugfix improved <code>VirtualMachine</code> validations, ensuring defaults are applied before any attempt to validate.</li> <li>[PR #9267][fossedihelm] This version of KubeVirt includes upgraded virtualization technology based on libvirt 9.0.0 and QEMU 7.2.0.</li> <li>[PR #9197][kubevirt-bot] Fix addvolume not rejecting adding existing volume source, fix removevolume allowing to remove non hotpluggable volume</li> <li>[PR #9120][0xFelix] Fix access to portforwarding on VMs/VMIs with the cluster roles kubevirt.io:admin and kubevirt.io:edit</li> <li>[PR #9116][EdDev] Allow the specification of the ACPI Index on a network interface.</li> <li>[PR #8774][avlitman] Added new Virtual machines CPU metrics:</li> <li>[PR #9087][zhuchenwang] Open <code>/dev/vhost-vsock</code> explicitly to ensure that the right vsock module is loaded</li> <li>[PR #9020][feitnomore] Adding support for status/scale subresources so that VirtualMachinePool now supports HorizontalPodAutoscaler</li> <li>[PR #9085][0xFelix] virtctl: Add options to infer instancetype and preference when creating a VM</li> <li>[PR #8917][xpivarc] Kubevirt can be configured with Seccomp profile. It now ships a custom profile for the launcher.</li> <li>[PR #9054][enp0s3] do not inject LimitRange defaults into VMI</li> <li>[PR #7862][vladikr] Store the finalized VMI migration status in the migration objects.</li> <li>[PR #8878][0xFelix] Add 'create vm' command to virtctl</li> <li>[PR #9048][jean-edouard] DisableCustomSELinuxPolicy feature gate introduced to disable our custom SELinux policy</li> <li>[PR #8953][awels] VMExport now has endpoint containing entire VM definition.</li> <li>[PR #8976][iholder101] Fix podman CRI detection</li> <li>[PR #9043][iholder101] Adjust operator functional tests to custom images specification</li> <li>[PR #8875][machadovilaca] Rename migration metrics removing 'total' keyword</li> <li>[PR #9040][lyarwood] <code>inferFromVolume</code> now uses labels instead of annotations to lookup default instance type and preference details from a referenced <code>Volume</code>. This has changed in order to provide users with a way of looking up suitably decorated resources through these labels before pointing to them within the <code>VirtualMachine</code>.</li> <li>[PR #9039][orelmisan] client-go: Added context to additional VirtualMachineInstance's methods.</li> <li>[PR #9018][orelmisan] client-go: Added context to additional VirtualMachineInstance's methods.</li> <li>[PR #9025][akalenyu] BugFix: Hotplug pods have hardcoded resource req which don't comply with LimitRange maxLimitRequestRatio of 1</li> <li>[PR #8908][orelmisan] client-go: Added context to some of VirtualMachineInstance's methods.</li> <li>[PR #6863][rmohr] The install strategy job will respect the infra node placement from now on</li> <li>[PR #8948][iholder101] Bugfix: virt-handler socket leak</li> <li>[PR #8649][acardace] KubeVirt is now able to run VMs inside restricted namespaces.</li> <li>[PR #8992][iholder101] Align with k8s fix for default limit range requirements</li> <li>[PR #8889][rmohr] Add basic TLS encryption support for vsock websocket connections</li> <li>[PR #8660][huyinhou] Fix remoteAddress field in virt-api log being truncated when it is an ipv6 address</li> <li>[PR #8961][rmohr] Bump distroless base images</li> <li>[PR #8952][rmohr] Fix read-only sata disk validation</li> <li>[PR #8657][fossedihelm] Use an increasingly exponential backoff before retrying to start the VM, when an I/O error occurs.</li> <li>[PR #8480][lyarwood] New <code>inferFromVolume</code> attributes have been introduced to the <code>{Instancetype,Preference}Matchers</code> of a <code>VirtualMachine</code>. When provided the <code>Volume</code> referenced by the attribute is checked for the following annotations with which to populate the <code>{Instancetype,Preference}Matchers</code>:</li> <li>[PR #7762][VirrageS] Service <code>kubevirt-prometheus-metrics</code> now sets <code>ClusterIP</code> to <code>None</code> to make it a headless service.</li> <li>[PR #8599][machadovilaca] Change KubevirtVmHighMemoryUsage threshold from 20MB to 50MB</li> <li>[PR #7761][VirrageS] imagePullSecrets field has been added to KubeVirt CR to support deployments form private registries</li> <li>[PR #8887][iholder101] Bugfix: use virt operator image if provided</li> <li>[PR #8750][jordigilh] Fixes an issue that prevented running real time workloads in non-root configurations due to libvirt's dependency on CAP_SYS_NICE to change the vcpu's thread's scheduling and priority to FIFO and 1. The change of priority and scheduling is now executed in the virt-launcher for both root and non-root configurations, removing the dependency in libvirt.</li> <li>[PR #8845][lyarwood] An empty <code>Timer</code> is now correctly omitted from <code>Clock</code> fixing bug #8844.</li> <li>[PR #8842][andreabolognani] The virt-launcher pod no longer needs the SYS_PTRACE capability.</li> <li>[PR #8734][alicefr] Change libguestfs-tools image using root appliance in qcow2 format</li> <li>[PR #8764][ShellyKa13] Add list of included and excluded volumes in vmSnapshot</li> <li>[PR #8811][iholder101] Custom components: support gs</li> <li>[PR #8770][dhiller] Add Ginkgo V2 Serial decorator to serial tests as preparation to simplify parallel vs. serial test run logic</li> <li>[PR #8808][acardace] Apply migration backoff only for evacuation migrations.</li> <li>[PR #8525][jean-edouard] CR option mediatedDevicesTypes is deprecated in favor of mediatedDeviceTypes</li> <li>[PR #8792][iholder101] Expose new custom components env vars to csv-generator and manifest-templator</li> <li>[PR #8701][enp0s3] Consider the ParallelOutboundMigrationsPerNode when evicting VMs</li> <li>[PR #8740][iholder101] Fix: Align Reenlightenment flows between converter.go and template.go</li> <li>[PR #8530][acardace] Use exponential backoff for failing migrations</li> <li>[PR #8720][0xFelix] The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced</li> <li>[PR #8458][iholder101] Introduce support for clones with a snapshot source (e.g. clone snapshot -&gt; VM)</li> <li>[PR #8716][rhrazdil] Add overhead of interface with Passt binding when no ports are specified</li> <li>[PR #8619][fossedihelm] virt-launcher: use <code>virtqemud</code> daemon instead of <code>libvirtd</code></li> <li>[PR #8736][knopt] Added more precise rest_client_request_latency_seconds histogram buckets</li> <li>[PR #8624][zhuchenwang] Add the REST API to be able to talk to the application in the guest VM via VSOCK.</li> <li>[PR #8625][AlonaKaplan] iptables are no longer used by masquerade binding. Nodes with iptables only won't be able to run VMs with masquerade binding.</li> <li>[PR #8673][iholder101] Allow specifying custom images for core components</li> <li>[PR #8622][jean-edouard] Built with golang 1.19</li> <li>[PR #8336][alicefr] Flag for setting the guestfs uid and gid</li> <li>[PR #8667][huyinhou] connect VM vnc failed when virt-launcher work directory is not /</li> <li>[PR #8368][machadovilaca] Use collector to set migration metrics</li> <li>[PR #8558][xpivarc] Bug-fix: LimitRange integration now works when VMI is missing namespace</li> <li>[PR #8404][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 8.7.0, QEMU 7.1.0 and CentOS Stream 9.</li> <li>[PR #8652][akalenyu] BugFix: Exporter pod does not comply with restricted PSA</li> <li>[PR #8563][xpivarc] Kubevirt now runs with nonroot user by default</li> <li>[PR #8442][kvaps] Add Deckhouse to the Adopters list</li> <li>[PR #8546][zhuchenwang] Provides the Vsock feature for KubeVirt VMs.</li> <li>[PR #8598][acardace] VMs configured with hugepages can now run using the default container_t SELinux type</li> <li>[PR #8594][kylealexlane] Fix permission denied on on selinux relabeling on some kernel versions</li> <li>[PR #8521][akalenyu] Add an option to specify a TTL for VMExport objects</li> <li>[PR #7918][machadovilaca] Add alerts for VMs unhealthy states</li> <li>[PR #8516][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> <li>[PR #7772][jean-edouard] The SELinux policy for virt-launcher is down to 4 rules, 1 for hugepages and 3 for virtiofs.</li> <li>[PR #8402][jean-edouard] Most VMIs now run under the SELinux type container_t</li> <li>[PR #8513][alromeros] [Bug-fix] Fix error handling in virtctl image-upload</li> </ul>"},{"location":"release_notes/#v0581","title":"v0.58.1","text":"<p>Released on: Thu Feb 11 00:08:46 2023 +0000</p> <ul> <li>[PR #9203][jean-edouard] Most VMIs now run under the SELinux type container_t</li> <li>[PR #9191][kubevirt-bot] Default RBAC for clone and export</li> <li>[PR #9150][kubevirt-bot] Fix access to portforwarding on VMs/VMIs with the cluster roles kubevirt.io:admin and kubevirt.io:edit</li> <li>[PR #9128][kubevirt-bot] Rename migration metrics removing 'total' keyword</li> <li>[PR #9034][akalenyu] BugFix: Hotplug pods have hardcoded resource req which don't comply with LimitRange maxLimitRequestRatio of 1</li> <li>[PR #9002][iholder101] Bugfix: virt-handler socket leak</li> <li>[PR #8907][kubevirt-bot] Bugfix: use virt operator image if provided</li> <li>[PR #8784][kubevirt-bot] Use exponential backoff for failing migrations</li> <li>[PR #8816][iholder101] Expose new custom components env vars to csv-generator, manifest-templator and gs</li> <li>[PR #8798][iholder101] Fix: Align Reenlightenment flows between converter.go and template.go</li> <li>[PR #8731][kubevirt-bot] Allow specifying custom images for core components</li> <li>[PR #8785][0xFelix] The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced</li> <li>[PR #8806][kubevirt-bot] Consider the ParallelOutboundMigrationsPerNode when evicting VMs</li> <li>[PR #8738][machadovilaca] Use collector to set migration metrics</li> <li>[PR #8747][kubevirt-bot] Add alerts for VMs unhealthy states</li> <li>[PR #8685][kubevirt-bot] BugFix: Exporter pod does not comply with restricted PSA</li> <li>[PR #8647][akalenyu] BugFix: Add an option to specify a TTL for VMExport objects</li> <li>[PR #8609][kubevirt-bot] Fix permission denied on on selinux relabeling on some kernel versions</li> <li>[PR #8578][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> </ul>"},{"location":"release_notes/#v0580","title":"v0.58.0","text":"<p>Released on: Thu Oct 13 00:24:51 2022 +0000</p> <ul> <li>[PR #8578][rhrazdil] When using Passt binding, virl-launcher has unprivileged_port_start set to 0, so that passt may bind to all ports.</li> <li>[PR #8463][Barakmor1] Improve metrics documentation</li> <li>[PR #8282][akrejcir] Improves instancetype and preference controller revisions. This is a backwards incompatible change and introduces a new v1alpha2 api for instancetype and preferences.</li> <li>[PR #8272][jean-edouard] No more empty section in the kubevirt-cr manifest</li> <li>[PR #8536][qinqon] Don't show a failure if ConfigDrive cloud init has UserDataSecretRef and not NetworkDataSecretRef</li> <li>[PR #8375][xpivarc] Virtiofs can be used with Nonroot feature gate</li> <li>[PR #8465][rmohr] Add a vnc screenshot REST endpoint and a \"virtctl vnc screenshot\" command for UI and script integration</li> <li>[PR #8418][alromeros] Enable automatic token generation for VirtualMachineExport objects</li> <li>[PR #8488][0xFelix] virtctl: Be less verbose when using the local ssh client</li> <li>[PR #8396][alicefr] Add group flag for setting the gid and fsgroup in guestfs</li> <li>[PR #8476][iholder-redhat] Allow setting virt-operator log verbosity through Kubevirt CR</li> <li>[PR #8366][rthallisey] Move KubeVirt to a 15 week release cadence</li> <li>[PR #8479][arnongilboa] Enable DataVolume GC by default in cluster-deploy</li> <li>[PR #8474][vasiliy-ul] Fixed migration failure of VMs with containerdisks on systems with containerd</li> <li>[PR #8316][ShellyKa13] Fix possible race when deleting unready vmsnapshot and the vm remaining frozen</li> <li>[PR #8436][xpivarc] Kubevirt is able to run with restricted Pod Security Standard enabled with an automatic escalation of namespace privileges.</li> <li>[PR #8197][alromeros] Add vmexport command to virtctl</li> <li>[PR #8252][fossedihelm] Add <code>tlsConfiguration</code> to Kubevirt Configuration</li> <li>[PR #8431][rmohr] Fix shadow status updates and periodic status updates on VMs, performed by the snapshot controller</li> <li>[PR #8359][iholder-redhat] [Bugfix]: HyperV Reenlightenment VMIs should be able to start when TSC Frequency is not exposed</li> <li>[PR #8330][jean-edouard] Important: If you use docker with SELinux enabled, set the <code>DockerSELinuxMCSWorkaround</code> feature gate before upgrading</li> <li>[PR #8401][machadovilaca] Rename metrics to follow the naming convention</li> </ul>"},{"location":"release_notes/#v0570","title":"v0.57.0","text":"<p>Released on: Mon Sep 12 14:00:44 2022 +0000</p> <ul> <li>[PR #8129][mlhnono68] Fixes virtctl to support connection to clusters proxied by RANCHER or having special paths</li> <li>[PR #8337][0xFelix] virtctl's native SSH client is now useable in the Windows console without workarounds</li> <li>[PR #8257][awels] VirtualMachineExport now supports VM export source type.</li> <li>[PR #8367][vladikr] fix the guest memory conversion by setting it to resources.requests.memory when guest memory is not explicitly provided</li> <li>[PR #7990][ormergi] Deprecate SR-IOV live migration feature gate.</li> <li>[PR #8069][lyarwood] The VirtualMachineInstancePreset resource has been deprecated ahead of removal in a future release. Users should instead use the VirtualMachineInstancetype and VirtualMachinePreference resources to encapsulate any shared resource or preferences characteristics shared by their VirtualMachines.</li> <li>[PR #8326][0xFelix] virtctl: Do not log wrapped ssh command by default</li> <li>[PR #8325][rhrazdil] Enable route_localnet sysctl option for masquerade binding at virt-handler</li> <li>[PR #8159][acardace] Add support for USB disks</li> <li>[PR #8006][lyarwood] <code>AutoattachInputDevice</code> has been added to <code>Devices</code> allowing an <code>Input</code> device to be automatically attached to a <code>VirtualMachine</code> on start up.  <code>PreferredAutoattachInputDevice</code> has also been added to <code>DevicePreferences</code> allowing users to control this behaviour with a set of preferences.</li> <li>[PR #8134][arnongilboa] Support DataVolume garbage collection</li> <li>[PR #8157][StefanKro] TrilioVault for Kubernetes now supports KubeVirt for backup and recovery.</li> <li>[PR #8273][alaypatel07] add server-side validations for spec.topologySpreadConstraints during object creation</li> <li>[PR #8049][alicefr] Set RunAsNonRoot as default for the guestfs pod</li> <li>[PR #8107][awels] Allow VirtualMachineSnapshot as a VirtualMachineExport source</li> <li>[PR #7846][janeczku] Added support for configuring topology spread constraints for virtual machines.</li> <li>[PR #8215][alaypatel07] support validation for spec.affinity fields during vmi creation</li> <li>[PR #8071][oshoval] Relax networkInterfaceMultiqueue semantics: multi queue will configure only what it can (virtio interfaces).</li> <li>[PR #7549][akrejcir] Added new API subresources to expand instancetype and preference.</li> </ul>"},{"location":"release_notes/#v0560","title":"v0.56.0","text":"<p>Released on: Thu Aug 18 20:10:29 2022 +0000</p> <ul> <li>[PR #7599][iholder-redhat] Introduce a mechanism to abort non-running migrations - fixes \"Unable to cancel live-migration if virt-launcher pod in pending state\" bug</li> <li>[PR #8027][alaypatel07] Wait deletion to succeed all the way till objects are finalized in perfscale tests</li> <li>[PR #8198][rmohr] Improve path handling for non-root virt-launcher workloads</li> <li>[PR #8136][iholder-redhat] Fix cgroups unit tests: mock out underlying runc cgroup manager</li> <li>[PR #8047][iholder-redhat] Deprecate live migration feature gate</li> <li>[PR #7986][iholder-redhat] [Bug-fix]: Windows VM with WSL2 guest fails to migrate</li> <li>[PR #7814][machadovilaca] Add VMI filesystem usage metrics</li> <li>[PR #7849][AlonaKaplan] [TECH PREVIEW] Introducing passt - a new approach to user-mode networking for virtual machines</li> <li>[PR #7991][ShellyKa13] Virtctl memory dump with create flag to create a new pvc</li> <li>[PR #8039][lyarwood] The flavor API and associated CRDs of <code>VirtualMachine{Flavor,ClusterFlavor}</code> are renamed to instancetype and <code>VirtualMachine{Instancetype,ClusterInstancetype}</code>.</li> <li>[PR #8112][AlonaKaplan] Changing the default of <code>virtctl expose</code> <code>ip-family</code> parameter to be empty value instead of IPv4.</li> <li>[PR #8073][orenc1] Bump runc to v1.1.2</li> <li>[PR #8092][Barakmor1] Bump the version of emicklei/go-restful from 2.15.0 to 2.16.0</li> <li>[PR #8053][alromeros] [Bug-fix]: Fix mechanism to fetch fs overhead when CDI resource has a different name</li> <li>[PR #8035][0xFelix] Add option to wrap local scp client to scp command</li> <li>[PR #7981][lyarwood] Conflicts will now be raised when using flavors if the <code>VirtualMachine</code> defines any <code>CPU</code> or <code>Memory</code> resource requests.</li> <li>[PR #8068][awels] Set cache mode to match regular disks on hotplugged disks.</li> </ul>"},{"location":"release_notes/#v0550","title":"v0.55.0","text":"<p>Released on: Thu Jul 14 16:33:25 2022 +0000</p> <ul> <li>[PR #7336][iholder-redhat] Introduce clone CRD, controller and API</li> <li>[PR #7791][iholder-redhat] Introduction of an initial deprecation policy</li> <li>[PR #7875][lyarwood] <code>ControllerRevisions</code> of any <code>VirtualMachineFlavorSpec</code> or <code>VirtualMachinePreferenceSpec</code> are stored during the initial start of a <code>VirtualMachine</code> and used for subsequent restarts ensuring changes to the original <code>VirtualMachineFlavor</code> or <code>VirtualMachinePreference</code> do not modify the <code>VirtualMachine</code> and the <code>VirtualMachineInstance</code> it creates.</li> <li>[PR #8011][fossedihelm] Increase virt-launcher memory overhead</li> <li>[PR #7963][qinqon] Bump alpine_with_test_tooling</li> <li>[PR #7881][ShellyKa13] Enable memory dump to be included in VMSnapshot</li> <li>[PR #7926][qinqon] tests: Move main clean function to global AfterEach and create a VM per each infra_test.go Entry.</li> <li>[PR #7845][janeczku] Fixed a bug that caused <code>make generate</code> to fail when API code comments contain backticks. (#7844, @janeczku)</li> <li>[PR #7932][marceloamaral] Addition of kubevirt_vmi_migration_phase_transition_time_from_creation_seconds metric to monitor how long it takes to transition a VMI Migration object to a specific phase from creation time.</li> <li>[PR #7879][marceloamaral] Faster VM phase transitions thanks to an increased virt-controller QPS/Burst</li> <li>[PR #7807][acardace] make cloud-init 'instance-id' persistent across reboots</li> <li>[PR #7928][iholder-redhat] bugfix: node-labeller now removes \"host-model-cpu.node.kubevirt.io/\" and \"host-model-required-features.node.kubevirt.io/\" prefixes</li> <li>[PR #7841][jean-edouard] Non-root VMs will now migrate to root VMs after a cluster disables non-root.</li> <li>[PR #7933][akalenyu] BugFix: Fix vm restore in case of restore size bigger then PVC requested size</li> <li>[PR #7919][lyarwood] Device preferences are now applied to any default network interfaces or missing volume disks added to a <code>VirtualMachineInstance</code> at runtime.</li> <li>[PR #7910][qinqon] tests: Create the expected readiness probe instead of liveness</li> <li>[PR #7732][acardace] Prevent virt-handler from starting a migration twice</li> <li>[PR #7594][alicefr] Enable to run libguestfs-tools pod to run as noroot user</li> <li>[PR #7811][raspbeep] User now gets information about the type of commands which the guest agent does not support.</li> <li>[PR #7590][awels] VMExport allows filesystem PVCs to be exported as either disks or directories.</li> <li>[PR #7683][alicefr] Add --command and --local-ssh-opts\" options to virtctl ssh to execute remote command using local ssh method</li> </ul>"},{"location":"release_notes/#v0540","title":"v0.54.0","text":"<p>Released on: Wed Jun 8 14:15:43 2022 +0000</p> <ul> <li>[PR #7757][orenc1] new alert for excessive number of VMI migrations in a period of time.</li> <li>[PR #7517][ShellyKa13] Add virtctl Memory Dump command</li> <li>[PR #7801][VirrageS] Empty (<code>nil</code> values) of <code>Address</code> and <code>Driver</code> fields in XML will be omitted.</li> <li>[PR #7475][raspbeep] Adds the reason of a live-migration failure to a recorded event in case EvictionStrategy is set but live-migration is blocked due to its limitations.</li> <li>[PR #7739][fossedihelm] Allow <code>virtualmachines/migrate</code> subresource to admin/edit users</li> <li>[PR #7618][lyarwood] The requirement to define a <code>Disk</code> or <code>Filesystem</code> for each <code>Volume</code> associated with a <code>VirtualMachine</code> has been removed. Any <code>Volumes</code> without a <code>Disk</code> or <code>Filesystem</code> defined will have a <code>Disk</code> defined within the <code>VirtualMachineInstance</code> at runtime.</li> <li>[PR #7529][xpivarc] NoReadyVirtController and NoReadyVirtOperator should be properly fired.</li> <li>[PR #7465][machadovilaca] Add metrics for migrations and respective phases</li> <li>[PR #7592][akalenyu] BugFix: virtctl guestfs incorrectly assumes image name</li> </ul>"},{"location":"release_notes/#v0531","title":"v0.53.1","text":"<p>Released on: Tue May 17 14:55:54 2022 +0000</p> <ul> <li>[PR #7749][kubevirt-bot] NoReadyVirtController and NoReadyVirtOperator should be properly fired.</li> </ul>"},{"location":"release_notes/#v0530","title":"v0.53.0","text":"<p>Released on: Mon May 9 14:02:20 2022 +0000</p> <ul> <li>[PR #7533][akalenyu] Add several VM snapshot metrics</li> <li>[PR #7574][rmohr] Pull in cdi dependencies with minimized transitive dependencies to ease API adoption</li> <li>[PR #7318][iholder-redhat] Snapshot restores now support restoring to a target VM different than the source</li> <li>[PR #7474][borod108] Added the following metrics for live migration: kubevirt_migrate_vmi_data_processed_bytes, kubevirt_migrate_vmi_data_remaining_bytes, kubevirt_migrate_vmi_dirty_memory_rate_bytes</li> <li>[PR #7441][rmohr] Add <code>virtctl scp</code> to ease copying files from and to VMs and VMIs</li> <li>[PR #7265][rthallisey] Support steady-state job types in the load-generator tool</li> <li>[PR #7544][fossedihelm] Upgraded go version to 1.17.8</li> <li>[PR #7582][acardace] Fix failed reported migrations when actually they were successful.</li> <li>[PR #7546][0xFelix] Update virtio-container-disk to virtio-win version 0.1.217-1</li> <li>[PR #7530][iholder-redhat] [External Kernel Boot]: Disallow kernel args without providing custom kernel</li> <li>[PR #7493][davidvossel] Adds new EvictionStrategy \"External\" for blocking eviction which is handled by an external controller</li> <li>[PR #7563][akalenyu] Switch VolumeSnapshot to v1</li> <li>[PR #7406][acardace] Reject <code>LiveMigrate</code> as a workload-update strategy if the <code>LiveMigration</code> feature gate is not enabled.</li> <li>[PR #7103][jean-edouard] Non-persistent vTPM now supported. Keep in mind that the state of the TPM is wiped after each shutdown. Do not enable Bitlocker!</li> <li>[PR #7277][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 8.0.0 and QEMU 6.2.0.</li> <li>[PR #7130][Barakmor1] Add field to kubevirtCR to set Prometheus ServiceMonitor object's namespace</li> <li>[PR #7401][iholder-redhat] virt-api deployment is now scalable - replicas are determined by the number of nodes in the cluster</li> <li>[PR #7500][awels] BugFix: Fixed RBAC for admin/edit user to allow virtualmachine/addvolume and removevolume. This allows for persistent disks</li> <li>[PR #7328][apoorvajagtap] Don't ignore --identity-file when setting --local-ssh=true on <code>virtctl ssh</code></li> <li>[PR #7469][xpivarc] Users can now enable the NonRoot feature gate instead of NonRootExperimental</li> <li>[PR #7451][fossedihelm] Reduce virt-launcher memory usage by splitting monitoring and launcher processes</li> </ul>"},{"location":"release_notes/#v0520","title":"v0.52.0","text":"<p>Released on: Fri Apr 8 16:17:56 2022 +0000</p> <ul> <li>[PR #7024][fossedihelm] Add an warning message if the client and server virtctl versions are not aligned</li> <li>[PR #7486][rmohr] Move stable.txt location to a more appropriate path</li> <li>[PR #7372][saschagrunert] Fixed <code>KubeVirtComponentExceedsRequestedMemory</code> alert complaining about many-to-many matching not allowed.</li> <li>[PR #7426][iholder-redhat] Add warning for manually determining core-component replica count in Kubevirt CR</li> <li>[PR #7424][maiqueb] Provide interface binding types descriptions, which will be featured in the KubeVirt API.</li> <li>[PR #7422][orelmisan] Fixed setting custom guest pciAddress and bootOrder parameter(s) to a list of SR-IOV NICs.</li> <li>[PR #7421][rmohr] Fix knowhosts file corruption for virtctl ssh</li> <li>[PR #6854][rmohr] Make virtctl ssh work with ssh-rsa+ preauthentication</li> <li>[PR #7267][iholder-redhat] Applied migration configurations can now be found in VMI's status</li> <li>[PR #7321][iholder-redhat] [Migration Policies]: precedence to VMI labels over Namespace labels</li> <li>[PR #7326][oshoval] The Ginkgo dependency has been upgraded to v2.1.3 (major version upgrade)</li> <li>[PR #7361][SeanKnight] Fixed a bug that prevents virtctl from working with clusters accessed via Rancher authentication proxy, or any other cluster where the server URL contains a path component. (#3760)</li> <li>[PR #7255][tyleraharrison] Users are now able to specify <code>--address [ip_address]</code> when using <code>virtctl vnc</code> rather than only using 127.0.0.1</li> <li>[PR #7275][enp0s3] Add observedGeneration to virt-operator to have a race-free way to detect KubeVirt config rollouts</li> <li>[PR #7233][xpivarc] Bug fix: Successfully aborted migrations should be reported now</li> <li>[PR #7158][AlonaKaplan] Add masquerade VMs support to single stack IPv6.</li> <li>[PR #7227][rmohr] Remove VMI informer from virt-api to improve scaling characteristics of virt-api</li> <li>[PR #7288][raspbeep] Users now don't need to specify container for <code>kubectl logs &lt;vmi-pod&gt;</code> and <code>kubectl exec &lt;vmi-pod&gt;</code>.</li> <li>[PR #6709][xpivarc] Workloads will be migrated to nonroot implementation if NonRoot feature gate is set. (Except VirtioFS)</li> <li>[PR #7241][lyarwood] Fixed a bug that prevents only a unattend.xml configmap or secret being provided as contents for a sysprep disk. (#7240, @lyarwood)</li> </ul>"},{"location":"release_notes/#v0510","title":"v0.51.0","text":"<p>Released on: Tue Mar 8 21:06:59 2022 +0000</p> <ul> <li>[PR #7102][machadovilaca] Add Virtual Machine name label to virt-launcher pod</li> <li>[PR #7139][davidvossel] Fixes inconsistent VirtualMachinePool VM/VMI updates by using controller revisions</li> <li>[PR #6754][jean-edouard] New and resized disks are now always 1MiB-aligned</li> <li>[PR #7086][acardace] Add 'EvictionStrategy' as a cluster-wide setting in the KubeVirt CR</li> <li>[PR #7232][rmohr] Properly format the PDB scale event during migrations</li> <li>[PR #7223][Barakmor1] Add a name label to virt-operator pods</li> <li>[PR #7221][davidvossel] RunStrategy: Once - allows declaring a VM should run once to a finalized state</li> <li>[PR #7091][EdDev] SR-IOV interfaces are now reported in the VMI status even without an active guest-agent.</li> <li>[PR #7169][rmohr] Improve device plugin de-registration in virt-handler and some test stabilizations</li> <li>[PR #6604][alicefr] Add shareable option to identify if the disk is shared with other VMs</li> <li>[PR #7144][davidvossel] Garbage collect finalized migration objects only leaving the most recent 5 objects</li> <li>[PR #6110][xpivarc] [Nonroot] SRIOV is now available.</li> </ul>"},{"location":"release_notes/#v0500","title":"v0.50.0","text":"<p>Released on: Wed Feb 9 18:01:08 2022 +0000</p> <ul> <li>[PR #7056][fossedihelm] Update k8s dependencies to 0.23.1</li> <li>[PR #7135][davidvossel] Switch from reflects.DeepEquals to equality.Semantic.DeepEquals() across the entire project</li> <li>[PR #7052][sradco] Updated recording rule \"kubevirt_vm_container_free_memory_bytes\"</li> <li>[PR #7000][iholder-redhat] Adds a possibility to override default libvirt log filters though VMI annotations</li> <li>[PR #7064][davidvossel] Fixes issue associated with blocked uninstalls when VMIs exist during removal</li> <li>[PR #7097][iholder-redhat] [Bug fix] VMI with kernel boot stuck on \"Terminating\" status if more disks are defined</li> <li>[PR #6700][VirrageS] Simplify replacing <code>time.Ticker</code> in agent poller and fix default values for <code>qemu-*-interval</code> flags</li> <li>[PR #6581][ormergi] SRIOV network interfaces are now hot-plugged when disconnected manually or due to aborted migrations.</li> <li>[PR #6924][EdDev] Support for legacy GPU definition is removed. Please see https://kubevirt.io/user-guide/virtual_machines/host-devices on how to define host-devices.</li> <li>[PR #6735][uril] The command <code>migrate_cancel</code> was added to virtctl. It cancels an active VM migration.</li> <li>[PR #6883][rthallisey] Add instance-type to cloud-init metadata</li> <li>[PR #6999][maya-r] When expanding disk images, take the minimum between the request and the capacity - avoid using the full underlying file system on storage like NFS, local.</li> <li>[PR #6946][vladikr] Numa information of an assigned device will be presented in the devices metadata</li> <li>[PR #6042][iholder-redhat] Fully support cgroups v2, include a new cohesive package and perform major refactoring.</li> <li>[PR #6968][vladikr] Added Writeback disk cache support</li> <li>[PR #6995][sradco] Alert OrphanedVirtualMachineImages name was changed to OrphanedVirtualMachineInstances.</li> <li>[PR #6923][rhrazdil] Fix issue with ssh being unreachable on VMIs with Istio proxy</li> <li>[PR #6821][jean-edouard] Migrating VMIs that contain dedicated CPUs will now have properly dedicated CPUs on target</li> <li>[PR #6793][oshoval] Add infoSource field to vmi.status.interfaces.</li> </ul>"},{"location":"release_notes/#v0490","title":"v0.49.0","text":"<p>Released on: Tue Jan 11 17:27:09 2022 +0000</p> <ul> <li>[PR #7004][iholder-redhat] Bugfix: Avoid setting block migration for volumes used by read-only disks</li> <li>[PR #6959][enp0s3] generate event when target pod enters unschedulable phase</li> <li>[PR #6888][assafad] Added common labels into alert definitions</li> <li>[PR #6166][vasiliy-ul] Experimental support of AMD SEV</li> <li>[PR #6980][vasiliy-ul] Updated the dependencies to include the fix for CVE-2021-43565 (KubeVirt is not affected)</li> <li>[PR #6944][iholder-redhat] Remove disabling TLS configuration from Live Migration Policies</li> <li>[PR #6800][jean-edouard] CPU pinning doesn't require hardware-assisted virtualization anymore</li> <li>[PR #6501][ShellyKa13] Use virtctl image-upload to upload archive content</li> <li>[PR #6918][iholder-redhat] Bug fix: Unscheduable host-model VMI alert is now properly triggered</li> <li>[PR #6796][Barakmor1] 'kubevirt-operator' changed to 'virt-operator' on 'managed-by' label in kubevirt's components made by virt-operator</li> <li>[PR #6036][jean-edouard] Migrations can now be done over a dedicated multus network</li> <li>[PR #6933][erkanerol] Add a new lane for monitoring tests</li> <li>[PR #6949][jean-edouard] KubeVirt components should now be successfully removed on CR deletion, even when using only 1 replica for virt-api and virt-controller</li> <li>[PR #6954][maiqueb] Update the <code>virtctl</code> exposed services <code>IPFamilyPolicyType</code> default to <code>IPFamilyPolicyPreferDualStack</code></li> <li>[PR #6931][fossedihelm] added DryRun to AddVolumeOptions and RemoveVolumeOptions</li> <li>[PR #6379][nunnatsa] Fix issue https://bugzilla.redhat.com/show_bug.cgi?id=1945593</li> <li>[PR #6399][iholder-redhat] Introduce live migration policies that allow system-admins to have fine-grained control over migration configuration for different sets of VMs.</li> <li>[PR #6880][iholder-redhat] Add full Podman support for <code>make</code> and <code>make test</code></li> <li>[PR #6702][acardace] implement virt-handler canary upgrade and rollback for faster and safer rollouts</li> <li>[PR #6717][davidvossel] Introducing the VirtualMachinePools feature for managing stateful VMs at scale</li> <li>[PR #6698][rthallisey] Add tracing to the virt-controller work queue</li> <li>[PR #6762][fossedihelm] added DryRun mode to virtcl to migrate command</li> <li>[PR #6891][rmohr] Fix \"Make raw terminal failed: The handle is invalid?\" issue with \"virtctl console\" when not executed in a pty</li> <li>[PR #6783][rmohr] Skip SSH RSA auth if no RSA key was explicitly provided and not key exists at the default location</li> </ul>"},{"location":"release_notes/#v0481","title":"v0.48.1","text":"<p>Released on: Wed Dec 15 15:11:55 2021 +0000</p> <ul> <li>[PR #6900][kubevirt-bot] Skip SSH RSA auth if no RSA key was explicitly provided and not key exists at the default location</li> <li>[PR #6902][kubevirt-bot] Fix \"Make raw terminal failed: The handle is invalid?\" issue with \"virtctl console\" when not executed in a pty</li> </ul>"},{"location":"release_notes/#v0480","title":"v0.48.0","text":"<p>Released on: Mon Dec 6 18:26:51 2021 +0000</p> <ul> <li>[PR #6670][futuretea] Added 'virtctl soft-reboot' command to reboot the VMI.</li> <li>[PR #6861][orelmisan] virtctl errors are written to stderr instead of stdout</li> <li>[PR #6836][enp0s3] Added PHASE and VMI columns for the 'kubectl get vmim' CLI output</li> <li>[PR #6784][nunnatsa] kubevirt-config configMap is no longer supported for KubeVirt configuration</li> <li>[PR #6839][ShellyKa13] fix restore of VM with RunStrategy</li> <li>[PR #6533][zcahana] Paused VMIs are now marked as unready even when no readinessProbe is specified</li> <li>[PR #6858][rmohr] Fix a nil pointer in virtctl in combination with some external auth plugins</li> <li>[PR #6780][fossedihelm] Add PatchOptions to the Patch request of the VirtualMachineInstanceInterface</li> <li>[PR #6773][iholder-redhat] alert if migration for VMI with host-model CPU is stuck since no node is suitable</li> <li>[PR #6714][rhrazdil] Shorten timeout for Istio proxy detection</li> <li>[PR #6725][fossedihelm] added DryRun mode to virtcl for pause and unpause commands</li> <li>[PR #6737][davidvossel] Pending migration target pods timeout after 5 minutes when unschedulable</li> <li>[PR #6814][fossedihelm] Changed some terminology to be more inclusive</li> <li>[PR #6649][Barakmor1] Designate the apps.kubevirt.io/component label for KubeVirt components.</li> <li>[PR #6650][victortoso] Introduces support to ich9 or ac97 sound devices</li> <li>[PR #6734][Barakmor1] replacing the command that extract libvirtd's pid  to avoid this error:</li> <li>[PR #6802][rmohr] Maintain a separate api package which synchronizes to kubevirt.io/api for better third party integration with client-gen</li> <li>[PR #6730][zhhray] change kubevrit cert secret type from Opaque to kubernetes.io/tls</li> <li>[PR #6508][oshoval] Add missing domain to guest search list, in case subdomain is used.</li> <li>[PR #6664][vladikr] enable the display and ramfb for vGPUs by default</li> <li>[PR #6710][iholder-redhat] virt-launcher fix - stop logging successful shutdown when it isn't true</li> <li>[PR #6162][vladikr] KVM_HINTS_REALTIME will always be set when dedicatedCpusPlacement is requested</li> <li>[PR #6772][zcahana] Bugfix: revert #6565 which prevented upgrades to v0.47.</li> <li>[PR #6722][zcahana] Remove obsolete scheduler.alpha.kubernetes.io/critical-pod annotation</li> <li>[PR #6723][acardace] remove stale pdbs created by &lt; 0.41.1 virt-controller</li> <li>[PR #6721][iholder-redhat] Set default CPU model in VMI spec, even if not defined in KubevirtCR</li> <li>[PR #6713][zcahana] Report WaitingForVolumeBinding VM status when PVC/DV-type volumes reference unbound PVCs</li> <li>[PR #6681][fossedihelm] Users can use --dry-run flag</li> <li>[PR #6663][jean-edouard] The number of virt-api and virt-controller replicas is now configurable in the CSV</li> <li>[PR #5981][maya-r] Always resize disk.img files to the largest size at boot.</li> </ul>"},{"location":"release_notes/#v0471","title":"v0.47.1","text":"<p>Released on: Thu Nov 11 15:52:59 2021 +0000</p> <ul> <li>[PR #6775][kubevirt-bot] Bugfix: revert #6565 which prevented upgrades to v0.47.</li> <li>[PR #6703][mhenriks] Fix BZ 2018521 - On upgrade VirtualMachineSnapshots going to Failed</li> <li>[PR #6511][knopt] Fixed virt-api significant memory usage when using Cluster Profiler with large KubeVirt deployments. (#6478, @knopt)</li> <li>[PR #6629][awels] BugFix: Hotplugging more than one block device would cause IO error (#6564)</li> <li>[PR #6657][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.6.0 and QEMU 6.0.0.</li> <li>[PR #6565][Barakmor1] 'kubevirt-operator' changed to 'virt-operator' on 'managed-by' label in kubevirt's components made by virt-operator</li> <li>[PR #6642][ShellyKa13] Include hot-plugged disks in a Online VM Snapshot</li> <li>[PR #6513][brybacki] Adds force-bind flag to virtctl imageupload</li> <li>[PR #6588][erkanerol] Fix recording rules based on up metrics</li> <li>[PR #6575][davidvossel] VM controller now syncs VMI conditions to corresponding VM object</li> <li>[PR #6661][rmohr] Make the kubevirt api compatible with client-gen to make selecting compatible k8s golang dependencies easier</li> <li>[PR #6535][rmohr] Migrations use digests to reference containerDisks and kernel boot images to ensure disk consistency</li> <li>[PR #6651][ormergi] Kubevirt Conformance plugin now supports passing tests images registry.</li> <li>[PR #6589][iholder-redhat] custom kernel / initrd to boot from is now pre-pulled which improves stability</li> <li>[PR #6199][ormergi] Kubevirt Conformance plugin now supports passing image tag or digest</li> <li>[PR #6477][zcahana] Report DataVolumeError VM status when referenced a DataVolume indicates an error</li> <li>[PR #6593][rhrazdil] Removed python dependencies from virt-launcher and virt-handler containers</li> <li>[PR #6026][akrejcir] Implemented minimal VirtualMachineFlavor functionality.</li> <li>[PR #6570][erkanerol] Use honorLabels instead of labelDrop for namespace label on metrics</li> <li>[PR #6182][jordigilh] adds support for real time workloads</li> <li>[PR #6177][rmohr] Switch the node base images to centos8 stream</li> <li>[PR #6171][zcahana] Report ErrorPvcNotFound/ErrorDataVolumeNotFound VM status when PVC/DV-type volumes reference non-existent objects</li> <li>[PR #6437][VirrageS] Fix deprecated use of watch API to prevent reporting incorrect metrics.</li> <li>[PR #6482][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6375][dhiller] Rely on kubevirtci installing cdi during testing</li> </ul>"},{"location":"release_notes/#v0461","title":"v0.46.1","text":"<p>Released on: Tue Oct 19 15:41:10 2021 +0000</p> <ul> <li>[PR #6557][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> </ul>"},{"location":"release_notes/#v0460","title":"v0.46.0","text":"<p>Released on: Fri Oct 8 21:12:33 2021 +0000</p> <ul> <li>[PR #6425][awels] Hotplug disks are possible when iothreads are enabled.</li> <li>[PR #6297][acardace] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6464][awels] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6465][salanki] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> <li>[PR #6458][vladikr] Tagged SR-IOV interfaces will now appear in the config drive metadata</li> <li>[PR #6446][brybacki] Access mode for virtctl image upload is now optional. This version of virtctl now requires CDI v1.34 or greater</li> <li>[PR #6391][zcahana] Cleanup obsolete permissions from virt-operator's ClusterRole</li> <li>[PR #6419][rthallisey] Fix virt-controller panic caused by lots of deleted VMI events</li> <li>[PR #5972][kwiesmueller] Add a <code>ssh</code> command to <code>virtctl</code> that can be used to open SSH sessions to VMs/VMIs.</li> <li>[PR #6403][jrife] Removed go module pinning to an old version (v0.3.0) of github.com/go-kit/kit</li> <li>[PR #6367][brybacki] virtctl imageupload now uses DataVolume.spec.storage</li> <li>[PR #6198][iholder-redhat] Fire a Prometheus alert when a lot of REST failures are detected in virt-api</li> <li>[PR #6211][davidvossel] cluster-profiler pprof gathering tool and corresponding \"ClusterProfiler\" feature gate</li> <li>[PR #6323][vladikr] switch live migration to use unix sockets</li> <li>[PR #6374][vladikr] Fix the default setting of CPU requests on vmipods</li> <li>[PR #6283][rthallisey] Record the time it takes to delete a VMI and expose it as a metric</li> <li>[PR #6251][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6377][rmohr] Don't fail on failed selinux relabel attempts if selinux is permissive</li> <li>[PR #6308][awels] BugFix: hotplug was broken when using it with a hostpath volume that was on a separate device.</li> <li>[PR #6186][davidvossel] Add resource and verb labels to rest_client_requests_total metric</li> </ul>"},{"location":"release_notes/#v0451","title":"v0.45.1","text":"<p>Released on: Tue Oct 19 15:39:42 2021 +0000</p> <ul> <li>[PR #6537][kubevirt-bot] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> <li>[PR #6556][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6480][kubevirt-bot] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6384][kubevirt-bot] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> </ul>"},{"location":"release_notes/#v0450","title":"v0.45.0","text":"<p>Released on: Wed Sep 8 13:56:47 2021 +0000</p> <ul> <li>[PR #6191][marceloamaral] Addition of perfscale-load-generator to perform stress tests to evaluate the control plane</li> <li>[PR #6248][VirrageS] Reduced logging in hot paths</li> <li>[PR #6079][weihanglo] Hotplug volume can be unplugged at anytime and reattached after a VM restart.</li> <li>[PR #6101][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6204][sradco] This PR adds to each alert the runbook url that points to a runbook that provides additional details on each alert and how to mitigate it.</li> <li>[PR #5974][vladikr] a list of desired mdev types can now be provided in KubeVirt CR to kubevirt to configure these devices on relevant nodes</li> <li>[PR #6147][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #6161][ashleyschuett] Remove HostDevice validation on VMI creation</li> <li>[PR #6078][zcahana] Report ErrImagePull/ImagePullBackOff VM status when image pull errors occur</li> <li>[PR #6176][kwiesmueller] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #6047][ShellyKa13] Add phases to the vm snapshot api, specifically a failure phase</li> <li>[PR #6138][ansijain] NA</li> </ul>"},{"location":"release_notes/#v0443","title":"v0.44.3","text":"<p>Released on: Tue Oct 19 15:38:22 2021 +0000</p> <ul> <li>[PR #6518][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6532][kubevirt-bot] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6536][kubevirt-bot] Fix corrupted DHCP Gateway Option from local DHCP server, leading to rejected IP configuration on Windows VMs.</li> </ul>"},{"location":"release_notes/#v0442","title":"v0.44.2","text":"<p>Released on: Thu Oct 7 12:55:34 2021 +0000</p> <ul> <li>[PR #6479][kubevirt-bot] BugFix: Fixed hotplug race between kubelet and virt-handler when virt-launcher dies unexpectedly.</li> <li>[PR #6392][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6251][rmohr] Better place vcpu threads on host cpus to form more efficient passthrough architectures</li> <li>[PR #6344][kubevirt-bot] BugFix: hotplug was broken when using it with a hostpath volume that was on a separate device.</li> <li>[PR #6263][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6207][kubevirt-bot] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #6101][rmohr] Make k8s client rate limits configurable</li> <li>[PR #6249][kubevirt-bot] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> </ul>"},{"location":"release_notes/#v0441","title":"v0.44.1","text":"<p>Released on: Thu Aug 12 12:28:02 2021 +0000</p> <ul> <li>[PR #6219][kubevirt-bot] Add phases to the vm snapshot api, specifically a failure phase</li> </ul>"},{"location":"release_notes/#v0440","title":"v0.44.0","text":"<p>Released on: Mon Aug 9 14:20:14 2021 +0000</p> <ul> <li>[PR #6058][acardace] Fix virt-launcher exit pod race condition</li> <li>[PR #6035][davidvossel] Addition of perfscale-audit tool for auditing performance of control plane during stress tests</li> <li>[PR #6145][acardace] virt-launcher: disable unencrypted TCP socket for libvirtd.</li> <li>[PR #6163][davidvossel] Handle qemu processes in defunc (zombie) state</li> <li>[PR #6105][ashleyschuett] Add VirtualMachineInstancesPerNode to KubeVirt CR under Spec.Configuration</li> <li>[PR #6104][zcahana] Report FailedUnschedulable VM status when scheduling errors occur</li> <li>[PR #5905][davidvossel] VM CrashLoop detection and Exponential Backoff</li> <li>[PR #6070][acardace] Initiate Live-Migration using a unix socket (exposed by virt-handler) instead of an additional TCP&lt;-&gt;Unix migration proxy started by virt-launcher</li> <li>[PR #5728][vasiliy-ul] Live migration of VMs with hotplug volumes is now enabled</li> <li>[PR #6109][rmohr] Fix virt-controller SCC: Reflect the need for NET_BIND_SERVICE in the virt-controller SCC.</li> <li>[PR #5942][ShellyKa13] Integrate guest agent to online VM snapshot</li> <li>[PR #6034][ashleyschuett] Go version updated to version 1.16.6</li> <li>[PR #6040][yuhaohaoyu] Improved debuggability by keeping the environment of a failed VMI alive.</li> <li>[PR #6068][dhiller] Add check that not all tests have been skipped</li> <li>[PR #6041][xpivarc] [Experimental] Virt-launcher can run as non-root user</li> <li>[PR #6062][iholder-redhat] replace dead \"stress\" binary with new, maintained, \"stress-ng\" binary</li> <li>[PR #6029][mhenriks] CDI to 1.36.0 with DataSource support</li> <li>[PR #4089][victortoso] Add support to USB Redirection with usbredir</li> <li>[PR #5946][vatsalparekh] Add guest-agent based ping probe</li> <li>[PR #6005][acardace] make containerDisk validation memory usage limit configurable</li> <li>[PR #5791][zcahana] Added a READY column to the tabular output of \"kubectl get vm/vmi\"</li> <li>[PR #6006][awels] DataVolumes created by DataVolumeTemplates will follow the associated VMs priority class.</li> <li>[PR #5982][davidvossel] Reduce vmi Update collisions (http code 409) during startup</li> <li>[PR #5891][akalenyu] BugFix: Pending VMIs when creating concurrent bulk of VMs backed by WFFC DVs</li> <li>[PR #5925][rhrazdil] Fix issue with Windows VMs not being assigned IP address configured in network-attachment-definition IPAM.</li> <li>[PR #6007][rmohr] Fix: The bandwidth limitation on migrations is no longer ignored. Caution: The default bandwidth limitation of 64Mi is changed to \"unlimited\" to not break existing installations.</li> <li>[PR #4944][kwiesmueller] Add <code>/portforward</code> subresource to <code>VirtualMachine</code> and <code>VirtualMachineInstance</code> that can tunnel TCP traffic through the API Server using a websocket stream.</li> <li>[PR #5402][alicefr] Integration of libguestfs-tools and added new command <code>guestfs</code> to virtctl</li> <li>[PR #5953][ashleyschuett] Allow Failed VMs to be stopped when using <code>--force --gracePeriod 0</code></li> <li>[PR #5876][mlsorensen] KubeVirt CR supports specifying a runtime class for virt-launcher pods via 'launcherRuntimeClass'.</li> </ul>"},{"location":"release_notes/#v0431","title":"v0.43.1","text":"<p>Released on: Tue Oct 19 15:36:32 2021 +0000</p> <ul> <li>[PR #6555][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6052][kubevirt-bot] make containerDisk validation memory usage limit configurable</li> </ul>"},{"location":"release_notes/#v0430","title":"v0.43.0","text":"<p>Released on: Fri Jul 9 15:46:22 2021 +0000</p> <ul> <li>[PR #5952][mhenriks] Use CDI beta API. CDI v1.20.0 is now the minimum requirement for kubevirt.</li> <li>[PR #5846][rmohr] Add \"spec.cpu.numaTopologyPassthrough\" which allows emulating a host-alligned virtual numa topology for high performance</li> <li>[PR #5894][rmohr] Add <code>spec.migrations.disableTLS</code> to the KubeVirt CR to allow disabling encrypted migrations. They stay secure by default.</li> <li>[PR #5649][awels] Enhancement: remove one attachment pod per disk limit (behavior on upgrade with running VM with hotplugged disks is undefined)</li> <li>[PR #5742][rmohr] VMIs which choose evictionStrategy <code>LifeMigrate</code> and request the <code>invtsc</code> cpuflag are now live-migrateable</li> <li>[PR #5911][dhiller] Bumps kubevirtci, also suppresses kubectl.sh output to avoid confusing checks</li> <li>[PR #5863][xpivarc] Fix: ioerrors don't cause crash-looping of notify server</li> <li>[PR #5867][mlsorensen] New build target added to export virt-* images as a tar archive.</li> <li>[PR #5766][davidvossel] Addition of kubevirt_vmi_phase_transition_seconds_since_creation to monitor how long it takes to transition a VMI to a specific phase from creation time.</li> <li>[PR #5823][dhiller] Change default branch to <code>main</code> for <code>kubevirt/kubevirt</code> repository</li> <li>[PR #5763][nunnatsa] Fix bug 1945589: Prevent migration of VMIs that uses virtiofs</li> <li>[PR #5827][mlsorensen] Auto-provisioned disk images on empty PVCs now leave 128KiB unused to avoid edge cases that run the volume out of space.</li> <li>[PR #5849][davidvossel] Fixes event recording causing a segfault in virt-controller</li> <li>[PR #5797][rhrazdil] Add serviceAccountDisk automatically when Istio is enabled in VMI annotations</li> <li>[PR #5723][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5806][mlsorensen] configmap, secret, and cloud-init raw disks now work when underlying node storage has 4k blocks.</li> <li>[PR #5623][iholder-redhat] [bugfix]: Allow migration of VMs with host-model CPU to migrate only for compatible nodes</li> <li>[PR #5716][rhrazdil] Fix issue with virt-launcher becoming <code>NotReady</code> after migration when Istio is used.</li> <li>[PR #5778][ashleyschuett] Update ca-bundle if it is unable to be parsed</li> <li>[PR #5787][acardace] migrated references of authorization/v1beta1 to authorization/v1</li> <li>[PR #5461][rhrazdil] Add support for Istio proxy when no explicit ports are specified on masquerade interface</li> <li>[PR #5751][acardace] EFI VMIs with secureboot disabled can now be booted even when only OVMF_CODE.secboot.fd and OVMF_VARS.fd are present in the virt-launcher image</li> <li>[PR #5629][andreyod] Support starting Virtual Machine with its guest CPU paused using <code>virtctl start --paused</code></li> <li>[PR #5725][dhiller] Generate REST API coverage report after functional tests</li> <li>[PR #5758][davidvossel] Fixes kubevirt_vmi_phase_count to include all phases, even those that occur before handler hand off.</li> <li>[PR #5745][ashleyschuett] Alert with resource usage exceeds resource requests</li> <li>[PR #5759][mhenriks] Update CDI to 1.34.1</li> <li>[PR #5038][kwiesmueller] Add exec command to VM liveness and readinessProbe executed through the qemu-guest-agent.</li> <li>[PR #5431][alonSadan] Add NFT and IPTables rules to allow port-forward to non-declared ports on the VMI. Declaring ports on VMI will limit</li> </ul>"},{"location":"release_notes/#v0422","title":"v0.42.2","text":"<p>Released on: Tue Oct 19 15:34:37 2021 +0000</p> <ul> <li>[PR #6554][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5887][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5907][kubevirt-bot] Fix: ioerrors don't cause crash-looping of notify server</li> <li>[PR #5871][maiqueb] Fix: do not override with the DHCP server advertising IP with the gateway info.</li> <li>[PR #5875][kubevirt-bot] Update ca-bundle if it is unable to be parsed</li> </ul>"},{"location":"release_notes/#v0421","title":"v0.42.1","text":"<p>Released on: Thu Jun 10 01:31:52 2021 +0000</p> <ul> <li>[PR #5738][rmohr] Stop releasing jinja2 templates of our operator. Kustomize is the preferred way for customizations.</li> <li>[PR #5691][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #5558][ormergi] Drop virt-launcher SYS_RESOURCE capability</li> <li>[PR #5694][davidvossel] Fixes null pointer dereference in migration controller</li> <li>[PR #5416][iholder-redhat] [feature] support booting VMs from a custom kernel/initrd images with custom kernel arguments</li> <li>[PR #5495][iholder-redhat] Go version updated to version 1.16.1.</li> <li>[PR #5502][rmohr] Add downwardMetrics volume to expose a limited set of hots metrics to guests</li> <li>[PR #5601][maya-r] Update libvirt-go to 7.3.0</li> <li>[PR #5661][davidvossel] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5652][rmohr] Automatically discover kube-prometheus installations and configure kubevirt monitoring</li> <li>[PR #5631][davidvossel] Expand backport policy to include logging and debug fixes</li> <li>[PR #5528][zcahana] Introduced a \"status.printableStatus\" field in the VirtualMachine CRD. This field is now displayed in the tabular output of \"kubectl get vm\".</li> <li>[PR #5200][rhrazdil] Add support for Istio proxy traffic routing with masquerade interface. nftables is required for this feature.</li> <li>[PR #5560][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5514][rhrazdil] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5583][dhiller] Reenable coverage</li> <li>[PR #5129][davidvossel] Gracefully shutdown virt-api connections and ensure zero exit code under normal shutdown conditions</li> <li>[PR #5582][dhiller] Fix flaky unit tests</li> <li>[PR #5600][davidvossel] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #5564][omeryahud] virtctl rename support is dropped</li> <li>[PR #5585][iholder-redhat] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5595][zcahana] Fixes adoption of orphan DataVolumes</li> <li>[PR #5566][davidvossel] Release branches are now cut on the first business day of the month rather than the first day.</li> <li>[PR #5108][Omar007] Fixes handling of /proc//mountpoint by working on the device information instead of mount information <li>[PR #5250][mlsorensen] Controller health checks will no longer actively test connectivity to the Kubernetes API. They will rely in health of their watches to determine if they have API connectivity.</li> <li>[PR #5563][ashleyschuett] Set KubeVirt resources flags in the KubeVirt CR</li> <li>[PR #5328][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li>"},{"location":"release_notes/#v0420","title":"v0.42.0","text":"<p>Released on: Tue Jun 8 12:09:49 2021 +0000</p> <ul> <li>[PR #5738][rmohr] Stop releasing jinja2 templates of our operator. Kustomize is the preferred way for customizations.</li> <li>[PR #5691][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #5558][ormergi] Drop virt-launcher SYS_RESOURCE capability</li> <li>[PR #5694][davidvossel] Fixes null pointer dereference in migration controller</li> <li>[PR #5416][iholder-redhat] [feature] support booting VMs from a custom kernel/initrd images with custom kernel arguments</li> <li>[PR #5495][iholder-redhat] Go version updated to version 1.16.1.</li> <li>[PR #5502][rmohr] Add downwardMetrics volume to expose a limited set of hots metrics to guests</li> <li>[PR #5601][maya-r] Update libvirt-go to 7.3.0</li> <li>[PR #5661][davidvossel] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5652][rmohr] Automatically discover kube-prometheus installations and configure kubevirt monitoring</li> <li>[PR #5631][davidvossel] Expand backport policy to include logging and debug fixes</li> <li>[PR #5528][zcahana] Introduced a \"status.printableStatus\" field in the VirtualMachine CRD. This field is now displayed in the tabular output of \"kubectl get vm\".</li> <li>[PR #5200][rhrazdil] Add support for Istio proxy traffic routing with masquerade interface. nftables is required for this feature.</li> <li>[PR #5560][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5514][rhrazdil] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5583][dhiller] Reenable coverage</li> <li>[PR #5129][davidvossel] Gracefully shutdown virt-api connections and ensure zero exit code under normal shutdown conditions</li> <li>[PR #5582][dhiller] Fix flaky unit tests</li> <li>[PR #5600][davidvossel] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #5564][omeryahud] virtctl rename support is dropped</li> <li>[PR #5585][iholder-redhat] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5595][zcahana] Fixes adoption of orphan DataVolumes</li> <li>[PR #5566][davidvossel] Release branches are now cut on the first business day of the month rather than the first day.</li> <li>[PR #5108][Omar007] Fixes handling of /proc//mountpoint by working on the device information instead of mount information <li>[PR #5250][mlsorensen] Controller health checks will no longer actively test connectivity to the Kubernetes API. They will rely in health of their watches to determine if they have API connectivity.</li> <li>[PR #5563][ashleyschuett] Set KubeVirt resources flags in the KubeVirt CR</li> <li>[PR #5328][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li>"},{"location":"release_notes/#v0414","title":"v0.41.4","text":"<p>Released on: Tue Oct 19 15:31:59 2021 +0000</p> <ul> <li>[PR #6573][acardace] mutate migration PDBs instead of creating an additional one for the duration of the migration.</li> <li>[PR #6517][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6333][acardace] Fix virt-launcher exit pod race condition</li> <li>[PR #6401][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #6147][rmohr] Fix rbac permissions for freeze/unfreeze, addvolume/removevolume, guestosinfo, filesystemlist and userlist</li> <li>[PR #5673][kubevirt-bot] Improved logging around VM/VMI shutdown and restart</li> <li>[PR #6227][kwiesmueller] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> </ul>"},{"location":"release_notes/#v0413","title":"v0.41.3","text":"<p>Released on: Thu Aug 12 16:35:43 2021 +0000</p> <ul> <li>[PR #6196][ashleyschuett] Allow multiple shutdown events to ensure the event is received by ACPI</li> <li>[PR #6194][kubevirt-bot] Allow Failed VMs to be stopped when using <code>--force --gracePeriod 0</code></li> <li>[PR #6039][akalenyu] BugFix: Pending VMIs when creating concurrent bulk of VMs backed by WFFC DVs</li> <li>[PR #5917][davidvossel] Fixes event recording causing a segfault in virt-controller</li> <li>[PR #5886][ashleyschuett] Allow virtctl to stop VM and ignore the graceful shutdown period</li> <li>[PR #5866][xpivarc] Fix: Kubevirt build with golang 1.14+ will not fail on validation of container disk with memory allocation error</li> <li>[PR #5873][kubevirt-bot] Update ca-bundle if it is unable to be parsed</li> <li>[PR #5822][kubevirt-bot] migrated references of authorization/v1beta1 to authorization/v1</li> <li>[PR #5704][davidvossel] Fix virt-controller clobbering in progress vmi migration state during virt handler handoff</li> <li>[PR #5707][kubevirt-bot] Fixes null pointer dereference in migration controller</li> <li>[PR #5685][stu-gott] [bugfix] - reject VM defined with volume with no matching disk</li> <li>[PR #5670][stu-gott] Validation/Mutation webhooks now explicitly define a 10 second timeout period</li> <li>[PR #5653][kubevirt-bot] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5644][kubevirt-bot] Fix live-migration failing when VM with masquarade iface has explicitly specified any of these ports: 22222, 49152, 49153</li> <li>[PR #5646][kubevirt-bot] virtctl rename support is dropped</li> </ul>"},{"location":"release_notes/#v0412","title":"v0.41.2","text":"<p>Released on: Wed Jul 28 12:13:19 2021 -0400</p>"},{"location":"release_notes/#v0411","title":"v0.41.1","text":"<p>Released on: Wed Jul 28 12:08:42 2021 -0400</p>"},{"location":"release_notes/#v0410","title":"v0.41.0","text":"<p>Released on: Wed May 12 14:30:49 2021 +0000</p> <ul> <li>[PR #5586][kubevirt-bot] This version of KubeVirt includes upgraded virtualization technology based on libvirt 7.0.0 and QEMU 5.2.0.</li> <li>[PR #5344][ashleyschuett] Reconcile PrometheusRules and ServiceMonitor resources</li> <li>[PR #5542][andreyod] Add startStrategy field to VMI spec to allow Virtual Machine start in paused state.</li> <li>[PR #5459][ashleyschuett] Reconcile service resource</li> <li>[PR #5520][ashleyschuett] Reconcile required labels and annotations on ConfigMap resources</li> <li>[PR #5533][rmohr] Fix <code>docker save</code> and <code>docker push</code> issues with released kubevirt images</li> <li>[PR #5428][oshoval] virt-launcher now populates domain's guestOS info and interfaces status according guest agent also when doing periodic resyncs.</li> <li>[PR #5410][ashleyschuett] Reconcile ServiceAccount resources</li> <li>[PR #5109][Omar007] Add support for specifying a logical and physical block size for disk devices</li> <li>[PR #5471][ashleyschuett] Reconcile APIService resources</li> <li>[PR #5513][ashleyschuett] Reconcile Secret resources</li> <li>[PR #5496][davidvossel] Improvements to migration proxy logging</li> <li>[PR #5376][ashleyschuett] Reconcile CustomResourceDefinition resources</li> <li>[PR #5435][AlonaKaplan] Support dual stack service on \"virtctl expose\"-</li> <li>[PR #5425][davidvossel] Fixes VM restart during eviction when EvictionStrategy=LiveMigrate</li> <li>[PR #5423][ashleyschuett] Add resource requests to virt-controller, virt-api, virt-operator and virt-handler</li> <li>[PR #5343][erkanerol] Some cleanups and small additions to the storage metrics</li> <li>[PR #4682][stu-gott] Updated Guest Agent Version compatibility check. The new approach is much more accurate.</li> <li>[PR #5485][rmohr] Fix fallback to iptables if nftables is not used on the host on arm64</li> <li>[PR #5426][rmohr] Fix fallback to iptables if nftables is not used on the host</li> <li>[PR #5403][tiraboschi] Added a kubevirt_ prefix to several recording rules and metrics</li> <li>[PR #5241][stu-gott] Introduced Duration and RenewBefore parameters for cert rotation. Previous values are now deprecated.</li> <li>[PR #5463][acardace] Fixes upgrades from KubeVirt v0.36</li> <li>[PR #5456][zhlhahaha] Enable arm64 cross-compilation</li> <li>[PR #3310][davidvossel] Doc outlines our Kubernetes version compatibility commitment</li> <li>[PR #3383][EdDev] Add <code>vmIPv6NetworkCIDR</code> under <code>NetworkSource.pod</code> to support custom IPv6 CIDR for the vm network when using masquerade binding.</li> <li>[PR #3415][zhlhahaha] Make kubevirt code fit for arm64 support. No testing is at this stage performed against arm64 at this point.</li> <li>[PR #5147][xpivarc] Remove CAP_NET_ADMIN from the virt-launcher pod(second take).</li> <li>[PR #5351][awels] Support hotplug with virtctl using addvolume and removevolume commands</li> <li>[PR #5050][ashleyschuett] Fire Prometheus Alert when a vmi is orphaned for more than an hour</li> </ul>"},{"location":"release_notes/#v0401","title":"v0.40.1","text":"<p>Released on: Tue Oct 19 13:33:33 2021 +0000</p> <ul> <li>[PR #6598][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #6287][kubevirt-bot] Fix goroutine leak in virt-handler, potentially causing issues with a high turnover of VMIs.</li> <li>[PR #5559][kubevirt-bot] Fix <code>docker save</code> issues with kubevirt images</li> <li>[PR #5500][kubevirt-bot] Support hotplug with virtctl using addvolume and removevolume commands</li> </ul>"},{"location":"release_notes/#v0400","title":"v0.40.0","text":"<p>Released on: Mon Apr 19 12:25:41 2021 +0000</p> <ul> <li>[PR #5467][rmohr] Fixes upgrades from KubeVirt v0.36</li> <li>[PR #5350][jean-edouard] Removal of entire <code>permittedHostDevices</code> section will now remove all user-defined host device plugins.</li> <li>[PR #5242][jean-edouard] Creating more than 1 migration at the same time for a given VMI will now fail</li> <li>[PR #4907][vasiliy-ul] Initial cgroupv2 support</li> <li>[PR #5324][jean-edouard] Default feature gates can now be defined in the provider configuration.</li> <li>[PR #5006][alicefr] Add discard=unmap option</li> <li>[PR #5022][davidvossel] Fixes race condition between operator adding service and webhooks that can result in installs/uninstalls failing</li> <li>[PR #5310][ashleyschuett] Reconcile CRD resources</li> <li>[PR #5102][iholder-redhat] Go version updated to 1.14.14</li> <li>[PR #4746][ashleyschuett] Reconcile Deployments, DaemonSets, MutatingWebhookConfigurations and ValidatingWebhookConfigurations</li> <li>[PR #5037][ormergi] Hot-plug SR-IOV VF interfaces to VM's post a successful migration.</li> <li>[PR #5269][mlsorensen] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> <li>[PR #5138][davidvossel] virt-handler now waits up to 5 minutes for all migrations on the node to complete before shutting down.</li> <li>[PR #5191][yuvalturg] Added a metric for monitoring CPU affinity</li> <li>[PR #5215][xphyr] Enable detection of Intel GVT-g vGPU.</li> <li>[PR #4760][rmohr] Make virt-handler heartbeat more efficient and robust: Only one combined PATCH and no need to detect different cluster types anymore.</li> <li>[PR #5091][iholder-redhat] QEMU SeaBios debug logs are being seen as part of virt-launcher log.</li> <li>[PR #5221][rmohr] Remove  workload placement validation webhook which blocks placement updates when VMIs are running</li> <li>[PR #5128][yuvalturg] Modified memory related metrics by adding several new metrics and splitting the swap traffic bytes metric</li> <li>[PR #5084][ashleyschuett] Add validation to CustomizeComponents object on the KubeVirt resource</li> <li>[PR #5182][davidvossel] New [release-blocker] functional test marker to signify tests that can never be disabled before making a release</li> <li>[PR #5137][davidvossel] Added our policy around release branch backporting in docs/release-branch-backporting.md</li> <li>[PR #5096][yuvalturg] Modified networking metrics by adding new metrics, splitting existing ones by rx/tx and using the device alias for the interface name when available</li> <li>[PR #5088][awels] Hotplug works with hostpath storage.</li> <li>[PR #4908][dhiller] Move travis tag and master builds to kubevirt prow.</li> <li>[PR #4741][EdDev] Allow live migration for SR-IOV VM/s without preserving the VF interfaces.</li> </ul>"},{"location":"release_notes/#v0392","title":"v0.39.2","text":"<p>Released on: Tue Oct 19 13:29:33 2021 +0000</p> <ul> <li>[PR #6597][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5854][rthallisey] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> <li>[PR #5561][kubevirt-bot] Fix <code>docker save</code> issues with kubevirt images</li> </ul>"},{"location":"release_notes/#v0391","title":"v0.39.1","text":"<p>Released on: Tue Apr 13 12:10:13 2021 +0000</p>"},{"location":"release_notes/#v0390","title":"v0.39.0","text":"<p>Released on: Wed Mar 10 14:51:58 2021 +0000</p> <ul> <li>[PR #5010][jean-edouard] Migrated VMs stay persistent and can therefore survive S3, among other things.</li> <li>[PR #4952][ashleyschuett] Create warning NodeUnresponsive event if a node is running a VMI pod but not a virt-handler pod</li> <li>[PR #4686][davidvossel] Automated workload updates via new KubeVirt WorkloadUpdateStrategy API</li> <li>[PR #4886][awels] Hotplug support for WFFC datavolumes.</li> <li>[PR #5026][AlonaKaplan] virt-launcher, masquerade binding - prefer nft over iptables.</li> <li>[PR #4921][borod108] Added support for Sysprep in the API. A user can now add a answer file through a ConfigMap or a Secret. The User Guide is updated accordingly. /kind feature</li> <li>[PR #4874][ormergi] Add new feature-gate SRIOVLiveMigration,</li> <li>[PR #4917][iholder-redhat] Now it is possible to enable QEMU SeaBios debug logs setting virt-launcher log verbosity to be greater than 5.</li> <li>[PR #4966][arnongilboa] Solve virtctl \"Error when closing file ... file already closed\" that shows after successful image upload</li> <li>[PR #4489][salanki] Fix a bug where a disk.img file was created on filesystems mounted via Virtio-FS</li> <li>[PR #4982][xpivarc] Fixing handling of transient domain</li> <li>[PR #4984][ashleyschuett] Change customizeComponents.patches such that '*' resourceName or resourceType matches all, all fields of a patch (type, patch, resourceName, resourceType) are now required.</li> <li>[PR #4972][vladikr] allow disabling pvspinlock to support older guest kernels</li> <li>[PR #4927][yuhaohaoyu] Fix of XML and JSON marshalling/unmarshalling for user defined device alias names which can make migrations fail.</li> <li>[PR #4552][rthallisey] VMs using bridged networking will survive a kubelet restart by having kubevirt create a dummy interface on the virt-launcher pods, so that some Kubernetes CNIs, that have implemented the <code>CHECK</code> RPC call, will not cause VMI pods to enter a failed state.</li> <li>[PR #4883][iholder-redhat] Bug fixed: Enabling libvirt debug logs only if debugLogs label value is \"true\", disabling otherwise.</li> <li>[PR #4840][alicefr] Generate k8s events on IO errors</li> <li>[PR #4940][vladikr] permittedHostDevices will support both upper and lowercase letters in the device ID</li> </ul>"},{"location":"release_notes/#v0382","title":"v0.38.2","text":"<p>Released on: Tue Oct 19 13:24:57 2021 +0000</p> <ul> <li>[PR #6596][jean-edouard] VMs with cloud-init data should now properly migrate from older KubeVirt versions</li> <li>[PR #5853][rthallisey] Prometheus metrics scraped from virt-handler are now served from the VMI informer cache, rather than calling back to the Kubernetes API for VMI information.</li> </ul>"},{"location":"release_notes/#v0381","title":"v0.38.1","text":"<p>Released on: Mon Feb 8 19:00:24 2021 +0000</p> <ul> <li>[PR #4870][qinqon] Bump k8s deps to 0.20.2</li> <li>[PR #4571][yuvalturg] Added os, workflow and flavor labels to the kubevirt_vmi_phase_count metric</li> <li>[PR #4659][salanki] Fixed an issue where non-root users inside a guest could not write to a Virtio-FS mount.</li> <li>[PR #4844][xpivarc] Fixed limits/requests to accept int again</li> <li>[PR #4850][rmohr] virtio-scsi now respects the useTransitionalVirtio flag instead of assigning a virtio version depending on the machine layout</li> <li>[PR #4672][vladikr] allow increasing logging verbosity of infra components in KubeVirt CR</li> <li>[PR #4838][rmohr] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> <li>[PR #4806][rmohr] Make the mutating webhooks for VMIs and VMs  required to avoid letting entities into the cluster which are not properly defaulted</li> <li>[PR #4779][brybacki] Error message on virtctl image-upload to WaitForFirstConsumer DV</li> <li>[PR #4749][davidvossel] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> <li>[PR #4772][jean-edouard] Faster VMI phase transitions thanks to an increased number of VMI watch threads in virt-controller</li> <li>[PR #4730][rmohr] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> </ul>"},{"location":"release_notes/#v0380","title":"v0.38.0","text":"<p>Released on: Mon Feb 8 13:15:32 2021 +0000</p> <ul> <li>[PR #4870][qinqon] Bump k8s deps to 0.20.2</li> <li>[PR #4571][yuvalturg] Added os, workflow and flavor labels to the kubevirt_vmi_phase_count metric</li> <li>[PR #4659][salanki] Fixed an issue where non-root users inside a guest could not write to a Virtio-FS mount.</li> <li>[PR #4844][xpivarc] Fixed limits/requests to accept int again</li> <li>[PR #4850][rmohr] virtio-scsi now respects the useTransitionalVirtio flag instead of assigning a virtio version depending on the machine layout</li> <li>[PR #4672][vladikr] allow increasing logging verbosity of infra components in KubeVirt CR</li> <li>[PR #4838][rmohr] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> <li>[PR #4806][rmohr] Make the mutating webhooks for VMIs and VMs  required to avoid letting entities into the cluster which are not properly defaulted</li> <li>[PR #4779][brybacki] Error message on virtctl image-upload to WaitForFirstConsumer DV</li> <li>[PR #4749][davidvossel] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> <li>[PR #4772][jean-edouard] Faster VMI phase transitions thanks to an increased number of VMI watch threads in virt-controller</li> <li>[PR #4730][rmohr] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> </ul>"},{"location":"release_notes/#v0372","title":"v0.37.2","text":"<p>Released on: Wed Jan 27 17:49:36 2021 +0000</p> <ul> <li>[PR #4872][kubevirt-bot] Add spec.domain.devices.useVirtioTransitional boolean to support virtio-transitional for old guests</li> <li>[PR #4855][kubevirt-bot] Fix an issue where it may not be able to update the KubeVirt CR after creation for up to minutes due to certificate propagation delays</li> </ul>"},{"location":"release_notes/#v0371","title":"v0.37.1","text":"<p>Released on: Thu Jan 21 16:20:52 2021 +0000</p> <ul> <li>[PR #4842][kubevirt-bot] KUBEVIRT_CLIENT_GO_SCHEME_REGISTRATION_VERSION env var for specifying exactly what client-go scheme version is registered</li> </ul>"},{"location":"release_notes/#v0370","title":"v0.37.0","text":"<p>Released on: Mon Jan 18 17:57:03 2021 +0000</p> <ul> <li>[PR #4654][AlonaKaplan] Introduce virt-launcher DHCPv6 server.</li> <li>[PR #4669][kwiesmueller] Add nodeSelector to kubevirt components restricting them to run on linux nodes only.</li> <li>[PR #4648][davidvossel] Update libvirt base container to be based of packages in rhel-av 8.3</li> <li>[PR #4653][qinqon] Allow configure cloud-init with networkData only.</li> <li>[PR #4644][ashleyschuett] Operator validation webhook will deny updates to the workloads object of the KubeVirt CR if there are running VMIs</li> <li>[PR #3349][davidvossel] KubeVirt v1 GA api</li> <li>[PR #4645][maiqueb] Re-introduce the CAP_NET_ADMIN, to allow migration of VMs already having it.</li> <li>[PR #4546][yuhaohaoyu] Failure detection and handling for VM with EFI Insecure Boot in KubeVirt environments where EFI Insecure Boot is not supported by design.</li> <li>[PR #4625][awels] virtctl upload now shows error when specifying access mode of ReadOnlyMany</li> <li>[PR #4396][xpivarc] KubeVirt is now explainable!</li> <li>[PR #4517][danielBelenky] Fix guest agent reporting.</li> </ul>"},{"location":"release_notes/#v0362","title":"v0.36.2","text":"<p>Released on: Mon Feb 22 10:20:40 2021 -0500</p>"},{"location":"release_notes/#v0361","title":"v0.36.1","text":"<p>Released on: Tue Jan 19 12:30:33 2021 +0100</p>"},{"location":"release_notes/#v0360","title":"v0.36.0","text":"<p>Released on: Wed Dec 16 14:30:37 2020 +0000</p> <ul> <li>[PR #4667][kubevirt-bot] Update libvirt base container to be based of packages in rhel-av 8.3</li> <li>[PR #4634][kubevirt-bot] Failure detection and handling for VM with EFI Insecure Boot in KubeVirt environments where EFI Insecure Boot is not supported by design.</li> <li>[PR #4647][kubevirt-bot] Re-introduce the CAP_NET_ADMIN, to allow migration of VMs already having it.</li> <li>[PR #4627][kubevirt-bot] Fix guest agent reporting.</li> <li>[PR #4458][awels] It is now possible to hotplug DataVolume and PVC volumes into a running Virtual Machine.</li> <li>[PR #4025][brybacki] Adds a special handling for DataVolumes in WaitForFirstConsumer state to support CDI's delayed binding mode.</li> <li>[PR #4217][mfranczy] Set only an IP address for interfaces reported by qemu-guest-agent. Previously that was CIDR.</li> <li>[PR #4195][davidvossel] AccessCredentials API for dynamic user/password and ssh public key injection</li> <li>[PR #4335][oshoval] VMI status displays SRIOV interfaces with their network name only when they have originally</li> <li>[PR #4408][andreabolognani] This version of KubeVirt includes upgraded virtualization technology based on libvirt 6.6.0 and QEMU 5.1.0.</li> <li>[PR #4514][ArthurSens] <code>domain</code> label removed from metric <code>kubevirt_vmi_memory_unused_bytes</code></li> <li>[PR #4542][danielBelenky] Fix double migration on node evacuation</li> <li>[PR #4506][maiqueb] Remove CAP_NET_ADMIN from the virt-launcher pod.</li> <li>[PR #4501][AlonaKaplan] CAP_NET_RAW removed from virt-launcher.</li> <li>[PR #4488][salanki] Disable Virtio-FS metadata cache to prevent OOM conditions on the host.</li> <li>[PR #3937][vladikr] Generalize host devices assignment. Provides an interface between kubevirt and external device plugins. Provides a mechanism for accesslisting host devices.</li> <li>[PR #4443][rmohr] All kubevirt webhooks support now dry-runs.</li> </ul>"},{"location":"release_notes/#v0350","title":"v0.35.0","text":"<p>Released on: Mon Nov 9 13:08:27 2020 +0000</p> <ul> <li>[PR #4409][vladikr] Increase the static memory overhead by 10Mi</li> <li>[PR #4272][maiqueb] Add <code>ip-family</code> to the <code>virtctl expose</code> command.</li> <li>[PR #4398][rmohr] VMIs reflect deleted stuck virt-launcher pods with the \"PodTerminating\" reason in the ready condition. The VMIRs detects this reason and immediately creates replacement VMIs.</li> <li>[PR #4393][salanki] Disable legacy service links in <code>virt-launcher</code> Pods to speed up Pod instantiation and decrease Kubelet load in namespaces with many services.</li> <li>[PR #2935][maiqueb] Add the macvtap bind mechanism.</li> <li>[PR #4132][mstarostik] fixes a bug that prevented unique device name allocation when configuring both SCSI and SATA drives</li> <li>[PR #3257][xpivarc] Added support of <code>kubectl explain</code> for Kubevirt resources.</li> <li>[PR #4288][ezrasilvera] Adding DownwardAPI volumes type</li> <li>[PR #4233][maya-r] Update base image used for pods to Fedora 31.</li> <li>[PR #4192][xpivarc] We now run gosec in Kubevirt</li> <li>[PR #4328][stu-gott] Version 2.x QEMU guest agents are supported.</li> <li>[PR #4289][AlonaKaplan] Masquerade binding - set the virt-launcher pod interface MTU on the bridge.</li> <li>[PR #4300][maiqueb] Update the NetworkInterfaceMultiqueue openAPI documentation to better specify its semantics within KubeVirt.</li> <li>[PR #4277][awels] PVCs populated by DVs are now allowed as volumes.</li> <li>[PR #4265][dhiller] Fix virtctl help text when running as a plugin</li> <li>[PR #4273][dhiller] Only run Travis build for PRs against release branches</li> </ul>"},{"location":"release_notes/#v0342","title":"v0.34.2","text":"<p>Released on: Tue Nov 17 08:13:22 2020 -0500</p>"},{"location":"release_notes/#v0341","title":"v0.34.1","text":"<p>Released on: Mon Nov 16 08:22:56 2020 -0500</p>"},{"location":"release_notes/#v0340","title":"v0.34.0","text":"<p>Released on: Wed Oct 7 13:59:50 2020 +0300</p> <ul> <li>[PR #4315][kubevirt-bot] PVCs populated by DVs are now allowed as volumes.</li> <li>[PR #3837][jean-edouard] VM interfaces with no <code>bootOrder</code> will no longer be candidates for boot when using the BIOS bootloader, as documented</li> <li>[PR #3879][ashleyschuett] KubeVirt should now be configured through the KubeVirt CR <code>configuration</code> key. The usage of the kubevirt-config configMap will be deprecated in the future.</li> <li>[PR #4074][stu-gott] Fixed bug preventing non-admin users from pausing/unpausing VMs</li> <li>[PR #4252][rhrazdil] Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1853911</li> <li>[PR #4016][ashleyschuett] Allow for post copy VMI migrations</li> <li>[PR #4235][davidvossel] Fixes timeout failure that occurs when pulling large containerDisk images</li> <li>[PR #4263][rmohr] Add readiness and liveness probes to virt-handler, to clearly indicate readiness</li> <li>[PR #4248][maiqueb] always compile KubeVirt with selinux support on pure go builds.</li> <li>[PR #4012][danielBelenky] Added support for the eviction API for VMIs with eviction strategy. This enables VMIs to be live-migrated when the node is drained or when the descheduler wants to move a VMI to a different node.</li> <li>[PR #4075][ArthurSens] Metric kubevirt_vmi_vcpu_seconds' state label is now exposed as a human-readable state instead of an integer</li> <li>[PR #4162][vladikr] introduce a cpuAllocationRatio config parameter to normalize the number of CPUs requested for a pod, based on the number of vCPUs</li> <li>[PR #4177][maiqueb] Use vishvananda/netlink instead of songgao/water to create tap devices.</li> <li>[PR #4092][stu-gott] Allow specifying nodeSelectors, affinity and tolerations to control where KubeVirt components will run</li> <li>[PR #3927][ArthurSens] Adds new metric kubevirt_vmi_memory_unused_bytes</li> <li>[PR #3493][vladikr] virtio-fs is being added as experimental, protected by a feature-gate that needs to be enabled in the kubevirt config by the administrator</li> <li>[PR #4193][mhenriks] Add snapshot.kubevirt.io to admin/edit/view roles</li> <li>[PR #4149][qinqon] Bump kubevirtci to k8s-1.19</li> <li>[PR #3471][crobinso] Allow hiding that the VM is running on KVM, so that Nvidia graphics cards can be passed through</li> <li>[PR #4115][phoracek] Add conformance automation and manifest publishing</li> <li>[PR #3733][davidvossel] each PRs description.</li> <li>[PR #4082][mhenriks] VirtualMachineRestore API and implementation</li> <li>[PR #4154][davidvossel] Fixes issue with Service endpoints not being updated properly in place during KubeVirt updates.</li> <li>[PR #3289][vatsalparekh] Add option to run only VNC Proxy in virtctl</li> <li>[PR #4027][alicefr] Added memfd as default memory backend for hugepages. This introduces the new annotation kubevirt.io/memfd to disable memfd as default and fallback to the previous behavior.</li> <li>[PR #3612][ashleyschuett] Adds <code>customizeComponents</code> to the kubevirt api</li> <li>[PR #4029][cchengleo] Fix an issue which prevented virt-operator from installing monitoring resources in custom namespaces.</li> <li>[PR #4031][rmohr] Initial support for sonobuoy for conformance testing</li> </ul>"},{"location":"release_notes/#v0330","title":"v0.33.0","text":"<p>Released on: Tue Sep 15 14:46:00 2020 +0000</p> <ul> <li>[PR #3226][vatsalparekh] Added tests to verify custom pciAddress slots and function</li> <li>[PR #4048][davidvossel] Improved reliability for failed migration retries</li> <li>[PR #3585][mhenriks] \"virtctl image-upload pvc ...\" will create the PVC if it does not exist</li> <li>[PR #3945][xpivarc] KubeVirt is now being built with Go1.13.14</li> <li>[PR #3845][ArthurSens] action required: The domain label from VMI metrics is being removed and may break dashboards that use the domain label to identify VMIs. Use name and namespace labels instead</li> <li>[PR #4011][dhiller] ppc64le arch has been disabled for the moment, see https://github.com/kubevirt/kubevirt/issues/4037</li> <li>[PR #3875][stu-gott] Resources created by KubeVirt are now labelled more clearly in terms of relationship and role.</li> <li>[PR #3791][ashleyschuett] make node as kubevirt.io/schedulable=false on virt-handler restart</li> <li>[PR #3998][vladikr] the local provider is usable again.</li> <li>[PR #3290][maiqueb] Have virt-handler (KubeVirt agent) create the tap devices on behalf of the virt-launchers.</li> <li>[PR #3957][AlonaKaplan] virt-launcher support Ipv6 on dual stack cluster.</li> <li>[PR #3952][davidvossel] Fixes rare situation where vmi may not properly terminate if failure occurs before domain starts.</li> <li>[PR #3973][xpivarc] Fixes VMs with clock.timezone set.</li> <li>[PR #3923][danielBelenky] Add support to configure QEMU I/O mode for VMIs</li> <li>[PR #3889][rmohr] The status fields for our CRDs are now protected on normal PATCH and PUT operations.The /status subresource is now used where possible for status updates.</li> <li>[PR #3568][xpivarc] Guest swap metrics available</li> </ul>"},{"location":"release_notes/#v0320","title":"v0.32.0","text":"<p>Released on: Tue Aug 11 19:21:56 2020 +0000</p> <ul> <li>[PR #3921][vladikr] use correct memory units in libvirt xml</li> <li>[PR #3893][davidvossel] Adds recurring period that rsyncs virt-launcher domains with virt-handler</li> <li>[PR #3880][sgarbour] Better error message when input parameters are not the expected number of parameters for each argument. Help menu will popup in case the number of parameters is incorrect.</li> <li>[PR #3785][xpivarc] Vcpu wait metrics available</li> <li>[PR #3642][vatsalparekh] Add a way to update VMI Status with latest Pod IP for Masquerade bindings</li> <li>[PR #3636][ArthurSens] Adds kubernetes metadata.labels as VMI metrics' label</li> <li>[PR #3825][awels] Virtctl now prints error messages from the response body on upload errors.</li> <li>[PR #3830][davidvossel] Fixes re-establishing domain notify client connections when domain notify server restarts due to an error event.</li> <li>[PR #3778][danielBelenky] Do not emit a SyncFailed event if we fail to sync a VMI in a final state</li> <li>[PR #3803][andreabolognani] Not sure what to write here (see above)</li> <li>[PR #2694][rmohr] Use native go libraries for selinux to not rely on python-selinux tools like semanage, which are not always present.</li> <li>[PR #3692][victortoso] QEMU logs can now be fetched from outside the pod</li> <li>[PR #3738][enp0s3] Restrict creation of VMI if it has labels that are used internally by Kubevirt components.</li> <li>[PR #3725][danielBelenky] The tests binary is now part of the release and can be consumed from the GitHub release page.</li> <li>[PR #3684][rmohr] Log if critical devices, like kvm, which virt-handler wants to expose are not present on the node.</li> <li>[PR #3166][petrkotas] Introduce new virtctl commands:</li> <li>[PR #3708][andreabolognani] Make qemu work on GCE by pulling in a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1822682</li> </ul>"},{"location":"release_notes/#v0310","title":"v0.31.0","text":"<p>Released on: Thu Jul 9 16:08:18 2020 +0300</p> <ul> <li>[PR 3690][davidvossel] Update go-grpc dependency to v1.30.0 in order to improve stability</li> <li>[PR 3628][AlonaKaplan] Avoid virt-handler crash in case of virt-launcher network configuration error</li> <li>[PR 3635][jean-edouard] The \"HostDisk\" feature gate has to be enabled to use hostDisks</li> <li>[PR 3641][vatsalparekh] Reverts kubevirt/kubevirt#3488 because CI seems to have merged it without all tests passing</li> <li>[PR 3488][vatsalparekh] Add a way to update VMI Status with latest Pod IP for Masquerade bindings</li> <li>[PR 3406][tomob] If a PVC was created by a DataVolume, it cannot be used as a Volume Source for a VM. The owning DataVolume has to be used instead.</li> <li>[PR 3566][kraxel] added: TigerVNC support for linux &amp; windows</li> <li>[PR 3529][jean-edouard] Enabling EFI will also enable Secure Boot, which requires SMM to be enabled.</li> <li>[PR 3455][ashleyschuett] Add KubevirtConfiguration, MigrationConfiguration, DeveloperConfiguration and NetworkConfiguration to API-types</li> <li>[PR 3520][rmohr] Fix hot-looping on the  VMI sync-condition if errors happen during the Scheduled phase of a VMI</li> <li>[PR 3220][mhenriks] API and controller/webhook for VirtualMachineSnapshots</li> </ul>"},{"location":"release_notes/#v0307","title":"v0.30.7","text":"<p>Released on: Mon Oct 26 11:57:21 2020 -0400</p>"},{"location":"release_notes/#v0306","title":"v0.30.6","text":"<p>Released on: Wed Aug 12 10:55:31 2020 +0200</p>"},{"location":"release_notes/#v0305","title":"v0.30.5","text":"<p>Released on: Fri Jul 17 05:26:37 2020 -0400</p>"},{"location":"release_notes/#v0304","title":"v0.30.4","text":"<p>Released on: Fri Jul 10 07:44:00 2020 -0400</p>"},{"location":"release_notes/#v0303","title":"v0.30.3","text":"<p>Released on: Tue Jun 30 17:39:42 2020 -0400</p>"},{"location":"release_notes/#v0302","title":"v0.30.2","text":"<p>Released on: Thu Jun 25 17:05:59 2020 -0400</p>"},{"location":"release_notes/#v0301","title":"v0.30.1","text":"<p>Released on: Tue Jun 16 13:10:17 2020 -0400</p>"},{"location":"release_notes/#v0300","title":"v0.30.0","text":"<p>Released on: Fri Jun 5 12:19:57 2020 +0200</p> <ul> <li>Tests: Many more test fixes</li> <li>Security: Introduce a custom SELinux policy for virt-launcher</li> <li>More user friendly IPv6 default CIDR for IPv6 addresses</li> <li>Fix OpenAPI compatibility issues by switching to openapi-gen</li> <li>Improved support for EFI boot (configurable OVMF path and test fixes)</li> <li>Improved VMI IP reporting</li> <li>Support propagation of annotations from VMI to pods</li> <li>Support for more fine grained (NET_RAW( capability granting to virt-launcher</li> <li>Support for eventual consistency with DataVolumes</li> </ul>"},{"location":"release_notes/#v0292","title":"v0.29.2","text":"<p>Released on: Mon May 25 21:15:30 2020 +0200</p>"},{"location":"release_notes/#v0291","title":"v0.29.1","text":"<p>Released on: Tue May 19 10:03:27 2020 +0200</p>"},{"location":"release_notes/#v0290","title":"v0.29.0","text":"<p>Released on: Wed May 6 15:01:57 2020 +0200</p> <ul> <li>Tests: Many many test fixes</li> <li>Tests: Many more test fixes</li> <li>CI: Add lane with SELinux enabled</li> <li>CI: Drop PPC64 support for now</li> <li>Drop Genie support</li> <li>Drop the use of hostPaths in the virt-launcher for improved security</li> <li>Support priority classes for important components</li> <li>Support IPv6 over masquerade binding</li> <li>Support certificate rotations based on shared secrets</li> <li>Support for VM ready condition</li> <li>Support for advanced node labelling (supported CPU Families and machine types)</li> </ul>"},{"location":"release_notes/#v0280","title":"v0.28.0","text":"<p>Released on: Thu Apr 9 23:01:29 2020 +0200</p> <ul> <li>CI: Try to discover flaky tests before merge</li> <li>Fix the use of priorityClasses</li> <li>Fix guest memory overhead calculation</li> <li>Fix SR-IOV device overhead requirements</li> <li>Fix loading of tun module during virt-handler initialization</li> <li>Fixes for several test cases</li> <li>Fixes to support running with container_t</li> <li>Support for renaming a VM</li> <li>Support ioEmulator thread pinning</li> <li>Support a couple of alerts for virt-handler</li> <li>Support for filesystem listing using the guest agent</li> <li>Support for retrieving data from the guest agent</li> <li>Support for device role tagging</li> <li>Support for assigning devices to the PCI root bus</li> <li>Support for guest overhead override</li> <li>Rewrite container-disk in C to in order to reduce it's memory footprint</li> </ul>"},{"location":"release_notes/#v0270","title":"v0.27.0","text":"<p>Released on: Fri Mar 6 22:40:34 2020 +0100</p> <ul> <li>Support for more guest agent informations in the API</li> <li>Support setting priorityClasses on VMs</li> <li>Support for additional control plane alerts via prometheus</li> <li>Support for io and emulator thread pinning</li> <li>Support setting a custom SELinux type for the launcher</li> <li>Support to perform network configurations from handler instead of launcher</li> <li>Support to opt-out of auto attaching the serial console</li> <li>Support for different uninstall strategies for data protection</li> <li>Fix to let qemu run in the qemu group</li> <li>Fix guest agent connectivity check after i.e. live migrations</li> </ul>"},{"location":"release_notes/#v0265","title":"v0.26.5","text":"<p>Released on: Tue Apr 14 15:07:04 2020 -0400</p>"},{"location":"release_notes/#v0264","title":"v0.26.4","text":"<p>Released on: Mon Mar 30 03:43:48 2020 +0200</p>"},{"location":"release_notes/#v0263","title":"v0.26.3","text":"<p>Released on: Tue Mar 10 08:57:27 2020 -0400</p>"},{"location":"release_notes/#v0262","title":"v0.26.2","text":"<p>Released on: Tue Mar 3 12:31:56 2020 -0500</p>"},{"location":"release_notes/#v0261","title":"v0.26.1","text":"<p>Released on: Fri Feb 14 20:42:46 2020 +0100</p>"},{"location":"release_notes/#v0260","title":"v0.26.0","text":"<p>Released on: Fri Feb 7 09:40:07 2020 +0100</p> <ul> <li>Fix incorrect ownerReferences to avoid VMs getting GCed</li> <li>Fixes for several tests</li> <li>Fix greedy permissions around Secrets by delegating them to kubelet</li> <li>Fix OOM infra pod by increasing it's memory request</li> <li>Clarify device support around live migrations</li> <li>Support for an uninstall strategy to protect workloads during uninstallation</li> <li>Support for more prometheus metrics and alert rules</li> <li>Support for testing SRIOV connectivity in functional tests</li> <li>Update Kubernetes client-go to 1.16.4</li> <li>FOSSA fixes and status</li> </ul>"},{"location":"release_notes/#v0250","title":"v0.25.0","text":"<p>Released on: Mon Jan 13 20:37:15 2020 +0100</p> <ul> <li>CI: Support for Kubernetes 1.17</li> <li>Support emulator thread pinning</li> <li>Support virtctl restart --force</li> <li>Support virtctl migrate to trigger live migrations from the CLI</li> </ul>"},{"location":"release_notes/#v0240","title":"v0.24.0","text":"<p>Released on: Tue Dec 3 15:34:34 2019 +0100</p> <ul> <li>CI: Support for Kubernetes 1.15</li> <li>CI: Support for Kubernetes 1.16</li> <li>Add and fix a couple of test cases</li> <li>Support for pause and unpausing VMs</li> <li>Update of libvirt to 5.6.0</li> <li>Fix bug related to parallel scraping of Prometheus endpoints</li> <li>Fix to reliably test VNC</li> </ul>"},{"location":"release_notes/#v0233","title":"v0.23.3","text":"<p>Released on: Tue Jan 21 13:17:20 2020 -0500</p>"},{"location":"release_notes/#v0232","title":"v0.23.2","text":"<p>Released on: Fri Jan 10 10:36:36 2020 -0500</p>"},{"location":"release_notes/#v0231","title":"v0.23.1","text":"<p>Released on: Thu Nov 28 09:36:41 2019 +0100</p>"},{"location":"release_notes/#v0230","title":"v0.23.0","text":"<p>Released on: Mon Nov 4 16:42:54 2019 +0100</p> <ul> <li>Guest OS Information is available under the VMI status now</li> <li>Updated to Go 1.12.8 and latest bazel</li> <li>Updated go-yaml to v2.2.4, which has a ddos vulnerability fixed</li> <li>Cleaned up and fixed CRD scheme registration</li> <li>Several bug fixes</li> <li>Many CI improvements (e.g. more logs in case of test failures)</li> </ul>"},{"location":"release_notes/#v0220","title":"v0.22.0","text":"<p>Released on: Thu Oct 10 18:55:08 2019 +0200</p> <ul> <li>Support for Nvidia GPUs and vGPUs exposed by Nvidia Kubevirt Device Plugin.</li> <li>VMIs now successfully start if they get a 0xfe prefixed MAC address assigned from the pod network</li> <li>Removed dependency on host semanage in SELinux Permissive mode</li> <li>Some changes as result of entering the CNCF sandbox (DCO check, FOSSA check, best practice badge)</li> <li>Many bug fixes and improvements in several areas</li> <li>CI: Introduced a OKD 4 test lane</li> <li>CI: Many improved tests resulting in less flakiness</li> </ul>"},{"location":"release_notes/#v0210","title":"v0.21.0","text":"<p>Released on: Mon Sep 9 09:59:08 2019 +0200</p> <ul> <li>CI: Support for Kubernetes 1.14</li> <li>Many bug fixes in several areas</li> <li>Support for <code>virtctl migrate</code></li> <li>Support configurable number of controller threads</li> <li>Support to opt-out of bridge binding for podNetwork</li> <li>Support for OpenShift Prometheus monitoring</li> <li>Support for setting more SMBIOS fields</li> <li>Improved containerDisk memory usage and speed</li> <li>Fix CRI-O memory limit</li> <li>Drop spc_t from launcher</li> <li>Add feature gates to security sensitive features</li> </ul>"},{"location":"release_notes/#v0208","title":"v0.20.8","text":"<p>Released on: Thu Oct 3 12:03:40 2019 +0200</p>"},{"location":"release_notes/#v0207","title":"v0.20.7","text":"<p>Released on: Fri Sep 27 15:21:56 2019 +0200</p>"},{"location":"release_notes/#v0206","title":"v0.20.6","text":"<p>Released on: Wed Sep 11 06:09:47 2019 -0400</p>"},{"location":"release_notes/#v0205","title":"v0.20.5","text":"<p>Released on: Thu Sep 5 17:48:59 2019 +0200</p>"},{"location":"release_notes/#v0204","title":"v0.20.4","text":"<p>Released on: Mon Sep 2 18:55:35 2019 +0200</p>"},{"location":"release_notes/#v0203","title":"v0.20.3","text":"<p>Released on: Tue Aug 27 16:58:15 2019 +0200</p>"},{"location":"release_notes/#v0202","title":"v0.20.2","text":"<p>Released on: Tue Aug 20 15:51:07 2019 +0200</p>"},{"location":"release_notes/#v0201","title":"v0.20.1","text":"<p>Released on: Fri Aug 9 19:48:17 2019 +0200</p> <ul> <li>Container disks are now secure and they are not copied anymore on every start. Old container disks can still be used in the same secure way, but new container disks can't be used on older kubevirt releases</li> <li>Create specific SecurityContextConstraints on OKD instead of using the privileged SCC</li> <li>Added clone authorization check for DataVolumes with PVC source</li> <li>The sidecar feature is feature-gated now</li> <li>Use container image shasums instead of tags for KubeVirt deployments</li> <li>Protect control plane components against voluntary evictions with a PodDisruptionBudget of MinAvailable=1</li> <li>Replaced hardcoded <code>virtctl</code> by using the basename of the call, this enables nicer output when installed via krew plugin package manager</li> <li>Added RNG device to all Fedora VMs in tests and examples (newer kernels might block bootimg while waiting for entropy)</li> <li>The virtual memory is now set to match the memory limit, if memory limit is specified and guest memory is not</li> <li>Support nftable for CoreOS</li> <li>Added a block-volume flag to the virtctl image-upload command</li> <li>Improved virtctl console/vnc data flow</li> <li>Removed DataVolumes feature gate in favor of auto-detecting CDI support</li> <li>Removed SR-IOV feature gate, it is enabled by default now</li> <li>VMI-related metrics have been renamed from <code>kubevirt_vm_</code> to <code>kubevirt_vmi_</code> to better reflect their purpose</li> <li>Added metric to report the VMI count</li> <li>Improved integration with HCO by adding a CSV generator tool and modified KubeVirt CR conditions</li> <li>CI Improvements:</li> <li>Added dedicated SR-IOV test lane</li> <li>Improved log gathering</li> <li>Reduced amount of flaky tests</li> </ul>"},{"location":"release_notes/#v0200","title":"v0.20.0","text":"<p>Released on: Fri Aug 9 16:42:41 2019 +0200</p> <ul> <li>container Disks are now secure and they are not copied anymore on every start. Old container Disks can still be used in the same secure way, but new container Disks can't be used on older kubevirt releases</li> <li>Create specific SecurityContextConstraints on OKD instead of using the privileged SCC</li> <li>Added clone authorization check for DataVolumes with PVC source</li> <li>The sidecar feature is feature-gated now</li> <li>Use container image shasum's instead of tags for KubeVirt deployments</li> <li>Protect control plane components against voluntary evictions with a PodDisruptionBudget of MinAvailable=1</li> <li>Replaced hardcoded <code>virtctl</code> by using the basename of the call, this enables nicer output when installed via krew plugin package manager</li> <li>Added RNG device to all Fedora VMs in tests and examples (newer kernels might block boot img while waiting for entropy)</li> <li>The virtual memory is now set to match the memory limit, if memory limit is specified and guest memory is not</li> <li>Support nftable for CoreOS</li> <li>Added a block-volume flag to the virtctl image-upload command</li> <li>Improved virtctl console/vnc data flow</li> <li>Removed DataVolumes feature gate in favor of auto-detecting CDI support</li> <li>Removed SR-IOV feature gate, it is enabled by default now</li> <li>VMI-related metrics have been renamed from <code>kubevirt_vm_</code> to <code>kubevirt_vmi_</code> to better reflect their purpose</li> <li>Added metric to report the VMI count</li> <li>Improved integration with HCO by adding a CSV generator tool and modified KubeVirt CR conditions</li> <li>CI Improvements:</li> <li>Added dedicated SR-IOV test lane</li> <li>Improved log gathering</li> <li>Reduced amount of flaky tests</li> </ul>"},{"location":"release_notes/#v0190","title":"v0.19.0","text":"<p>Released on: Fri Jul 5 12:52:16 2019 +0200</p> <ul> <li>Fixes when run on kind</li> <li>Fixes for sub-resource RBAC</li> <li>Limit pod network interface bindings</li> <li>Many additional bug fixes in many areas</li> <li>Additional test cases for updates, disk types, live migration with NFS</li> <li>Additional test cases for memory over-commit, block storage, cpu manager, headless mode</li> <li>Improvements around HyperV</li> <li>Improved error handling for runStrategies</li> <li>Improved update procedure</li> <li>Improved network metrics reporting (packets and errors)</li> <li>Improved guest overhead calculation</li> <li>Improved SR-IOV test suite</li> <li>Support for live migration auto-converge</li> <li>Support for config-drive disks</li> <li>Support for setting a pullPolicy con container Disks</li> <li>Support for unprivileged VMs when using SR-IOV</li> <li>Introduction of a project security policy</li> </ul>"},{"location":"release_notes/#v0181","title":"v0.18.1","text":"<p>Released on: Thu Jun 13 12:00:56 2019 +0200</p>"},{"location":"release_notes/#v0180","title":"v0.18.0","text":"<p>Released on: Wed Jun 5 22:25:09 2019 +0200</p> <ul> <li>Build: Use of go modules</li> <li>CI: Support for Kubernetes 1.13</li> <li>Countless test cases fixes and additions</li> <li>Several smaller bug fixes</li> <li>Improved upgrade documentation</li> </ul>"},{"location":"release_notes/#v0174","title":"v0.17.4","text":"<p>Released on: Tue Jun 25 07:49:12 2019 -0400</p>"},{"location":"release_notes/#v0173","title":"v0.17.3","text":"<p>Released on: Wed Jun 19 12:00:45 2019 -0400</p>"},{"location":"release_notes/#v0172","title":"v0.17.2","text":"<p>Released on: Wed Jun 5 08:12:04 2019 -0400</p>"},{"location":"release_notes/#v0171","title":"v0.17.1","text":"<p>Released on: Tue Jun 4 14:41:10 2019 -0400</p>"},{"location":"release_notes/#v0170","title":"v0.17.0","text":"<p>Released on: Mon May 6 16:18:01 2019 +0200</p> <ul> <li>Several test case additions</li> <li>Improved virt-controller node distribution</li> <li>Improved support between version migrations</li> <li>Support for a configurable MachineType default</li> <li>Support for live-migration of a VM on node taints</li> <li>Support for VM swap metrics</li> <li>Support for versioned virt-launcher / virt-handler communication</li> <li>Support for HyperV flags</li> <li>Support for different VM run strategies (i.e manual and rerunOnFailure)</li> <li>Several fixes for live-migration (TLS support, protected pods)</li> </ul>"},{"location":"release_notes/#v0163","title":"v0.16.3","text":"<p>Released on: Thu May 2 23:51:08 2019 +0200</p>"},{"location":"release_notes/#v0162","title":"v0.16.2","text":"<p>Released on: Fri Apr 26 12:24:33 2019 +0200</p>"},{"location":"release_notes/#v0161","title":"v0.16.1","text":"<p>Released on: Tue Apr 23 19:31:19 2019 +0200</p>"},{"location":"release_notes/#v0160","title":"v0.16.0","text":"<p>Released on: Fri Apr 5 23:18:22 2019 +0200</p> <ul> <li>Bazel fixes</li> <li>Initial work to support upgrades (not finalized)</li> <li>Initial support for HyperV features</li> <li>Support propagation of MAC addresses to multus</li> <li>Support live migration cancellation</li> <li>Support for table input devices</li> <li>Support for generating OLM metadata</li> <li>Support for triggering VM live migration on node taints</li> </ul>"},{"location":"release_notes/#v0150","title":"v0.15.0","text":"<p>Released on: Tue Mar 5 10:35:08 2019 +0100</p> <ul> <li>CI: Several fixes</li> <li>Fix configurable number of KVM devices</li> <li>Narrow virt-handler permissions</li> <li>Use bazel for development builds</li> <li>Support for live migration with shared and non-shared disks</li> <li>Support for live migration progress tracking</li> <li>Support for EFI boot</li> <li>Support for libvirt 5.0</li> <li>Support for extra DHCP options</li> <li>Support for a hook to manipulate cloud-init metadata</li> <li>Support setting a VM serial number</li> <li>Support for exposing infra and VM metrics</li> <li>Support for a tablet input device</li> <li>Support for extra CPU flags</li> <li>Support for ignition metadata</li> <li>Support to set a default CPU model</li> <li>Update to go 1.11.5</li> </ul>"},{"location":"release_notes/#v0140","title":"v0.14.0","text":"<p>Released on: Mon Feb 4 22:04:14 2019 +0100</p> <ul> <li>CI: Several stabilizing fixes</li> <li>docs: Document the KubeVirt Razor</li> <li>build: golang update</li> <li>Update to Kubernetes 1.12</li> <li>Update CDI</li> <li>Support for Ready and Created Operator conditions</li> <li>Support (basic) EFI</li> <li>Support for generating cloud-init network-config</li> </ul>"},{"location":"release_notes/#v0137","title":"v0.13.7","text":"<p>Released on: Mon Oct 28 17:02:35 2019 -0400</p>"},{"location":"release_notes/#v0136","title":"v0.13.6","text":"<p>Released on: Wed Sep 25 17:19:44 2019 +0200</p>"},{"location":"release_notes/#v0135","title":"v0.13.5","text":"<p>Released on: Thu Aug 1 11:25:00 2019 -0400</p>"},{"location":"release_notes/#v0134","title":"v0.13.4","text":"<p>Released on: Thu Aug 1 09:52:35 2019 -0400</p>"},{"location":"release_notes/#v0133","title":"v0.13.3","text":"<p>Released on: Mon Feb 4 15:46:48 2019 -0500</p>"},{"location":"release_notes/#v0132","title":"v0.13.2","text":"<p>Released on: Thu Jan 24 23:24:06 2019 +0100</p>"},{"location":"release_notes/#v0131","title":"v0.13.1","text":"<p>Released on: Thu Jan 24 11:16:20 2019 +0100</p>"},{"location":"release_notes/#v0130","title":"v0.13.0","text":"<p>Released on: Tue Jan 15 08:26:25 2019 +0100</p> <ul> <li>CI: Fix virt-api race</li> <li>API: Remove volumeName from disks</li> </ul>"},{"location":"release_notes/#v0120","title":"v0.12.0","text":"<p>Released on: Fri Jan 11 22:22:02 2019 +0100</p> <ul> <li>Introduce a KubeVirt Operator for KubeVirt life-cycle management</li> <li>Introduce dedicated kubevirt namespace</li> <li>Support VMI ready conditions</li> <li>Support vCPU threads and sockets</li> <li>Support scale and HPA for VMIRs</li> <li>Support to pass NTP related DHCP options</li> <li>Support guest IP address reporting via qemu guest agent</li> <li>Support for live migration with shared storage</li> <li>Support scheduling of VMs based on CPU family</li> <li>Support masquerade network interface binding</li> </ul>"},{"location":"release_notes/#v0111","title":"v0.11.1","text":"<p>Released on: Thu Dec 13 10:21:56 2018 +0200</p>"},{"location":"release_notes/#v0110","title":"v0.11.0","text":"<p>Released on: Thu Dec 6 10:15:51 2018 +0100</p> <ul> <li>API: registryDisk got renamed to containerDisk</li> <li>CI: User OKD 3.11</li> <li>Fix: Tolerate if the PVC has less capacity than expected</li> <li>Aligned to use ownerReferences</li> <li>Update to libvirt-4.10.0</li> <li>Support for VNC on MAC OSX</li> <li>Support for network SR-IOV interfaces</li> <li>Support for custom DHCP options</li> <li>Support for VM restarts via a custom endpoint</li> <li>Support for liveness and readiness probes</li> </ul>"},{"location":"release_notes/#v0100","title":"v0.10.0","text":"<p>Released on: Thu Nov 8 15:21:34 2018 +0100</p> <ul> <li>Support for vhost-net</li> <li>Support for block multi-queue</li> <li>Support for custom PCI addresses for virtio devices</li> <li>Support for deploying KubeVirt to a custom namespace</li> <li>Support for ServiceAccount token disks</li> <li>Support for multus backed networks</li> <li>Support for genie backed networks</li> <li>Support for kuryr backed networks</li> <li>Support for block PVs</li> <li>Support for configurable disk device caches</li> <li>Support for pinned IO threads</li> <li>Support for virtio net multi-queue</li> <li>Support for image upload (depending on CDI)</li> <li>Support for custom entity lists with more VM details (custom columns)</li> <li>Support for IP and MAC address reporting of all vNICs</li> <li>Basic support for guest agent status reporting</li> <li>More structured logging</li> <li>Better libvirt error reporting</li> <li>Stricter CR validation</li> <li>Better ownership references</li> <li>Several test improvements</li> </ul>"},{"location":"release_notes/#v096","title":"v0.9.6","text":"<p>Released on: Thu Nov 22 17:14:18 2018 +0100</p>"},{"location":"release_notes/#v095","title":"v0.9.5","text":"<p>Released on: Thu Nov 8 09:57:48 2018 +0100</p>"},{"location":"release_notes/#v094","title":"v0.9.4","text":"<p>Released on: Wed Nov 7 08:22:14 2018 -0500</p>"},{"location":"release_notes/#v093","title":"v0.9.3","text":"<p>Released on: Mon Oct 22 09:04:02 2018 -0400</p>"},{"location":"release_notes/#v092","title":"v0.9.2","text":"<p>Released on: Thu Oct 18 12:14:09 2018 +0200</p>"},{"location":"release_notes/#v091","title":"v0.9.1","text":"<p>Released on: Fri Oct 5 09:01:51 2018 +0200</p>"},{"location":"release_notes/#v090","title":"v0.9.0","text":"<p>Released on: Thu Oct 4 14:42:28 2018 +0200</p> <ul> <li>CI: NetworkPolicy tests</li> <li>CI: Support for an external provider (use a preconfigured cluster for tests)</li> <li>Fix virtctl console issues with CRI-O</li> <li>Support to initialize empty PVs</li> <li>Support for basic CPU pinning</li> <li>Support for setting IO Threads</li> <li>Support for block volumes</li> <li>Move preset logic to mutating webhook</li> <li>Introduce basic metrics reporting using prometheus metrics</li> <li>Many stabilizing fixes in many places</li> </ul>"},{"location":"release_notes/#v080","title":"v0.8.0","text":"<p>Released on: Thu Sep 6 14:25:22 2018 +0200</p> <ul> <li>Support for DataVolume</li> <li>Support for a subprotocol for web browser terminals</li> <li>Support for virtio-rng</li> <li>Support disconnected VMs</li> <li>Support for setting host model</li> <li>Support for host CPU passthrough</li> <li>Support setting a vNICs mac and PCI address</li> <li>Support for memory over-commit</li> <li>Support booting from network devices</li> <li>Use less devices by default, aka disable unused ones</li> <li>Improved VMI shutdown status</li> <li>More logging to improve debugability</li> <li>A lot of small fixes, including typos and documentation fixes</li> <li>Race detection in tests</li> <li>Hook improvements</li> <li>Update to use Fedora 28 (includes updates of dependencies like libvirt and   qemu)</li> <li>Move CI to support Kubernetes 1.11</li> </ul>"},{"location":"release_notes/#v070","title":"v0.7.0","text":"<p>Released on: Wed Jul 4 17:41:33 2018 +0200</p> <ul> <li>CI: Move test storage to hostPath</li> <li>CI: Add support for Kubernetes 1.10.4</li> <li>CI: Improved network tests for multiple-interfaces</li> <li>CI: Drop Origin 3.9 support</li> <li>CI: Add test for testing templates on Origin</li> <li>VM to VMI rename</li> <li>VM affinity and anti-affinity</li> <li>Add awareness for multiple networks</li> <li>Add hugepage support</li> <li>Add device-plugin based kvm</li> <li>Add support for setting the network interface model</li> <li>Add (basic and initial) Kubernetes compatible networking approach (SLIRP)</li> <li>Add role aggregation for our roles</li> <li>Add support for setting a disks serial number</li> <li>Add support for specifying the CPU model</li> <li>Add support for setting an network interfaces MAC address</li> <li>Relocate binaries for FHS conformance</li> <li>Logging improvements</li> <li>Template fixes</li> <li>Fix OpenShift CRD validation</li> <li>virtctl: Improve vnc logging improvements</li> <li>virtctl: Add expose</li> <li>virtctl: Use PATCH instead of PUT</li> </ul>"},{"location":"release_notes/#v064","title":"v0.6.4","text":"<p>Released on: Tue Aug 21 17:29:28 2018 +0300</p>"},{"location":"release_notes/#v063","title":"v0.6.3","text":"<p>Released on: Mon Jul 30 16:14:22 2018 +0200</p>"},{"location":"release_notes/#v062","title":"v0.6.2","text":"<p>Released on: Wed Jul 4 17:49:37 2018 +0200</p> <ul> <li>Binary relocation for packaging</li> <li>QEMU Process detection</li> <li>Role aggregation</li> <li>CPU Model selection</li> <li>VM Rename fix</li> </ul>"},{"location":"release_notes/#v061","title":"v0.6.1","text":"<p>Released on: Mon Jun 18 17:07:48 2018 -0400</p>"},{"location":"release_notes/#v060","title":"v0.6.0","text":"<p>Released on: Mon Jun 11 09:30:28 2018 +0200</p> <ul> <li>A range of flakiness reducing test fixes</li> <li>Vagrant setup got deprecated</li> <li>Updated Docker and CentOS versions</li> <li>Add Kubernetes 1.10.3 to test matrix</li> <li>A couple of ginkgo concurrency fixes</li> <li>A couple of spelling fixes</li> <li>A range if infra updates</li> <li>Use /dev/kvm if possible, otherwise fallback to emulation</li> <li>Add default view/edit/admin RBAC Roles</li> <li>Network MTU fixes</li> <li>CD-ROM drives are now read-only</li> <li>Secrets can now be correctly referenced on VMs</li> <li>Add disk boot ordering</li> <li>Add virtctl version</li> <li>Add virtctl expose</li> <li>Fix virtual machine memory calculations</li> <li>Add basic virtual machine Network API</li> </ul>"},{"location":"release_notes/#v050","title":"v0.5.0","text":"<p>Released on: Fri May 4 18:25:32 2018 +0200</p> <ul> <li>Better controller health signaling</li> <li>Better virtctl error messages</li> <li>Improvements to enable CRI-O support</li> <li>Run CI on stable OpenShift</li> <li>Add test coverage for multiple PVCs</li> <li>Improved controller life-cycle guarantees</li> <li>Add Webhook validation</li> <li>Add tests coverage for node eviction</li> <li>OfflineVirtualMachine status improvements</li> <li>RegistryDisk API update</li> </ul>"},{"location":"release_notes/#v041","title":"v0.4.1","text":"<p>Released on: Thu Apr 12 11:46:09 2018 +0200</p> <ul> <li>VM shutdown fixes and tests</li> <li>Functional test for CRD validation</li> <li>Windows VM test</li> <li>DHCP link-local change</li> </ul>"},{"location":"release_notes/#v040","title":"v0.4.0","text":"<p>Released on: Fri Apr 6 16:40:31 2018 +0200</p> <ul> <li>Fix several networking issues</li> <li>Add and enable OpenShift support to CI</li> <li>Add conditional Windows tests (if an image is present)</li> <li>Add subresources for console access</li> <li>virtctl config alignment with kubectl</li> <li>Fix API reference generation</li> <li>Stable UUIDs for OfflineVirtualMachines</li> <li>Build virtctl for MacOS and Windows</li> <li>Set default architecture to x86_64</li> <li>Major improvement to the CI infrastructure (all containerized)</li> <li>virtctl convenience functions for starting and stopping a VM</li> </ul>"},{"location":"release_notes/#v030","title":"v0.3.0","text":"<p>Released on: Thu Mar 8 10:21:57 2018 +0100</p> <ul> <li>Kubernetes compatible networking</li> <li>Kubernetes compatible PV based storage</li> <li>VirtualMachinePresets support</li> <li>OfflineVirtualMachine support</li> <li>RBAC improvements</li> <li>Switch to q35 machine type by default</li> <li>A large number of test and CI fixes</li> <li>Ephemeral disk support</li> </ul>"},{"location":"release_notes/#v020","title":"v0.2.0","text":"<p>Released on: Fri Jan 5 16:30:45 2018 +0100</p> <ul> <li>VM launch and shutdown flow improvements</li> <li>VirtualMachine API redesign</li> <li>Removal of HAProxy</li> <li>Redesign of VNC/Console access</li> <li>Initial support for different vagrant providers</li> </ul>"},{"location":"release_notes/#v010","title":"v0.1.0","text":"<p>Released on: Fri Dec 8 20:43:06 2017 +0100</p> <ul> <li>Many API improvements for a proper OpenAPI reference</li> <li>Add watchdog support</li> <li>Drastically improve the deployment on non-vagrant setups</li> <li>Dropped nodeSelectors</li> <li>Separated inner component deployment from edge component deployment</li> <li>Created separate manifests for developer, test, and release deployments</li> <li>Moved components to kube-system namespace</li> <li>Improved and unified flag parsing</li> </ul>"},{"location":"release_notes/#v004","title":"v0.0.4","text":"<p>Released on: Tue Nov 7 11:51:45 2017 +0100</p> <ul> <li>Add support for node affinity to VM.Spec</li> <li>Add OpenAPI specification</li> <li>Drop swagger 1.2 specification</li> <li>virt-launcher refactoring</li> <li>Leader election mechanism for virt-controller</li> <li>Move from glide to dep for dependency management</li> <li>Improve virt-handler synchronization loops</li> <li>Add support for running the functional tests on oVirt infrastructure</li> <li>Several tests fixes (spice, cleanup, ...)</li> <li>Add console test tool</li> <li>Improve libvirt event notification</li> </ul>"},{"location":"release_notes/#v003","title":"v0.0.3","text":"<p>Released on: Fri Oct 6 10:21:16 2017 +0200</p> <ul> <li>Containerized binary builds</li> <li>Socket based container detection</li> <li>cloud-init support</li> <li>Container based ephemeral disk support</li> <li>Basic RBAC profile</li> <li>client-go updates</li> <li>Rename of VM to VirtualMachine</li> <li>Introduction of VirtualMachineReplicaSet</li> <li>Improved migration events</li> <li>Improved API documentation</li> </ul>"},{"location":"release_notes/#v002","title":"v0.0.2","text":"<p>Released on: Mon Sep 4 21:12:46 2017 +0200</p> <ul> <li>Usage of CRDs</li> <li>Moved libvirt to a pod</li> <li>Introduction of <code>virtctl</code></li> <li>Use glide instead of govendor</li> <li>Container based ephemeral disks</li> <li>Contributing guide improvements</li> <li>Support for Kubernetes Namespaces</li> </ul>"},{"location":"cluster_admin/activating_feature_gates/","title":"Activating feature gates","text":"<p>KubeVirt has a set of features that are not mature enough to be enabled by default. As such, they are protected by a Kubernetes concept called feature gates.</p>"},{"location":"cluster_admin/activating_feature_gates/#how-to-activate-a-feature-gate","title":"How to activate a feature gate","text":"<p>You can activate a specific feature gate directly in KubeVirt's CR, by provisioning the following yaml, which uses the <code>LiveMigration</code> feature gate as an example: <pre><code>cat &lt;&lt; END &gt; enable-feature-gate.yaml\n---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration: \n      featureGates:\n        - LiveMigration\nEND\n\nkubectl apply -f enable-feature-gate.yaml\n</code></pre></p> <p>Alternatively, the existing kubevirt CR can be altered: <pre><code>kubectl edit kubevirt kubevirt -n kubevirt\n</code></pre></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - DataVolumes\n        - LiveMigration\n</code></pre> <p>Note: the name of the feature gates is case sensitive.</p> <p>The snippet above assumes KubeVirt is installed in the <code>kubevirt</code> namespace. Change the namespace to suite your installation.</p>"},{"location":"cluster_admin/activating_feature_gates/#list-of-feature-gates","title":"List of feature gates","text":"<p>The list of feature gates (which evolve in time) can be checked directly from the source code.</p>"},{"location":"cluster_admin/annotations_and_labels/","title":"Annotations and labels","text":"<p>KubeVirt builds on and exposes a number of labels and annotations that either are used for internal implementation needs or expose useful information to API users. This page documents the labels and annotations that may be useful for regular API consumers. This page intentionally does not list labels and annotations that are merely part of internal implementation.</p> <p>Note: Annotations and labels that are not specific to KubeVirt are also documented here.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtio","title":"kubevirt.io","text":"<p>Example: <code>kubevirt.io=virt-launcher</code></p> <p>Used on: Pod</p> <p>This label marks resources that belong to KubeVirt. An optional value may indicate which specific KubeVirt component a resource belongs to. This label may be used to list all resources that belong to KubeVirt, for example, to uninstall it from a cluster.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtioschedulable","title":"kubevirt.io/schedulable","text":"<p>Example: <code>kubevirt.io/schedulable=true</code></p> <p>Used on: Node</p> <p>This label declares whether a particular node is available for scheduling virtual machine instances on it.</p>"},{"location":"cluster_admin/annotations_and_labels/#kubevirtioheartbeat","title":"kubevirt.io/heartbeat","text":"<p>Example: <code>kubevirt.io/heartbeat=2018-07-03T20:07:25Z</code></p> <p>Used on: Node</p> <p>This annotation is regularly updated by virt-handler to help determine if a particular node is alive and hence should be available for new virtual machine instance scheduling.</p>"},{"location":"cluster_admin/api_validation/","title":"API Validation","text":"<p>The KubeVirt VirtualMachineInstance API is implemented using a Kubernetes Custom Resource Definition (CRD). Because of this, KubeVirt is able to leverage a couple of features Kubernetes provides in order to perform validation checks on our API as objects created and updated on the cluster.</p>"},{"location":"cluster_admin/api_validation/#how-api-validation-works","title":"How API Validation Works","text":""},{"location":"cluster_admin/api_validation/#crd-openapiv3-schema","title":"CRD OpenAPIv3 Schema","text":"<p>The KubeVirt API is registered with Kubernetes at install time through a series of CRD definitions. KubeVirt includes an OpenAPIv3 schema in these definitions which indicates to the Kubernetes Apiserver some very basic information about our API, such as what fields are required and what type of data is expected for each value.</p> <p>This OpenAPIv3 schema validation is installed automatically and requires no thought on the users part to enable.</p>"},{"location":"cluster_admin/api_validation/#admission-control-webhooks","title":"Admission Control Webhooks","text":"<p>The OpenAPIv3 schema validation is limited. It only validates the general structure of a KubeVirt object looks correct. It does not however verify that the contents of that object make sense.</p> <p>With OpenAPIv3 validation alone, users can easily make simple mistakes (like not referencing a volume's name correctly with a disk) and the cluster will still accept the object. However, the VirtualMachineInstance will of course not start if these errors in the API exist. Ideally we'd like to catch configuration issues as early as possible and not allow an object to even be posted to the cluster if we can detect there's a problem with the object's Spec.</p> <p>In order to perform this advanced validation, KubeVirt implements its own admission controller which is registered with kubernetes as an admission controller webhook. This webhook is registered with Kubernetes at install time. As KubeVirt objects are posted to the cluster, the Kubernetes API server forwards Creation requests to our webhook for validation before persisting the object into storage.</p> <p>Note however that the KubeVirt admission controller requires features to be enabled on the cluster in order to be enabled.</p>"},{"location":"cluster_admin/api_validation/#enabling-kubevirt-admission-controller-on-kubernetes","title":"Enabling KubeVirt Admission Controller on Kubernetes","text":"<p>When provisioning a new Kubernetes cluster, ensure that both the MutatingAdmissionWebhook and ValidatingAdmissionWebhook values are present in the Apiserver's --admission-control cli argument.</p> <p>Below is an example of the --admission-control values we use during development</p> <pre><code>--admission-control='Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota'\n</code></pre> <p>Note that the old --admission-control flag was deprecated in 1.10 and replaced with --enable-admission-plugins. MutatingAdmissionWebhook and ValidatingAdmissionWebhook are enabled by default.</p>"},{"location":"cluster_admin/api_validation/#enabling-kubevirt-admission-controller-on-okd","title":"Enabling KubeVirt Admission Controller on OKD","text":"<p>OKD also requires the admission control webhooks to be enabled at install time. The process is slightly different though. With OKD, we enable webhooks using an admission plugin.</p> <p>These admission control plugins can be configured in openshift-ansible by setting the following value in ansible inventory file.</p> <pre><code>openshift_master_admission_plugin_config={\"ValidatingAdmissionWebhook\":{\"configuration\":{\"kind\": \"DefaultAdmissionConfig\",\"apiVersion\": \"v1\",\"disable\": false}},\"MutatingAdmissionWebhook\":{\"configuration\":{\"kind\": \"DefaultAdmissionConfig\",\"apiVersion\": \"v1\",\"disable\": false}}}\n</code></pre>"},{"location":"cluster_admin/authorization/","title":"Authorization","text":"<p>KubeVirt authorization is performed using Kubernetes's Resource Based Authorization Control system (RBAC). RBAC allows cluster admins to grant access to cluster resources by binding RBAC roles to users.</p> <p>For example, an admin creates an RBAC role that represents the permissions required to create a VirtualMachineInstance. The admin can then bind that role to users in order to grant them the permissions required to launch a VirtualMachineInstance.</p> <p>With RBAC roles, admins can grant users targeted access to various KubeVirt features.</p>"},{"location":"cluster_admin/authorization/#kubevirt-default-rbac-clusterroles","title":"KubeVirt Default RBAC ClusterRoles","text":"<p>KubeVirt comes with a set of predefined RBAC ClusterRoles that can be used to grant users permissions to access KubeVirt Resources.</p>"},{"location":"cluster_admin/authorization/#default-view-role","title":"Default View Role","text":"<p>The kubevirt.io:view ClusterRole gives users permissions to view all KubeVirt resources in the cluster. The permissions to create, delete, modify or access any KubeVirt resources beyond viewing the resource's spec are not included in this role. This means a user with this role could see that a VirtualMachineInstance is running, but neither shutdown nor gain access to that VirtualMachineInstance via console/VNC.</p>"},{"location":"cluster_admin/authorization/#default-edit-role","title":"Default Edit Role","text":"<p>The kubevirt.io:edit ClusterRole gives users permissions to modify all KubeVirt resources in the cluster. For example, a user with this role can create new VirtualMachineInstances, delete VirtualMachineInstances, and gain access to both console and VNC.</p>"},{"location":"cluster_admin/authorization/#default-admin-role","title":"Default Admin Role","text":"<p>The kubevirt.io:admin ClusterRole grants users full permissions to all KubeVirt resources, including the ability to delete collections of resources.</p> <p>The admin role also grants users access to view and modify the KubeVirt runtime config. This config exists within the Kubevirt Custom Resource under the <code>configuration</code> key in the namespace the KubeVirt operator is running.</p> <p>NOTE Users are only guaranteed the ability to modify the kubevirt runtime configuration if a ClusterRoleBinding is used. A RoleBinding will work to provide kubevirt CR access only if the RoleBinding targets the same namespace that the kubevirt CR exists in.</p>"},{"location":"cluster_admin/authorization/#binding-default-clusterroles-to-users","title":"Binding Default ClusterRoles to Users","text":"<p>The KubeVirt default ClusterRoles are granted to users by creating either a ClusterRoleBinding or RoleBinding object.</p>"},{"location":"cluster_admin/authorization/#binding-within-all-namespaces","title":"Binding within All Namespaces","text":"<p>With a ClusterRoleBinding, users receive the permissions granted by the role across all namespaces.</p>"},{"location":"cluster_admin/authorization/#binding-within-single-namespace","title":"Binding within Single Namespace","text":"<p>With a RoleBinding, users receive the permissions granted by the role only within a targeted namespace.</p>"},{"location":"cluster_admin/authorization/#extending-kubernetes-default-roles-with-kubevirt-permissions","title":"Extending Kubernetes Default Roles with KubeVirt permissions","text":"<p>The aggregated ClusterRole Kubernetes feature facilitates combining multiple ClusterRoles into a single aggregated ClusterRole. This feature is commonly used to extend the default Kubernetes roles with permissions to access custom resources that do not exist in the Kubernetes core.</p> <p>In order to extend the default Kubernetes roles to provide permission to access KubeVirt resources, we need to add the following labels to the KubeVirt ClusterRoles.</p> <pre><code>kubectl label clusterrole kubevirt.io:admin rbac.authorization.k8s.io/aggregate-to-admin=true\nkubectl label clusterrole kubevirt.io:edit rbac.authorization.k8s.io/aggregate-to-edit=true\nkubectl label clusterrole kubevirt.io:view rbac.authorization.k8s.io/aggregate-to-view=true\n</code></pre> <p>By adding these labels, any user with a RoleBinding or ClusterRoleBinding involving one of the default Kubernetes roles will automatically gain access to the equivalent KubeVirt roles as well.</p> <p>More information about aggregated cluster roles can be found here</p>"},{"location":"cluster_admin/authorization/#creating-custom-rbac-roles","title":"Creating Custom RBAC Roles","text":"<p>If the default KubeVirt ClusterRoles are not expressive enough, admins can create their own custom RBAC roles to grant user access to KubeVirt resources. The creation of a RBAC role is inclusive only, meaning there's no way to deny access. Instead access is only granted.</p> <p>Below is an example of what KubeVirt's default admin ClusterRole looks like. A custom RBAC role can be created by reducing the permissions in this example role.</p> <pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n  name: my-custom-rbac-role\n  labels:\n    kubevirt.io: \"\"\nrules:\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/console\n      - virtualmachineinstances/vnc\n    verbs:\n      - get\n  - apiGroups:\n      - kubevirt.io\n    resources:\n      - virtualmachineinstances\n      - virtualmachines\n      - virtualmachineinstancepresets\n      - virtualmachineinstancereplicasets\n    verbs:\n      - get\n      - delete\n      - create\n      - update\n      - patch\n      - list\n      - watch\n      - deletecollection\n</code></pre>"},{"location":"cluster_admin/confidential_computing/","title":"Confidential computing","text":""},{"location":"cluster_admin/confidential_computing/#amd-secure-encrypted-virtualization-sev","title":"AMD Secure Encrypted Virtualization (SEV)","text":"<p>FEATURE STATE: KubeVirt v0.49.0 (experimental support)</p> <p>Secure Encrypted Virtualization (SEV) is a feature of AMD's EPYC CPUs that allows the memory of a virtual machine to be encrypted on the fly.</p> <p>KubeVirt supports running confidential VMs on AMD EPYC hardware with SEV feature.</p>"},{"location":"cluster_admin/confidential_computing/#preconditions","title":"Preconditions","text":"<p>In order to run an SEV guest the following condition must be met:</p> <ul> <li><code>WorkloadEncryptionSEV</code> feature gate must be enabled.</li> <li>The guest must support UEFI boot</li> <li>SecureBoot must be disabled for the guest VM</li> </ul>"},{"location":"cluster_admin/confidential_computing/#running-an-sev-guest","title":"Running an SEV guest","text":"<p>SEV memory encryption can be requested by setting the <code>spec.domain.launchSecurity.sev</code> element in the VMI definition:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    launchSecurity:\n      sev: {}\n    firmware:\n      bootloader:\n        efi:\n          secureBoot: false\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre>"},{"location":"cluster_admin/confidential_computing/#current-limitations","title":"Current limitations","text":"<ul> <li>SEV-encrypted VMs cannot contain directly-accessible host devices (that is, PCI passthrough)</li> <li>Live Migration is not supported</li> <li>The VMs are not attested</li> </ul>"},{"location":"cluster_admin/customize_components/","title":"Customize components","text":""},{"location":"cluster_admin/customize_components/#customize-kubevirt-components","title":"Customize KubeVirt Components","text":""},{"location":"cluster_admin/customize_components/#customize-components-using-patches","title":"Customize components using patches","text":"<p> If the patch created is invalid KubeVirt will not be able to update or deploy the system. This is intended for special use cases and should not be used unless you know what you are doing.</p> <p>Valid resource types are: Deployment, DaemonSet, Service, ValidatingWebhookConfiguraton, MutatingWebhookConfiguration, APIService, and CertificateSecret. More information can be found in the API spec.</p> <p>Example customization patch: <pre><code>---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  certificateRotateStrategy: {}\n  configuration: {}\n  customizeComponents:\n    patches:\n    - resourceType: Deployment\n      resourceName: virt-controller\n      patch: '[{\"op\": \"remove\", \"path\": \"/spec/template/spec/containers/0/livenessProbe\"}]'\n      type: json\n    - resourceType: Deployment\n      resourceName: virt-controller\n      patch: '{\"metadata\":{\"annotations\":{\"patch\": \"true\"}}}'\n      type: strategic\n</code></pre></p> <p>The above example will update the <code>virt-controller</code> deployment to have an annotation in it's metadata that says <code>patch: true</code> and will remove the livenessProbe from the container definition.</p>"},{"location":"cluster_admin/customize_components/#customize-flags","title":"Customize Flags","text":"<p> If the flags are invalid or become invalid on update the component will not be able to run</p> <p>By using the customize flag option, whichever component the flags are to be applied to, all default flags will be removed and only the flags specified will be used. The available resources to change the flags on are <code>api</code>, <code>controller</code> and <code>handler</code>. You can find our more details about the API in the API spec.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  certificateRotateStrategy: {}\n  configuration: {}\n  customizeComponents:\n    flags:\n      api:\n        v: \"5\"\n        port: \"8443\"\n        console-server-port: \"8186\"\n        subresources-only: \"true\"\n</code></pre> <p>The above example would produce a <code>virt-api</code> pod with the following command</p> <pre><code>...\nspec:\n  ....\n  container:\n  - name: virt-api\n    command:\n    - virt-api\n    - --v\n    - \"5\"\n    - --console-server-port\n    - \"8186\"\n    - --port\n    - \"8443\"\n    - --subresources-only\n    - \"true\"\n    ...\n</code></pre>"},{"location":"cluster_admin/device_status_on_Arm64/","title":"Device Status on Arm64","text":"<p>This page is based on https://github.com/kubevirt/kubevirt/issues/8916</p> Devices Description Status on Arm64 DisableHotplug supported Disks sata/ virtio bus support virtio bus Watchdog i6300esb not supported UseVirtioTransitional virtio-transitional supported Interfaces e1000/ virtio-net-device support virtio-net-device Inputs tablet virtio/usb bus supported AutoattachPodInterface connect to /net/tun (devices.kubevirt.io/tun) supported AutoattachGraphicsDevice create a virtio-gpu device / vga device support virtio-gpu AutoattachMemBalloon virtio-balloon-pci-non-transitional supported AutoattachInputDevice auto add tablet supported Rng virtio-rng-pci-non-transitional host:/dev/urandom supported BlockMultiQueue \"driver\":\"virtio-blk-pci-non-transitional\",\"num-queues\":$cpu_number supported NetworkInterfaceMultiQueue -netdev tap,fds=21:23:24:25,vhost=on,vhostfds=26:27:28:29,id=hostua-default#fd number equals to queue number supported GPUs not verified Filesystems virtiofs, vhost-user-fs-pci, need to enable featuregate: ExperimentalVirtiofsSupport supported ClientPassthrough https://www.linaro.org/blog/kvm-pciemsi-passthrough-armarm64/on x86_64, iommu need to be enabled not verified Sound ich9/ ac97 not supported TPM tpm-tis-devicehttps://qemu.readthedocs.io/en/latest/specs/tpm.html supported Sriov vfio-pci not verified"},{"location":"cluster_admin/feature_gate_status_on_Arm64/","title":"Feature Gate Status on Arm64","text":"<p>This page is based on https://github.com/kubevirt/kubevirt/issues/9749 It records the feature gate status on Arm64 platform. Here is the explanation of the status:</p> <ul> <li>Supported: the feature gate support on Arm64 platform.</li> <li>Not supported yet: there are some dependencies of the feature gate not support Arm64, so this feature does not support for now. We may support the dependencies in the future.</li> <li>Not supported: The feature gate is not support on Arm64.</li> <li>Not verified: The feature has not been verified yet.</li> </ul> FEATURE GATE STATUS NOTES ExpandDisksGate Not supported yet CDI is needed CPUManager Supported use taskset to do CPU pinning, do not support kvm-hint-dedicated (this is only works on x86 platform) NUMAFeatureGate Not supported yet Need to support Hugepage on Arm64 IgnitionGate Supported This feature is only used for CoreOS/RhCOS LiveMigrationGate Supported Verified live migration with masquerade network SRIOVLiveMigrationGate Not verified Need two same Machine and SRIOV device HypervStrictCheckGate Not supported Hyperv does not work on Arm64 SidecarGate Supported GPUGate Not verified Need GPU device HostDevicesGate Not verified Need GPU or sound card SnapshotGate Supported Need snapshotter support https://github.com/kubernetes-csi/external-snapshotter VMExportGate Partially supported Need snapshotter support https://kubevirt.io/user-guide/operations/export_api/, support exporting pvc, not support exporting DataVolumes and MemoryDump which rely on CDI HotplugVolumesGate Not supported yet Rely on datavolume and CDI HostDiskGate Supported VirtIOFSGate Supported MacvtapGate Not supported yet quay.io/kubevirt/macvtap-cni not support Arm64, https://github.com/kubevirt/macvtap-cni#deployment PasstGate Supported VM have same ip with pods; start a process for network /usr/bin/passt --runas 107 -e -t 8080 DownwardMetricsFeatureGate need more information It used to let guest get host information, failed on both Arm64 and x86_64. The block is successfully attached and can see the following information: <code>-blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-private/downwardapi-disks/vhostmd0\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"}</code>But unable to get information via <code>vm-dump-metrics</code>:<code>LIBMETRICS: read_mdisk(): Unable to read metrics disk</code><code>LIBMETRICS: get_virtio_metrics(): Unable to export metrics: open(/dev/virtio-ports/org.github.vhostmd.1) No such file or directory</code><code>LIBMETRICS: get_virtio_metrics(): Unable to read metrics</code> NonRootDeprecated Supported NonRoot Supported Root Supported ClusterProfiler Supported WorkloadEncryptionSEV Not supported SEV is only available on x86_64 VSOCKGate Supported HotplugNetworkIfacesGate Not supported yet Need to setup multus-cni and multus-dynamic-networks-controller: https://github.com/k8snetworkplumbingwg/multus-cni <code>cat ./deployments/multus-daemonset-thick.yml \\| kubectl apply -f -</code>https://github.com/k8snetworkplumbingwg/multus-dynamic-networks-controller <code>kubectl apply -f manifests/dynamic-networks-controller.yaml</code> Currently, the image ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick does not support Arm64 server. For more information please refer to https://github.com/k8snetworkplumbingwg/multus-cni/pull/1027. CommonInstancetypesDeploymentGate Not supported yet Support of common-instancetypes instancetypes needs to be tested, common-instancetypes preferences for ARM workloads are still missing"},{"location":"cluster_admin/gitops/","title":"Managing KubeVirt with GitOps","text":"<p>The GitOps way uses Git repositories as a single source of truth to deliver infrastructure as code. Automation is employed to keep the desired and the live state of clusters in sync at all times. This means any change to a repository is automatically applied to one or more clusters while changes to a cluster will be automatically reverted to the state described in the single source of truth.</p> <p>With GitOps the separation of testing and production environments, improving the availability of applications and working with multi-cluster environments becomes considerably easier.</p>"},{"location":"cluster_admin/gitops/#demo-repository","title":"Demo repository","text":"<p>A demo with detailed explanation on how to manage KubeVirt with GitOps can be found here.</p> <p>The demo is using Open Cluster Management and ArgoCD to deploy KubeVirt and virtual  machines across multiple clusters.</p>"},{"location":"cluster_admin/installation/","title":"Installation","text":"<p>KubeVirt is a virtualization add-on to Kubernetes and this guide assumes that a Kubernetes cluster is already installed.</p> <p>If installed on OKD, the web console is extended for management of virtual machines.</p>"},{"location":"cluster_admin/installation/#requirements","title":"Requirements","text":"<p>A few requirements need to be met before you can begin:</p> <ul> <li>Kubernetes cluster or derivative     (such as OpenShift)     based on a one of the latest three Kubernetes releases that are     out at the time the KubeVirt release is made.</li> <li>Kubernetes apiserver must have <code>--allow-privileged=true</code> in order to run KubeVirt's privileged DaemonSet.</li> <li><code>kubectl</code> client utility</li> </ul>"},{"location":"cluster_admin/installation/#container-runtime-support","title":"Container Runtime Support","text":"<p>KubeVirt is currently supported on the following container runtimes:</p> <ul> <li>containerd</li> <li>crio (with runv)</li> </ul> <p>Other container runtimes, which do not use virtualization features, should work too. However, the mentioned ones are the main target.</p>"},{"location":"cluster_admin/installation/#integration-with-apparmor","title":"Integration with AppArmor","text":"<p>In most of the scenarios, KubeVirt can run normally on systems with AppArmor. However, there are several known use cases that may require additional user interaction.</p> <ul> <li> <p>On a system with AppArmor enabled, the locally installed profiles     may block the execution of the KubeVirt privileged containers. That     usually results in initialization failure of the <code>virt-handler</code>     pod:</p> <pre><code>$ kubectl get pods -n kubevirt\nNAME                               READY   STATUS       RESTARTS         AGE\nvirt-api-77df5c4f87-7mqv4          1/1     Running      1 (17m ago)      27m\nvirt-api-77df5c4f87-wcq44          1/1     Running      1 (17m ago)      27m\nvirt-controller-749d8d99d4-56gb7   1/1     Running      1 (17m ago)      27m\nvirt-controller-749d8d99d4-78j6x   1/1     Running      1 (17m ago)      27m\nvirt-handler-4w99d                 0/1     Init:Error   14 (5m18s ago)   27m\nvirt-operator-564f568975-g9wh4     1/1     Running      1 (17m ago)      31m\nvirt-operator-564f568975-wnpz8     1/1     Running      1 (17m ago)      31m\n\n$ kubectl logs -n kubevirt virt-handler-4w99d virt-launcher\nerror: failed to get emulator capabilities\n\nerror: internal error: Failed to start QEMU binary /usr/libexec/qemu-kvm for probing: libvirt: error : cannot execute binary /usr/libexec/qemu-kvm: Permission denied\n\n$ journalctl -b | grep DEN\n...\nMay 18 16:44:20 debian audit[6316]: AVC apparmor=\"DENIED\" operation=\"exec\" profile=\"libvirtd\" name=\"/usr/libexec/qemu-kvm\" pid=6316 comm=\"rpc-worker\" requested_mask=\"x\" denied_mask=\"x\" fsuid=107 ouid=0\nMay 18 16:44:20 debian kernel: audit: type=1400 audit(1652888660.539:39): apparmor=\"DENIED\" operation=\"exec\" profile=\"libvirtd\" name=\"/usr/libexec/qemu-kvm\" pid=6316 comm=\"rpc-worker\" requested_mask=\"x\" denied_mask=\"x\" fsuid=107 ouid=0\n...\n</code></pre> <p>Here, the host AppArmor profile for <code>libvirtd</code> does not allow the execution of the <code>/usr/libexec/qemu-kvm</code> binary. In the future this will hopefully work out of the box (tracking issue), but until then there are a couple of possible workarounds.</p> <p>The first (and simplest) one is to remove the libvirt package from the host: assuming the host is a dedicated Kubernetes node, you likely won't need it anyway.</p> <p>If you actually need libvirt to be present on the host, then you can add the following rule to the AppArmor profile for libvirtd (usually <code>/etc/apparmor.d/usr.sbin.libvirtd</code>):</p> <pre><code># vim /etc/apparmor.d/usr.sbin.libvirtd\n...\n/usr/libexec/qemu-kvm PUx,\n...\n# apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd # or systemctl reload apparmor.service\n</code></pre> </li> <li> <p>The default AppArmor profile used by the container runtimes usually     denies <code>mount</code> call for the workloads. That may prevent from     running VMs with VirtIO-FS.     This is a known issue.     The current workaround is to run such a VM as <code>unconfined</code> by adding the     following annotation to the VM or VMI object:</p> <pre><code>annotations:\n  container.apparmor.security.beta.kubernetes.io/compute: unconfined\n</code></pre> </li> </ul>"},{"location":"cluster_admin/installation/#validate-hardware-virtualization-support","title":"Validate Hardware Virtualization Support","text":"<p>Hardware with virtualization support is recommended. You can use virt-host-validate to ensure that your hosts are capable of running virtualization workloads:</p> <pre><code>$ virt-host-validate qemu\n  QEMU: Checking for hardware virtualization                                 : PASS\n  QEMU: Checking if device /dev/kvm exists                                   : PASS\n  QEMU: Checking if device /dev/kvm is accessible                            : PASS\n  QEMU: Checking if device /dev/vhost-net exists                             : PASS\n  QEMU: Checking if device /dev/net/tun exists                               : PASS\n...\n</code></pre>"},{"location":"cluster_admin/installation/#selinux-support","title":"SELinux support","text":"<p>SELinux-enabled nodes need Container-selinux installed. The minimum version is documented inside the kubevirt/kubevirt repository, in docs/getting-started.md, under \"SELinux support\".</p> <p>For (older) release branches that don't specify a container-selinux version, version 2.170.0 or newer is recommended.</p>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-kubernetes","title":"Installing KubeVirt on Kubernetes","text":"<p>KubeVirt can be installed using the KubeVirt operator, which manages the lifecycle of all the KubeVirt core components. Below is an example of how to install KubeVirt's latest official release. It supports to deploy KubeVirt on both x86_64 and Arm64 platforms.</p> <pre><code># Point at latest release\n$ export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)\n# Deploy the KubeVirt operator\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml\n# Create the KubeVirt CR (instance deployment request) which triggers the actual installation\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml\n# wait until all KubeVirt components are up\n$ kubectl -n kubevirt wait kv kubevirt --for condition=Available\n</code></pre> <p>If hardware virtualization is not available, then a software emulation fallback can be enabled using by setting in the KubeVirt CR <code>spec.configuration.developerConfiguration.useEmulation</code> to <code>true</code> as follows:</p> <pre><code>$ kubectl edit -n kubevirt kubevirt kubevirt\n</code></pre> <p>Add the following to the <code>kubevirt.yaml</code> file</p> <pre><code>spec:\n    ...\n    configuration:\n    developerConfiguration:\n        useEmulation: true\n</code></pre> <p>Note: Prior to release v0.20.0 the condition for the <code>kubectl wait</code> command was named \"Ready\" instead of \"Available\"</p> <p>Note: Prior to KubeVirt 0.34.2 a ConfigMap called <code>kubevirt-config</code> in the install-namespace was used to configure KubeVirt. Since 0.34.2 this method is deprecated. The configmap still has precedence over <code>configuration</code> on the CR exists, but it will not receive future updates and you should migrate any custom configurations to <code>spec.configuration</code> on the KubeVirt CR.</p> <p>All new components will be deployed under the <code>kubevirt</code> namespace:</p> <pre><code>kubectl get pods -n kubevirt\nNAME                                           READY     STATUS        RESTARTS   AGE\nvirt-api-6d4fc3cf8a-b2ere                      1/1       Running       0          1m\nvirt-controller-5d9fc8cf8b-n5trt               1/1       Running       0          1m\nvirt-handler-vwdjx                             1/1       Running       0          1m\n...\n</code></pre>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-okd","title":"Installing KubeVirt on OKD","text":"<p>The following SCC needs to be added prior KubeVirt deployment:</p> <pre><code>$ oc adm policy add-scc-to-user privileged -n kubevirt -z kubevirt-operator\n</code></pre> <p>Once privileges are granted, the KubeVirt can be deployed as described above.</p>"},{"location":"cluster_admin/installation/#web-user-interface-on-okd","title":"Web user interface on OKD","text":"<p>No additional steps are required to extend OKD's web console for KubeVirt.</p> <p>The virtualization extension is automatically enabled when KubeVirt deployment is detected.</p>"},{"location":"cluster_admin/installation/#from-service-catalog-as-an-apb","title":"From Service Catalog as an APB","text":"<p>You can find KubeVirt in the OKD Service Catalog and install it from there. In order to do that please follow the documentation in the KubeVirt APB repository.</p>"},{"location":"cluster_admin/installation/#installing-kubevirt-on-k3os","title":"Installing KubeVirt on k3OS","text":"<p>The following configuration needs to be added to all nodes prior KubeVirt deployment:</p> <pre><code>k3os:\n  modules:\n  - kvm\n  - vhost_net\n</code></pre> <p>Once nodes are restarted with this configuration, the KubeVirt can be deployed as described above.</p>"},{"location":"cluster_admin/installation/#installing-the-daily-developer-builds","title":"Installing the Daily Developer Builds","text":"<p>KubeVirt releases daily a developer build from the current main branch. One can see when the last release happened by looking at our nightly-build-jobs.</p> <p>To install the latest developer build, run the following commands:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator.yaml\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr.yaml\n</code></pre> <p>To find out which commit this build is based on, run:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest)\n$ curl https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/commit\nd358cf085b5a86cc4fa516215f8b757a4e61def2\n</code></pre>"},{"location":"cluster_admin/installation/#arm64-developer-builds","title":"ARM64 developer builds","text":"<p>ARM64 developer builds can be installed like this:</p> <pre><code>$ LATEST=$(curl -L https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/latest-arm64)\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-operator-arm64.yaml\n$ kubectl apply -f https://storage.googleapis.com/kubevirt-prow/devel/nightly/release/kubevirt/kubevirt/${LATEST}/kubevirt-cr-arm64.yaml\n</code></pre>"},{"location":"cluster_admin/installation/#deploying-from-source","title":"Deploying from Source","text":"<p>See the Developer Getting Started Guide to understand how to build and deploy KubeVirt from source.</p>"},{"location":"cluster_admin/installation/#installing-network-plugins-optional","title":"Installing network plugins (optional)","text":"<p>KubeVirt alone does not bring any additional network plugins, it just allows user to utilize them. If you want to attach your VMs to multiple networks (Multus CNI) or have full control over L2 (OVS CNI), you need to deploy respective network plugins. For more information, refer to OVS CNI installation guide.</p> <p>Note: KubeVirt Ansible network playbook installs these plugins by default.</p>"},{"location":"cluster_admin/installation/#restricting-kubevirt-components-node-placement","title":"Restricting KubeVirt components node placement","text":"<p>You can restrict the placement of the KubeVirt components across your  cluster nodes by editing the KubeVirt CR:</p> <ul> <li>The placement of the KubeVirt control plane components (virt-controller, virt-api)   is governed by the <code>.spec.infra.nodePlacement</code> field in the KubeVirt CR.</li> <li>The placement of the virt-handler DaemonSet pods (and consequently, the placement of the    VM workloads scheduled to the cluster) is governed by the <code>.spec.workloads.nodePlacement</code>   field in the KubeVirt CR.</li> </ul> <p>For each of these <code>.nodePlacement</code> objects, the <code>.affinity</code>, <code>.nodeSelector</code> and <code>.tolerations</code> sub-fields can be configured. See the description in the API reference for further information about using these fields.</p> <p>For example, to restrict the virt-controller and virt-api pods to only run on the control-plane nodes:</p> <pre><code>kubectl patch -n kubevirt kubevirt kubevirt --type merge --patch '{\"spec\": {\"infra\": {\"nodePlacement\": {\"nodeSelector\": {\"node-role.kubernetes.io/control-plane\": \"\"}}}}}'\n</code></pre> <p>To restrict the virt-handler pods to only run on nodes with the \"region=primary\" label:</p> <pre><code>kubectl patch -n kubevirt kubevirt kubevirt --type merge --patch '{\"spec\": {\"workloads\": {\"nodePlacement\": {\"nodeSelector\": {\"region\": \"primary\"}}}}}'\n</code></pre>"},{"location":"cluster_admin/ksm/","title":"KSM Management","text":"<p>Kernel Samepage Merging (KSM) allows de-duplication of memory. KSM tries to find identical Memory Pages and merge those to free memory.</p> <p>Further Information: - KSM (Kernel Samepage Merging) feature - Kernel Same-page Merging (KSM)</p>"},{"location":"cluster_admin/ksm/#enabling-ksm-through-kubevirt-cr","title":"Enabling KSM through KubeVirt CR","text":"<p>KSM can be enabled on nodes by <code>spec.configuration.ksmConfiguration</code> in the KubeVirt CR. <code>ksmConfiguration</code> instructs on which nodes KSM will be enabled, exposing a <code>nodeLabelSelector</code>. <code>nodeLabelSelector</code> is a LabelSelector and defines the filter, based on the node labels. If a node's labels match the label selector term, then on that node, KSM will be enabled.  </p> <p>NOTE If <code>nodeLabelSelector</code> is nil KSM will not be enabled on any nodes. Empty <code>nodeLabelSelector</code> will enable KSM on every node.  </p>"},{"location":"cluster_admin/ksm/#examples","title":"Examples:","text":"<ul> <li> <p>Enabling KSM on nodes in which the hostname is <code>node01</code> or <code>node03</code>: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector:\n        matchExpressions:\n          - key: kubernetes.io/hostname\n            operator: In\n            values:\n              - node01\n              - node03\n</code></pre></p> </li> <li> <p>Enabling KSM on nodes with labels <code>kubevirt.io/first-label: true</code>, <code>kubevirt.io/second-label: true</code>: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector:\n        matchLabels:\n          kubevirt.io/first-label: \"true\"\n          kubevirt.io/second-label: \"true\"\n</code></pre></p> </li> <li> <p>Enabling KSM on every node: <pre><code>spec:\n  configuration:\n    ksmConfiguration:\n      nodeLabelSelector: {}\n</code></pre></p> </li> </ul>"},{"location":"cluster_admin/ksm/#annotation-and-restore-mechanism","title":"Annotation and restore mechanism","text":"<p>On those nodes where KubeVirt enables the KSM via configuration, an annotation will be added (<code>kubevirt.io/ksm-handler-managed</code>). This annotation is an internal record to keep track of which nodes are currently  managed by virt-handler, so that it is possible to distinguish which nodes should be restored in case of future ksmConfiguration changes.</p> <p>Let's imagine this scenario:</p> <ol> <li>There are 3 nodes in the cluster and one of them(<code>node01</code>) has KSM externally enabled.</li> <li>An admin patches the KubeVirt CR adding a ksmConfiguration which enables ksm for <code>node02</code> and <code>node03</code>.</li> <li>After a while, an admin patches again the KubeVirt CR deleting the ksmConfiguration.</li> </ol> <p>Thanks to the annotation, the virt-handler is able to disable ksm on only those nodes where it itself had enabled it(<code>node02</code> <code>node03</code>), leaving the others unchanged (<code>node01</code>).</p>"},{"location":"cluster_admin/ksm/#node-labelling","title":"Node labelling","text":"<p>KubeVirt can discover on which nodes KSM is enabled and will mark them with a special label (<code>kubevirt.io/ksm-enabled</code>) with value <code>true</code>. This label can be used to schedule the vms in nodes with KSM enabled or not. <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: testvm\n    spec:\n      nodeSelector:\n        kubevirt.io/ksm-enabled: \"true\"\n      [...]\n</code></pre></p>"},{"location":"cluster_admin/migration_policies/","title":"Migration Policies","text":"<p>Migration policies provides a new way of applying migration configurations to Virtual Machines. The policies can refine Kubevirt CR's <code>MigrationConfiguration</code> that sets the cluster-wide migration configurations. This way, the cluster-wide settings serve as a default that can be refined (i.e. changed, removed or added) by the migration policy.</p> <p>Please bear in mind that migration policies are in version <code>v1alpha1</code>. This means that this API is not fully stable yet and that APIs may change in the future.</p>"},{"location":"cluster_admin/migration_policies/#overview","title":"Overview","text":"<p>KubeVirt supports Live Migrations of Virtual Machine workloads. Before migration policies were introduced, migration settings could be configurable only on the cluster-wide scope by editing KubevirtCR's spec or more specifically MigrationConfiguration CRD.</p> <p>Several aspects (although not all) of migration behaviour that can be customized are: - Bandwidth - Auto-convergence - Post/Pre-copy - Max number of parallel migrations - Timeout</p> <p>Migration policies generalize the concept of defining migration configurations, so it would be possible to apply different configurations to specific groups of VMs.</p> <p>Such capability can be useful for a lot of different use cases on which there is a need to differentiate between different workloads. Differentiation of different configurations could be needed because different workloads are considered to be in different priorities, security segregation, workloads with different requirements, help to converge workloads which aren't migration-friendly, and many other reasons.</p>"},{"location":"cluster_admin/migration_policies/#api-examples","title":"API Examples","text":""},{"location":"cluster_admin/migration_policies/#migration-configurations","title":"Migration Configurations","text":"<p>Currently the MigrationPolicy spec will only include the following configurations from KubevirtCR's MigrationConfiguration (in the future more configurations that aren't part of Kubevirt CR are intended to be added): <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n    allowAutoConverge: true\n    bandwidthPerMigration: 217Ki\n    completionTimeoutPerGiB: 23\n    allowPostCopy: false\n</code></pre></p> <p>All above fields are optional. When omitted, the configuration will be applied as defined in KubevirtCR's MigrationConfiguration. This way, KubevirtCR will serve as a configurable set of defaults for both VMs that are not bound to any MigrationPolicy and VMs that are bound to a MigrationPolicy that does not define all fields of the configurations.</p>"},{"location":"cluster_admin/migration_policies/#matching-policies-to-vms","title":"Matching Policies to VMs","text":"<p>Next in the spec are the selectors that define the group of VMs on which to apply the policy. The options to do so are the following.</p> <p>This policy applies to the VMs in namespaces that have all the required labels: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true       # Matches a key and a value \n</code></pre></p> <p>This policy applies for the VMs that have all the required labels: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    virtualMachineInstanceSelector:\n      workload-type: db       # Matches a key and a value \n</code></pre></p> <p>It is also possible to combine the previous two: <pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\n  spec:\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true\n    virtualMachineInstanceSelector:\n      workload-type: db\n</code></pre></p>"},{"location":"cluster_admin/migration_policies/#full-manifest","title":"Full Manifest:","text":"<pre><code>apiVersion: migrations.kubevirt.io/v1alpha1\nkind: MigrationPolicy\nmetadata:\n  name: my-awesome-policy\nspec:\n  # Migration Configuration\n  allowAutoConverge: true\n  bandwidthPerMigration: 217Ki\n  completionTimeoutPerGiB: 23\n  allowPostCopy: false\n\n  # Matching to VMs\n  selectors:\n    namespaceSelector:\n      hpc-workloads: true\n    virtualMachineInstanceSelector:\n      workload-type: db\n</code></pre>"},{"location":"cluster_admin/migration_policies/#policies-precedence","title":"Policies' Precedence","text":"<p>It is possible that multiple policies apply to the same VMI. In such cases, the precedence is in the same order as the bullets above (VMI labels first, then namespace labels). It is not allowed to define two policies with the exact same selectors.</p> <p>If multiple policies apply to the same VMI: * The most detailed policy will be applied, that is, the policy with the highest number of matching labels</p> <ul> <li>If multiple policies match to a VMI with the same number of matching labels, the policies will be sorted by the lexicographic order of the matching labels keys. The first one in this order will be applied.</li> </ul>"},{"location":"cluster_admin/migration_policies/#example","title":"Example","text":"<p>For example, let's imagine a VMI with the following labels:</p> <ul> <li> <p>size: small</p> </li> <li> <p>os: fedora</p> </li> <li> <p>gpu: nvidia</p> </li> </ul> <p>And let's say the namespace to which the VMI belongs contains the following labels:</p> <ul> <li> <p>priority: high</p> </li> <li> <p>bandwidth: medium</p> </li> <li> <p>hpc-workload: true</p> </li> </ul> <p>The following policies are listed by their precedence (high to low):</p> <p>1) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high, bandwidth: medium}</code></p> <ul> <li>Matching labels: 4, First key in lexicographic order: <code>bandwidth</code>.</li> </ul> <p>2) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high, hpc-workload:true}</code></p> <ul> <li>Matching labels: 4, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>3) VMI labels: <code>{size: small, gpu: nvidia}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>Matching labels: 3, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>4) VMI labels: <code>{size: small}</code>, Namespace labels: <code>{priority:high, hpc-workload:true}</code></p> <ul> <li>Matching labels: 3, First key in lexicographic order: <code>hpc-workload</code>.</li> </ul> <p>5) VMI labels: <code>{gpu: nvidia}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>Matching labels: 2, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>6) VMI labels: <code>{gpu: nvidia}</code>, Namespace labels: <code>{}</code></p> <ul> <li>Matching labels: 1, First key in lexicographic order: <code>gpu</code>.</li> </ul> <p>7) VMI labels: <code>{gpu: intel}</code>, Namespace labels: <code>{priority:high}</code></p> <ul> <li>VMI label does not match - policy cannot be applied.</li> </ul>"},{"location":"cluster_admin/node_maintenance/","title":"Node maintenance","text":"<p>Before removing a kubernetes node from the cluster, users will want to ensure that VirtualMachineInstances have been gracefully terminated before powering down the node. Since all VirtualMachineInstances are backed by a Pod, the recommended method of evicting VirtualMachineInstances is to use the kubectl drain command, or in the case of OKD the oc adm drain command.</p>"},{"location":"cluster_admin/node_maintenance/#evict-all-vms-from-a-node","title":"Evict all VMs from a Node","text":"<p>Select the node you'd like to evict VirtualMachineInstances from by identifying the node from the list of cluster nodes.</p> <p><code>kubectl get nodes</code></p> <p>The following command will gracefully terminate all VMs on a specific node. Replace <code>&lt;node-name&gt;</code> with the name of the node where the eviction should occur.</p> <p><code>kubectl drain &lt;node-name&gt; --delete-local-data --ignore-daemonsets=true --force --pod-selector=kubevirt.io=virt-launcher</code></p> <p>Below is a break down of why each argument passed to the drain command is required.</p> <ul> <li> <p><code>kubectl drain &lt;node-name&gt;</code> is selecting a specific node as a target     for the eviction</p> </li> <li> <p><code>--delete-local-data</code> is a required flag that is necessary for     removing any pod that utilizes an emptyDir volume. The     VirtualMachineInstance Pod does use emptyDir volumes, however the     data in those volumes are ephemeral which means it is safe to delete     after termination.</p> </li> <li> <p><code>--ignore-daemonsets=true</code> is a required flag because every node     running a VirtualMachineInstance will also be running our helper     DaemonSet called virt-handler. DaemonSets are not allowed to be     evicted using kubectl drain. By default, if this command     encounters a DaemonSet on the target node, the command will fail.     This flag tells the command it is safe to proceed with the eviction     and to just ignore DaemonSets.</p> </li> <li> <p><code>--force</code> is a required flag because VirtualMachineInstance pods are     not owned by a ReplicaSet or DaemonSet controller. This means     kubectl can't guarantee that the pods being terminated on the target     node will get re-scheduled replacements placed else where in the     cluster after the pods are evicted. KubeVirt has its own controllers     which manage the underlying VirtualMachineInstance pods. Each     controller behaves differently to a VirtualMachineInstance being     evicted. That behavior is outlined further down in this document.</p> </li> <li> <p><code>--pod-selector=kubevirt.io=virt-launcher</code> means only     VirtualMachineInstance pods managed by KubeVirt will be removed from     the node.</p> </li> </ul>"},{"location":"cluster_admin/node_maintenance/#evict-all-vms-and-pods-from-a-node","title":"Evict all VMs and Pods from a Node","text":"<p>By removing the <code>-pod-selector</code> argument from the previous command, we can issue the eviction of all Pods on a node. This command ensures Pods associated with VMs as well as all other Pods are evicted from the target node.</p> <p><code>kubectl drain &lt;node name&gt; --delete-local-data --ignore-daemonsets=true --force</code></p>"},{"location":"cluster_admin/node_maintenance/#evacuate-vmis-via-live-migration-from-a-node","title":"Evacuate VMIs via Live Migration from a Node","text":"<p>If the <code>LiveMigration</code> feature gate is enabled, it is possible to specify an <code>evictionStrategy</code> on VMIs which will react with live-migrations on specific taints on nodes. The following snippet on a VMI or the VMI templates in a VM ensures that the VMI is migrated during node eviction:</p> <pre><code>spec:\n  evictionStrategy: LiveMigrate\n</code></pre> <p>Here a full VMI:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  evictionStrategy: LiveMigrate\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: cloudinitdisk\n    cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n</code></pre> Behind the scenes a PodDisruptionBudget is created for each VMI which has an evictionStrategy defined. This ensures that evictions are be blocked on these VMIs and that we can guarantee that a VMI will be migrated instead of shut off.</p> <p>Note Prior to v0.34 the drain process with live migrations was detached from the <code>kubectl drain</code> itself and required in addition specifying a special taint on the nodes: <code>kubectl taint nodes foo kubevirt.io/drain=draining:NoSchedule</code>. This is no longer needed. The taint will still be respected if provided but is obsolete.</p>"},{"location":"cluster_admin/node_maintenance/#re-enabling-a-node-after-eviction","title":"Re-enabling a Node after Eviction","text":"<p>The kubectl drain will result in the target node being marked as unschedulable. This means the node will not be eligible for running new VirtualMachineInstances or Pods.</p> <p>If it is decided that the target node should become schedulable again, the following command must be run.</p> <p><code>kubectl uncordon &lt;node name&gt;</code></p> <p>or in the case of OKD.</p> <p><code>oc adm uncordon &lt;node name&gt;</code></p>"},{"location":"cluster_admin/node_maintenance/#shutting-down-a-node-after-eviction","title":"Shutting down a Node after Eviction","text":"<p>From KubeVirt's perspective, a node is safe to shutdown once all VirtualMachineInstances have been evicted from the node. In a multi-use cluster where VirtualMachineInstances are being scheduled alongside other containerized workloads, it is up to the cluster admin to ensure all other pods have been safely evicted before powering down the node.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachine-evictions","title":"VirtualMachine Evictions","text":"<p>The eviction of any VirtualMachineInstance that is owned by a VirtualMachine set to running=true will result in the VirtualMachineInstance being re-scheduled to another node.</p> <p>The VirtualMachineInstance in this case will be forced to power down and restart on another node. In the future once KubeVirt introduces live migration support, the VM will be able to seamlessly migrate to another node during eviction.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachineinstancereplicaset-eviction-behavior","title":"VirtualMachineInstanceReplicaSet Eviction Behavior","text":"<p>The eviction of VirtualMachineInstances owned by a VirtualMachineInstanceReplicaSet will result in the VirtualMachineInstanceReplicaSet scheduling replacements for the evicted VirtualMachineInstances on other nodes in the cluster.</p>"},{"location":"cluster_admin/node_maintenance/#virtualmachineinstance-eviction-behavior","title":"VirtualMachineInstance Eviction Behavior","text":"<p>VirtualMachineInstances not backed by either a VirtualMachineInstanceReplicaSet or an VirtualMachine object will not be re-scheduled after eviction.</p>"},{"location":"cluster_admin/operations_on_Arm64/","title":"Arm64 Operations","text":"<p>This page summarizes all operations that are not supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hotplug-network-interfaces","title":"Hotplug Network Interfaces","text":"<p>Hotplug Network Interfaces are not supported on Arm64, because the image ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot-thick does not support for the Arm64 platform. For more information please refer to https://github.com/k8snetworkplumbingwg/multus-cni/pull/1027.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hotplug-volumes","title":"Hotplug Volumes","text":"<p>Hotplug Volumes are not supported on Arm64, because the Containerized Data Importer is not supported on Arm64 for now.</p>"},{"location":"cluster_admin/operations_on_Arm64/#hugepages-support","title":"Hugepages support","text":"<p>Hugepages feature is not supported on Arm64. The hugepage mechanism differs between X86_64 and Arm64. Now we only verify KubeVirt on 4k pagesize systems.</p>"},{"location":"cluster_admin/operations_on_Arm64/#containerized-data-importer","title":"Containerized Data Importer","text":"<p>For now, we have not supported this project on Arm64, but it is in our plan.</p>"},{"location":"cluster_admin/operations_on_Arm64/#export-api","title":"Export API","text":"<p>Export API is partially supported on the Arm64 platform. As CDI is not supported yet, the export of DataVolumes and MemoryDump are not supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#virtual-machine-memory-dump","title":"Virtual machine memory dump","text":"<p>As explained above, MemoryDump requires CDI, and is not yet supported on Arm64.</p>"},{"location":"cluster_admin/operations_on_Arm64/#mediated-devices-and-virtual-gpus","title":"Mediated devices and virtual GPUs","text":"<p>This is not verified on Arm64 platform.</p>"},{"location":"cluster_admin/scheduler/","title":"KubeVirt Scheduler","text":"<p>Scheduling is the process of matching Pods/VMs to Nodes. By default, the scheduler used is  kube-scheduler. Further details can be found at Kubernetes Scheduler Documentation.</p> <p>Custom schedulers can be used if the default scheduler does not satisfy your needs. For instance, you might want to schedule VMs using a load aware scheduler such as Trimaran Schedulers.</p>"},{"location":"cluster_admin/scheduler/#creating-a-custom-scheduler","title":"Creating a Custom Scheduler","text":"<p>KubeVirt is compatible with custom schedulers. The configuration steps are described in the Official Kubernetes  Documentation. Please note, the Kubernetes version KubeVirt is running on and the Kubernetes version used to build the custom scheduler have to match. To get the Kubernetes version KubeVirt is running on, you can run the following command:</p> <pre><code>$ kubectl version\nClient Version: version.Info{Major:\"1\", Minor:\"22\", GitVersion:\"v1.22.13\", GitCommit:\"a43c0904d0de10f92aa3956c74489c45e6453d6e\", GitTreeState:\"clean\", BuildDate:\"2022-08-17T18:28:56Z\", GoVersion:\"go1.16.15\", Compiler:\"gc\", Platform:\"linux/amd64\"}\nServer Version: version.Info{Major:\"1\", Minor:\"22\", GitVersion:\"v1.22.13\", GitCommit:\"a43c0904d0de10f92aa3956c74489c45e6453d6e\", GitTreeState:\"clean\", BuildDate:\"2022-08-17T18:23:45Z\", GoVersion:\"go1.16.15\", Compiler:\"gc\", Platform:\"linux/amd64\"}\n</code></pre> <p>Pay attention to the <code>Server</code> line.  In this case, the Kubernetes version is <code>v1.22.13</code>. You have to checkout the matching Kubernetes version and build the Kubernetes project:</p> <pre><code>$ cd kubernetes\n$ git checkout v1.22.13\n$ make\n</code></pre> <p>Then, you can follow the configuration steps described here. Additionally, the ClusterRole <code>system:kube-scheduler</code> needs permissions to use the verbs <code>watch</code>, <code>list</code> and <code>get</code> on StorageClasses.</p> <pre><code>- apiGroups:                                                                                                   \n  - storage.k8s.io                                                                                             \n  resources:                                                                                                   \n  - storageclasses                                                                                             \n  verbs:                                                                                                       \n  - watch                                                                                                      \n  - list                                                                                                       \n  - get \n</code></pre>"},{"location":"cluster_admin/scheduler/#scheduling-vms-with-the-custom-scheduler","title":"Scheduling VMs with the Custom Scheduler","text":"<p>The second scheduler should be up and running. You can check it with:</p> <pre><code>$ kubectl get all -n kube-system\n</code></pre> <p>The deployment <code>my-scheduler</code> should be up and running if everything is setup properly. In order to launch the VM using the custom scheduler, you need to set the <code>SchedulerName</code> in the VM's spec to <code>my-scheduler</code>. Here is an example VM definition:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\nspec:\n  running: true\n  template:\n    spec:\n      schedulerName: my-scheduler\n      domain:\n        devices:\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          rng: {}\n        resources:\n          requests:\n            memory: 1Gi\n      terminationGracePeriodSeconds: 180\n      volumes:\n        - containerDisk:\n            image: quay.io/containerdisks/fedora:latest\n          name: containerdisk\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: fedora\n              user: fedora\n          name: cloudinitdisk\n</code></pre> In case the specified <code>SchedulerName</code> does not match any existing scheduler, the <code>virt-launcher</code> pod will stay in state Pending,  until the specified scheduler can be found. You can check if the VM has been scheduled using the <code>my-scheduler</code> checking the <code>virt-launcher</code> pod events associated with the VM. The pod should have been scheduled with <code>my-scheduler</code>.</p> <pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vm-fedora-dpc87    2/2     Running   0          24m\n\n$ kubectl describe pod virt-launcher-vm-fedora-dpc87\n[...] \nEvents:\n  Type    Reason     Age   From              Message\n  ----    ------     ----  ----              -------\n  Normal  Scheduled  21m   my-scheduler  Successfully assigned default/virt-launcher-vm-fedora-dpc87 to node01\n[...]\n</code></pre>"},{"location":"cluster_admin/tekton_tasks/","title":"KubeVirt Tekton","text":""},{"location":"cluster_admin/tekton_tasks/#prerequisites","title":"Prerequisites","text":"<ul> <li>Tekton</li> <li>KubeVirt</li> <li>CDI</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#kubevirt-tekton-tasks","title":"KubeVirt Tekton Tasks","text":""},{"location":"cluster_admin/tekton_tasks/#what-are-kubevirt-tekton-tasks","title":"What are KubeVirt Tekton Tasks?","text":"<p>KubeVirt-specific Tekton Tasks, which are focused on:</p> <ul> <li>Creating and managing resources (VMs, DataVolumes)</li> <li>Executing commands in VMs</li> <li>Manipulating disk images with libguestfs tools</li> </ul> <p>KubeVirt Tekton Tasks and example Pipelines are available in artifacthub.io from where you can easily deploy them to your cluster.</p>"},{"location":"cluster_admin/tekton_tasks/#existing-tasks","title":"Existing Tasks","text":""},{"location":"cluster_admin/tekton_tasks/#create-virtual-machines","title":"Create Virtual Machines","text":"<ul> <li>create-vm-from-manifest - create a VM from provided manifest or with virtctl.</li> <li>create-vm-from-template - create a VM from template (works only on OpenShift).</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#utilize-templates","title":"Utilize Templates","text":"<ul> <li>copy-template - Copies the given template and creates a new one (works only on OpenShift).</li> <li>modify-vm-template - Modifies a template with user provided data (works only on OpenShift).</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#modify-data-objects","title":"Modify Data Objects","text":"<ul> <li>modify-data-object - Creates / modifies / deletes a datavolume / datasource</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#generate-ssh-keys","title":"Generate SSH Keys","text":"<ul> <li>generate-ssh-keys - Generates a private and public key pair, and injects it into a VM.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#execute-commands-in-virtual-machines","title":"Execute commands in Virtual Machines","text":"<ul> <li>execute-in-vm - Execute commands over SSH in a VM.</li> <li>cleanup-vm - Execute commands and/or stop/delete VMs.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#manipulate-pvcs-with-libguestfs-tools","title":"Manipulate PVCs with libguestfs tools","text":"<ul> <li>disk-virt-customize - execute virt-customize commands in PVCs.</li> <li>disk-virt-sysprep- execute virt-sysprep commands in PVCs.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#wait-for-virtual-machine-instance-status","title":"Wait for Virtual Machine Instance Status","text":"<ul> <li>wait-for-vmi-status - Waits for a VMI to be running.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#modify-windows-iso","title":"Modify Windows iso","text":"<ul> <li>modify-windows-iso-file - modifies windows iso (replaces prompt bootloader with no-prompt bootloader) and replaces original iso    in PVC with updated one. This helps with automated installation of Windows in EFI boot mode. By default Windows in EFI boot mode    uses a prompt bootloader, which will not continue with the boot process until a key is pressed. By replacing it with the non-prompt    bootloader no key press is required to boot into the Windows installer.</li> </ul>"},{"location":"cluster_admin/tekton_tasks/#example-pipeline","title":"Example Pipeline","text":"<p>All these Tasks can be used for creating Pipelines. We prepared example Pipelines which show what can you do with the KubeVirt Tasks.</p> <ul> <li> <p>Windows efi installer - This Pipeline will prepare a Windows 10/11/2k22 datavolume with virtio drivers installed. User has to provide a working link to a Windows 10/11/2k22 iso file. The Pipeline is suitable for Windows versions, which requires EFI (e.g. Windows 10/11/2k22). More information about Pipeline can be found here</p> </li> <li> <p>Windows customize - This Pipeline will install a SQL server or a VS Code in a Windows VM. More information about Pipeline can be found here</p> </li> </ul> <p>Note</p> <ul> <li>If you define a different namespace for Pipelines and a different namespace for Tasks, you will have to create a cluster resolver object. </li> <li>By default, example Pipelines create the resulting datavolume in the <code>kubevirt-os-images</code> namespace. </li> <li>In case you would like to create resulting datavolume in different namespace (by specifying <code>baseDvNamespace</code> attribute in Pipeline), additional RBAC permissions will be required (list of all required RBAC permissions can be found here). </li> <li>In case you would like to live migrate the VM while a given Pipeline is running, the following prerequisities must be met </li> </ul>"},{"location":"cluster_admin/unresponsive_nodes/","title":"Unresponsive nodes","text":"<p>KubeVirt has its own node daemon, called virt-handler. In addition to the usual k8s methods of detecting issues on nodes, the virt-handler daemon has its own heartbeat mechanism. This allows for fine-tuned error handling of VirtualMachineInstances.</p>"},{"location":"cluster_admin/unresponsive_nodes/#virt-handler-heartbeat","title":"virt-handler heartbeat","text":"<p><code>virt-handler</code> periodically tries to update the <code>kubevirt.io/schedulable</code> label and the <code>kubevirt.io/heartbeat</code> annotation on the node it is running on:</p> <pre><code>$ kubectl get nodes -o yaml\napiVersion: v1\nitems:\n- apiVersion: v1\n  kind: Node\n  metadata:\n    annotations:\n      kubevirt.io/heartbeat: 2018-11-05T09:42:25Z\n    creationTimestamp: 2018-11-05T08:55:53Z\n    labels:\n      beta.kubernetes.io/arch: amd64\n      beta.kubernetes.io/os: linux\n      cpumanager: \"false\"\n      kubernetes.io/hostname: node01\n      kubevirt.io/schedulable: \"true\"\n      node-role.kubernetes.io/control-plane: \"\"\n</code></pre> <p>If a <code>VirtualMachineInstance</code> gets scheduled, the scheduler is only considering nodes where <code>kubevirt.io/schedulable</code> is <code>true</code>. This can be seen when looking on the corresponding pod of a <code>VirtualMachineInstance</code>:</p> <pre><code>$ kubectl get pods  virt-launcher-vmi-nocloud-ct6mr -o yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  [...]\nspec:\n  [...]\n  nodeName: node01\n  nodeSelector:\n    kubevirt.io/schedulable: \"true\"\n  [...]\n</code></pre> <p>In case there is a communication issue or the host goes down, <code>virt-handler</code> can't update its labels and annotations any-more. Once the last <code>kubevirt.io/heartbeat</code> timestamp is older than five minutes, the KubeVirt node-controller kicks in and sets the <code>kubevirt.io/schedulable</code> label to <code>false</code>. As a consequence no more VMIs will be schedule to this node until virt-handler is connected again.</p>"},{"location":"cluster_admin/unresponsive_nodes/#deleting-stuck-vmis-when-virt-handler-is-unresponsive","title":"Deleting stuck VMIs when virt-handler is unresponsive","text":"<p>In cases where <code>virt-handler</code> has some issues but the node is in general fine, a <code>VirtualMachineInstance</code> can be deleted as usual via <code>kubectl delete vmi &lt;myvm&gt;</code>. Pods of a <code>VirtualMachineInstance</code> will be told by the cluster-controllers they should shut down. As soon as the Pod is gone, the <code>VirtualMachineInstance</code> will be moved to <code>Failed</code> state, if <code>virt-handler</code> did not manage to update it's heartbeat in the meantime. If <code>virt-handler</code> could recover in the meantime, <code>virt-handler</code> will move the <code>VirtualMachineInstance</code> to failed state instead of the cluster-controllers.</p>"},{"location":"cluster_admin/unresponsive_nodes/#deleting-stuck-vmis-when-the-whole-node-is-unresponsive","title":"Deleting stuck VMIs when the whole node is unresponsive","text":"<p>If the whole node is unresponsive, deleting a <code>VirtualMachineInstance</code> via <code>kubectl delete vmi &lt;myvmi&gt;</code> alone will never remove the <code>VirtualMachineInstance</code>. In this case all pods on the unresponsive node need to be force-deleted: First make sure that the node is really dead. Then delete all pods on the node via a force-delete: <code>kubectl delete pod --force --grace-period=0 &lt;mypod&gt;</code>.</p> <p>As soon as the pod disappears and the heartbeat from virt-handler timed out, the VMIs will be moved to <code>Failed</code> state. If they were already marked for deletion they will simply disappear. If not, they can be deleted and will disappear almost immediately.</p>"},{"location":"cluster_admin/unresponsive_nodes/#timing-considerations","title":"Timing considerations","text":"<p>It takes up to five minutes until the KubeVirt cluster components can detect that virt-handler is unhealthy. During that time-frame it is possible that new VMIs are scheduled to the affected node. If virt-handler is not capable of connecting to these pods on the node, the pods will sooner or later go to failed state. As soon as the cluster finally detects the issue, the VMIs will be set to failed by the cluster.</p>"},{"location":"cluster_admin/updating_and_deletion/","title":"Updating and deletion","text":""},{"location":"cluster_admin/updating_and_deletion/#updating-kubevirt-control-plane","title":"Updating KubeVirt Control Plane","text":"<p>Zero downtime rolling updates are supported starting with release <code>v0.17.0</code> onward. Updating from any release prior to the KubeVirt <code>v0.17.0</code> release is not supported.</p> <p>Note: Updating is only supported from N-1 to N release.</p> <p>Updates are triggered one of two ways.</p> <ol> <li> <p>By changing the imageTag value in the KubeVirt CR's spec.</p> <p>For example, updating from <code>v0.17.0-alpha.1</code> to <code>v0.17.0</code> is as simple as patching the KubeVirt CR with the <code>imageTag: v0.17.0</code> value. From there the KubeVirt operator will begin the process of rolling out the new version of KubeVirt. Existing VM/VMIs will remain uninterrupted both during and after the update succeeds.</p> <pre><code>$ kubectl patch kv kubevirt -n kubevirt --type=json -p '[{ \"op\": \"add\", \"path\": \"/spec/imageTag\", \"value\": \"v0.17.0\" }]'\n</code></pre> </li> <li> <p>Or, by updating the kubevirt operator if no imageTag value is set.</p> <p>When no imageTag value is set in the kubevirt CR, the system assumes that the version of KubeVirt is locked to the version of the operator. This means that updating the operator will result in the underlying KubeVirt installation being updated as well.</p> <pre><code>$ export RELEASE=v0.26.0\n$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml\n</code></pre> </li> </ol> <p>The first way provides a fine granular approach where you have full control over what version of KubeVirt is installed independently of what version of the KubeVirt operator you might be running. The second approach allows you to lock both the operator and operand to the same version.</p> <p>Newer KubeVirt may require additional or extended RBAC rules. In this case, the #1 update method may fail, because the virt-operator present in the cluster doesn't have these RBAC rules itself. In this case, you need to update the <code>virt-operator</code> first, and then proceed to update kubevirt. See this issue for more details.</p>"},{"location":"cluster_admin/updating_and_deletion/#updating-kubevirt-workloads","title":"Updating KubeVirt Workloads","text":"<p>Workload updates are supported as an opt in feature starting with <code>v0.39.0</code></p> <p>By default, when KubeVirt is updated this only involves the control plane components. Any existing VirtualMachineInstance (VMI) workloads that are running before an update occurs remain 100% untouched. The workloads continue to run and are not interrupted as part of the default update process.</p> <p>It's important to note that these VMI workloads do involve components such as libvirt, qemu, and virt-launcher, which can optionally be updated during the KubeVirt update process as well. However that requires opting in to having virt-operator perform automated actions on workloads.</p> <p>Opting in to VMI updates involves configuring the <code>workloadUpdateStrategy</code> field on the KubeVirt CR. This field controls the methods virt-operator will use to when updating the VMI workload pods.</p> <p>There are two methods supported.</p> <p>LiveMigrate: Which results in VMIs being updated by live migrating the virtual machine guest into a new pod with all the updated components enabled.</p> <p>Evict:  Which results in the VMI's pod being shutdown. If the VMI is controlled by a higher level VirtualMachine object with <code>runStrategy: always</code>, then a new VMI will spin up in a new pod with updated components.</p> <p>The least disruptive way to update VMI workloads is to use LiveMigrate. Any VMI workload that is not live migratable will be left untouched. If live migration is not enabled in the cluster, then the only option available for virt-operator managed VMI updates is the Evict method.</p> <p>Example: Enabling VMI workload updates via LiveMigration</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - LiveMigrate\n</code></pre> <p>Example: Enabling VMI workload updates via Evict with batch tunings</p> <p>The batch tunings allow configuring how quickly VMI's are evicted. In large clusters, it's desirable to ensure that VMI's are evicted in batches in order to distribute load.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - Evict\n    batchEvictionSize: 10\n    batchEvictionInterval: \"1m\"\n</code></pre> <p>Example: Enabling VMI workload updates with both LiveMigrate and Evict</p> <p>When both LiveMigrate and Evict are specified, then any workloads which are live migratable will be guaranteed to be live migrated. Only workloads which are not live migratable will be evicted.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  imagePullPolicy: IfNotPresent\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n      - LiveMigrate\n      - Evict\n    batchEvictionSize: 10\n    batchEvictionInterval: \"1m\"\n</code></pre>"},{"location":"cluster_admin/updating_and_deletion/#deleting-kubevirt","title":"Deleting KubeVirt","text":"<p>To delete the KubeVirt you should first to delete <code>KubeVirt</code> custom resource and then delete the KubeVirt operator.</p> <pre><code>$ export RELEASE=v0.17.0\n$ kubectl delete -n kubevirt kubevirt kubevirt --wait=true # --wait=true should anyway be default\n$ kubectl delete apiservices v1.subresources.kubevirt.io # this needs to be deleted to avoid stuck terminating namespaces\n$ kubectl delete mutatingwebhookconfigurations virt-api-mutator # not blocking but would be left over\n$ kubectl delete validatingwebhookconfigurations virt-operator-validator # not blocking but would be left over\n$ kubectl delete validatingwebhookconfigurations virt-api-validator # not blocking but would be left over\n$ kubectl delete -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml --wait=false\n</code></pre> <p>Note: If by mistake you deleted the operator first, the KV custom resource will get stuck in the <code>Terminating</code> state, to fix it, delete manually finalizer from the resource.</p> <p>Note: The <code>apiservice</code> and the <code>webhookconfigurations</code> need to be deleted manually due to a bug.</p> <pre><code>$ kubectl -n kubevirt patch kv kubevirt --type=json -p '[{ \"op\": \"remove\", \"path\": \"/metadata/finalizers\" }]'\n</code></pre>"},{"location":"cluster_admin/virtual_machines_on_Arm64/","title":"Virtual Machines on Arm64","text":"<p>This page summaries all unsupported Virtual Machines configurations and different default setups on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#virtual-hardware","title":"Virtual hardware","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#machine-type","title":"Machine Type","text":"<p>Currently, we only support one machine type, <code>virt</code>, which is set by default.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#biosuefi","title":"BIOS/UEFI","text":"<p>On Arm64 platform, we only support UEFI boot which is set by default. UEFI secure boot is not supported.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#cpu","title":"CPU","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#node-labeller","title":"Node-labeller","text":"<p>Currently, Node-labeller is partially supported on Arm64 platform. It does not yet support parsing virsh_domcapabilities.xml and capabilities.xml, and extracting related information such as CPU features.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#model","title":"Model","text":"<p><code>host-passthrough</code> is the only model that supported on Arm64. The CPU model is set by default on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#clock","title":"Clock","text":"<p><code>kvm</code> and <code>hyperv</code> timers are not supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#video-and-graphics-device","title":"Video and Graphics Device","text":"<p>We do not support vga devices but use virtio-gpu by default.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#hugepages","title":"Hugepages","text":"<p>Hugepages are not supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#resources-requests-and-limits","title":"Resources Requests and Limits","text":"<p>CPU pinning is supported on Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#numa","title":"NUMA","text":"<p>As Hugepages are a precondition of the NUMA feature, and Hugepages are not enabled on the Arm64 platform, the NUMA feature does not work on Arm64.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#disks-and-volumes","title":"Disks and Volumes","text":"<p>Arm64 only supports virtio and scsi disk bus types.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#interface-and-networks","title":"Interface and Networks","text":""},{"location":"cluster_admin/virtual_machines_on_Arm64/#macvlan","title":"macvlan","text":"<p>We do not support <code>macvlan</code> network because the project https://github.com/kubevirt/macvtap-cni does not support Arm64.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#sriov","title":"SRIOV","text":"<p>This class of devices is not verified on the Arm64 platform.</p>"},{"location":"cluster_admin/virtual_machines_on_Arm64/#liveness-and-readiness-probes","title":"Liveness and Readiness Probes","text":"<p><code>Watchdog</code> device is not supported on Arm64 platform.</p>"},{"location":"compute/client_passthrough/","title":"Client Passthrough","text":"<p>KubeVirt included support for redirecting devices from the client's machine to the VMI with the support of virtctl command.</p>"},{"location":"compute/client_passthrough/#usb-redirection","title":"USB Redirection","text":"<p>Support for redirection of client's USB device was introduced in release v0.44. This feature is not enabled by default. To enable it, add an empty <code>clientPassthrough</code> under devices, as such:</p> <pre><code>spec:\n  domain:\n    devices:\n      clientPassthrough: {}\n</code></pre> <p>This configuration currently adds 4 USB slots to the VMI that can only be used with virtctl.</p> <p>There are two ways of redirecting the same USB devices: Either using its device's vendor and product information or the actual bus and device address information. In Linux, you can gather this info with <code>lsusb</code>, a redacted example below:</p> <pre><code>&gt; lsusb\nBus 002 Device 008: ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2/50\nBus 001 Device 003: ID 13d3:5406 IMC Networks Integrated Camera\nBus 001 Device 010: ID 0781:55ae SanDisk Corp. Extreme 55AE\n</code></pre>"},{"location":"compute/client_passthrough/#using-vendor-and-product","title":"Using Vendor and Product","text":"<p>Redirecting the Kingston storage device. <pre><code>virtctl usbredir 0951:1666 vmi-name\n</code></pre></p>"},{"location":"compute/client_passthrough/#using-bus-and-device-address","title":"Using Bus and Device address","text":"<p>Redirecting the integrated camera <pre><code>virtctl usbredir 01-03 vmi-name\n</code></pre></p>"},{"location":"compute/client_passthrough/#requirements-for-virtctl-usbredir","title":"Requirements for <code>virtctl usbredir</code>","text":"<p>The <code>virtctl</code> command uses an application called <code>usbredirect</code> to handle client's USB device by unplugging the device from the Client OS and channeling the communication between the device and the VMI.</p>"},{"location":"compute/client_passthrough/#usbredirect","title":"usbredirect","text":"<p>The <code>usbredirect</code> binary comes from the usbredir project and is supported by most Linux distros. You can either fetch the latest release or MSI installer for Windows support.</p>"},{"location":"compute/client_passthrough/#permissions","title":"Permissions","text":"<p>Managing USB devices requires privileged access in most Operation Systems. The user running <code>virtctl usbredir</code> would need to be privileged or run it in a privileged manner (e.g: with <code>sudo</code>)</p>"},{"location":"compute/client_passthrough/#windows-support","title":"Windows support","text":"<ul> <li>Redirecting USB devices on Windows requires the installation of UsbDk.</li> <li>Be sure to have <code>usbredirect</code> included in the PATH Enviroment Variable.</li> </ul>"},{"location":"compute/cpu_hotplug/","title":"CPU Hotplug","text":"<p>The CPU hotplug feature was introduced in KubeVirt v1.0, making it possible to configure the VM workload to allow for adding or removing virtual CPUs while the VM is running.</p>"},{"location":"compute/cpu_hotplug/#abstract","title":"Abstract","text":"<p>A virtual CPU (vCPU) is the CPU that is seen to the Guest VM OS. A VM owner can manage the amount of vCPUs from the VM spec template using the CPU topology fields (<code>spec.template.spec.domain.cpu</code>). The <code>cpu</code> object has the integers <code>cores,sockets,threads</code> so that the virtual CPU is calculated by the following formula: <code>cores * sockets * threads</code>. </p> <p>Before CPU hotplug was introduced, the VM owner could change these integers in the VM template while the VM is running, and they were staged until the next boot cycle. With CPU hotplug, it is possible to patch the <code>sockets</code> integer in the VM template and the change will take effect right away. </p> <p>Per each new socket that is hot-plugged, the amount of new vCPUs that would be seen by the guest is <code>cores * threads</code>, since the overall calculation of vCPUs is <code>cores * sockets * threads</code>. </p>"},{"location":"compute/cpu_hotplug/#configuration","title":"Configuration","text":""},{"location":"compute/cpu_hotplug/#enable-feature-gate","title":"Enable feature-gate","text":"<p>In order to enable CPU hotplug we need to add the <code>VMLiveUpdateFeatures</code> feature gate in Kubevirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"compute/cpu_hotplug/#configure-the-workload-update-strategy","title":"Configure the workload update strategy","text":"<p>Current implementation of the hotplug process requires the VM to live-migrate. The migration will be triggered automatically by the workload updater. The workload update strategy in the KubeVirt CR must be configured with <code>LiveMigrate</code>, as follows:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre>"},{"location":"compute/cpu_hotplug/#configure-the-vm-rollout-strategy","title":"Configure the VM rollout strategy","text":"<p>Hotplug requires a VM rollout strategy of <code>LiveUpdate</code>, so that the changes made to the VM object propagate to the VMI without a restart. This is also done in the KubeVirt CR configuration:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre> <p>More information can be found on the VM Rollout Strategies page</p>"},{"location":"compute/cpu_hotplug/#optional-set-maximum-sockets-or-hotplug-ratio","title":"[OPTIONAL] Set maximum sockets or hotplug ratio","text":"<p>You can explicitly set the maximum amount of sockets in three ways:</p> <ol> <li>with a value VM level</li> <li>with a value at the cluster level</li> <li>with a ratio at the cluster level (<code>maxSockets = ratio * sockets</code>).</li> </ol> <p>Note: the third way (cluster-level ratio) will also affect other quantitative hotplug resources like memory.</p> <p> VM level </p> <p> Cluster level value </p> <p> Cluster level ratio </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nspec:\n  template:\n    spec:\n      domain:\n        cpu:\n          maxSockets: 8\n</code></pre> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxCpuSockets: 8\n</code></pre> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxHotplugRatio: 4\n</code></pre> <p>The VM-level configuration will take precedence over the cluster-wide configuration.</p>"},{"location":"compute/cpu_hotplug/#hotplug-process","title":"Hotplug process","text":"<p>Let's assume we have a running VM with the 4 vCPUs, which were configured with <code>sockets:4 cores:1 threads:1</code> In the VMI status we can observe the current CPU topology the VM is running with:</p> <p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\n...\nstatus:\n  currentCPUTopology:\n    cores: 1\n    sockets: 4\n    threads: 1\n</code></pre> Now we want to hotplug another socket, by patching the VM object:</p> <p><pre><code>kubectl patch vm vm-cirros --type='json' \\\n-p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/domain/cpu/sockets\", \"value\": 5}]'\n</code></pre> We can observe the CPU hotplug process in the VMI status:</p> <pre><code>status:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: null\n    status: \"True\"\n    type: LiveMigratable\n  - lastProbeTime: null\n    lastTransitionTime: null\n    status: \"True\"\n    type: HotVCPUChange\n  currentCPUTopology:\n    cores: 1\n    sockets: 4\n    threads: 1\n</code></pre> <p>Please note the condition <code>HotVCPUChange</code> that indicates the hotplug process is taking place. Also you can notice the VirtualMachineInstanceMigration object that was created for the VM in subject:</p> <p><pre><code>NAME                             PHASE     VMI\nkubevirt-workload-update-kflnl   Running   vm-cirros\n</code></pre> When the hotplug process has completed, the <code>currentCPUTopology</code> will be updated with the new number of sockets and the migration is marked as successful.</p> <pre><code>#kubectl get vmi vm-cirros -oyaml\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vm-cirros\nspec:\n  domain:\n    cpu:\n      cores: 1\n      sockets: 5\n      threads: 1\n...\n...\nstatus:\n  currentCPUTopology:\n    cores: 1\n    sockets: 5\n    threads: 1\n\n\n#kubectl get vmim -l kubevirt.io/vmi-name=vm-cirros\nNAME                             PHASE       VMI\nkubevirt-workload-update-cgdgd   Succeeded   vm-cirros\n</code></pre>"},{"location":"compute/cpu_hotplug/#limitations","title":"Limitations","text":"<ul> <li>VPCU hotplug is currently not supported by ARM64 architecture.</li> <li>Current hotplug implementation involves live-migration of the VM workload.</li> </ul>"},{"location":"compute/dedicated_cpu_resources/","title":"Dedicated CPU resources","text":"<p>Certain workloads, requiring a predictable latency and enhanced performance during its execution would benefit from obtaining dedicated CPU resources. KubeVirt, relying on the Kubernetes CPU manager, is able to pin guest's vCPUs to the host's pCPUs.</p>"},{"location":"compute/dedicated_cpu_resources/#kubernetes-cpu-manager","title":"Kubernetes CPU manager","text":"<p>Kubernetes CPU manager is a mechanism that affects the scheduling of workloads, placing it on a host which can allocate <code>Guaranteed</code> resources and pin certain Pod's containers to host pCPUs, if the following requirements are met:</p> <ul> <li>Pod's QoS is Guaranteed<ul> <li>resources requests and limits are equal</li> <li>all containers in the Pod express CPU and memory requirements</li> </ul> </li> <li>Requested number of CPUs is an Integer</li> </ul> <p>Additional information: </p> <ul> <li>Enabling the CPU manager on Kubernetes</li> <li>Enabling the CPU manager on OKD</li> <li>Kubernetes blog explaining the feature</li> </ul>"},{"location":"compute/dedicated_cpu_resources/#requesting-dedicated-cpu-resources","title":"Requesting dedicated CPU resources","text":"<p>Setting <code>spec.domain.cpu.dedicatedCpuPlacement</code> to <code>true</code> in a VMI spec will indicate the desire to allocate dedicated CPU resource to the VMI</p> <p>Kubevirt will verify that all the necessary conditions are met, for the Kubernetes CPU manager to pin the virt-launcher container to dedicated host CPUs. Once, virt-launcher is running, the VMI's vCPUs will be pinned to the pCPUS that has been dedicated for the virt-launcher container.</p> <p>Expressing the desired amount of VMI's vCPUs can be done by either setting the guest topology in <code>spec.domain.cpu</code> (<code>sockets</code>, <code>cores</code>, <code>threads</code>) or <code>spec.domain.resources.[requests/limits].cpu</code> to a whole number integer ([1-9]+) indicating the number of vCPUs requested for the VMI. Number of vCPUs is counted as <code>sockets * cores * threads</code> or if <code>spec.domain.cpu</code> is empty then it takes value from <code>spec.domain.resources.requests.cpu</code> or <code>spec.domain.resources.limits.cpu</code>.</p> <p>Note: Users should not specify both <code>spec.domain.cpu</code> and <code>spec.domain.resources.[requests/limits].cpu</code></p> <p>Note: <code>spec.domain.resources.requests.cpu</code> must be equal to <code>spec.domain.resources.limits.cpu</code></p> <p>Note: Multiple cpu-bound microbenchmarks show a significant performance advantage when using <code>spec.domain.cpu.sockets</code> instead of <code>spec.domain.cpu.cores</code>.</p> <p>All inconsistent requirements will be rejected.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      sockets: 2\n      cores: 1\n      threads: 1\n      dedicatedCpuPlacement: true\n    resources:\n      limits:\n        memory: 2Gi\n[...]\n</code></pre> <p>OR</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      dedicatedCpuPlacement: true\n    resources:\n      limits:\n        cpu: 2\n        memory: 2Gi\n[...]\n</code></pre>"},{"location":"compute/dedicated_cpu_resources/#requesting-dedicated-cpu-for-qemu-emulator","title":"Requesting dedicated CPU for QEMU emulator","text":"<p>A number of QEMU threads, such as QEMU main event loop, async I/O operation completion, etc., also execute on the same physical CPUs as the VMI's vCPUs. This may affect the expected latency of a vCPU. In order to enhance the real-time support in KubeVirt and provide improved latency, KubeVirt will allocate an additional dedicated CPU, exclusively for the emulator thread, to which it will be pinned. This will effectively \"isolate\" the emulator thread from the vCPUs of the VMI. In case <code>ioThreadsPolicy</code> is set to <code>auto</code> IOThreads will also be \"isolated\" and placed on the same physical CPU as the QEMU emulator thread.</p> <p>This functionality can be enabled by specifying <code>isolateEmulatorThread: true</code> inside VMI spec's <code>Spec.Domain.CPU</code> section. Naturally, this setting has to be specified in a combination with a <code>dedicatedCpuPlacement: true</code>.</p> <p>Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    cpu:\n      dedicatedCpuPlacement: true\n      isolateEmulatorThread: true\n    resources:\n      limits:\n        cpu: 2\n        memory: 2Gi\n</code></pre>"},{"location":"compute/dedicated_cpu_resources/#compute-nodes-with-smt-enabled","title":"Compute Nodes with SMT Enabled","text":"<p>When the following conditions are met:</p> <ul> <li>The compute node has SMT enabled</li> <li>Kubelet's CPUManager policy is set to static - full-pcpus-only</li> <li>The VM is configured to have an even number of CPUs</li> <li><code>dedicatedCpuPlacement</code> and <code>isolateEmulatorThread</code> are enabled</li> </ul> <p>The VM is scheduled, but rejected by the kubelet with the following event: <pre><code>SMT Alignment Error: requested 3 cpus not multiple cpus per core = 2\n</code></pre></p> <p>In order to address this issue:</p> <ol> <li>Enable the <code>AlignCPUs</code> feature gate in the KubeVirt CR.</li> <li>Add the following annotation to the Kubevirt CR:</li> </ol> <pre><code>alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity:\n</code></pre> <p>KubeVirt will then add one or two dedicated CPUs for the emulator threads, in a way that completes the total CPU count to be even.</p>"},{"location":"compute/dedicated_cpu_resources/#identifying-nodes-with-a-running-cpu-manager","title":"Identifying nodes with a running CPU manager","text":"<p>At this time, Kubernetes doesn't label the nodes that has CPU manager running on it.</p> <p>KubeVirt has a mechanism to identify which nodes has the CPU manager running and manually add a <code>cpumanager=true</code> label. This label will be removed when KubeVirt will identify that CPU manager is no longer running on the node. This automatic identification should be viewed as a temporary workaround until Kubernetes will provide the required functionality. Therefore, this feature should be manually enabled by activating the <code>CPUManager</code> feature gate to the KubeVirt CR.</p> <p>When automatic identification is disabled, cluster administrator may manually add the above label to all the nodes when CPU Manager is running.</p> <ul> <li> <p>Nodes' labels are view-able: <code>kubectl describe nodes</code></p> </li> <li> <p>Administrators may manually label a missing node:     <code>kubectl label node [node_name] cpumanager=true</code></p> </li> </ul>"},{"location":"compute/dedicated_cpu_resources/#sidecar-containers-and-cpu-allocation-overhead","title":"Sidecar containers and CPU allocation overhead","text":"<p>Note: In order to run sidecar containers, KubeVirt requires the <code>Sidecar</code> feature gate to be enabled in KubeVirt's CR.</p> <p>According to the Kubernetes CPU manager model, in order the POD would reach the required QOS level <code>Guaranteed</code>, all containers in the POD must express CPU and memory requirements. At this time, Kubevirt often uses a sidecar container to mount VMI's registry disk. It also uses a sidecar container of it's hooking mechanism. These additional resources can be viewed as an overhead and should be taken into account when calculating a node capacity.</p> <p>Note: The current defaults for sidecar's resources: <code>CPU: 200m</code> <code>Memory: 64M</code> As the CPU resource is not expressed as a whole number, CPU manager will not attempt to pin the sidecar container to a host CPU.</p>"},{"location":"compute/host-devices/","title":"Host Devices Assignment","text":"<p>KubeVirt provides a mechanism for assigning host devices to a virtual machine. This mechanism is generic and allows various types of PCI devices, such as accelerators (including GPUs) or any other devices attached to a PCI bus, to be assigned. It also allows Linux Mediated devices, such as pre-configured virtual GPUs to be assigned using the same mechanism.</p>"},{"location":"compute/host-devices/#host-preparation-for-pci-passthrough","title":"Host preparation for PCI Passthrough","text":"<ul> <li> <p>Host Devices passthrough requires the virtualization extension and the IOMMU extension (Intel VT-d or AMD IOMMU) to be enabled in the BIOS.</p> </li> <li> <p>To enable IOMMU, depending on the CPU type, a host should be booted with an additional kernel parameter, <code>intel_iommu=on</code> for Intel and <code>amd_iommu=on</code> for AMD.</p> </li> </ul> <p>Append these parameters to the end of the GRUB_CMDLINE_LINUX line in the grub configuration file.</p> <pre><code># vi /etc/default/grub\n...\nGRUB_CMDLINE_LINUX=\"nofb splash=quiet console=tty0 ... intel_iommu=on\n...\n\n# grub2-mkconfig -o /boot/grub2/grub.cfg\n\n# reboot\n</code></pre> <ul> <li>The vfio-pci kernel module should be enabled on the host. <pre><code># modprobe vfio-pci\n</code></pre></li> </ul>"},{"location":"compute/host-devices/#preparation-of-pci-devices-for-passthrough","title":"Preparation of PCI devices for passthrough","text":"<p>At this time, KubeVirt is only able to assign PCI devices that are using the <code>vfio-pci</code> driver. To prepare a specific device for device assignment, it should first be unbound from its original driver and bound to the <code>vfio-pci</code> driver.</p> <ul> <li>Find the PCI address of the desired device:</li> </ul> <pre><code>$ lspci -DD|grep NVIDIA\n0000.65:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)\n</code></pre> <ul> <li>Bind that device to the <code>vfio-pci</code> driver: <pre><code>echo 0000:65:00.0 &gt; /sys/bus/pci/drivers/nvidia/unbind\necho \"vfio-pci\" &gt; /sys/bus/pci/devices/0000\\:65\\:00.0/driver_override\necho 0000:65:00.0 &gt; /sys/bus/pci/drivers/vfio-pci/bind\n</code></pre></li> </ul>"},{"location":"compute/host-devices/#preparation-of-mediated-devices-such-as-vgpu","title":"Preparation of mediated devices such as vGPU","text":"<p>In general, configuration of a Mediated devices (mdevs), such as vGPUs, should be done according to the vendor directions.  KubeVirt can now facilitate the creation of the mediated devices / vGPUs on the cluster nodes. This assumes that the required vendor driver is already installed on the nodes. See the Mediated devices and virtual GPUs to learn more about this functionality.</p> <p>Once the mdev is configured, KubeVirt will be able to discover and use it for device assignment.</p>"},{"location":"compute/host-devices/#listing-permitted-devices","title":"Listing permitted devices","text":"<p>Administrators can control which host devices are exposed and permitted to be used in the cluster. Permitted host devices in the cluster will need to be allowlisted in KubeVirt CR by its <code>vendor:product</code> selector for PCI devices or mediated device names.</p> <pre><code>configuration:\n  permittedHostDevices:\n    pciHostDevices:\n    - pciVendorSelector: \"10DE:1EB8\"\n      resourceName: \"nvidia.com/TU104GL_Tesla_T4\"\n      externalResourceProvider: true\n    - pciVendorSelector: \"8086:6F54\"\n      resourceName: \"intel.com/qat\"\n    mediatedDevices:\n    - mdevNameSelector: \"GRID T4-1Q\"\n      resourceName: \"nvidia.com/GRID_T4-1Q\"\n</code></pre> <ul> <li> <p><code>pciVendorSelector</code> is a PCI vendor ID and product ID tuple in the form <code>vendor_id:product_id</code>.  This tuple can identify specific types of devices on a host. For example, the identifier <code>10de:1eb8</code>, shown above, can be found using <code>lspci</code>.</p> <pre><code>$ lspci -nnv|grep -i nvidia\n65:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)\n</code></pre> </li> <li> <p><code>mdevNameSelector</code> is a name of a Mediated device type that can identify specific types of Mediated devices on a host.</p> <p>You can see what mediated types a given PCI device supports by examining the contents of <code>/sys/bus/pci/devices/SLOT:BUS:DOMAIN.FUNCTION/mdev_supported_types/TYPE/name</code>. For example, if you have an NVIDIA T4 GPU on your system, and you substitute in the <code>SLOT</code>, <code>BUS</code>, <code>DOMAIN</code>, and <code>FUNCTION</code> values that are correct for your system into the above path name, you will see that a <code>TYPE</code> of <code>nvidia-226</code> contains the selector string <code>GRID T4-2A</code> in its <code>name</code> file.</p> <p>Taking <code>GRID T4-2A</code> and specifying it as the <code>mdevNameSelector</code> allows KubeVirt to find a corresponding mediated device by matching it against <code>/sys/class/mdev_bus/SLOT:BUS:DOMAIN.FUNCTION/$mdevUUID/mdev_type/name</code> for some values of <code>SLOT:BUS:DOMAIN.FUNCTION</code> and <code>$mdevUUID</code>.</p> </li> <li> <p>External providers: <code>externalResourceProvider</code> field indicates that this resource is being provided by an external device plugin. In this case, KubeVirt will only permit the usage of this device in the cluster but will leave the allocation and monitoring to an external device plugin.</p> </li> </ul>"},{"location":"compute/host-devices/#starting-a-virtual-machine","title":"Starting a Virtual Machine","text":"<p>Host devices can be assigned to virtual machines via the <code>gpus</code> and <code>hostDevices</code> fields.  The <code>deviceNames</code> can reference both PCI and Mediated device resource names.</p> <pre><code>kind: VirtualMachineInstance\nspec:\n  domain:\n    devices:\n      gpus:\n      - deviceName: nvidia.com/TU104GL_Tesla_T4\n        name: gpu1\n      - deviceName: nvidia.com/GRID_T4-1Q\n        name: gpu2\n      hostDevices:\n      - deviceName: intel.com/qat\n        name: quickaccess1\n</code></pre>"},{"location":"compute/host-devices/#nvme-pci-passthrough","title":"NVMe PCI passthrough","text":"<p>In order to passthrough an NVMe device the procedure is very similar to the gpu case. The device needs to be listed under the <code>permittedHostDevice</code> and under <code>hostDevices</code> in the VM declaration. </p> <p>Currently, the KubeVirt device plugin doesn't allow the user to select a specific device by specifying the address. Therefore, if multiple NVMe devices with the same vendor and product id exist in the cluster, they could be randomly assigned to a VM. If the devices are not on the same node, then the nodeSelector mitigates the issue.</p> <p>Example:</p> <p>Modify the <code>permittedHostDevice</code></p> <pre><code>configuration:\n  permittedHostDevices:\n    pciHostDevices:\n    - pciVendorSelector: 8086:5845\n      resourceName: devices.kubevirt.io/nvme\n</code></pre> <p>VMI declaration: <pre><code>kind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-nvme\n  name: vmi-nvme\nspec:\n  nodeSelector: \n    kubernetes.io/hostname: node03   # &lt;--\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      hostDevices:  # &lt;--\n      - name: nvme  # &lt;--\n        deviceName: devices.kubevirt.io/nvme  # &lt;--\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p>"},{"location":"compute/host-devices/#usb-host-passthrough","title":"USB Host Passthrough","text":"<p>Since KubeVirt v1.1, we can provide USB devices that are plugged in a Node to the VM running in the same Node.</p>"},{"location":"compute/host-devices/#requirements","title":"Requirements","text":"<p>Cluster admin privilege to edit the KubeVirt CR in order to:</p> <ul> <li>Enable the <code>HostDevices</code> feature gate</li> <li>Edit the <code>permittedHostDevices</code> configuration to expose node USB devices to the cluster</li> </ul>"},{"location":"compute/host-devices/#exposing-usb-devices","title":"Exposing USB Devices","text":"<p>In order to assign USB devices to your VMI, you'll need to expose those devices to the cluster under a resource name. The device allowlist can be edited in KubeVirt CR under <code>configuration.permittedHostDevices.usb</code>.</p> <p>For this example, we will use the <code>kubevirt.io/storage</code> resource name for the device with <code>vendor: \"46f4\"</code> and <code>product: \"0001\"</code> <sup>1</sup>.</p> <pre><code>spec:\n  configuration:\n    permittedHostDevices:\n      usb:\n        - resourceName: kubevirt.io/storage\n          selectors:\n            - vendor: \"46f4\"\n              product: \"0001\"\n</code></pre> <p>After adding the <code>usb</code> configuration under <code>permittedHostDevices</code> to the KubeVirt CR, KubeVirt's device-plugin will expose this resource name and you can use it in your VMI.</p>"},{"location":"compute/host-devices/#adding-usb-to-your-vm","title":"Adding USB to your VM","text":"<p>Now, in the VMI configuration, you can add the <code>devices.hostDevices.deviceName</code> and reference the resource name provided in the previous step, and also give it a local <code>name</code>, for example:</p> <pre><code>spec:\n  domain:\n    devices:\n      hostDevices:\n      - deviceName: kubevirt.io/storage\n        name: usb-storage\n</code></pre> <p>You can find a working example, which uses QEMU's emulated USB storage, under examples/vmi-usb.yaml.</p>"},{"location":"compute/host-devices/#bundle-of-usb-devices","title":"Bundle of USB devices","text":"<p>You might be interested to redirect more than one USB device to a VMI, for example, a keyboard, a mouse and a smartcard device. The KubeVirt CR supports assigning multiple USB devices under the same resource name, so you could do:</p> <pre><code>spec:\n  configuration:\n    permittedHostDevices:\n      usb:\n        - resourceName: kubevirt.io/peripherals\n          selectors:\n            - vendor: \"045e\"\n              product: \"07a5\"\n            - vendor: \"062a\"\n              product: \"4102\"\n            - vendor: \"072f\"\n              product: \"b100\"\n</code></pre> <p>Adding to the VMI configuration:</p> <pre><code>spec:\n  domain:\n    devices:\n      hostDevices:\n      - deviceName: kubevirt.io/peripherals\n        name: local-peripherals \n</code></pre> <p>Note that all USB devices need to be present in order for the assignment to work.</p> <ol> <li> <p>Note that you can easily find the <code>vendor:product</code> value with the <code>lsusb</code> command.\u00a0\u21a9</p> </li> </ol>"},{"location":"compute/hugepages/","title":"Hugepages support","text":"<p>For hugepages support you need at least Kubernetes version <code>1.9</code>.</p>"},{"location":"compute/hugepages/#enable-feature-gate","title":"Enable feature-gate","text":"<p>To enable hugepages on Kubernetes, check the official documentation.</p> <p>To enable hugepages on OKD, check the official documentation.</p>"},{"location":"compute/hugepages/#pre-allocate-hugepages-on-a-node","title":"Pre-allocate hugepages on a node","text":"<p>To pre-allocate hugepages on boot time, you will need to specify hugepages under kernel boot parameters <code>hugepagesz=2M hugepages=64</code> and restart your machine.</p> <p>You can find more about hugepages under official documentation.</p>"},{"location":"compute/live_migration/","title":"Live Migration","text":"<p>Live migration is a process during which a running Virtual Machine Instance moves to another compute node while the guest workload continues to run and remain accessible.</p>"},{"location":"compute/live_migration/#enabling-the-live-migration-support","title":"Enabling the live-migration support","text":"<p>Live migration is enabled by default in recent versions of KubeVirt. Versions prior to v0.56, it must be enabled in the feature gates. The feature gates field in the KubeVirt CR must be expanded by adding the <code>LiveMigration</code> to it.</p>"},{"location":"compute/live_migration/#limitations","title":"Limitations","text":"<ul> <li> <p>Virtual machines using a PersistentVolumeClaim (PVC) must have a   shared ReadWriteMany (RWX) access mode to be live migrated.</p> </li> <li> <p>Live migration is not allowed with a pod network binding of bridge   interface type   ()</p> </li> <li> <p>Live migration requires ports <code>49152, 49153</code> to be available in the virt-launcher pod.   If these ports are explicitly specified in masquarade interface, live migration will not function.</p> </li> </ul>"},{"location":"compute/live_migration/#initiate-live-migration","title":"Initiate live migration","text":"<p>Live migration is initiated by posting a VirtualMachineInstanceMigration (VMIM) object to the cluster. The example below starts a migration process for a virtual machine instance <code>vmi-fedora</code></p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\n</code></pre>"},{"location":"compute/live_migration/#using-virtctl-to-initiate-live-migration","title":"Using virtctl to initiate live migration","text":"<p>Live migration can also be initiated using virtctl <pre><code>    virtctl migrate vmi-fedora\n</code></pre></p>"},{"location":"compute/live_migration/#migration-status-reporting","title":"Migration Status Reporting","text":""},{"location":"compute/live_migration/#condition-and-migration-method","title":"Condition and migration method","text":"<p>When starting a virtual machine instance, it has also been calculated whether the machine is live migratable. The result is being stored in the VMI <code>VMI.status.conditions</code>. The calculation can be based on multiple parameters of the VMI, however, at the moment, the calculation is largely based on the <code>Access Mode</code> of the VMI volumes. Live migration is only permitted when the volume access mode is set to <code>ReadWriteMany</code>. Requests to migrate a non-LiveMigratable VMI will be rejected.</p> <p>The reported <code>Migration Method</code> is also being calculated during VMI start. <code>BlockMigration</code> indicates that some of the VMI disks require copying from the source to the destination. <code>LiveMigration</code> means that only the instance memory will be copied.</p> <pre><code>Status:\n  Conditions:\n    Status: True\n    Type: LiveMigratable\n  Migration Method: BlockMigration\n</code></pre>"},{"location":"compute/live_migration/#migration-status","title":"Migration Status","text":"<p>The migration progress status is being reported in the VMI <code>VMI.status</code>. Most importantly, it indicates whether the migration has been <code>Completed</code> or if it <code>Failed</code>.</p> <p>Below is an example of a successful migration.</p> <pre><code>Migration State:\n    Completed:        true\n    End Timestamp:    2019-03-29T03:37:52Z\n    Migration Config:\n      Completion Timeout Per GiB:  800\n      Progress Timeout:             150\n    Migration UID:                  c64d4898-51d3-11e9-b370-525500d15501\n    Source Node:                    node02\n    Start Timestamp:                2019-03-29T04:02:47Z\n    Target Direct Migration Node Ports:\n      35001:                      0\n      41068:                      49152\n      38284:                      49153\n    Target Node:                  node01\n    Target Node Address:          10.128.0.46\n    Target Node Domain Detected:  true\n    Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq\n</code></pre>"},{"location":"compute/live_migration/#canceling-a-live-migration","title":"Canceling a live migration","text":"<p>Live migration can also be canceled by simply deleting the migration object. A successfully aborted migration will indicate that the abort has been requested <code>Abort Requested</code>, and that it succeeded: <code>Abort Status: Succeeded</code>. The migration in this case will be <code>Completed</code> and <code>Failed</code>.</p> <pre><code>Migration State:\n    Abort Requested:  true\n    Abort Status:     Succeeded\n    Completed:        true\n    End Timestamp:    2019-03-29T04:02:49Z\n    Failed:           true\n    Migration Config:\n      Completion Timeout Per GiB:  800\n      Progress Timeout:             150\n    Migration UID:                  57a693d6-51d7-11e9-b370-525500d15501\n    Source Node:                    node02\n    Start Timestamp:                2019-03-29T04:02:47Z\n    Target Direct Migration Node Ports:\n      39445:                      0\n      43345:                      49152\n      44222:                      49153\n    Target Node:                  node01\n    Target Node Address:          10.128.0.46\n    Target Node Domain Detected:  true\n    Target Pod:                   virt-launcher-testvmimcbjgw6zrzcmp8wpddvztvzm7x2k6cjbdgktwv8tkq\n</code></pre>"},{"location":"compute/live_migration/#using-virtctl-to-cancel-a-live-migration","title":"Using virtctl to cancel a live migration","text":"<p>Live migration can also be canceled using virtctl, by specifying the name of a VMI which is currently being migrated <pre><code>    virtctl migrate-cancel vmi-fedora\n</code></pre></p>"},{"location":"compute/live_migration/#changing-cluster-wide-migration-limits","title":"Changing Cluster Wide Migration Limits","text":"<p>KubeVirt puts some limits in place, so that migrations don't overwhelm the cluster. By default, it is configured to only run <code>5</code> migrations in parallel with an additional limit of a maximum of <code>2</code> outbound migrations per node. Finally, every migration is limited to a bandwidth of <code>64MiB/s</code>.</p> <p>These values can be changed in the <code>kubevirt</code> CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    migrations:\n      parallelMigrationsPerCluster: 5\n      parallelOutboundMigrationsPerNode: 2\n      bandwidthPerMigration: 64Mi\n      completionTimeoutPerGiB: 800\n      progressTimeout: 150\n      disableTLS: false\n      nodeDrainTaintKey: \"kubevirt.io/drain\"\n      allowAutoConverge: false\n      allowPostCopy: false\n      unsafeMigrationOverride: false\n</code></pre> <p>Bear in mind that most of these configuration can be overridden and fine-tuned to a specified group of VMs. For more information, please see Migration Policies.</p>"},{"location":"compute/live_migration/#understanding-different-migration-strategies","title":"Understanding different migration strategies","text":"<p>Live migration is a complex process. During a migration, the source VM needs to transfer its whole state (mainly RAM) to the target VM. If there are enough resources available, such as network bandwidth and CPU power, migrations should converge nicely. If this is not the scenario, however, the migration might get stuck without an ability to progress.</p> <p>The main factor that affects migrations from the guest perspective is its <code>dirty rate</code>, which is the rate by which the VM dirties memory. Guests with high dirty rate lead to a race during migration. On the one hand, memory would be transferred continuously to the target, and on the other, the same memory would get dirty by the guest. On such scenarios, one could consider to use more advanced migration strategies.</p> <p>Let's explain the 3 supported migration strategies as of today.</p>"},{"location":"compute/live_migration/#pre-copy","title":"Pre-copy","text":"<p>Pre-copy is the default strategy. It should be used for most cases.</p> <p>The way it works is as following:</p> <ol> <li>The target VM is created, but the guest keeps running on the source VM.</li> <li>The source starts sending chunks of VM state (mostly memory) to the target. This continues until   all of the state has been transferred to the target.</li> <li>The guest starts executing on the target VM.</li> <li>The source VM is being removed.</li> </ol> <p>Pre-copy is the safest and fastest strategy for most cases. Furthermore, it can be easily cancelled, can utilize multithreading, and more. If there is no real reason to use another strategy, this is definitely the strategy to go with.</p> <p>However, on some cases migrations might not converge easily, that is, by the time the chunk of source VM state would be received by the target VM, it would already be mutated by the source VM (which is the VM the guest executes on). There are many reasons for migrations to fail converging, such as a high dirty-rate or low resources like network bandwidth and CPU. On such scenarios, see the following alternative strategies below.</p>"},{"location":"compute/live_migration/#post-copy","title":"Post-copy","text":"<p>The way post-copy migrations work is as following:</p> <ol> <li>The target VM is created.</li> <li>The guest is being run on the target VM.</li> <li>The source starts sending chunks of VM state (mostly memory) to the target.</li> <li>When the guest, running on the target VM, would access memory:<ol> <li>If the memory exists on the target VM, the guest can access it.</li> <li>Otherwise, the target VM asks for a chunk of memory from the source VM.</li> </ol> </li> <li>Once all of the memory state is updated at the target VM, the source VM is being removed.</li> </ol> <p>The main idea here is that the guest starts to run immediately on the target VM. This approach has advantages and disadvantages:</p> <p>advantages:</p> <ul> <li>The same memory chunk is never being transferred twice. This is possible due to the fact that with post-copy it doesn't matter that a page had been dirtied since the guest is already running on the target VM.</li> <li>This means that a high dirty-rate has much less effect.</li> <li>Consumes less network bandwidth.</li> </ul> <p>disadvantages:</p> <ul> <li>When using post-copy, the VM state has no one source of truth. When the guest (running on the target VM) writes to memory, this memory is one part of the guest's state, but some other parts of it may still be updated only at the source VM. This situation is generally dangerous, since, for  example, if either the target or guest VMs crash the state cannot be recovered.</li> <li>Slow warmup: when the guest starts executing, no memory is present at the target VM. Therefore, the guest would have to wait for a lot of memory in a short period of time.</li> <li>Slower than pre-copy on most cases.</li> <li>Harder to cancel a migration.</li> </ul>"},{"location":"compute/live_migration/#auto-converge","title":"Auto-converge","text":"<p>Auto-converge is a technique to help pre-copy migrations converge faster without changing the core algorithm of how the migration works.</p> <p>Since a high dirty-rate is usually the most significant factor for migrations to not converge, auto-converge simply throttles the guest's CPU. If the migration would converge fast enough, the guest's CPU would not be throttled or throttled negligibly. But, if the migration would not converge fast enough, the CPU would be throttled more and more as time goes.</p> <p>This technique dramatically increases the probability of the migration converging eventually.</p>"},{"location":"compute/live_migration/#using-a-different-network-for-migrations","title":"Using a different network for migrations","text":"<p>Live migrations can be configured to happen on a different network than the one Kubernetes is configured to use. That potentially allows for more determinism, control and/or bandwidth, depending on use-cases.</p>"},{"location":"compute/live_migration/#creating-a-migration-network-on-a-cluster","title":"Creating a migration network on a cluster","text":"<p>A separate physical network is required, meaning that every node on the cluster has to have at least 2 NICs, and the NICs that will be used for migrations need to be interconnected, i.e. all plugged to the same switch. The examples below assume that <code>eth1</code> will be used for migrations.</p> <p>It is also required for the Kubernetes cluster to have multus installed.</p> <p>If the desired network doesn't include a DHCP server, then whereabouts will be needed as well.</p> <p>Finally, a NetworkAttachmentDefinition needs to be created in the namespace where KubeVirt is installed. Here is an example: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: migration-network\n  namespace: kubevirt\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"migration-bridge\",\n      \"type\": \"macvlan\",\n      \"master\": \"eth1\",\n      \"mode\": \"bridge\",\n      \"ipam\": {\n        \"type\": \"whereabouts\",\n        \"range\": \"10.1.1.0/24\"\n      }\n    }'\n</code></pre></p>"},{"location":"compute/live_migration/#configuring-kubevirt-to-migrate-vmis-over-that-network","title":"Configuring KubeVirt to migrate VMIs over that network","text":"<p>This is just a matter of adding the name of the NetworkAttachmentDefinition to the KubeVirt CR, like so: <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n      - LiveMigration\n    migrations:\n      network: migration-network\n</code></pre></p> <p>That change will trigger a restart of the virt-handler pods, as they get connected to that new network.</p> <p>From now on, migrations will happen over that network.</p>"},{"location":"compute/live_migration/#configuring-kubevirtci-for-testing-migration-networks","title":"Configuring KubeVirtCI for testing migration networks","text":"<p>Developers and people wanting to test the feature before deploying it on a real cluster might want to configure a dedicated migration network in KubeVirtCI.</p> <p>KubeVirtCI can simply be configured to include a virtual secondary network, as well as automatically install multus and whereabouts. The following environment variables just have to be declared before running <code>make cluster-up</code>: <pre><code>export KUBEVIRT_NUM_NODES=2;\nexport KUBEVIRT_NUM_SECONDARY_NICS=1;\nexport KUBEVIRT_DEPLOY_ISTIO=true;\nexport KUBEVIRT_WITH_CNAO=true\n</code></pre></p>"},{"location":"compute/live_migration/#migration-timeouts","title":"Migration timeouts","text":"<p>Depending on the type, the live migration process will copy virtual machine memory pages and disk blocks to the destination. During this process non-locked pages and blocks are being copied and become free for the instance to use again. To achieve a successful migration, it is assumed that the instance will write to the free pages and blocks (pollute the pages) at a lower rate than these are being copied.</p>"},{"location":"compute/live_migration/#completion-time","title":"Completion time","text":"<p>In some cases the virtual machine can write to different memory pages / disk blocks at a higher rate than these can be copied, which will prevent the migration process from completing in a reasonable amount of time. In this case, live migration will be aborted if it is running for a long period of time. The timeout is calculated base on the size of the VMI, it's memory and the ephemeral disks that are needed to be copied. The configurable parameter <code>completionTimeoutPerGiB</code>, which defaults to 800s is the time for GiB of data to wait for the migration to be completed before aborting it. A VMI with 8Gib of memory will time out after 6400 seconds.</p>"},{"location":"compute/live_migration/#progress-timeout","title":"Progress timeout","text":"<p>Live migration will also be aborted when it will be noticed that copying memory doesn't make any progress. The time to wait for live migration to make progress in transferring data is configurable by <code>progressTimeout</code> parameter, which defaults to 150s</p>"},{"location":"compute/live_migration/#disabling-secure-migrations","title":"Disabling secure migrations","text":"<p>FEATURE STATE: KubeVirt v0.43</p> <p>Sometimes it may be desirable to disable TLS encryption of migrations to improve performance. Use <code>disableTLS</code> to do that:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"LiveMigration\"\n    migrationConfiguration:\n      disableTLS: true\n</code></pre> <p>Note: While this increases performance it may allow MITM attacks. Be careful.</p>"},{"location":"compute/mediated_devices_configuration/","title":"Mediated devices and virtual GPUs","text":""},{"location":"compute/mediated_devices_configuration/#configuring-mediated-devices-and-virtual-gpus","title":"Configuring mediated devices and virtual GPUs","text":"<p>KubeVirt aims to facilitate the configuration of mediated devices on large clusters. Administrators can use the <code>mediatedDevicesConfiguration</code> API in the KubeVirt CR to create or remove mediated devices in a declarative way, by providing a list of the desired mediated device types that they expect to be configured in the cluster.</p> <p>You can also include the <code>nodeMediatedDeviceTypes</code> option to provide a more specific configuration that targets a specific node or a group of nodes directly with a node selector. The <code>nodeMediatedDeviceTypes</code> option must be used in combination with <code>mediatedDevicesTypes</code> in order to override the global configuration set in the <code>mediatedDevicesTypes</code> section.</p> <p>KubeVirt will use the provided configuration to automatically create the relevant mdev/vGPU devices on nodes that can support it.</p> <p>Currently, a single mdev type per card will be configured. The maximum amount of instances of the selected mdev type will be configured per card.</p> <p>Note: Some vendors, such as NVIDIA, require a driver to be installed on the nodes to provide mediated devices, including vGPUs.</p> <p>Example snippet of a KubeVirt CR configuration that includes both <code>nodeMediatedDeviceTypes</code> and <code>mediatedDevicesTypes</code>: <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-222\n      - nvidia-228\n      nodeMediatedDeviceTypes:\n      - nodeSelector:\n          kubernetes.io/hostname: nodeName\n        mediatedDevicesTypes:\n        - nvidia-234\n</code></pre></p>"},{"location":"compute/mediated_devices_configuration/#configuration-scenarios","title":"Configuration scenarios","text":""},{"location":"compute/mediated_devices_configuration/#example-large-cluster-with-multiple-cards-on-each-node","title":"Example: Large cluster with multiple cards on each node","text":"<p>On nodes with multiple cards that can support similar vGPU types, the relevant desired types will be created in a round-robin manner.</p> <p>For example, considering the following KubeVirt CR configuration:</p> <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-222\n      - nvidia-228\n      - nvidia-105\n      - nvidia-108\n</code></pre> <p>This cluster has nodes with two different PCIe cards:</p> <ol> <li> <p>Nodes with 3 Tesla T4 cards, where each card can support multiple devices types:</p> <ul> <li>nvidia-222</li> <li>nvidia-223</li> <li>nvidia-228</li> <li>...</li> </ul> </li> <li> <p>Nodes with 2 Tesla V100 cards, where each card can support multiple device types:</p> <ul> <li>nvidia-105</li> <li>nvidia-108</li> <li>nvidia-217</li> <li>nvidia-299</li> <li>...</li> </ul> </li> </ol> <p>KubeVirt will then create the following devices:</p> <ol> <li>Nodes with 3 Tesla T4 cards will be configured with:<ul> <li>16 vGPUs of type nvidia-222 on card 1</li> <li>2 vGPUs of type nvidia-228 on card 2</li> <li>16 vGPUs of type nvidia-222 on card 3</li> </ul> </li> <li>Nodes with 2 Tesla V100 cards will be configured with:<ul> <li>16 vGPUs of type nvidia-105 on card 1</li> <li>2 vGPUs of type nvidia-108 on card 2</li> </ul> </li> </ol>"},{"location":"compute/mediated_devices_configuration/#example-single-card-on-a-node-multiple-desired-vgpu-types-are-supported","title":"Example: Single card on a node, multiple desired vGPU types are supported","text":"<p>When nodes only have a single card, the first supported type from the list will be configured.</p> <p>For example, consider the following list of desired types, where nvidia-223 and nvidia-224 are supported:</p> <p><pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-223\n      - nvidia-224\n</code></pre> In this case, nvidia-223 will be configured on the node because it is the first supported type in the list.</p>"},{"location":"compute/mediated_devices_configuration/#overriding-configuration-on-a-specifc-node","title":"Overriding configuration on a specifc node","text":"<p>To override the global configuration set by <code>mediatedDevicesTypes</code>, include the <code>nodeMediatedDeviceTypes</code> option, specifying the node selector and the <code>mediatedDevicesTypes</code> that you want to override for that node.</p>"},{"location":"compute/mediated_devices_configuration/#example-overriding-the-configuration-for-a-specific-node-in-a-large-cluster-with-multiple-cards-on-each-node","title":"Example: Overriding the configuration for a specific node in a large cluster with multiple cards on each node","text":"<p>In this example, the KubeVirt CR includes the <code>nodeMediatedDeviceTypes</code> option to override the global configuration specifically for node 2, which will only use the nvidia-234 type.</p> <pre><code>spec:\n  configuration:\n    mediatedDevicesConfiguration:\n      mediatedDevicesTypes:\n      - nvidia-230\n      - nvidia-223\n      - nvidia-224\n    nodeMediatedDeviceTypes:\n    - nodeSelector:\n        kubernetes.io/hostname: node2  \n      mediatedDevicesTypes:\n      - nvidia-234\n</code></pre> <p>The cluster has two nodes that both have 3 Tesla T4 cards.</p> <ul> <li>Each card can support a long list of types, including:<ul> <li>nvidia-222</li> <li>nvidia-223</li> <li>nvidia-224</li> <li>nvidia-230</li> <li>...</li> </ul> </li> </ul> <p>KubeVirt will then create the following devices:</p> <ol> <li>Node 1<ul> <li>type nvidia-230 on card 1</li> <li>type nvidia-223 on card 2</li> </ul> </li> <li>Node 2<ul> <li>type nvidia-234 on card 1 and card 2</li> </ul> </li> </ol> <p>Node 1 has been configured in a round-robin manner based on the global configuration but node 2 only uses the nvidia-234 that was specified for it.</p>"},{"location":"compute/mediated_devices_configuration/#updating-and-removing-vgpu-types","title":"Updating and Removing vGPU types","text":"<p>Changes made to the <code>mediatedDevicesTypes</code> section of the KubeVirt CR will trigger a re-evaluation of the configured mdevs/vGPU types on the cluster nodes.</p> <p>Any change to the node labels that match the <code>nodeMediatedDeviceTypes</code> nodeSelector in the KubeVirt CR will trigger a similar re-evaluation.</p> <p>Consequently, mediated devices will be reconfigured or entirely removed based on the updated configuration.</p>"},{"location":"compute/mediated_devices_configuration/#assigning-vgpumdev-to-a-virtual-machine","title":"Assigning vGPU/MDEV to a Virtual Machine","text":"<p>See the Host Devices Assignment to learn how to consume the newly created mediated devices/vGPUs.</p>"},{"location":"compute/memory_dump/","title":"Virtual machine memory dump","text":"<p>Kubevirt now supports getting a VM memory dump for analysis purposes. The Memory dump can be used to diagnose, identify and resolve issues in the VM. Typically providing information about the last state of the programs, applications and system before they were terminated or crashed.</p> <p>Note This memory dump is not used for saving VM state and resuming it later.</p>"},{"location":"compute/memory_dump/#prerequisites","title":"Prerequisites","text":""},{"location":"compute/memory_dump/#hot-plug-feature-gate","title":"Hot plug Feature Gate","text":"<p>The memory dump process mounts a PVC to the virt-launcher in order to get the output in that PVC, hence the hot plug volumes feature gate must be enabled. The feature gates field in the KubeVirt CR must be expanded by adding the <code>HotplugVolumes</code> to it.</p>"},{"location":"compute/memory_dump/#virtctl-support","title":"Virtctl support","text":""},{"location":"compute/memory_dump/#get-memory-dump","title":"Get memory dump","text":"<p>Now lets assume we have a running VM and the name of the VM is 'my-vm'. We can either dump to an existing pvc, or request one to be created.</p>"},{"location":"compute/memory_dump/#existing-pvc","title":"Existing PVC","text":"<p>The size of the PVC must be big enough to hold the memory dump. The calculation is (VMMemorySize + 100Mi) * FileSystemOverhead, Where <code>VMMemorySize</code> is the memory size, 100Mi is reserved space for the memory dump overhead and <code>FileSystemOverhead</code> is the value used to adjust requested PVC size with the filesystem overhead. also the PVC must have a <code>FileSystem</code> volume mode.</p> <p>Example for such PVC:</p> <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: my-pvc\nspec:\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 2Gi\n  storageClassName: rook-ceph-block\n  volumeMode: Filesystem\n</code></pre> <p>We can get a memory dump of the VM to the PVC by using the 'memory-dump get' command available with virtctl</p> <pre><code>$ virtctl memory-dump get my-vm --claim-name=my-pvc\n</code></pre>"},{"location":"compute/memory_dump/#on-demand-pvc","title":"On demand PVC","text":"<p>For on demand PVC, we need to add <code>--create-claim</code> flag to the virtctl request:</p> <pre><code>$ virtctl memory-dump get my-vm --claim-name=new-pvc --create-claim\n</code></pre> <p>A PVC with size big enough for the dump will be created. We can also request specific storage class and access mode with appropriate flags.</p>"},{"location":"compute/memory_dump/#download-memory-dump","title":"Download memory dump","text":"<p>By adding the <code>--output</code> flag, the memory will be dumped to the PVC and then downloaded to the given output path.</p> <pre><code>$ virtctl memory-dump get myvm --claim-name=memoryvolume --create-claim --output=memoryDump.dump.gz\n</code></pre> <p>For downloading the last memory dump from the PVC associated with the VM, without triggering another memory dump, use the memory dump download command.</p> <pre><code>$ virtctl memory-dump download myvm --output=memoryDump.dump.gz\n</code></pre> <p>For downloading a memory dump from a PVC already disassociated from the VM you can use the virtctl vmexport command</p>"},{"location":"compute/memory_dump/#monitoring-the-memory-dump","title":"Monitoring the memory dump","text":"<p>Information regarding the memory dump process will be available on the VM's status section <pre><code>memoryDumpRequest:\n  claimName: memory-dump\n  phase: Completed\n  startTimestamp: \"2022-03-29T11:00:04Z\"\n  endTimestamp: \"2022-03-29T11:00:09Z\"\n  fileName: my-vm-my-pvc-20220329-110004\n</code></pre></p> <p>During the process the volumeStatus on the VMI will be updated with the process information such as the attachment pod information and messages, if all goes well once the process is completed, the PVC is unmounted from the virt-launcher pod and the volumeStatus is deleted. A memory dump annotation will be added to the PVC with the memory dump file name.</p>"},{"location":"compute/memory_dump/#retriggering-the-memory-dump","title":"Retriggering the memory dump","text":"<p>Getting a new memory dump to the same PVC is possible without the need to use any flag: <pre><code>$ virtctl memory-dump get my-vm\n</code></pre></p> <p>Note Each memory-dump command will delete the previous dump in that PVC.</p> <p>In order to get a memory dump to a different PVC you need to 'remove' the current memory-dump PVC and then do a new get with the new PVC name.</p>"},{"location":"compute/memory_dump/#remove-memory-dump","title":"Remove memory dump","text":"<p>As mentioned in order to remove the associated memory dump PVC you need to run a 'memory-dump remove' command. This will allow you to replace the current PVC and get the memory dump to a new one.</p> <pre><code>$ virtctl memory-dump remove my-vm\n</code></pre>"},{"location":"compute/memory_dump/#handle-the-memory-dump","title":"Handle the memory dump","text":"<p>Once the memory dump process is completed the PVC will hold the output. You can manage the dump in one of the following ways: - Download the memory dump - Create a pod with troubleshooting tools that will mount the PVC and inspect it within the pod. - Include the memory dump in the VM Snapshot (will include both the memory dump and the disks) to save a snapshot of the VM in that point of time and inspect it when needed. (The VM Snapshot can be exported and downloaded).</p> <p>The output of the memory dump can be inspected with memory analysis tools for example Volatility3</p>"},{"location":"compute/memory_hotplug/","title":"Memory Hotplug","text":"<p>Memory hotplug was introduced in KubeVirt version 1.1, enabling the dynamic resizing of the amount of memory available to a running VM.</p>"},{"location":"compute/memory_hotplug/#limitations","title":"Limitations","text":"<ul> <li>Memory hotplug is currently only supported on the x86_64 architecture.</li> <li>Current hotplug implementation involves live-migration of the VM workload.</li> </ul>"},{"location":"compute/memory_hotplug/#configuration","title":"Configuration","text":""},{"location":"compute/memory_hotplug/#enable-feature-gate","title":"Enable feature-gate","text":"<p>To use memory hotplug we need to add the <code>VMLiveUpdateFeatures</code> feature gate in the KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"compute/memory_hotplug/#configure-the-workload-update-strategy","title":"Configure the Workload Update Strategy","text":"<p>Configure <code>LiveMigrate</code> as <code>workloadUpdateStrategy</code> in the KubeVirt CR, since the current implementation of the hotplug process requires the VM to live-migrate.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre>"},{"location":"compute/memory_hotplug/#configure-the-vm-rollout-strategy","title":"Configure the VM rollout strategy","text":"<p>Finally, set the VM rollout strategy to <code>LiveUpdate</code>, so that the changes made to the VM object propagate to the VMI without a restart. This is also done in the KubeVirt CR configuration:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre> <p>NOTE: If memory hotplug is enabled/disabled on an already running VM, a reboot is necessary for the changes to take effect.</p> <p>More information can be found on the VM Rollout Strategies page.</p>"},{"location":"compute/memory_hotplug/#optional-set-a-cluster-wide-maximum-amount-of-memory","title":"[OPTIONAL] Set a cluster-wide maximum amount of memory","text":"<p>You can set the maximum amount of memory for the guest using a cluster level setting in the KubeVirt CR.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    liveUpdateConfiguration:\n      maxGuest: 8Gi\n</code></pre> <p>The VM-level configuration will take precedence over the cluster-wide one.</p>"},{"location":"compute/memory_hotplug/#memory-hotplug-in-action","title":"Memory Hotplug in Action","text":"<p>First we enable the <code>VMLiveUpdateFeatures</code> feature gate, set the rollout strategy to <code>LiveUpdate</code> and set <code>LiveMigrate</code> as <code>workloadUpdateStrategy</code> in the KubeVirt CR.</p> <pre><code>$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates\", \"value\": [\"VMLiveUpdateFeatures\"]}]' --type='json'\n$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/vmRolloutStrategy\", \"value\": \"LiveUpdate\"}]' --type='json'\n$ kubectl --namespace kubevirt patch kv kubevirt -p='[{\"op\": \"add\", \"path\": \"/spec/workloadUpdateStrategy/workloadUpdateMethods\", \"value\": [\"LiveMigrate\"]}]' --type='json'\n</code></pre> <p>Now we create a VM with memory hotplug enabled.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-cirros\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        memory:\n          maxGuest: 2Gi\n          guest: 128Mi\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/alpine-container-disk-demo:devel\n        name: containerdisk\n</code></pre> <p>The Virtual Machine will automatically start and once booted it will report the currently available memory to the guest in the <code>status.memory</code> field inside the VMI.</p> <p><pre><code>$ kubectl get vmi vm-cirros -o json | jq .status.memory\n</code></pre> <pre><code>{\n  \"guestAtBoot\": \"128Mi\",\n  \"guestCurrent\": \"128Mi\",\n  \"guestRequested\": \"128Mi\"\n}\n</code></pre></p> <p>Since the Virtual Machine is now running we can patch the VM object to double the available guest memory so that we'll go from 128Mi to 256Mi.</p> <pre><code>$ kubectl patch vm vm-cirros -p='[{\"op\": \"replace\", \"path\": \"/spec/template/spec/domain/memory/guest\", \"value\": \"256Mi\"}]' --type='json'\n</code></pre> <p>After the hotplug request is processed and the Virtual Machine is live migrated, the new amount of memory should be available to the guest and visible in the VMI object.</p> <p><pre><code>$ kubectl get vmi vm-cirros -o json | jq .status.memory\n</code></pre> <pre><code>{\n  \"guestAtBoot\": \"128Mi\",\n  \"guestCurrent\": \"256Mi\",\n  \"guestRequested\": \"256Mi\"\n}\n</code></pre></p>"},{"location":"compute/node_assignment/","title":"Node assignment","text":"<p>You can constrain the VM to only run on specific nodes or to prefer running on specific nodes:</p> <ul> <li>nodeSelector</li> <li>Affinity and anti-affinity</li> <li>Taints and Tolerations</li> </ul>"},{"location":"compute/node_assignment/#nodeselector","title":"nodeSelector","text":"<p>Setting <code>spec.nodeSelector</code> requirements, constrains the scheduler to only schedule VMs on nodes, which contain the specified labels. In the following example the vmi contains the labels <code>cpu: slow</code> and <code>storage: fast</code>:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>Thus the scheduler will only schedule the vmi to nodes which contain these labels in their metadata. It works exactly like the Pods <code>nodeSelector</code>. See the Pod nodeSelector Documentation for more examples.</p>"},{"location":"compute/node_assignment/#affinity-and-anti-affinity","title":"Affinity and anti-affinity","text":"<p>The <code>spec.affinity</code> field allows specifying hard- and soft-affinity for VMs. It is possible to write matching rules against workloads (VMs and Pods) and Nodes. Since VMs are a workload type based on Pods, Pod-affinity affects VMs as well.</p> <p>An example for <code>podAffinity</code> and <code>podAntiAffinity</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  affinity:\n    podAffinity:\n      requiredDuringSchedulingIgnoredDuringExecution:\n      - labelSelector:\n          matchExpressions:\n          - key: security\n            operator: In\n            values:\n            - S1\n        topologyKey: failure-domain.beta.kubernetes.io/zone\n    podAntiAffinity:\n      preferredDuringSchedulingIgnoredDuringExecution:\n      - weight: 100\n        podAffinityTerm:\n          labelSelector:\n            matchExpressions:\n            - key: security\n              operator: In\n              values:\n              - S2\n          topologyKey: kubernetes.io/hostname\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>Affinity and anti-affinity works exactly like the Pods <code>affinity</code>. This includes <code>podAffinity</code>, <code>podAntiAffinity</code>, <code>nodeAffinity</code> and <code>nodeAntiAffinity</code>. See the Pod affinity and anti-affinity Documentation for more examples and details.</p>"},{"location":"compute/node_assignment/#taints-and-tolerations","title":"Taints and Tolerations","text":"<p>Affinity as described above, is a property of VMs that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite - they allow a node to repel a set of VMs.</p> <p>Taints and tolerations work together to ensure that VMs are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any VMs that do not tolerate the taints. Tolerations are applied to VMs, and allow (but do not require) the VMs to schedule onto nodes with matching taints.</p> <p>You add a taint to a node using kubectl taint. For example,</p> <pre><code>kubectl taint nodes node1 key=value:NoSchedule\n</code></pre> <p>An example for <code>tolerations</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-ephemeral\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  nodeSelector:\n    cpu: slow\n    storage: fast\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  tolerations:\n  - key: \"key\"\n    operator: \"Equal\"\n    value: \"value\"\n    effect: \"NoSchedule\"\n</code></pre>"},{"location":"compute/node_assignment/#node-balancing-with-descheduler","title":"Node balancing with Descheduler","text":"<p>In some cases we might need to rebalance the cluster on current scheduling policy and load conditions. Descheduler can find pods, which violates e.g. scheduling decisions and evict them based on descheduler policies. Kubevirt VMs are handled as pods with local storage, so by default, descheduler will not evict them. But it can be easily overridden by adding special annotation to the VMI template in the VM:</p> <pre><code>spec:\n  template:\n    metadata:\n      annotations:\n        descheduler.alpha.kubernetes.io/evict: true\n</code></pre> <p>This annotation will cause, that the descheduler will be able to evict the VM's pod which can then be scheduled by scheduler on different nodes. A VirtualMachine will never restart or re-create a VirtualMachineInstance until the current instance of the VirtualMachineInstance is deleted from the cluster.</p>"},{"location":"compute/node_assignment/#live-update","title":"Live update","text":"<p>When the VM rollout strategy is set to <code>LiveUpdate</code>, changes to a VM's node selector or affinities will dynamically propagate to the VMI (unless the <code>RestartRequired</code> condition is set). Changes to tolerations will not dynamically propagate, and will trigger a <code>RestartRequired</code> condition if changed on a running VM.</p> <p>Modifications of the node selector / affinities will only take effect on next migration, the change alone will not trigger one.</p>"},{"location":"compute/node_overcommit/","title":"Node overcommit","text":"<p>KubeVirt does not yet support classical Memory Overcommit Management or Memory Ballooning. In other words VirtualMachineInstances can't give back memory they have allocated. However, a few other things can be tweaked to reduce the memory footprint and overcommit the per-VMI memory overhead.</p>"},{"location":"compute/node_overcommit/#remove-the-graphical-devices","title":"Remove the Graphical Devices","text":"<p>First the safest option to reduce the memory footprint, is removing the graphical device from the VMI by setting <code>spec.domain.devices.autottachGraphicsDevice</code> to <code>false</code>. See the video and graphics device documentation for further details and examples.</p> <p>This will save a constant amount of <code>16MB</code> per VirtualMachineInstance but also disable VNC access.</p>"},{"location":"compute/node_overcommit/#overcommit-the-guest-overhead","title":"Overcommit the Guest Overhead","text":"<p>Before you continue, make sure you make yourself comfortable with the Out of Resource Management of Kubernetes.</p> <p>Every VirtualMachineInstance requests slightly more memory from Kubernetes than what was requested by the user for the Operating System. The additional memory is used for the per-VMI overhead consisting of our infrastructure which is wrapping the actual VirtualMachineInstance process.</p> <p>In order to increase the VMI density on the node, it is possible to not request the additional overhead by setting <code>spec.domain.resources.overcommitGuestOverhead</code> to <code>true</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      overcommitGuestOverhead: true\n      requests:\n        memory: 1024M\n[...]\n</code></pre> <p>This will work fine for as long as most of the VirtualMachineInstances will not request the whole memory. That is especially the case if you have short-lived VMIs. But if you have long-lived VirtualMachineInstances or do extremely memory intensive tasks inside the VirtualMachineInstance, your VMIs will use all memory they are granted sooner or later.</p>"},{"location":"compute/node_overcommit/#overcommit-guest-memory","title":"Overcommit Guest Memory","text":"<p>The third option is real memory overcommit on the VMI. In this scenario the VMI is explicitly told that it has more memory available than what is requested from the cluster by setting <code>spec.domain.memory.guest</code> to a value higher than <code>spec.domain.resources.requests.memory</code>.</p> <p>The following definition requests <code>1024MB</code> from the cluster but tells the VMI that it has <code>2048MB</code> of memory available:</p> <pre><code>apiVersion: kubevirt.io/v1alpha3\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      overcommitGuestOverhead: true\n      requests:\n        memory: 1024M\n    memory:\n      guest: 2048M\n[...]\n</code></pre> <p>For as long as there is enough free memory available on the node, the VMI can happily consume up to <code>2048MB</code>. This VMI will get the <code>Burstable</code> resource class assigned by Kubernetes (See QoS classes in Kubernetes for more details). The same eviction rules like for Pods apply to the VMI in case the node gets under memory pressure.</p> <p>Implicit memory overcommit is disabled by default. This means that when memory request is not specified, it is set to match <code>spec.domain.memory.guest</code>. However, it can be enabled using <code>spec.configuration.developerConfiguration.memoryOvercommit</code> in the <code>kubevirt</code> CR. For example, by setting <code>memoryOvercommit: \"150\"</code> we define that when memory request is not explicitly set, it will be implicitly set to achieve memory overcommit of 150%. For instance, when <code>spec.domain.memory.guest: 3072M</code>, memory request is set to 2048M, if omitted. Note that the actual memory request depends on additional configuration options like OvercommitGuestOverhead.</p>"},{"location":"compute/node_overcommit/#configuring-the-memory-pressure-behavior-of-nodes","title":"Configuring the memory pressure behavior of nodes","text":"<p>If the node gets under memory pressure, depending on the <code>kubelet</code> configuration the virtual machines may get killed by the OOM handler or by the <code>kubelet</code> itself. It is possible to tweak that behaviour based on the requirements of your VirtualMachineInstances by:</p> <ul> <li>Configuring Soft Eviction     Thresholds</li> <li>Configuring Hard Eviction     Thresholds</li> <li>Requesting the right QoS class for VirtualMachineInstances</li> <li>Setting <code>--system-reserved</code> and <code>--kubelet-reserved</code></li> <li>Enabling KSM</li> <li>Enabling swap</li> </ul>"},{"location":"compute/node_overcommit/#configuring-soft-eviction-thresholds","title":"Configuring Soft Eviction Thresholds","text":"<p>Note: Soft Eviction will effectively shutdown VirtualMachineInstances. They are not paused, hibernated or migrated. Further, Soft Eviction is disabled by default.</p> <p>If configured, VirtualMachineInstances get evicted once the available memory falls below the threshold specified via <code>--eviction-soft</code> and the VirtualmachineInstance is given the chance to perform a shutdown of the VMI within a timespan specified via <code>--eviction-max-pod-grace-period</code>. The flag <code>--eviction-soft-grace-period</code> specifies for how long a soft eviction condition must be held before soft evictions are triggered.</p> <p>If set properly according to the demands of the VMIs, overcommitting should only lead to soft evictions in rare cases for some VMIs. They may even get re-scheduled to the same node with less initial memory demand. For some workload types, this can be perfectly fine and lead to better overall memory-utilization.</p>"},{"location":"compute/node_overcommit/#configuring-hard-eviction-thresholds","title":"Configuring Hard Eviction Thresholds","text":"<p>Note: If unspecified, the kubelet will do hard evictions for Pods once <code>memory.available</code> falls below <code>100Mi</code>.</p> <p>Limits set via <code>--eviction-hard</code> will lead to immediate eviction of VirtualMachineInstances or Pods. This stops VMIs without a grace period and is comparable with power-loss on a real computer.</p> <p>If the hard limit is hit, VMIs may from time to time simply be killed. They may be re-scheduled to the same node immediately again, since they start with less memory consumption again. This can be a simple option, if the memory threshold is only very seldom hit and the work performed by the VMIs is reproducible or it can be resumed from some checkpoints.</p>"},{"location":"compute/node_overcommit/#requesting-the-right-qos-class-for-virtualmachineinstances","title":"Requesting the right QoS Class for VirtualMachineInstances","text":"<p>Different QoS classes get assigned to Pods and VirtualMachineInstances based on the <code>requests.memory</code> and <code>limits.memory</code>. KubeVirt right now supports the QoS classes <code>Burstable</code> and <code>Guaranteed</code>. <code>Burstable</code> VMIs are evicted before <code>Guaranteed</code> VMIs.</p> <p>This allows creating two classes of VMIs:</p> <ul> <li>One type can have equal <code>requests.memory</code> and <code>limits.memory</code> set     and therefore gets the <code>Guaranteed</code> class assigned. This one will     not get evicted and should never run into memory issues, but is more     demanding.</li> <li>One type can have no <code>limits.memory</code> or a <code>limits.memory</code> which is     greater than <code>requests.memory</code> and therefore gets the <code>Burstable</code>     class assigned. These VMIs will be evicted first.</li> </ul>"},{"location":"compute/node_overcommit/#setting-system-reserved-and-kubelet-reserved","title":"Setting <code>--system-reserved</code> and <code>--kubelet-reserved</code>","text":"<p>It may be important to reserve some memory for other daemons (not DaemonSets) which are running on the same node (ssh, dhcp servers, etc). The reservation can be done with the <code>--system reserved</code> switch. Further for the Kubelet and Docker a special flag called <code>--kubelet-reserved</code> exists.</p>"},{"location":"compute/node_overcommit/#enabling-ksm","title":"Enabling KSM","text":"<p>The KSM (Kernel same-page merging) daemon can be started on the node. Depending on its tuning parameters it can more or less aggressively try to merge identical pages between applications and VirtualMachineInstances. The more aggressive it is configured the more CPU it will use itself, so the memory overcommit advantages comes with a slight CPU performance hit.</p> <p>Config file tuning allows changes to scanning frequency (how often will KSM activate) and aggressiveness (how many pages per second will it scan).</p>"},{"location":"compute/node_overcommit/#enabling-swap","title":"Enabling Swap","text":"<p>Note: This will definitely make sure that your VirtualMachines can't crash or get evicted from the node but it comes with the cost of pretty unpredictable performance once the node runs out of memory and the kubelet may not detect that it should evict Pods to increase the performance again.</p> <p>Enabling swap is in general not recommended on Kubernetes right now. However, it can be useful in combination with KSM, since KSM merges identical pages over time. Swap allows the VMIs to successfully allocate memory which will then effectively never be used because of the later de-duplication done by KSM.</p>"},{"location":"compute/node_overcommit/#node-cpu-allocation-ratio","title":"Node CPU allocation ratio","text":"<p>KubeVirt runs Virtual Machines in a Kubernetes Pod. This pod requests a certain amount of CPU time from the host. On the other hand, the Virtual Machine is being created with a certain amount of vCPUs. The number of vCPUs may not necessarily correlate to the number of requested CPUs by the POD.  Depending on the QOS of the POD, vCPUs can be scheduled on a variable amount of physical CPUs; this depends on the available CPU resources on a node. When there are fewer available CPUs on the node as the requested vCPU, vCPU will be over committed.</p> <p>By default, each pod requests 100mil of CPU time. The CPU requested on the pod sets the cgroups cpu.shares which serves as a priority for the scheduler to provide CPU time for vCPUs in this POD. As the number of vCPUs increases, this will reduce the amount of CPU time each vCPU may get when competing with other processes on the node or other Virtual Machine Instances with a lower amount of vCPUs.</p> <p>The <code>cpuAllocationRatio</code> comes to normalize the amount of CPU time the POD will request based on the number of vCPUs. For example, POD CPU request = number of vCPUs * 1/cpuAllocationRatio When cpuAllocationRatio is set to 1, a full amount of vCPUs will be requested for the POD.</p> <p>Note: In Kubernetes, one full core is 1000 of CPU time More Information</p> <p>Administrators can change this ratio by updating the KubeVirt CR</p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      cpuAllocationRatio: 10\n</code></pre>"},{"location":"compute/numa/","title":"NUMA","text":"<p>FEATURE STATE: KubeVirt v0.43</p> <p>NUMA support in KubeVirt is at this stage limited to a small set of special use-cases and will improve over time together with improvements made to Kubernetes.</p> <p>In general, the goal is to map the host NUMA topology as efficiently as possible to the Virtual Machine topology to improve the performance.</p> <p>The following NUMA mapping strategies can be used:</p> <ul> <li>GuestMappingPassthrough</li> </ul>"},{"location":"compute/numa/#preconditions","title":"Preconditions","text":"<p>In order to use current NUMA support, the following preconditions must be met:</p> <ul> <li>Dedicated CPU Resources must be configured.</li> <li>Hugepages need to be allocatable on target   nodes.</li> <li>The <code>NUMA</code> feature gate   must be enabled.</li> </ul>"},{"location":"compute/numa/#guestmappingpassthrough","title":"GuestMappingPassthrough","text":"<p>GuestMappingPassthrough will pass through the node numa topology to the guest. The topology is based on the dedicated CPUs which the VMI got assigned from the kubelet via the CPU Manager. It can be requested by setting <code>spec.domain.cpu.numa.guestMappingPassthrough</code> on the VMI.</p> <p>Since KubeVirt does not know upfront which exclusive CPUs the VMI will get from the kubelet, there are some limitations:</p> <ul> <li>Guests may see different NUMA topologies when being rescheduled.</li> <li>The resulting NUMA topology may be asymmetrical.</li> <li>The VMI may fail to start on the node if not enough hugepages are available on   the assigned NUMA nodes.</li> </ul> <p>While this NUMA modelling strategy has its limitations, aligning the guest's NUMA architecture with the node's can be critical for high-performance applications.</p> <p>An example VMI may look like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: numavm\nspec:\n  domain:\n    cpu:\n      cores: 4\n      dedicatedCpuPlacement: true\n      numa:\n        guestMappingPassthrough: { }\n    devices:\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n        - disk:\n            bus: virtio\n          name: cloudinitdisk\n    resources:\n      requests:\n        memory: 64Mi\n    memory:\n      hugepages:\n        pageSize: 2Mi\n  volumes:\n    - containerDisk:\n        image: quay.io/kubevirt/cirros-container-disk-demo\n      name: containerdisk\n    - cloudInitNoCloud:\n        userData: |\n          #!/bin/sh\n          echo 'printed from cloud-init userdata'\n      name: cloudinitdisk\n</code></pre>"},{"location":"compute/numa/#running-real-time-workloads","title":"Running real-time workloads","text":""},{"location":"compute/numa/#overview","title":"Overview","text":"<p>It is possible to deploy Virtual Machines that run a real-time kernel and make use of libvirtd's guest cpu and memory optimizations that improve the overall latency. These changes leverage mostly on already available settings in KubeVirt, as we will see shortly, but the VMI manifest now exposes two new settings that instruct KubeVirt to configure the generated libvirt XML with the recommended tuning settings for running real-time workloads.</p> <p>To make use of the optimized settings, two new settings have been added to the VMI schema:</p> <ul> <li> <p><code>spec.domain.cpu.realtime</code>: When defined, it instructs KubeVirt to configure the linux scheduler for the VCPUS to run processes in FIFO scheduling policy (SCHED_FIFO) with priority 1. This setting guarantees that all processes running in the host will be executed with real-time priority.</p> </li> <li> <p><code>spec.domain.cpu.realtime.mask</code>: It defines which VCPUs assigned to the VM are used for real-time. If not defined, libvirt will define all VCPUS assigned to run processes in FIFO scheduling and in the highest priority (1).</p> </li> </ul>"},{"location":"compute/numa/#preconditions_1","title":"Preconditions","text":"<p>A prerequisite to running real-time workloads include locking resources in the cluster to allow the real-time VM exclusive usage. This translates into nodes, or node, that have been configured with a dedicated set of CPUs and also provides support for NUMA with a free number of hugepages of 2Mi or 1Gi size (depending on the configuration in the VMI). Additionally, the node must be configured to allow the scheduler to run processes with real-time policy.</p>"},{"location":"compute/numa/#nodes-capable-of-running-real-time-workloads","title":"Nodes capable of running real-time workloads","text":"<p>When the KubeVirt pods are deployed in a node, it will check if it is capable of running processes in real-time scheduling policy and label the node as real-time capable (kubevirt.io/realtime). If, on the other hand, the node is not able to deliver such capability, the label is not applied. To check which nodes are able to host real-time VM workloads run this command:</p> <pre><code>$&gt;kubectl get nodes -l kubevirt.io/realtime\nNAME         STATUS   ROLES              AGE    VERSION\nworker-0-0   Ready    worker             12d    v1.20.0+df9c838\n</code></pre> <p>Internally, the KubeVirt pod running in each node checks if the kernel setting <code>kernel.sched_rt_runtime_us</code> equals to -1, which grants processes to run in real-time scheduling policy for an unlimited amount of time.</p>"},{"location":"compute/numa/#configuring-a-vm-manifest","title":"Configuring a VM Manifest","text":"<p>Here is an example of a VM manifest that runs a custom fedora container disk configured to run with a real-time kernel. The settings have been configured for optimal efficiency.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: fedora-realtime\n  name: fedora-realtime\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: fedora-realtime\n    spec:\n      domain:\n        devices:\n          autoattachSerialConsole: true\n          autoattachMemBalloon: false\n          autoattachGraphicsDevice: false\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk \n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 1Gi\n            cpu: 2\n          limits:\n            memory: 1Gi\n            cpu: 2\n        cpu:\n          model: host-passthrough\n          dedicatedCpuPlacement: true\n          isolateEmulatorThread: true\n          ioThreadsPolicy: auto\n          features:\n            - name: tsc-deadline\n              policy: require\n          numa:\n            guestMappingPassthrough: {}\n          realtime: {}\n        memory:\n          hugepages:\n            pageSize: 1Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-realtime-container-disk:v20211008-22109a3\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n            bootcmd:\n              - tuned-adm profile realtime\n        name: cloudinitdisk\n</code></pre> <p>Breaking down the tuned sections, we have the following configuration:</p> <p>Devices:  - Disable the guest's memory balloon capability - Avoid attaching a graphics device, to reduce the number of interrupts to the kernel.</p> <pre><code>spec:\n  domain:\n    devices:\n      autoattachSerialConsole: true\n      autoattachMemBalloon: false\n      autoattachGraphicsDevice: false\n</code></pre> <p>CPU: - model: <code>host-passthrough</code> to allow the guest to see host CPU without masking any capability. - dedicated CPU Placement: The VM needs to have dedicated CPUs assigned to it. The Kubernetes CPU Manager takes care of this aspect. - isolatedEmulatorThread: to request an additional CPU to run the emulator on it, thus avoid using CPU cycles from the workload CPUs. - ioThreadsPolicy: Set to auto to let the dedicated IO thread to run in the same CPU as the emulator thread. - NUMA: defining <code>guestMappingPassthrough</code> enables NUMA support for this VM. - realtime: instructs the virt-handler to configure this VM for real-time workloads, such as configuring the VCPUS to use FIFO scheduler policy and set priority to 1. cpu: <pre><code>cpu:\n  model: host-passthrough\n  dedicatedCpuPlacement: true\n  isolateEmulatorThread: true\n  ioThreadsPolicy: auto\n  features:\n    - name: tsc-deadline\n      policy: require\n  numa:\n    guestMappingPassthrough: {}\n  realtime: {}\n</code></pre></p> <p>Memory - pageSize: allocate the pod's memory in hugepages of the given size, in this case of 1Gi. <pre><code>memory:\n  hugepages:\n    pageSize: 1Gi\n</code></pre></p>"},{"location":"compute/numa/#how-to-dedicate-vcpus-for-real-time-only","title":"How to dedicate VCPUS for real-time only","text":"<p>It is possible to pass a regular expression of the VCPUs to isolate to use real-time scheduling policy, by using the <code>realtime.mask</code> setting.</p> <pre><code>cpu:\n  numa:\n    guestMappingPassthrough: {}\n  realtime:\n    mask: \"0\"\n</code></pre> <p>When applied this configuration, KubeVirt will only set the first VCPU for real-time scheduler policy, leaving the remaining VCPUS to use the default scheduler policy. Other examples of valid masks are: - <code>0-3</code>: Use cores 0 to 3 for real-time scheduling, assuming that the VM has requested at least 3 cores. - <code>0-3,^1</code>: Use cores 0, 2 and 3 for real-time scheduling only, assuming that the VM has requested at least 3 cores.</p>"},{"location":"compute/numa/#additional-reading","title":"Additional Reading","text":"<p>Kubernetes provides additional NUMA components that may be relevant to your use-case but typically are not enabled by default. Please consult the Kubernetes documentation for details on configuration of these components.</p>"},{"location":"compute/numa/#topology-manager","title":"Topology Manager","text":"<p>Topology Manager provides optimizations related to CPU isolation, memory and device locality. It is useful, for example, where an SR-IOV network adaptor VF allocation needs to be aligned with a NUMA node.</p> <p>https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/</p>"},{"location":"compute/numa/#memory-manager","title":"Memory Manager","text":"<p>Memory Manager is analogous to CPU Manager. It is useful, for example, where you want to align hugepage allocations with a NUMA node. It works in conjunction with Topology Manager.</p> <p>The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod. The Memory Manager feeds the central manager (Topology Manager) with these affinity hints. Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.</p> <p>https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/</p>"},{"location":"compute/persistent_tpm_and_uefi_state/","title":"Persistent TPM and UEFI state","text":"<p>FEATURE STATE: KubeVirt v1.0.0</p> <p>For both TPM and UEFI, libvirt supports persisting data created by a virtual machine as files on the virtualization host. In KubeVirt, the virtualization host is the virt-launcher pod, which is ephemeral (created on VM start and destroyed on VM stop). As of v1.0.0, KubeVirt supports using a PVC to persist those files. KubeVirt usually refers to that storage area as \"backend storage\".</p>"},{"location":"compute/persistent_tpm_and_uefi_state/#backend-storage","title":"Backend storage","text":"<p>KubeVirt automatically creates backend storage PVCs for VMs that need it. However, the admin must first enable the <code>VMPersistentState</code> feature gate, and tell KubeVirt which storage class to use by setting the <code>vmStateStorageClass</code> configuration parameter in the KubeVirt Custom Resource (CR). The storage class must support read-write-many (RWX) in filesystem mode (FS). Here's an example of KubeVirt CR that sets both: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmStateStorageClass: \"nfs-csi\"\n    developerConfiguration:\n      featureGates:\n      - VMPersistentState\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#limitations","title":"Limitations","text":"<ul> <li>As mentioned above, the backend storage PVC can only be created using a storage class that supports RWX FS. There is ongoing work to support block storage in future versions of KubeVirt.</li> <li>Backend storage is currently incompatible with VM snapshot. It is planned to add snapshot support in the future.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#tpm-with-persistent-state","title":"TPM with persistent state","text":"<p>Since KubeVirt v0.53.0, a TPM device can be added to a VM (with just <code>tpm: {}</code>). However, the data stored in it does not persist across reboots. Support for persistence was added in v1.0.0 using a simple <code>persistent</code> boolean parameter that default to false, to preserve previous behavior. Of course, backend storage must first be configured before adding a persistent TPM to a VM. See above. Here's a portion of a VM definition that includes a persistent TPM: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm\nspec:\n  template:\n    spec:\n      domain:\n        devices:\n          tpm:\n            persistent: true\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#uses","title":"Uses","text":"<ul> <li>The Microsoft Windows 11 installer requires the presence of a TPM device, even though it doesn't use this. Persistence is not required in this case however.</li> <li>Some disk encryption software have optional/mandatory TPM support. For example, Bitlocker requires a persistent TPM device.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#notes","title":"Notes","text":"<ul> <li>The TPM device exposed to the virtual machine is fully emulated (vTPM). The worker nodes do not need to have a TPM device.</li> <li>When TPM persistence is enabled, the <code>tpm-crb</code> model is used (instead of <code>tpm-tis</code> for non-persistent vTPMs)</li> <li>A virtual TPM does not provide the same security guarantees as a physical one.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#efi-with-persistent-vars","title":"EFI with persistent VARS","text":"<p>EFI support is handled by libvirt using OVMF. OVMF data usually consists of 2 files, CODE and VARS. VARS is where persistent data from the guest can be stored. When EFI persistence is enabled on a VM, the VARS file will be persisted inside the backend storage. Of course, backend storage must first be configured before enabling EFI persistence on a VM. See above. Here's a portion of a VM definition that includes a persistent EFI: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm\nspec:\n  template:\n    spec:\n      domain:\n        firmware:\n          bootloader:\n            efi:\n              persistent: true\n</code></pre></p>"},{"location":"compute/persistent_tpm_and_uefi_state/#uses_1","title":"Uses","text":"<ul> <li>Preserving user-created Secure Boot certificates.</li> <li>Preserving EFI firmware settings, like language or display resolution.</li> </ul>"},{"location":"compute/persistent_tpm_and_uefi_state/#notes_1","title":"Notes","text":"<ul> <li>The boot entries/order can, and most likely will, get overriden by libvirt. This is to satisfy the VM specfications. Do not expect manual boot setting changes to persist.</li> </ul>"},{"location":"compute/resources_requests_and_limits/","title":"Resources requests and limits","text":"<p>In this document, we are talking about the resources values set on the virt-launcher compute container, referred to as \"the container\" below for simplicity.</p>"},{"location":"compute/resources_requests_and_limits/#cpu","title":"CPU","text":"<p>Note: dedicated CPUs (and isolated emulator thread) are ignored here as they have a dedicated page.</p>"},{"location":"compute/resources_requests_and_limits/#cpu-requests-on-the-container","title":"CPU requests on the container","text":"<ul> <li>By default, the container requests (1/cpuAllocationRatio) CPU per vCPU. The number of vCPUs is sockets * cores * threads, defaults to 1.</li> <li>cpuAllocationRatio defaults to 10 but can be changed in the CR.</li> <li>If a CPU limit is manually set on the VM(I) and no CPU request is, the CPU requests on the container will match the CPU limits</li> <li>Manually setting CPU requests on the VM(I) will override all of the above and be the CPU requests for the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#cpu-limits-on-the-container","title":"CPU limits on the container","text":"<ul> <li>By default, no CPU limit is set on the container</li> <li>If auto CPU limits is enabled (see next section), then the container will have a CPU limit of 1 per vCPU</li> <li>Manually setting CPU limits on the VM(I) will override all of the above and be the CPU limits for the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#auto-cpu-limits","title":"Auto CPU limits","text":"<p>KubeVirt provides two ways to automatically set CPU limits on VM(I)s:</p> <ul> <li>Enable the <code>AutoResourceLimitsGate</code> feature gate.</li> <li>Add the namespaceLabelSelector in the KubeVirt CR.</li> </ul> <p>In both cases, the VM(I) created will have a CPU limit of 1 per vCPU.</p>"},{"location":"compute/resources_requests_and_limits/#autoresourcelimitsgate-feature-gate","title":"AutoResourceLimitsGate feature gate","text":"<p>By enabling this feature gate, cpu limits will be added to the vmi if all the following conditions are true:</p> <ul> <li>The namespace where the VMI will be created has a ResourceQuota containing cpu limits.</li> <li>The VMI has no manually set cpu limits.</li> <li>The VMI is not requesting dedicated CPU.</li> </ul>"},{"location":"compute/resources_requests_and_limits/#autocpulimitnamespacelabelselector-configuration","title":"autoCPULimitNamespaceLabelSelector configuration","text":"<p>Cluster admins can define a label selector in the KubeVirt CR. Once that label selector is defined, if the creation namespace matches the selector, all VM(I)s created in it will have a CPU limits set.</p> <p>Example:</p> <ul> <li> <p>CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    autoCPULimitNamespaceLabelSelector:\n      matchLabels:\n        autoCpuLimit: \"true\"\n</code></pre></p> </li> <li> <p>Namespace: <pre><code>apiVersion: v1\nkind: Namespace\nmetadata:\n  labels:\n    autoCpuLimit: \"true\"\n    kubernetes.io/metadata.name: default\n  name: default\n</code></pre></p> </li> </ul>"},{"location":"compute/resources_requests_and_limits/#memory","title":"Memory","text":""},{"location":"compute/resources_requests_and_limits/#memory-requests-on-the-container","title":"Memory requests on the container","text":"<ul> <li>VM(I)s must specify a desired amount of memory, in either spec.domain.memory.guest or spec.domain.resources.requests.memory (ignoring hugepages, see the dedicated page). If both are set, the memory requests take precedence. A calculated amount of overhead will be added to it, forming the memory request value for the container.</li> </ul>"},{"location":"compute/resources_requests_and_limits/#memory-limits-on-the-container","title":"Memory limits on the container","text":"<ul> <li>By default, no memory limit is set on the container</li> <li>If auto memory limits is enabled (see next section), then the container will have a limit of 2x the requested memory.</li> <li>Manually setting a memory limit on the VM(I) will set the same value on the container</li> </ul>"},{"location":"compute/resources_requests_and_limits/#warnings","title":"Warnings","text":"<ul> <li>Memory limits have to be more than memory requests + overhead, otherwise the container will have memory requests &gt; limits and be rejected by Kubernetes.</li> <li>Memory usage bursts could lead to VM crashes when memory limits are set</li> </ul>"},{"location":"compute/resources_requests_and_limits/#auto-memory-limits","title":"Auto memory limits","text":"<p>KubeVirt provides a feature gate(<code>AutoResourceLimitsGate</code>) to automatically set memory limits on VM(I)s. By enabling this feature gate, memory limits will be added to the vmi if all the following conditions are true:</p> <ul> <li>The namespace where the VMI will be created has a ResourceQuota containing memory limits.</li> <li>The VMI has no manually set memory limits.</li> <li>The VMI is not requesting dedicated CPU.</li> </ul> <p>If all the previous conditions are true, the memory limits will be set to a value (<code>2x</code>) of the memory requests. This ratio can be adjusted, per namespace, by adding the annotation <code>alpha.kubevirt.io/auto-memory-limits-ratio</code>, with the desired custom value. For example, with <code>alpha.kubevirt.io/auto-memory-limits-ratio: 1.2</code>, the memory limits set will be equal to (<code>1.2x</code>) of the memory requests.</p>"},{"location":"compute/run_strategies/","title":"Run Strategies","text":""},{"location":"compute/run_strategies/#overview","title":"Overview","text":"<p>VirtualMachines have a <code>Running</code> setting that determines whether or not there should be a guest running or not. Because KubeVirt will always immediately restart a VirtualMachineInstance for VirtualMachines with <code>spec.running: true</code>, a simple boolean is not always enough to fully describe desired behavior. For instance, there are cases when a user would like the ability to shut down a guest from inside the virtual machine. With <code>spec.running: true</code>, KubeVirt would immediately restart the VirtualMachineInstance.</p>"},{"location":"compute/run_strategies/#runstrategy","title":"RunStrategy","text":"<p>To allow for greater variation of user states, the <code>RunStrategy</code> field has been introduced. This is mutually exclusive with <code>Running</code> as they have somewhat overlapping conditions. There are currently four RunStrategies defined:</p> <ul> <li> <p>Always: The system is tasked with keeping the VM in a running     state.     This is achieved by respawning a VirtualMachineInstance whenever     the current one terminated in a controlled (e.g. shutdown from     inside the guest) or uncontrolled (e.g. crash) way.     This behavior is equal to <code>spec.running: true</code>.</p> </li> <li> <p>RerunOnFailure: Similar to <code>Always</code>, except that the VM is only     restarted if it terminated in an uncontrolled way (e.g. crash)     and due to an infrastructure reason (i.e. the node crashed,     the KVM related process OOMed).     This allows a user to determine when the VM should be shut down     by initiating the shut down inside the guest.     Note: Guest sided crashes (i.e. BSOD) are not covered by this.     In such cases liveness checks or the use of a watchdog can help.</p> </li> <li> <p>Manual: The system will not automatically turn the VM on or off,     instead the user manually controlls the VM status by issuing     start, stop, and restart commands on the VirtualMachine     subresource endpoints.</p> </li> <li> <p>Halted: The system is asked to ensure that no VM is running.     This is achieved by stopping any VirtualMachineInstance that is     associated ith the VM. If a guest is already running, it will be     stopped.     This behavior is equal to <code>spec.running: false</code>.</p> </li> </ul> <p>Note: <code>RunStrategy</code> and <code>running</code> are mutually exclusive, because they can be contradictory. The API server will reject VirtualMachine resources that define both.</p>"},{"location":"compute/run_strategies/#virtctl","title":"Virtctl","text":"<p>The <code>start</code>, <code>stop</code> and <code>restart</code> methods of virtctl will invoke their respective subresources of VirtualMachines. This can have an effect on the runStrategy of the VirtualMachine as below:</p> RunStrategy start stop restart <p>Always</p> <p><code>-</code></p> <p><code>Halted</code></p> <p><code>Always</code></p> <p>RerunOnFailure</p> <p><code>RerunOnFailure</code></p> <p><code>RerunOnFailure</code></p> <p><code>RerunOnFailure</code></p> <p>Manual</p> <p><code>Manual</code></p> <p><code>Manual</code></p> <p><code>Manual</code></p> <p>Halted</p> <p><code>Always</code></p> <p><code>-</code></p> <p><code>-</code></p> <p>Table entries marked with <code>-</code> don't make sense, so won't have an effect on RunStrategy.</p>"},{"location":"compute/run_strategies/#runstrategy-examples","title":"RunStrategy Examples","text":""},{"location":"compute/run_strategies/#always","title":"Always","text":"<p>An example usage of the Always RunStrategy.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-cirros\n  name: vm-cirros\nspec:\n  runStrategy: Always\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-cirros\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n        name: containerdisk\n</code></pre>"},{"location":"compute/virtual_hardware/","title":"Virtual hardware","text":"<p>Fine-tuning different aspects of the hardware which are not device related (BIOS, mainboard, etc.) is sometimes necessary to allow guest operating systems to properly boot and reboot.</p>"},{"location":"compute/virtual_hardware/#machine-type","title":"Machine Type","text":"<p>QEMU is able to work with two different classes of chipsets for x86_64, so called machine types. The x86_64 chipsets are i440fx (also called pc) and q35. They are versioned based on qemu-system-${ARCH}, following the format <code>pc-${machine_type}-${qemu_version}</code>, e.g.<code>pc-i440fx-2.10</code> and <code>pc-q35-2.10</code>.</p> <p>KubeVirt defaults to QEMU's newest q35 machine type. If a custom machine type is desired, it is configurable through the following structure:</p> <p><pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    machine:\n      # This value indicates QEMU machine type.\n      type: pc-q35-2.10\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> Comparison of the machine types' internals can be found at QEMU wiki.</p>"},{"location":"compute/virtual_hardware/#biosuefi","title":"BIOS/UEFI","text":"<p>All virtual machines use BIOS by default for booting.</p> <p>It is possible to utilize UEFI/OVMF by setting a value via <code>spec.firmware.bootloader</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-alpine-efi\n  name: vmi-alpine-efi\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    features:\n      smm:\n        enabled: true\n    firmware:\n      # this sets the bootloader type\n      bootloader:\n        efi: {}\n</code></pre> <p>Enabling EFI automatically enables Secure Boot, unless the <code>secureBoot</code> field under <code>efi</code> is set to <code>false</code>. Secure Boot itself requires the SMM CPU feature to be enabled as above, which does not happen automatically, for security reasons.</p>"},{"location":"compute/virtual_hardware/#smbios-firmware","title":"SMBIOS Firmware","text":"<p>In order to provide a consistent view on the virtualized hardware for the guest OS, the SMBIOS UUID can be set to a constant value via <code>spec.firmware.uuid</code>:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    firmware:\n      # this sets the UUID\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n      serial: e4686d2c-6e8d-4335-b8fd-81bee22f4815\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p>In addition, the SMBIOS serial number can be set to a constant value via <code>spec.firmware.serial</code>, as demonstrated above.</p>"},{"location":"compute/virtual_hardware/#cpu","title":"CPU","text":"<p>Note: This is not related to scheduling decisions or resource assignment.</p>"},{"location":"compute/virtual_hardware/#topology","title":"Topology","text":"<p>Setting the number of CPU cores is possible via <code>spec.domain.cpu.cores</code>. The following VM will have a CPU with <code>3</code> cores:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the cores\n      cores: 3\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#labeling-nodes-with-cpu-models-and-cpu-features","title":"Labeling nodes with cpu models and cpu features","text":"<p>KubeVirt can create node selectors based on VM cpu models and features. With these node selectors, VMs will be scheduled on the nodes that support the matching VM cpu model and features.</p> <p>To properly label the node, user can use Kubevirt Node-labeller, which creates all  necessary labels or create node labels by himself.</p> <p>Kubevirt node-labeller creates 3 types of labels: cpu models, cpu features and kvm info. It uses libvirt to get all supported cpu models and cpu features on host and then Node-labeller creates labels from cpu models. </p> <p>Node-labeller supports obsolete list of cpu models and minimal baseline cpu model for features. Both features can be set via KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    obsoleteCPUModels:\n      486: true\n      pentium: true\n...\n</code></pre> <p>Obsolete cpus will not be inserted in labels. If KubeVirt CR doesn't  contain <code>obsoleteCPUModels</code> variable, Labeller sets default values (\"pentium, pentium2, pentium3, pentiumpro, coreduo, n270,  core2duo, Conroe, athlon, phenom, kvm32, kvm64, qemu32 and qemu64\").</p> <p>User can change obsoleteCPUModels by adding / removing cpu model in config map. Kubevirt then update nodes with new labels.</p> <p>For homogenous cluster / clusters without live migration enabled it's possible to disable the node labeler and avoid adding labels to the nodes by adding the  following annotation to the nodes:</p> <p><code>node-labeller.kubevirt.io/skip-node</code>.</p>"},{"location":"compute/virtual_hardware/#model","title":"Model","text":"<p>Note: If CPU model wasn't defined, the VM will have CPU model closest to one that used on the node where the VM is running.</p> <p>Note: CPU model is case sensitive.</p> <p>Setting the CPU model is possible via <code>spec.domain.cpu.model</code>. The following VM will have a CPU with the <code>Conroe</code> model:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the CPU model\n      model: Conroe\n...\n</code></pre> <p>You can check list of available models here.</p> <p>When CPUNodeDiscovery feature-gate is enabled and VM has cpu model, Kubevirt creates node selector with format: <code>cpu-model.node.kubevirt.io/&lt;cpuModel&gt;</code>, e.g. <code>cpu-model.node.kubevirt.io/Conroe</code>. When VM doesn\u2019t have cpu model, then no node selector is created.</p>"},{"location":"compute/virtual_hardware/#enabling-default-cluster-cpu-model","title":"Enabling default cluster cpu model","text":"<p>To enable the default cpu model, user may add the <code>cpuModel</code> field in the KubeVirt CR.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    cpuModel: \"EPYC\"\n...\n</code></pre> <p>Default CPU model is set when vmi doesn't have any cpu model. When vmi has cpu model set, then vmi's cpu model is preferred. When default cpu model is not set and vmi's cpu model is not set too, <code>host-model</code> will be set. Default cpu model can be changed when kubevirt is running. When CPUNodeDiscovery feature gate is enabled Kubevirt creates node selector with default cpu model.</p>"},{"location":"compute/virtual_hardware/#cpu-model-special-cases","title":"CPU model special cases","text":"<p>As special cases you can set <code>spec.domain.cpu.model</code> equals to: - <code>host-passthrough</code> to passthrough CPU from the node to the VM</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this passthrough the node CPU to the VM\n      model: host-passthrough\n...\n</code></pre> <ul> <li><code>host-model</code> to get CPU on the VM close to the node one</li> </ul> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this set the VM CPU close to the node one\n      model: host-model\n...\n</code></pre> <p>See the CPU API reference for more details.</p>"},{"location":"compute/virtual_hardware/#features","title":"Features","text":"<p>Setting CPU features is possible via <code>spec.domain.cpu.features</code> and can contain zero or more CPU features :</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    cpu:\n      # this sets the CPU features\n      features:\n      # this is the feature's name\n      - name: \"apic\"\n      # this is the feature's policy\n       policy: \"require\"\n...\n</code></pre> <p>Note: Policy attribute can either be omitted or contain one of the following policies: force, require, optional, disable, forbid.</p> <p>Note: In case a policy is omitted for a feature, it will default to require.</p> <p>Behaviour according to Policies:</p> <ul> <li>All policies will be passed to libvirt during virtual machine     creation.</li> <li>In case the feature gate \"CPUNodeDiscovery\" is enabled and the     policy is omitted or has \"require\" value, then the virtual machine     could be scheduled only on nodes that support this feature.</li> <li>In case the feature gate \"CPUNodeDiscovery\" is enabled and the     policy has \"forbid\" value, then the virtual machine would not be     scheduled on nodes that support this feature.</li> </ul> <p>Full description about features and policies can be found here.</p> <p>When CPUNodeDiscovery feature-gate is enabled Kubevirt creates node selector from cpu features with format: <code>cpu-feature.node.kubevirt.io/&lt;cpuFeature&gt;</code>, e.g. <code>cpu-feature.node.kubevirt.io/apic</code>. When VM doesn\u2019t have cpu feature, then no node selector is created.</p>"},{"location":"compute/virtual_hardware/#clock","title":"Clock","text":""},{"location":"compute/virtual_hardware/#guest-time","title":"Guest time","text":"<p>Sets the virtualized hardware clock inside the VM to a specific time. Available options are</p> <ul> <li> <p>utc</p> </li> <li> <p>timezone</p> </li> </ul> <p>See the Clock API Reference for all possible configuration options.</p>"},{"location":"compute/virtual_hardware/#utc","title":"utc","text":"<p>If <code>utc</code> is specified, the VM's clock will be set to UTC.</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      utc: {}\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#timezone","title":"timezone","text":"<p>If <code>timezone</code> is specified, the VM's clock will be set to the specified local time.</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      timezone: \"America/New York\"\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#timers","title":"Timers","text":"<ul> <li> <p>pit</p> </li> <li> <p>rtc</p> </li> <li> <p>kvm</p> </li> <li> <p>hyperv</p> </li> </ul> <p>A pretty common timer configuration for VMs looks like this:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    clock:\n      utc: {}\n      # here are the timer\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p><code>hpet</code> is disabled,<code>pit</code> and <code>rtc</code> are configured to use a specific <code>tickPolicy</code>. Finally, <code>hyperv</code> is made available too.</p> <p>See the Timer API Reference for all possible configuration options.</p> <p>Note: Timer can be part of a machine type. Thus it may be necessary to explicitly disable them. We may in the future decide to add them via cluster-level defaulting, if they are part of a QEMU machine definition.</p>"},{"location":"compute/virtual_hardware/#random-number-generator-rng","title":"Random number generator (RNG)","text":"<p>You may want to use entropy collected by your cluster nodes inside your guest. KubeVirt allows to add a <code>virtio</code> RNG device to a virtual machine as follows.</p> <pre><code>metadata:\n  name: vmi-with-rng\nspec:\n  domain:\n    devices:\n      rng: {}\n</code></pre> <p>For Linux guests, the <code>virtio-rng</code> kernel module should be loaded early in the boot process to acquire access to the entropy source. Other systems may require similar adjustments to work with the <code>virtio</code> RNG device.</p> <p>Note: Some guest operating systems or user payloads may require the RNG device with enough entropy and may fail to boot without it. For example, fresh Fedora images with newer kernels (4.16.4+) may require the <code>virtio</code> RNG device to be present to boot to login.</p>"},{"location":"compute/virtual_hardware/#video-and-graphics-device","title":"Video and Graphics Device","text":"<p>By default a minimal Video and Graphics device configuration will be applied to the VirtualMachineInstance. The video device is <code>vga</code> compatible and comes with a memory size of 16 MB. This device allows connecting to the OS via <code>vnc</code>.</p> <p>It is possible not attach it by setting <code>spec.domain.devices.autoattachGraphicsDevice</code> to <code>false</code>:</p> <pre><code>metadata:\n  name: myvmi\nspec:\n  domain:\n    devices:\n      autoattachGraphicsDevice: false\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimName: myclaim\n</code></pre> <p>VMIs without graphics and video devices are very often referenced as <code>headless</code> VMIs.</p> <p>If using a huge amount of small VMs this can be helpful to increase the VMI density per node, since no memory needs to be reserved for video.</p>"},{"location":"compute/virtual_hardware/#features_1","title":"Features","text":"<p>KubeVirt supports a range of virtualization features which may be tweaked in order to allow non-Linux based operating systems to properly boot. Most noteworthy are</p> <ul> <li> <p>acpi</p> </li> <li> <p>apic</p> </li> <li> <p>hyperv</p> </li> </ul> <p>A common feature configuration is shown by the following example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    # typical features\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    resources:\n      requests:\n        memory: 512M\n    devices:\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre> <p>See the Features API Reference for all available features and configuration options.</p>"},{"location":"compute/virtual_hardware/#resources-requests-and-limits","title":"Resources Requests and Limits","text":"<p>An optional resource request can be specified by the users to allow the scheduler to make a better decision in finding the most suitable Node to place the VM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  domain:\n    resources:\n      requests:\n        memory: \"1Gi\"\n        cpu: \"1\"\n      limits:\n        memory: \"2Gi\"\n        cpu: \"2\"\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre>"},{"location":"compute/virtual_hardware/#cpu_1","title":"CPU","text":"<p>Specifying CPU limits will determine the amount of cpu shares set on the control group the VM is running in, in other words, the amount of time the VM's CPUs can execute on the assigned resources when there is a competition for CPU resources.</p> <p>For more information please refer to how Pods with resource limits are run.</p>"},{"location":"compute/virtual_hardware/#memory-overhead","title":"Memory Overhead","text":"<p>Various VM resources, such as a video adapter, IOThreads, and supplementary system software, consume additional memory from the Node, beyond the requested memory intended for the guest OS consumption. In order to provide a better estimate for the scheduler, this memory overhead will be calculated and added to the requested memory.</p> <p>Please see how Pods with resource requests are scheduled for additional information on resource requests and limits.</p>"},{"location":"compute/virtual_hardware/#hugepages","title":"Hugepages","text":"<p>KubeVirt give you possibility to use hugepages as backing memory for your VM. You will need to provide desired amount of memory <code>resources.requests.memory</code> and size of hugepages to use <code>memory.hugepages.pageSize</code>, for example for x86_64 architecture it can be <code>2Mi</code>.</p> <pre><code>apiVersion: kubevirt.io/v1alpha1\nkind: VirtualMachine\nmetadata:\n  name: myvm\nspec:\n  domain:\n    resources:\n      requests:\n        memory: \"64Mi\"\n    memory:\n      hugepages:\n        pageSize: \"2Mi\"\n    disks:\n    - name: myimage\n      disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre> <p>In the above example the VM will have <code>64Mi</code> of memory, but instead of regular memory it will use node hugepages of the size of <code>2Mi</code>.</p>"},{"location":"compute/virtual_hardware/#limitations","title":"Limitations","text":"<ul> <li> <p>a node must have pre-allocated hugepages</p> </li> <li> <p>hugepages size cannot be bigger than requested memory</p> </li> <li> <p>requested memory must be divisible by hugepages size</p> </li> <li> <p>hugepages uses by default memfd. Memfd is supported from kernel &gt;= 4.14. If you run on an older host (e.g centos 7.9), it is required to disable memfd with the annotation <code>kubevirt.io/memfd: \"false\"</code> in the VMI metadata annotation.</p> </li> </ul>"},{"location":"compute/virtual_hardware/#input-devices","title":"Input Devices","text":""},{"location":"compute/virtual_hardware/#tablet","title":"Tablet","text":"<p>Kubevirt supports input devices. The only type which is supported is <code>tablet</code>. Tablet input device supports only <code>virtio</code> and <code>usb</code> bus. Bus can be empty. In that case, <code>usb</code> will be selected.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: myvm\nspec:\n  domain:\n    devices:\n      inputs:\n      - type: tablet\n        bus: virtio\n        name: tablet1\n      disks:\n      - name: myimage\n        disk: {}\n  volumes:\n    - name: myimage\n      persistentVolumeClaim:\n        claimname: myclaim\n</code></pre>"},{"location":"compute/vsock/","title":"VSOCK","text":"<p>VM Sockets (vsock) is a fast and efficient guest-host communication mechanism.</p>"},{"location":"compute/vsock/#background","title":"Background","text":"<p>Right now KubeVirt uses virtio-serial for local guest-host communication. Currently it used in KubeVirt by libvirt and qemu to communicate with the qemu-guest-agent. Virtio-serial can also be used by other agents, but it is a little bit cumbersome due to:</p> <ul> <li>A small set of ports on the virtio-serial device</li> <li>Low bandwidth</li> <li>No socket based communication possible, which requires every agent to establish their own protocols, or work with translation layers like SLIP to be able to use protocols like gRPC for reliable communication.</li> <li>No easy and supportable way to get a virtio-serial socket assigned and being able to access it without entering the virt-launcher pod.</li> <li>Due to the point above, privileges are required for services.</li> </ul> <p>With virtio-vsock we get support for easy guest-host communication which solves the above issues from a user/admin perspective.</p>"},{"location":"compute/vsock/#usage","title":"Usage","text":""},{"location":"compute/vsock/#feature-gate","title":"Feature Gate","text":"<p>To enable VSOCK in KubeVirt cluster, the user may expand the <code>featureGates</code> field in the KubeVirt CR by adding the <code>VSOCK</code> to it.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"VSOCK\"\n</code></pre> <p>Alternatively, users can edit an existing kubevirt CR:</p> <p><code>kubectl edit kubevirt kubevirt -n kubevirt</code></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"VSOCK\"\n</code></pre>"},{"location":"compute/vsock/#virtual-machine-instance","title":"Virtual Machine Instance","text":"<p>To attach VSOCK device to a Virtual Machine, the user has to add <code>autoattachVSOCK: true</code> in a <code>devices</code> section of Virtual Machine Instance specification:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-vsock\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      autoattachVSOCK: true\n</code></pre> <p>This will expose VSOCK device to the VM. The <code>CID</code> will be assigned randomly by <code>virt-controller</code>, and exposed to the Virtual Machine Instance status:</p> <pre><code>status:\n  VSOCKCID: 123\n</code></pre>"},{"location":"compute/vsock/#security","title":"Security","text":"<p>NOTE:  The <code>/dev/vhost-vsock</code> device is NOT NEEDED to connect or bind to a VSOCK socket.</p> <p>To make VSOCK feature secure, following measures are put in place:</p> <ul> <li>The whole VSOCK features will live behind a feature gate.</li> <li>By default the first 1024 ports of a vsock device are privileged. Services trying to bind to those require <code>CAP_NET_BIND_SERVICE</code> capability.</li> <li><code>AF_VSOCK</code> socket syscall gets blocked in containerd 1.7+ (containerd/containerd#7442). It is right now the responsibility of the vendor to ensure that the used CRI selects a default seccomp policy which blocks VSOCK socket calls in a similar way like it was done for containerd.</li> <li>CIDs are assigned by <code>virt-controller</code> and are unique per Virtual Machine Instance to ensure that <code>virt-handler</code> has an easy way of tracking the identity without races. While this still allows <code>virt-launcher</code> to fake-use an assigned CID, it eliminates possible assignment races which attackers could make use-of to redirect VSOCK calls.</li> </ul>"},{"location":"compute/windows_virtio_drivers/","title":"Windows virtio drivers","text":"<p>Purpose of this document is to explain how to install virtio drivers for Microsoft Windows running in a fully virtualized guest.</p>"},{"location":"compute/windows_virtio_drivers/#do-i-need-virtio-drivers","title":"Do I need virtio drivers?","text":"<p>Yes. Without the virtio drivers, you cannot use paravirtualized hardware properly. It would either not work, or will have a severe performance penalty.</p> <p>For more information about VirtIO and paravirtualization, see VirtIO and paravirtualization</p> <p>For more details on configuring your VirtIO driver please refer to Installing VirtIO driver on a new Windows virtual machine and Installing VirtIO driver on an existing Windows virtual machine.</p>"},{"location":"compute/windows_virtio_drivers/#which-drivers-i-need-to-install","title":"Which drivers I need to install?","text":"<p>There are usually up to 8 possible devices that are required to run Windows smoothly in a virtualized environment. KubeVirt currently supports only:</p> <ul> <li> <p>viostor, the block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>viorng, the entropy source driver, applies to PCI Device in the     Other devices group.</p> </li> <li> <p>NetKVM, the network driver, applies to Ethernet Controller in     the Other devices group. Available only if a virtio NIC is     configured.</p> </li> </ul> <p>Other virtio drivers, that exists and might be supported in the future:</p> <ul> <li> <p>Balloon, the balloon driver, applies to PCI Device in the Other     devices group</p> </li> <li> <p>vioserial, the paravirtual serial driver, applies to PCI Simple     Communications Controller in the Other devices group.</p> </li> <li> <p>vioscsi, the SCSI block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>qemupciserial, the emulated PCI serial driver, applies to PCI Serial     Port in the Other devices group.</p> </li> <li> <p>qxl, the paravirtual video driver, applied to Microsoft Basic     Display Adapter in the Display adapters group.</p> </li> <li> <p>pvpanic, the paravirtual panic driver, applies to Unknown device in     the Other devices group.</p> </li> </ul> <p>Note</p> <p>Some drivers are required in the installation phase. When you are installing Windows onto the virtio block storage you have to provide an appropriate virtio driver. Namely, choose viostor driver for your version of Microsoft Windows, eg. does not install XP driver when you run Windows 10.</p> <p>Other drivers can be installed after the successful windows installation. Again, please install only drivers matching your Windows version.</p>"},{"location":"compute/windows_virtio_drivers/#how-to-install-during-windows-install","title":"How to install during Windows install?","text":"<p>To install drivers before the Windows starts its install, make sure you have virtio-win package attached to your VirtualMachine as SATA CD-ROM. In the Windows installation, choose advanced install and load driver. Then please navigate to loaded Virtio CD-ROM and install one of viostor or vioscsi, depending on whichever you have set up.</p> <p>Step by step screenshots:</p> <p></p> <p></p> <p></p> <p></p> <p></p> <p></p>"},{"location":"compute/windows_virtio_drivers/#how-to-install-after-windows-install","title":"How to install after Windows install?","text":"<p>After windows install, please go to Device Manager. There you should see undetected devices in \"available devices\" section. You can install virtio drivers one by one going through this list.</p> <p></p> <p></p> <p></p> <p></p> <p>For more details on how to choose a proper driver and how to install the driver, please refer to the Windows Guest Virtual Machines on Red Hat Enterprise Linux 7.</p>"},{"location":"compute/windows_virtio_drivers/#how-to-obtain-virtio-drivers","title":"How to obtain virtio drivers?","text":"<p>The virtio Windows drivers are distributed in a form of containerDisk, which can be simply mounted to the VirtualMachine. The container image, containing the disk is located at: https://quay.io/repository/kubevirt/virtio-container-disk?tab=tags and the image be pulled as any other docker container:</p> <pre><code>docker pull quay.io/kubevirt/virtio-container-disk\n</code></pre> <p>However, pulling image manually is not required, it will be downloaded if not present by Kubernetes when deploying VirtualMachine.</p>"},{"location":"compute/windows_virtio_drivers/#attaching-to-virtualmachine","title":"Attaching to VirtualMachine","text":"<p>KubeVirt distributes virtio drivers for Microsoft Windows in a form of container disk. The package contains the virtio drivers and QEMU guest agent. The disk was tested on Microsoft Windows Server 2012. Supported Windows version is XP and up.</p> <p>The package is intended to be used as CD-ROM attached to the virtual machine with Microsoft Windows. It can be used as SATA CDROM during install phase or to provide drivers in an existing Windows installation.</p> <p>Attaching the virtio-win package can be done simply by adding ContainerDisk to you VirtualMachine.</p> <pre><code>spec:\n  domain:\n    devices:\n      disks:\n        - name: virtiocontainerdisk\n          # Any other disk you want to use, must go before virtioContainerDisk.\n          # KubeVirt boots from disks in order ther are defined.\n          # Therefore virtioContainerDisk, must be after bootable disk.\n          # Other option is to choose boot order explicitly:\n          #  - https://kubevirt.io/api-reference/v0.13.2/definitions.html#_v1_disk\n          # NOTE: You either specify bootOrder explicitely or sort the items in\n          #       disks. You can not do both at the same time.\n          # bootOrder: 2\n          cdrom:\n            bus: sata\nvolumes:\n  - containerDisk:\n      image: quay.io/kubevirt/virtio-container-disk\n    name: virtiocontainerdisk\n</code></pre> <p>Once you are done installing virtio drivers, you can remove virtio container disk by simply removing the disk from yaml specification and restarting the VirtualMachine.</p>"},{"location":"debug_virt_stack/debug/","title":"Debug","text":"<p>This page contains instructions on how to debug KubeVirt.</p> <p>This is useful to both KubeVirt developers and advanced users that would like to gain deep understanding on what's happening behind the scenes.</p>"},{"location":"debug_virt_stack/debug/#log-verbosity","title":"Log Verbosity","text":"<p>KubeVirt produces a lot of logging throughout its codebase. Some log entries have a verbosity level defined to them. The verbosity level that's defined for a log entry determines the minimum verbosity level in order to expose the log entry.</p> <p>In code, the log entry looks similar to: <code>log.Log.V(verbosity).Infof(\"...\")</code> while <code>verbosity</code> is the minimum verbosity level for this entry.</p> <p>For example, if the log verbosity for some log entry is <code>3</code>, then the log would be exposed only if the log verbosity is defined to be equal or greater than <code>3</code>, or else it would be filtered out.</p> <p>Currently, log verbosity can be defined per-component or per-node. The most updated API is detailed here.</p>"},{"location":"debug_virt_stack/debug/#setting-verbosity-per-kubevirt-component","title":"Setting verbosity per KubeVirt component","text":"<p>One way of raising log verbosity is to manually determine it for the different components in <code>KubeVirt</code> CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      logVerbosity:\n        virtLauncher: 2\n        virtHandler: 3\n        virtController: 4\n        virtAPI: 5\n        virtOperator: 6\n</code></pre></p> <p>This option is best for debugging specific components.</p>"},{"location":"debug_virt_stack/debug/#libvirt-virtqemudconf-set-log_filters-according-to-virt-launcher-log-verbosity","title":"libvirt virtqemud.conf set log_filters according to virt-launcher log Verbosity","text":"Verbosity level log_filters in virtqemud.conf 5 log_filters=\"3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 3:qemu.qemu_monitor 3:qemu.qemu_monitor_json 3:conf.domain_addr 1:*\" 6 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 3:qemu.qemu_monitor 1:* 7 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 3:util.threadjob 3:cpu.cpu 1:* 8 and above 3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:* <p>User can set self-defined log-filters via the annotations tag <code>kubevirt.io/libvirt-log-filters</code> in VMI configuration. e.g. <pre><code>kind: VirtualMachineInstance\nmetadata:\n  name: my-vmi\n  annotations:\n    kubevirt.io/libvirt-log-filters: \"3:remote 4:event 1:*\"\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#setting-verbosity-per-nodes","title":"Setting verbosity per nodes","text":"<p>Another way is to set verbosity level per node: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      nodeVerbosity:\n        \"node01\": 4\n        \"otherNodeName\": 6\n</code></pre></p> <p><code>nodeVerbosity</code> is essentially a map from string to int where the key is the node name and the value is the verbosity level. The verbosity level would be defined for all the different components in that node (e.g. <code>virt-handler</code>, <code>virt-launcher</code>, etc).</p>"},{"location":"debug_virt_stack/debug/#how-to-retrieve-kubevirt-components-logs","title":"How to retrieve KubeVirt components' logs","text":"<p>In Kubernetes, logs are defined at the Pod level. Therefore, first it's needed to list the Pods of KubeVirt's core components. In order to do that we can first list the Pods under KubeVirt's install namespace.</p> <p>For example: <pre><code>$&gt; kubectl get pods -n &lt;KubeVirt Install Namespace&gt;\nNAME                               READY   STATUS    RESTARTS   AGE\ndisks-images-provider-7gqbc        1/1     Running   0          32m\ndisks-images-provider-vg4kx        1/1     Running   0          32m\nvirt-api-57fcc4497b-7qfmc          1/1     Running   0          31m\nvirt-api-57fcc4497b-tx9nc          1/1     Running   0          31m\nvirt-controller-76c784655f-7fp6m   1/1     Running   0          30m\nvirt-controller-76c784655f-f4pbd   1/1     Running   0          30m\nvirt-handler-2m86x                 1/1     Running   0          30m\nvirt-handler-9qs6z                 1/1     Running   0          30m\nvirt-operator-7ccfdbf65f-q5snk     1/1     Running   0          32m\nvirt-operator-7ccfdbf65f-vllz8     1/1     Running   0          32m\n</code></pre></p> <p>Then, we can pick one of the pods and fetch its logs. For example: <pre><code>$&gt; kubectl logs -n &lt;KubeVirt Install Namespace&gt; virt-handler-2m86x | head -n8\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"set verbosity to 2\",\"pos\":\"virt-handler.go:453\",\"timestamp\":\"2022-04-17T08:58:37.373695Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"set verbosity to 2\",\"pos\":\"virt-handler.go:453\",\"timestamp\":\"2022-04-17T08:58:37.373726Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"setting rate limiter to 5 QPS and 10 Burst\",\"pos\":\"virt-handler.go:462\",\"timestamp\":\"2022-04-17T08:58:37.373782Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]\",\"pos\":\"cpu_plugin.go:96\",\"timestamp\":\"2022-04-17T08:58:37.390221Z\"}\n{\"component\":\"virt-handler\",\"level\":\"warning\",\"msg\":\"host model mode is expected to contain only one model\",\"pos\":\"cpu_plugin.go:103\",\"timestamp\":\"2022-04-17T08:58:37.390263Z\"}\n{\"component\":\"virt-handler\",\"level\":\"info\",\"msg\":\"node-labeller is running\",\"pos\":\"node_labeller.go:94\",\"timestamp\":\"2022-04-17T08:58:37.391011Z\"}\n</code></pre></p> <p>Obviously, for both examples above, <code>&lt;KubeVirt Install Namespace&gt;</code> needs to be replaced with the actual namespace KubeVirt is installed in.</p>"},{"location":"debug_virt_stack/debug/#kubevirt-pprof-profiler","title":"KubeVirt PProf Profiler","text":"<p>Using the <code>cluster-profiler</code> client tool, a developer can get the PProf profiling data for every component in the Kubevirt Control plane. Here is a user guide:</p>"},{"location":"debug_virt_stack/debug/#compile-cluster-profiler","title":"Compile <code>cluster-profiler</code>","text":"<p>Build from source code <pre><code>$ git clone https://github.com/kubevirt/kubevirt.git\n$ cd kubevirt/tools/cluster-profiler\n$ go build\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#enable-the-feature-gate","title":"Enable the feature gate","text":"<p>Add <code>ClusterProfiler</code> in KubeVirt config <pre><code>$ cat &lt;&lt; END &gt; enable-feature-gate.yaml\n\n---\napiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - ClusterProfiler\nEND\n\n$ kubectl apply -f enable-feature-gate.yaml\n</code></pre></p>"},{"location":"debug_virt_stack/debug/#do-the-profiling","title":"Do the profiling","text":"<p>Start CPU profiling <pre><code>$ cluster-profiler --cmd start\n\n2023/05/17 09:31:09 SUCCESS: started cpu profiling KubeVirt control plane\n</code></pre> Stop CPU profiling <pre><code>$ cluster-profiler --cmd stop\n\n2023/05/17 09:31:14 SUCCESS: stopped cpu profiling KubeVirt control plane\n</code></pre> Dump the pprof result <pre><code>$ cluster-profiler --cmd dump\n\n2023/05/17 09:31:18 Moving already existing \"cluster-profiler-results\" =&gt; \"cluster-profiler-results-old-67fq\"\nSUCCESS: Dumped PProf 6 results for KubeVirt control plane to [cluster-profiler-results]\n</code></pre> The PProf result can be found in the folder <code>cluster-profiler-results</code> <pre><code>$ tree cluster-profiler-results\n\ncluster-profiler-results\n\u251c\u2500\u2500 virt-api-5f96f84dcb-lkpb7\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-controller-5bbd9554d9-2f8j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-controller-5bbd9554d9-qct2w\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-handler-ccq6c\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u251c\u2500\u2500 virt-operator-cdc677b7-pg9j2\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 allocs.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 block.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 cpu.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 goroutine.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 heap.pprof\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 mutex.pprof\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 threadcreate.pprof\n\u2514\u2500\u2500 virt-operator-cdc677b7-pjqdx\n    \u251c\u2500\u2500 allocs.pprof\n    \u251c\u2500\u2500 block.pprof\n    \u251c\u2500\u2500 cpu.pprof\n    \u251c\u2500\u2500 goroutine.pprof\n    \u251c\u2500\u2500 heap.pprof\n    \u251c\u2500\u2500 mutex.pprof\n    \u2514\u2500\u2500 threadcreate.pprof\n</code></pre></p>"},{"location":"debug_virt_stack/launch-qemu-gdb/","title":"Launch QEMU with gdb and connect locally with gdb client","text":"<p>This guide is for cases where QEMU counters very early failures and it is hard to synchronize it in a later point in time.</p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#image-creation-and-pvc-population","title":"Image creation and PVC population","text":"<p>This scenario is a slight variation of the guide about starting strace, hence some of the details on the image build and the PVC population are simply skipped and explained in the other section.</p> <p>In this example, QEMU will be launched with <code>gdbserver</code> and later we will connect to it using a local <code>gdb</code> client.</p> <p>The wrapping script looks like: <pre><code>#!/bin/bash\n\nLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/gdbserver \\\n    localhost:1234 \\\n    /usr/libexec/qemu-kvm $@ &amp;\nprintf \"%d\" $(pgrep gdbserver) &gt; /run/libvirt/qemu/run/default_vmi-debug-tools.pid\n</code></pre></p> <p>First, we need to build and push the image with the wrapping script and the gdbserver: <pre><code>FROM quay.io/centos/centos:stream9 as build\n\nENV DIR /debug-tools\nENV DEBUGINFOD_URLS https://debuginfod.centos.org/\nRUN mkdir -p ${DIR}/logs\n\nRUN yum  install --installroot=${DIR} -y gdb-gdbserver &amp;&amp; yum clean all\n\nCOPY ./wrap_qemu_gdb.sh $DIR/wrap_qemu_gdb.sh\nRUN chmod 0755 ${DIR}/wrap_qemu_gdb.sh\nRUN chown 107:107 ${DIR}/wrap_qemu_gdb.sh\nRUN chown 107:107 ${DIR}/logs\n</code></pre></p> <p>Then, we can create and populate the <code>debug-tools</code> PVC as with did in the strace example: <pre><code>$ k apply -f debug-tools-pvc.yaml\npersistentvolumeclaim/debug-tools created\n$ kubectl  apply -f populate-job-pvc.yaml\njob.batch/populate-pvc created\n$ $ kubectl  get jobs\nNAME           COMPLETIONS   DURATION   AGE\npopulate-pvc   1/1           7s         2m12s\n</code></pre></p> <p>Configmap: <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;emulator&gt;/usr/libexec/qemu-kvm&lt;/emulator&gt;|&lt;emulator&gt;/var/run/debug/wrap_qemu_gdb.sh&lt;/emulator&gt;|\" $tempFile\n    cat $tempFile\n</code></pre></p> <p>As last step, we need to create the configmaps to modify the VM XML: <pre><code>$ kubectl apply -f configmap.yaml\nconfigmap/my-config-map created\n</code></pre></p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#build-client-image","title":"Build client image","text":"<p>In this scenario, we use an additional container image containing <code>gdb</code> and the same qemu binary as the target process to debug. This image will be run locally with <code>podman</code>.</p> <p>In order to build this image, we need to identify the image of the <code>virt-launcher</code> container we want to debug. Based on the KubeVirt installation, the namespace and the name of the KubeVirt CR could vary. In this example, we'll assume that KubeVirt CR is called <code>kubevirt</code> and installed in the <code>kubevirt</code> namespace.</p> <p>You can easily find out the right names in your cluster by searching with: <pre><code>$ kubectl get kubevirt -A\nNAMESPACE   NAME       AGE     PHASE\nkubevirt    kubevirt   3h11m   Deployed\n</code></pre></p> <p>The steps to build the image are:</p> <ol> <li> <p>Get the registry of the images of the KubeVirt installation: <pre><code>$ export registry=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.registry'|tr -d \"\\\"\")\n$ echo $registry\n\"registry:5000/kubevirt\"\n</code></pre></p> </li> <li> <p>Get the shasum of the virt-launcher image: <pre><code>$ export tag=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.virtLauncherSha'|tr -d \"\\\"\")\n$ echo $tag\n\"sha256:6c8b85eed8e83a4c70779836b246c057d3e882eb513f3ded0a02e0a4c4bda837\"\n</code></pre></p> </li> </ol> <p>Example of Dockerfile: <pre><code>ARG registry\nARG tag\nFROM ${registry}/kubevirt/virt-launcher${tag} AS launcher\nFROM quay.io/centos/centos:stream9 as build\n\nRUN yum  install -y gdb &amp;&amp; yum clean all\n\nCOPY --from=launcher /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm\n</code></pre></p> <ol> <li>Build the image by using the <code>registry</code> and the <code>tag</code> retrieved in the previous steps: <pre><code>$ podman build \\\n    -t gdb-client \\\n    --build-arg registry=$registry  \\\n    --build-arg tag=@$tag \\\n    -f Dockerfile.client .\n</code></pre></li> </ol> <p>Podman will replace the registry and tag arguments provided on the command line. In this way, we can specify the image registry and shasum for the KubeVirt version to debug.</p>"},{"location":"debug_virt_stack/launch-qemu-gdb/#run-the-vm-to-troubleshoot","title":"Run the VM to troubleshoot","text":"<p>For this example, we add an annotation to keep the virt-launcher pod running even if any errors occur: <pre><code>metadata:\n  annotations:\n    kubevirt.io/keep-launcher-alive-after-failure: \"true\"\n</code></pre></p> <p>Then, we can launch the VM: <pre><code>$ kubectl apply -f debug-vmi.yaml\nvirtualmachineinstance.kubevirt.io/vmi-debug-tools created\n$ kubectl  get vmi\nNAME              AGE   PHASE       IP    NODENAME   READY\nvmi-debug-tools   28s   Scheduled         node01     False\n$ kubectl  get po\nNAME                                  READY   STATUS      RESTARTS   AGE\npopulate-pvc-dnxld                    0/1     Completed   0          4m17s\nvirt-launcher-vmi-debug-tools-tfh28   4/4     Running     0          25s\n</code></pre></p> <p>The wrapping script starts the <code>gdbserver</code> and expose in the port <code>1234</code> inside the container. In order to be able to connect remotely to the gdbserver, we can use the command <code>kubectl port-forward</code> to expose the gdb port on our machine.</p> <pre><code>$ kubectl  port-forward virt-launcher-vmi-debug-tools-tfh28 1234\nForwarding from 127.0.0.1:1234 -&gt; 1234\nForwarding from [::1]:1234 -&gt; 1234\n</code></pre> <p>Finally, we can start the gbd client in the container: <pre><code>$ podman run -ti --network host gdb-client:latest\n$ gdb /usr/libexec/qemu-kvm -ex 'target remote localhost:1234'\nGNU gdb (GDB) Red Hat Enterprise Linux 10.2-12.el9\nCopyright (C) 2021 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\nType \"show copying\" and \"show warranty\" for details.\nThis GDB was configured as \"x86_64-redhat-linux-gnu\".\nType \"show configuration\" for configuration details.\nFor bug reporting instructions, please see:\n&lt;https://www.gnu.org/software/gdb/bugs/&gt;.\nFind the GDB manual and other documentation resources online at:\n    &lt;http://www.gnu.org/software/gdb/documentation/&gt;.\n\nFor help, type \"help\".\n--Type &lt;RET&gt; for more, q to quit, c to continue without paging--\nType \"apropos word\" to search for commands related to \"word\"...\nReading symbols from /usr/libexec/qemu-kvm...\n\nReading symbols from /root/.cache/debuginfod_client/26221a84fabd219a68445ad0cc87283e881fda15/debuginfo...\nRemote debugging using localhost:1234\nReading /lib64/ld-linux-x86-64.so.2 from remote target...\nwarning: File transfers from remote targets can be slow. Use \"set sysroot\" to access files locally instead.\nReading /lib64/ld-linux-x86-64.so.2 from remote target...\nReading symbols from target:/lib64/ld-linux-x86-64.so.2...\nDownloading separate debug info for /system-supplied DSO at 0x7ffc10eff000...\n0x00007f1a70225e70 in _start () from target:/lib64/ld-linux-x86-64.so.2\n</code></pre></p> <p>For simplicity, we started podman with the option <code>--network host</code> in this way, the container is able to access any port mapped on the host.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/","title":"Launch QEMU with strace","text":"<p>This guide explains how launch QEMU with a debugging tool in virt-launcher pod. This method can be useful to debug early failures or starting QEMU as a child of the debug tool relying on ptrace. The second point is particularly relevant when a process is operating in a non-privileged environment since otherwise, it would need root access to be able to ptrace the process.</p> <p>Ephemeral containers are among the emerging techniques to overcome the lack of debugging tool inside the original image. This solution does, however, come with a number of limitations.  For example, it is possible to spawn a new container inside the same pod of the application to debug and share the same PID namespace. Though they share the same PID namespace, KubeVirt's usage of unprivileged containers makes it, for example, impossible to ptrace a running container. Therefore, this technique isn't appropriate for our needs.</p> <p>Due to its security and image size reduction, KubeVirt container images are based on distroless containers. These kinds of images are extremely beneficial for deployments, but they are challenging to troubleshoot because there is no package management, which prevents the installation of additional tools on the flight.</p> <p>Wrapping the QEMU binary in a script is one practical method for debugging QEMU launched by Libvirt. This script launches the QEMU as a child of this process together with the debugging tool (such as strace or valgrind).</p> <p>The final part that needs to be added is the configuration for Libvirt to use the wrapped script rather than calling the QEMU program directly.</p> <p>It is possible to alter the generated XML with the help of KubeVirt sidecars. This allows us to use the wrapping script in place of the built-in emulator.</p> <p>The primary concept behind this configuration is that all of the additional tools, scripts, and final output files will be stored in a PerstistentVolumeClaim (PVC) that this guide refers to as <code>debug-tools</code>. The virt-launcher pod that we wish to debug will have this PVC attached to it.</p> <p>PVC: <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: debug-tools\nspec:\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Filesystem\n  resources:\n    requests:\n      storage: 1Gi\n</code></pre></p> <p>In this guide, we'll apply the above concepts to debug QEMU inside virt-launcher using strace without the need of build a custom virt-launcher image.</p> <p>You can see a full demo of this setup: </p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-bring-the-debug-tools-and-wrapping-script-into-distroless-containers","title":"How to bring the debug tools and wrapping script into distroless containers","text":"<p>This section provides an example of how to provide extra tools into the distroless container that will be supplied as a PVC using a Dockerfile. Although there are several ways to accomplish this, this covers a relatively simple technique. Alternatively, you could run a pod and manually populate the PVC by execing into the pod.</p> <p>Dockerfile: <pre><code>FROM quay.io/centos/centos:stream9 as build\n\nENV DIR /debug-tools\nRUN mkdir -p ${DIR}/logs\n\nRUN  yum install  --installroot=${DIR} -y strace &amp;&amp; yum clean all\n\nCOPY ./wrap_qemu_strace.sh $DIR/wrap_qemu_strace.sh\nRUN chmod 0755 ${DIR}/wrap_qemu_strace.sh\nRUN chown 107:107 ${DIR}/wrap_qemu_strace.sh\nRUN chown 107:107 ${DIR}/logs\n</code></pre></p> <p>The directory <code>debug-tools</code> stores the content that will be later copied inside the <code>debug-tools</code> PVC. We are essentially adding the missing utilities in the custom directory with <code>yum install --installroot=${DIR}}</code>, and the parent image matches with the parent images of virt-launcher.</p> <p>The <code>wrap_qemu_strace.sh</code> is the wrapping script that will be used to launch QEMU with <code>strace</code> similarly as the example with <code>valgrind</code>. <pre><code>#!/bin/bash\n\nLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/run/debug/usr/lib64 /var/run/debug/usr/bin/strace \\\n        -o /var/run/debug/logs/strace.out \\\n        /usr/libexec/qemu-kvm $@\n</code></pre></p> <p>It is important to set the dynamic library path <code>LD_LIBRARY_PATH</code> to the path where the PVC will be mounted in the virt-launcher container.</p> <p>Then, you will simply need to build the image and your debug setup is ready. The Dockerfle and the script <code>wrap_qemu_strace.sh</code> need to be in the same directory where you run the command. <pre><code>$ podman build -t debug .\n</code></pre></p> <p>The second step is to populate the PVC. This can be easily achieved using a kubernetes <code>Job</code> like: <pre><code>apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: populate-pvc\nspec:\n  template:\n    spec:\n      volumes:\n        - name: populate\n          persistentVolumeClaim:\n            claimName: debug-tools\n      containers:\n        - name: populate\n          image: registry:5000/debug:latest\n          command: [\"sh\", \"-c\", \"cp -r /debug-tools/* /vol\"]\n          imagePullPolicy: Always\n          volumeMounts:\n            - mountPath: \"/vol\"\n              name: populate\n      restartPolicy: Never\n  backoffLimit: 4\n</code></pre></p> <p>The image referenced in the <code>Job</code> is the image we built in the previous step. Once applied this and the job completed, the<code>debug-tools</code> PVC is ready to be used.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-start-qemu-launched-by-a-debugging-tool-eg-strace","title":"How to start qemu launched by a debugging tool (e.g strace)","text":"<p>This part is achieved by using ConfigMaps and a KubeVirt sidecar (more details in the section Using ConfigMap to run custom script).</p> <p>Configmap: <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;emulator&gt;/usr/libexec/qemu-kvm&lt;/emulator&gt;|&lt;emulator&gt;/var/run/debug/wrap_qemu_strace.sh&lt;/emulator&gt;|\" $tempFile\n    cat $tempFile\n</code></pre></p> <p>The script that replaces the QEMU binary with the wrapping script in the XML is stored in the configmap <code>my-config-map</code>. This script will run as a hook, as explained in full in the documentation for the KubeVirt sidecar.</p> <p>Once all the objects created, we can finally run the guest to debug.</p> <p>VMI: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    hooks.kubevirt.io/hookSidecars: '[{\"args\": [\"--version\", \"v1alpha2\"],\n    \"image\":\"registry:5000/kubevirt/sidecar-shim:devel\",\n    \"pvc\": {\"name\": \"debug-tools\",\"volumePath\": \"/debug\", \"sharedComputePath\": \"/var/run/debug\"},\n    \"configMap\": {\"name\": \"my-config-map\",\"key\": \"my_script.sh\", \"hookPath\": \"/usr/bin/onDefineDomain\"}}]'\n  labels:\n    special: vmi-debug-tools\n  name: vmi-debug-tools\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p> <p>The VMI example is a simply VM instance declaration and the interesting parts are the annotations for the hook: * <code>image</code> refers to the sidecar-shim already built and shipped with KubeVirt * <code>pvc</code> refers to the PVC populated with the debug setup. The <code>name</code> refers to the claim name, the <code>volumePath</code> is the path inside the sidecar container where the volume is mounted while the <code>sharedComputePath</code> is the path of the same volume inside the compute container. * <code>configMap</code> refers to the confimap containing the script to modify the XML for the wrapping script</p> <p>Once the VM is declared, the hook will modify the emulator section and Libvirt will call the wrapping script instead of QEMU directly.</p>"},{"location":"debug_virt_stack/launch-qemu-strace/#how-to-fetch-the-output","title":"How to fetch the output","text":"<p>The wrapping script configures <code>strace</code> to store the output in the PVC. In this way, it is possible to retrieve the output file in a later time, for example using an additional pod like:</p> <pre><code>apiVersion: v1\nkind: Pod\nmetadata:\n  name: fetch-logs\nspec:\n  securityContext:\n    runAsUser: 107\n    fsGroup: 107\n  volumes:\n    - name: populate\n      persistentVolumeClaim:\n        claimName: debug-tools\n  containers:\n    - name: populate\n      image: busybox:latest\n      command: [\"tail\", \"-f\", \"/dev/null\"]\n      volumeMounts:\n        - mountPath: \"/vol\"\n          name: populate\n</code></pre> <p>It is then possible to copy the file locally with: <pre><code>$ kubectl cp fetch-logs:/vol/logs/strace.out strace.out\n</code></pre></p>"},{"location":"debug_virt_stack/logging/","title":"Control libvirt logging for each component","text":"<p>Generally, cluster admins can control the log verbosity of each KubeVirt component in KubeVirt CR. For more details, please, check the KubeVirt documentation.</p> <p>Nonetheless, regular users can also adjust the qemu component logging to have a finer control over it. The annotation <code>kubevirt.io/libvirt-log-filters</code> enables you to modify each component's log level.</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    kubevirt.io/libvirt-log-filters: \"2:qemu.qemu_monitor 3:*\"\n  labels:\n    special: vmi-debug-tools\n  name: vmi-debug-tools\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  volumes:\n  - containerDisk:\n      image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n    name: cloudinitdisk\n</code></pre></p> <p>Then, it is possible to obtain the logs from the virt-launcher output:</p> <pre><code>$ kubectl get pods\nNAME                                  READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-debug-tools-fk64q   3/3     Running   0          64s\n$ kubectl  logs virt-launcher-vmi-debug-tools-fk64q\n[..]\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324640, \\\"microseconds\\\": 523652}, \\\"event\\\": \\\"NIC_RX_FILTER_CHANGED\\\", \\\"data\\\": {\\\"name\\\": \\\"ua-default\\\", \\\"path\\\": \\\"/machine/peripheral/ua-default/virtio-backend\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:40.523000Z\"}\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324644, \\\"microseconds\\\": 165626}, \\\"event\\\": \\\"VSERPORT_CHANGE\\\", \\\"data\\\": {\\\"open\\\": true, \\\"id\\\": \\\"channel0\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:44.165000Z\"}\n[..]\n{\"component\":\"virt-launcher\",\"level\":\"info\",\"msg\":\"QEMU_MONITOR_RECV_EVENT: mon=0x7faa8801f5d0 event={\\\"timestamp\\\": {\\\"seconds\\\": 1698324646, \\\"microseconds\\\": 707666}, \\\"event\\\": \\\"RTC_CHANGE\\\", \\\"data\\\": {\\\"offset\\\": 0, \\\"qom-path\\\": \\\"/machine/unattached/device[8]\\\"}}\",\"pos\":\"qemuMonitorJSONIOProcessLine:205\",\"subcomponent\":\"libvirt\",\"thread\":\"80\",\"timestamp\":\"2023-10-26T12:50:46.708000Z\"}\n[..]\n</code></pre> <p>The annotation enables the filter from the container creation. However, in certain cases you might desire to change the logging level dynamically once the container and libvirt have already been started. In this case, <code>virt-admin</code> comes to the rescue.</p> <p>Example: <pre><code>$ kubectl get pods\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters  \"1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util\"\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session  daemon-log-filters\n Logging filters: 1:*libvirt* 1:*qemu* 1:*conf* 1:*security* 3:*event* 3:*json* 3:*file* 3:*object* 1:*util*\n</code></pre></p> <p>Otherwise, if you prefer to redirect the output to a file and fetch it later, you can rely on <code>kubectl cp</code> to retrieve the file. In this case, we are saving the file in the <code>/var/run/libvirt</code> directory because the compute container has the permissions to write there.</p> <p>Example: <pre><code>$ kubectl get pods\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          26m\n$ kubectl  exec -ti virt-launcher-vmi-ephemeral-nqcld -- virt-admin  -c virtqemud:///session daemon-log-outputs \"1:file:/var/run/libvirt/libvirtd.log\"\n$ kubectl cp virt-launcher-vmi-ephemeral-nqcld:/var/run/libvirt/libvirtd.log libvirt-kubevirt.log\ntar: Removing leading `/' from member names\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/","title":"Privileged debugging on the node","text":"<p>This article describes the scenarios in which you can create privileged pods and have root access to the cluster nodes.</p> <p>With privileged pods, you may access devices in <code>/dev</code>, utilize host namespaces and ptrace processes that are running on the node, and use the <code>hostPath</code> volume to mount node directories in the container.</p> <p>A quick way to verify if you are allowed to create privileged pods is to create a sample pod with the <code>--dry-run=server</code> option, like:</p> <pre><code>$ kubectl apply -f debug-pod.yaml --dry-run=server\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#build-the-container-image","title":"Build the container image","text":"<p>KubeVirt uses distroless containers and those images don't have a package manager, for this reason it isn't possible to use the image as parent for installing additional packages.</p> <p>In certain debugging scenarios, the tools require to have exactly the same binary available. However, if the debug tools are operating in a different container, this can be especially difficult as the filesystems of the containers are isolated.</p> <p>This section will cover how to build a container image with the debug tools plus binaries of the KubeVirt version you want to debug.</p> <p>Based on your installation the namespace and the name of the KubeVirt CR could vary. In this example, we'll assume that KubeVirt CR is called <code>kubevirt</code> and installed in the <code>kubevirt</code> namespace. You can easily find out how it is called in your cluster by searching with <code>kubectl get kubevirt -A</code>. This is necessary as we need to retrieve the original <code>virt-launcher</code> image to have exactly the same QEMU binary we want to debug.</p> <p>Get the registry of the images of the KubeVirt installation: <pre><code>$ export registry=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.registry'|tr -d \"\\\"\")\n$ echo $registry\n\"registry:5000/kubevirt\"\n</code></pre></p> <p>Get the shasum of the virt-launcher image: <pre><code>$ export tag=$(kubectl get kubevirt kubevirt -n kubevirt  -o jsonpath='{.status.observedDeploymentConfig}' |jq '.virtLauncherSha'|tr -d \"\\\"\")\n$ echo $tag\n\"sha256:6c8b85eed8e83a4c70779836b246c057d3e882eb513f3ded0a02e0a4c4bda837\"\n</code></pre></p> <p>Dockerfile: <pre><code>ARG registry\nARG tag\nFROM ${registry}/kubevirt/virt-launcher${tag} AS launcher\n\nFROM quay.io/centos/centos:stream9\n\nRUN yum install -y \\\n        gdb \\\n        kernel-devel \\\n        qemu-kvm-tools \\\n        strace \\\n        systemtap-client \\\n        systemtap-devel \\\n    &amp;&amp; yum clean all\nCOPY --from=launcher / /\n</code></pre></p> <p>Then, we can build the image by using the <code>registry</code> and the <code>tag</code> retrieved in the previous steps: <pre><code>$ podman build \\\n    -t debug-tools \\\n    --build-arg registry=$registry  \\\n    --build-arg tag=@$tag \\\n    -f Dockerfile .\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/#deploy-the-privileged-debug-pod","title":"Deploy the privileged debug pod","text":"<p>This is an example that gives you a couple of suggestions how you can define your debugging pod:</p> <pre><code>kind: Pod\nmetadata:\n  name: node01-debug\nspec:\n  containers:\n  - command:\n    - /bin/sh\n    image: registry:5000/debug-tools:latest\n    imagePullPolicy: Always\n    name: debug\n    securityContext:\n      privileged: true\n      runAsUser: 0\n    stdin: true\n    stdinOnce: true\n    tty: true\n    volumeMounts:\n    - mountPath: /host\n      name: host\n    - mountPath: /usr/lib/modules\n      name: modules\n    - mountPath: /sys/kernel\n      name: sys-kernel\n  hostNetwork: true\n  hostPID: true\n  nodeName: node01\n  restartPolicy: Never\n  volumes:\n  - hostPath:\n      path: /\n      type: Directory\n    name: host\n  - hostPath:\n      path: /usr/lib/modules\n      type: Directory\n    name: modules\n  - hostPath:\n      path: /sys/kernel\n      type: Directory\n    name: sys-kernel\n</code></pre> <p>The <code>privileged</code> option is required to have access to mostly all the resources on the node.</p> <p>The <code>nodeName</code> ensures that the debugging pod will be scheduled on the desired node. In order to select the right now, you can use the <code>-owide</code> option with <code>kubectl get po</code> and this will report the nodes where the pod is running.</p> <p>Example: <pre><code> k get pods -owide\nNAME                                READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES\nlocal-volume-provisioner-4jtkb      1/1     Running   0          152m    10.244.196.129   node01   &lt;none&gt;           &lt;none&gt;\nnode01-debug                        1/1     Running   0          44m     192.168.66.101   node01   &lt;none&gt;           &lt;none&gt;\nvirt-launcher-vmi-ephemeral-xg98p   3/3     Running   0          2m54s   10.244.196.148   node01   &lt;none&gt;           1/1\n</code></pre></p> <p>In the <code>volumes</code> section, you can specify the directories you want to be directly mounted in the debugging container. For example, <code>/usr/lib/modules</code> is particularly useful if you need to load some kernel modules.</p> <p>Sharing the host pid namespace with the option <code>hostPID</code> allows you to see all the processes on the node and attach to it with tools like <code>gdb</code> and <code>strace</code>.</p> <p><code>exec</code>-ing into the pod gives you a shell with  privileged access to the node plus the tooling you installed into the image:</p> <pre><code>$ kubectl exec -ti debug -- bash\n</code></pre> <p>The following examples assume you have already execed into the <code>node01-debug</code> pod.</p>"},{"location":"debug_virt_stack/privileged-node-debugging/#validating-the-host-for-virtualization","title":"Validating the host for virtualization","text":"<p>The tool <code>vist-host-validate</code> is utility to validate the host to run libvirt hypervisor. This, for example, can be used to check if a particular node is kvm capable.</p> <p>Example: <pre><code>$  virt-host-validate\n  QEMU: Checking for hardware virtualization                                 : PASS\n  QEMU: Checking if device /dev/kvm exists                                   : PASS\n  QEMU: Checking if device /dev/kvm is accessible                            : PASS\n  QEMU: Checking if device /dev/vhost-net exists                             : PASS\n  QEMU: Checking if device /dev/net/tun exists                               : PASS\n  QEMU: Checking for cgroup 'cpu' controller support                         : PASS\n  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS\n  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS\n  QEMU: Checking for cgroup 'memory' controller support                      : PASS\n  QEMU: Checking for cgroup 'devices' controller support                     : PASS\n  QEMU: Checking for cgroup 'blkio' controller support                       : PASS\n  QEMU: Checking for device assignment IOMMU support                         : PASS\n  QEMU: Checking if IOMMU is enabled by kernel                               : PASS\n  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure\n</code></pre></p>"},{"location":"debug_virt_stack/privileged-node-debugging/#run-a-command-directly-on-the-node","title":"Run a command directly on the node","text":"<p>The debug container has in the volume section the host filesystem mounted under <code>/host</code>. This can be particularly useful if you want to access the node filesystem or execute a command directly on the host. However, the tool needs already to be present on the node.</p> <pre><code># chroot /host\nsh-5.1# cat /etc/os-release\nNAME=\"CentOS Stream\"\nVERSION=\"9\"\nID=\"centos\"\nID_LIKE=\"rhel fedora\"\nVERSION_ID=\"9\"\nPLATFORM_ID=\"platform:el9\"\nPRETTY_NAME=\"CentOS Stream 9\"\nANSI_COLOR=\"0;31\"\nLOGO=\"fedora-logo-icon\"\nCPE_NAME=\"cpe:/o:centos:centos:9\"\nHOME_URL=\"https://centos.org/\"\nBUG_REPORT_URL=\"https://bugzilla.redhat.com/\"\nREDHAT_SUPPORT_PRODUCT=\"Red Hat Enterprise Linux 9\"\nREDHAT_SUPPORT_PRODUCT_VERSION=\"CentOS Stream\"\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#attach-to-a-running-process-eg-strace-or-gdb","title":"Attach to a running process (e.g strace or gdb)","text":"<p>This requires the field <code>hostPID: true</code> in this way you are able to list all the processes running on the node.</p> <pre><code>$ ps -ef |grep qemu-kvm\nqemu       50122   49850  0 12:34 ?        00:00:25 /usr/libexec/qemu-kvm -name guest=default_vmi-ephemeral,debug-threads=on -S -object {\"qom-type\":\"secret\",\"id\":\"masterKey0\",\"format\":\"raw\",\"file\":\"/var/run/kubevirt-private/libvirt/qemu/lib/domain-1-default_vmi-ephemera/master-key.aes\"} -machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on -accel kvm -cpu Skylake-Client-IBRS,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,md-clear=on,stibp=on,flush-l1d=on,arch-capabilities=on,ssbd=on,xsaves=on,pdpe1gb=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,fb-clear=on,hle=off,rtm=off -m size=131072k -object {\"qom-type\":\"memory-backend-ram\",\"id\":\"pc.ram\",\"size\":134217728} -overcommit mem-lock=off -smp 1,sockets=1,dies=1,cores=1,threads=1 -object {\"qom-type\":\"iothread\",\"id\":\"iothread1\"} -uuid b56f06f0-07e9-4fe5-8913-18a14e83a4d1 -smbios type=1,manufacturer=KubeVirt,product=None,uuid=b56f06f0-07e9-4fe5-8913-18a14e83a4d1,family=KubeVirt -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=21,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device {\"driver\":\"pcie-root-port\",\"port\":16,\"chassis\":1,\"id\":\"pci.1\",\"bus\":\"pcie.0\",\"multifunction\":true,\"addr\":\"0x2\"} -device {\"driver\":\"pcie-root-port\",\"port\":17,\"chassis\":2,\"id\":\"pci.2\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x1\"} -device {\"driver\":\"pcie-root-port\",\"port\":18,\"chassis\":3,\"id\":\"pci.3\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x2\"} -device {\"driver\":\"pcie-root-port\",\"port\":19,\"chassis\":4,\"id\":\"pci.4\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x3\"} -device {\"driver\":\"pcie-root-port\",\"port\":20,\"chassis\":5,\"id\":\"pci.5\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x4\"} -device {\"driver\":\"pcie-root-port\",\"port\":21,\"chassis\":6,\"id\":\"pci.6\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x5\"} -device {\"driver\":\"pcie-root-port\",\"port\":22,\"chassis\":7,\"id\":\"pci.7\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x6\"} -device {\"driver\":\"pcie-root-port\",\"port\":23,\"chassis\":8,\"id\":\"pci.8\",\"bus\":\"pcie.0\",\"addr\":\"0x2.0x7\"} -device {\"driver\":\"pcie-root-port\",\"port\":24,\"chassis\":9,\"id\":\"pci.9\",\"bus\":\"pcie.0\",\"addr\":\"0x3\"} -device {\"driver\":\"virtio-scsi-pci-non-transitional\",\"id\":\"scsi0\",\"bus\":\"pci.5\",\"addr\":\"0x0\"} -device {\"driver\":\"virtio-serial-pci-non-transitional\",\"id\":\"virtio-serial0\",\"bus\":\"pci.6\",\"addr\":\"0x0\"} -blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt/container-disks/disk_0.img\",\"node-name\":\"libvirt-2-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"} -blockdev {\"node-name\":\"libvirt-2-format\",\"read-only\":true,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-2-storage\"} -blockdev {\"driver\":\"file\",\"filename\":\"/var/run/kubevirt-ephemeral-disks/disk-data/containerdisk/disk.qcow2\",\"node-name\":\"libvirt-1-storage\",\"cache\":{\"direct\":true,\"no-flush\":false},\"auto-read-only\":true,\"discard\":\"unmap\"} -blockdev {\"node-name\":\"libvirt-1-format\",\"read-only\":false,\"discard\":\"unmap\",\"cache\":{\"direct\":true,\"no-flush\":false},\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":\"libvirt-2-format\"} -device {\"driver\":\"virtio-blk-pci-non-transitional\",\"bus\":\"pci.7\",\"addr\":\"0x0\",\"drive\":\"libvirt-1-format\",\"id\":\"ua-containerdisk\",\"bootindex\":1,\"write-cache\":\"on\",\"werror\":\"stop\",\"rerror\":\"stop\"} -netdev {\"type\":\"tap\",\"fd\":\"22\",\"vhost\":true,\"vhostfd\":\"24\",\"id\":\"hostua-default\"} -device {\"driver\":\"virtio-net-pci-non-transitional\",\"host_mtu\":1480,\"netdev\":\"hostua-default\",\"id\":\"ua-default\",\"mac\":\"7e:cb:ba:c3:71:88\",\"bus\":\"pci.1\",\"addr\":\"0x0\",\"romfile\":\"\"} -add-fd set=0,fd=20,opaque=serial0-log -chardev socket,id=charserial0,fd=18,server=on,wait=off,logfile=/dev/fdset/0,logappend=on -device {\"driver\":\"isa-serial\",\"chardev\":\"charserial0\",\"id\":\"serial0\",\"index\":0} -chardev socket,id=charchannel0,fd=19,server=on,wait=off -device {\"driver\":\"virtserialport\",\"bus\":\"virtio-serial0.0\",\"nr\":1,\"chardev\":\"charchannel0\",\"id\":\"channel0\",\"name\":\"org.qemu.guest_agent.0\"} -audiodev {\"id\":\"audio1\",\"driver\":\"none\"} -vnc vnc=unix:/var/run/kubevirt-private/3a8f7774-7ec7-4cfb-97ce-581db52ee053/virt-vnc,audiodev=audio1 -device {\"driver\":\"VGA\",\"id\":\"video0\",\"vgamem_mb\":16,\"bus\":\"pcie.0\",\"addr\":\"0x1\"} -global ICH9-LPC.noreboot=off -watchdog-action reset -device {\"driver\":\"virtio-balloon-pci-non-transitional\",\"id\":\"balloon0\",\"free-page-reporting\":true,\"bus\":\"pci.8\",\"addr\":\"0x0\"} -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on\n$ gdb -p 50122 /usr/libexec/qemu-kvm\n</code></pre>"},{"location":"debug_virt_stack/privileged-node-debugging/#debugging-using-crictl","title":"Debugging using <code>crictl</code>","text":"<p><code>Crictl</code> is a cli for CRI runtimes and can be particularly useful to troubleshoot container failures (for a more detailed guide, please refer to this Kubernetes article).</p> <p>In this example, we'll concentrate to find where libvirt creates the files and directory in the <code>compute</code> container of the virt-launcher pod.</p> <pre><code>$ crictl ps |grep compute\n67bc7be3222da       5ef5ba25a087a80e204f28be6c9250bbf378fd87fa927085abd516188993d695                                                       25 minutes ago      Running             compute                   0                   7b045ea9f485f       virt-launcher-vmi-ephemeral-xg98p\n$ crictl inspect 67bc7be3222da\n[..]\n    \"mounts\": [\n      {\n      {\n        \"containerPath\": \"/var/run/libvirt\",\n        \"hostPath\": \"/var/lib/kubelet/pods/2ccc3e93-d1c3-4f22-bb31-321bfa74edf6/volumes/kubernetes.io~empty-dir/libvirt-runtime\",\n        \"propagation\": \"PROPAGATION_PRIVATE\",\n        \"readonly\": false,\n        \"selinuxRelabel\": true\n      },\n[..]\n$ ls /var/lib/kubelet/pods/2ccc3e93-d1c3-4f22-bb31-321bfa74edf6/volumes/kubernetes.io~empty-dir/libvirt-runtime/\ncommon      qemu         virtlogd-sock  virtqemud-admin-sock  virtqemud.conf\nhostdevmgr  virtlogd-admin-sock  virtlogd.pid   virtqemud-sock        virtqemud.pid\n</code></pre>"},{"location":"debug_virt_stack/virsh-commands/","title":"Execute virsh commands in virt-launcher pod","text":"<p>A powerful utility to check and troubleshoot the VM state is <code>virsh</code> and the utility is already installed in the <code>compute</code> container on the virt-launcher pod.</p> <p>For example, it possible to run any QMP commands.</p> <p>For a full list of QMP command, please refer to the QEMU documentation.</p> <pre><code>$ kubectl get po\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-xg98p   3/3     Running   0          44m\n$ kubectl exec -ti virt-launcher-vmi-debug-tools-fk64q -- bash\nbash-5.1$ virsh list\n Id   Name                      State\n-----------------------------------------\n 1    default_vmi-debug-tools   running\nbash-5.1$ virsh qemu-monitor-command default_vmi-debug-tools query-status --pretty\n{\n  \"return\": {\n    \"status\": \"running\",\n    \"singlestep\": false,\n    \"running\": true\n  },\n  \"id\": \"libvirt-439\"\n}\n$ virsh qemu-monitor-command default_vmi-debug-tools query-kvm --pretty\n{\n  \"return\": {\n    \"enabled\": true,\n    \"present\": true\n  },\n  \"id\": \"libvirt-438\"\n}\n</code></pre> <p>Another useful virsh command is the <code>qemu-monitor-event</code>. Once invoked, it observes and reports the QEMU events.</p> <p>The following example shows the events generated for pausing and unpausing the guest.</p> <pre><code>$ kubectl get po\nNAME                                READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-ephemeral-nqcld   3/3     Running   0          57m\n$ kubectl exec -ti virt-launcher-vmi-ephemeral-nqcld -- virsh qemu-monitor-event --pretty --loop\n</code></pre> <p>Then, you can, for example, pause and then unpause the guest and check the triggered events: <pre><code>$ virtctl pause vmi vmi-ephemeral\nVMI vmi-ephemeral was scheduled to pause\n $ virtctl unpause vmi vmi-ephemeral\nVMI vmi-ephemeral was scheduled to unpause\n</code></pre></p> <p>From the monitored events: <pre><code>$ kubectl exec -ti virt-launcher-vmi-ephemeral-nqcld -- virsh qemu-monitor-event --pretty --loop\nevent STOP at 1698405797.422823 for domain 'default_vmi-ephemeral': &lt;null&gt;\nevent RESUME at 1698405823.162458 for domain 'default_vmi-ephemeral': &lt;null&gt;\n</code></pre></p>"},{"location":"network/dns/","title":"DNS records","text":"<p>In order to create unique DNS records per VirtualMachineInstance, it is possible to set <code>spec.hostname</code> and <code>spec.subdomain</code>. If a subdomain is set and a headless service with a name, matching the subdomain, exists, kube-dns will create unique DNS entries for every VirtualMachineInstance which matches the selector of the service. Have a look at the DNS for Services and Pods documentation for additional information.</p> <p>The following example consists of a VirtualMachine and a headless Service which matches the labels and the subdomain of the VirtualMachineInstance:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vmi-fedora\n  labels:\n    expose: me\nspec:\n  hostname: \"myvmi\"\n  subdomain: \"mysubdomain\"\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-registry-disk-demo:latest\n  - cloudInitNoCloud:\n      userDataBase64: IyEvYmluL2Jhc2gKZWNobyAiZmVkb3JhOmZlZG9yYSIgfCBjaHBhc3N3ZAo=\n    name: cloudinitdisk\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: mysubdomain\nspec:\n  selector:\n    expose: me\n  clusterIP: None\n  ports:\n  - name: foo # Actually, no port is needed.\n    port: 1234\n    targetPort: 1234\n</code></pre> <p>As a consequence, when we enter the VirtualMachineInstance via e.g. <code>virtctl console vmi-fedora</code> and ping <code>myvmi.mysubdomain</code> we see that we find a DNS entry for <code>myvmi.mysubdomain.default.svc.cluster.local</code> which points to <code>10.244.0.57</code>, which is the IP of the VirtualMachineInstance (not of the Service):</p> <pre><code>[fedora@myvmi ~]$ ping myvmi.mysubdomain\nPING myvmi.mysubdomain.default.svc.cluster.local (10.244.0.57) 56(84) bytes of data.\n64 bytes from myvmi.mysubdomain.default.svc.cluster.local (10.244.0.57): icmp_seq=1 ttl=64 time=0.029 ms\n[fedora@myvmi ~]$ ip a\n2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel state UP group default qlen 1000\n    link/ether 0a:58:0a:f4:00:39 brd ff:ff:ff:ff:ff:ff\n    inet 10.244.0.57/24 brd 10.244.0.255 scope global dynamic eth0\n       valid_lft 86313556sec preferred_lft 86313556sec\n    inet6 fe80::858:aff:fef4:39/64 scope link\n       valid_lft forever preferred_lft forever\n</code></pre> <p>So <code>spec.hostname</code> and <code>spec.subdomain</code> get translated to a DNS A-record of the form <code>&lt;vmi.spec.hostname&gt;.&lt;vmi.spec.subdomain&gt;.&lt;vmi.metadata.namespace&gt;.svc.cluster.local</code>. If no <code>spec.hostname</code> is set, then we fall back to the VirtualMachineInstance name itself. The resulting DNS A-record looks like this then: <code>&lt;vmi.metadata.name&gt;.&lt;vmi.spec.subdomain&gt;.&lt;vmi.metadata.namespace&gt;.svc.cluster.local</code>.</p>"},{"location":"network/hotplug_interfaces/","title":"Hotplug Network Interfaces","text":"<p>Release: - v1.1.0: Alpha - v1.3.0: Beta</p> <p>KubeVirt supports hotplugging and unplugging network interfaces into a running Virtual Machine (VM). </p> <p>Hotplug is supported for interfaces using the <code>virtio</code> model connected through bridge binding  or SR-IOV binding.</p> <p>Hot-unplug is supported only for interfaces connected through bridge binding.</p>"},{"location":"network/hotplug_interfaces/#requirements","title":"Requirements","text":"<p>Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:</p> <ul> <li>Multus Dynamic Networks Controller:   this daemon will listen to annotation changes, and trigger Multus to configure   a new attachment for this pod.</li> <li>Multus CNI running as a thick plugin:   this Multus version exposes an endpoint to create attachments for a given pod   on demand.</li> </ul>"},{"location":"network/hotplug_interfaces/#enabling-network-interface-hotplug-support","title":"Enabling network interface hotplug support","text":"<p>Network interface hotplug support must be enabled via a feature gate. The feature gates array in the KubeVirt CR must feature <code>HotplugNICs</code>.</p>"},{"location":"network/hotplug_interfaces/#adding-an-interface-to-a-running-vm","title":"Adding an interface to a running VM","text":"<p>First start a VM. You can refer to the following example: <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          interfaces:\n          - masquerade: {}\n            name: defaultnetwork\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: defaultnetwork\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n</code></pre></p> <p>You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one: <pre><code>apiVersion: k8s.cni.cncf.io/v1\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: new-fancy-net\nspec:\n    config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"type\": \"bridge\",\n      \"mtu\": 1300,\n      \"name\":\"new-fancy-net\"\n    }'\n</code></pre> Please refer to the Multus documentation for more information.</p> <p>Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation by  editing the VM spec template and adding the desired interface and network: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: dyniface1\n          bridge: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: dyniface1\n      multus:\n        networkName: new-fancy-net\n ...\n</code></pre></p> <p>Note: <code>virtctl</code> <code>addinterface</code> and <code>removeinterface</code> commands are no longer available, hotplug/unplug interfaces is done by editing the VM spec template.</p> <p>The interface and network will be added to the corresponding VMI object as well by Kubevirt.</p> <p>You can now check the VMI status for the presence of this new interface: <pre><code>kubectl get vmi vm-fedora -ojsonpath=\"{ @.status.interfaces }\"\n</code></pre></p>"},{"location":"network/hotplug_interfaces/#removing-an-interface-from-a-running-vm","title":"Removing an interface from a running VM","text":"<p>Following the example above, the user can request an interface unplug operation by editing the VM spec template and set the desired interface state to <code>absent</code>: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n          - name: defaultnetwork\n            masquerade: {}\n          # set the interface state to absent \n          - name: dyniface1\n            state: absent\n            bridge: {}\n    networks:\n      - name: defaultnetwork\n        pod: {}\n      - name: dyniface1\n        multus:\n          networkName: new-fancy-net\n</code></pre> The interface in the corresponding VMI object will be set with state 'absent' as well by Kubevirt.</p> <p>Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.</p>"},{"location":"network/hotplug_interfaces/#migration-based-hotplug","title":"Migration based hotplug","text":"<p>In case your cluster doesn't run Multus as  thick plugin  and  Multus Dynamic Networks controller,  it's possible to hotplug an interface by migrating the VM.</p> <p>The actual attachment won't take place immediately, and the new interface will be available in the guest once the migration is completed.</p>"},{"location":"network/hotplug_interfaces/#add-new-interface","title":"Add new interface","text":"<p>Add the desired interface and network to the VM spec template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: dyniface1\n          bridge: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: dyniface1\n      multus:\n        networkName: new-fancy-net\n ...\n</code></pre></p> <p>At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the migration is completed the VM will have the new interface attached.</p> <p>Note: It is recommended to avoid performing migrations in parallel to a hotplug operation.  It is safer to assure hotplug succeeded or at least reached the VMI specification before issuing a migration.</p>"},{"location":"network/hotplug_interfaces/#remove-interface","title":"Remove interface","text":"<p>Set the desired interface state to <code>absent</code> in the VM spec template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n          - name: defaultnetwork\n            masquerade: {}\n          # set the interface state to absent \n          - name: dyniface1\n            state: absent\n            bridge: {}\n    networks:\n      - name: defaultnetwork\n        pod: {}\n      - name: dyniface1\n        multus:\n          networkName: new-fancy-net\n</code></pre></p> <p>At this point the subject interface should be detached from the guest but exist in the pod.</p> <p>Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm_1","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the VM is migrated, the interface will not exist in the migration target pod.</p> <p>Note: It is recommended to avoid performing migrations in parallel to an unplug operation. It is safer to assure unplug succeeded or at least reached the VMI specification before issuing a migration.</p>"},{"location":"network/hotplug_interfaces/#sr-iov-interfaces","title":"SR-IOV interfaces","text":"<p>Kubevirt supports hot-plugging of SR-IOV interfaces to running VMs.</p> <p>Similar to bridge binding interfaces, edit the VM spec template and add the desired SR-IOV interface and network: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: vm-fedora\ntemplate:\n  spec:\n    domain:\n      devices:\n        interfaces:\n        - name: defaultnetwork\n          masquerade: {}\n          # new interface\n        - name: sriov-net\n          sriov: {}\n    networks:\n    - name: defaultnetwork\n      pod: {}\n      # new network\n    - name: sriov-net\n      multus:\n        networkName: sriov-net-1\n ...\n</code></pre> Please refer to the Interface and Networks documentation for more information about SR-IOV networking.</p> <p>At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.</p>"},{"location":"network/hotplug_interfaces/#migrate-the-vm_2","title":"Migrate the VM","text":"<p><pre><code>cat &lt;&lt;EOF kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceMigration\nmetadata:\n  name: migration-job\nspec:\n  vmiName: vmi-fedora\nEOF\n</code></pre> Please refer to the Live Migration documentation for more information.</p> <p>Once the VM is migrated, the interface will not exist in the migration target pod. Due to limitation of Kubernetes device plugin API to allocate resources dynamically, the SR-IOV device plugin cannot allocate additional SR-IOV resources for Kubevirt to hotplug. Thus, SR-IOV interface hotplug is limited to migration based hotplug only, regardless of Multus \"thick\" version.</p>"},{"location":"network/hotplug_interfaces/#virtio-limitations","title":"Virtio Limitations","text":"<p>The hotplugged interfaces have <code>model: virtio</code>. This imposes several limitations: each interface will consume a PCI slot in the VM, and there are a total maximum of 32. Furthermore, other devices will also use these PCI slots (e.g. disks, guest-agent, etc).</p> <p>Kubevirt reserves resources for 4 interface to allow later hotplug operations. The actual maximum amount of available resources depends on the machine type (e.g. q35 adds another PCI slot). For more information on maximum limits, see libvirt documentation.</p> <p>Yet, upon a VM restart, the hotplugged interface will become part of the standard networks; this mitigates the maximum hotplug interfaces (per machine type) limitation.</p> <p>Note: The user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.</p>"},{"location":"network/interfaces_and_networks/","title":"Interfaces and Networks","text":"<p>Connecting a virtual machine to a network consists of two parts. First, networks are specified in <code>spec.networks</code>. Then, interfaces backed by the networks are added to the VM by specifying them in <code>spec.domain.devices.interfaces</code>.</p> <p>Each interface must have a corresponding network with the same name.</p> <p>An <code>interface</code> defines a virtual network interface of a virtual machine (also called a frontend). A <code>network</code> specifies the backend of an <code>interface</code> and declares which logical or physical device it is connected to (also called as backend).</p> <p>There are multiple ways of configuring an <code>interface</code> as well as a <code>network</code>.</p> <p>All possible configuration options are available in the Interface API Reference and Network API Reference.</p>"},{"location":"network/interfaces_and_networks/#backend","title":"Backend","text":"<p>Network backends are configured in <code>spec.networks</code>. A network must have a unique name. Additional fields declare which logical or physical device the network relates to.</p> <p>Each network should declare its type by defining one of the following fields:</p> Type Description <p><code>pod</code></p> <p>Default Kubernetes network</p> <p><code>multus</code></p> <p>Secondary network provided using Multus</p>"},{"location":"network/interfaces_and_networks/#pod","title":"pod","text":"<p>A <code>pod</code> network represents the default pod <code>eth0</code> interface configured by cluster network solution that is present in each pod.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n  networks:\n  - name: default\n    pod: {} # Stock pod network\n</code></pre>"},{"location":"network/interfaces_and_networks/#multus","title":"multus","text":"<p>It is also possible to connect VMIs to secondary networks using Multus. This assumes that multus is installed across your cluster and a corresponding <code>NetworkAttachmentDefinition</code> CRD was created.</p> <p>The following example defines a network which uses the bridge CNI plugin, which will connect the VMI to Linux bridge <code>br1</code>. Other CNI plugins such as ptp, ovs-cni, or Flannel might be used as well. For their installation and usage refer to the respective project documentation.</p> <p>First the <code>NetworkAttachmentDefinition</code> needs to be created. That is usually done by an administrator. Users can then reference the definition.</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: bridge-test\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"bridge-test\",\n      \"type\": \"bridge\",\n      \"bridge\": \"br1\",\n      \"disableContainerInterface\": true\n    }'\n</code></pre> <p>With following definition, the VMI will be connected to the default pod network and to the secondary Open vSwitch network.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n          bootOrder: 1   # attempt to boot from an external tftp server\n          dhcpOptions:\n            bootFileName: default_image.bin\n            tftpServerName: tftp.example.com\n        - name: ovs-net\n          bridge: {}\n          bootOrder: 2   # if first attempt failed, try to PXE-boot from this L2 networks\n  networks:\n  - name: default\n    pod: {} # Stock pod network\n  - name: ovs-net\n    multus: # Secondary multus network\n      networkName: ovs-vlan-100\n</code></pre> <p>It is also possible to define a multus network as the default pod network with Multus. A version of multus after this Pull Request is required (currently master).</p> <p>Note the following:</p> <ul> <li> <p>A multus default network and a pod network type are mutually     exclusive.</p> </li> <li> <p>The virt-launcher pod that starts the VMI will not have the pod     network configured.</p> </li> <li> <p>The multus delegate chosen as default must return at least one     IP address.</p> </li> </ul> <p>Create a <code>NetworkAttachmentDefinition</code> with IPAM.</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: bridge-test\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"bridge-test\",\n      \"type\": \"bridge\",\n      \"bridge\": \"br1\",\n      \"ipam\": {\n        \"type\": \"host-local\",\n        \"subnet\": \"10.250.250.0/24\"\n      }\n    }'\n</code></pre> <p>Define a VMI with a Multus network as the default.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: test1\n          bridge: {}\n  networks:\n  - name: test1\n    multus: # Multus network as default\n      default: true\n      networkName: bridge-test\n</code></pre>"},{"location":"network/interfaces_and_networks/#invalid-cnis-for-secondary-networks","title":"Invalid CNIs for secondary networks","text":"<p>The following list of CNIs is known not to work for bridge interfaces - which are most common for secondary interfaces.</p> <ul> <li> <p>macvlan</p> </li> <li> <p>ipvlan</p> </li> </ul> <p>The reason is similar: the bridge interface type moves the pod interface MAC address to the VM, leaving the pod interface with a different address. The aforementioned CNIs require the pod interface to have the original MAC address.</p> <p>These issues are tracked individually:</p> <ul> <li> <p>macvlan</p> </li> <li> <p>ipvlan</p> </li> </ul> <p>Feel free to discuss and / or propose fixes for them; we'd like to have these plugins as valid options on our ecosystem.</p>"},{"location":"network/interfaces_and_networks/#frontend","title":"Frontend","text":"<p>Network interfaces are configured in <code>spec.domain.devices.interfaces</code>. They describe properties of virtual interfaces as \"seen\" inside guest instances. The same network backend may be connected to a virtual machine in multiple different ways, each with their own connectivity guarantees and characteristics.</p> <p>Each interface should declare its type by defining on of the following fields:</p> Type Description <p><code>bridge</code></p> <p>Connect using a linux bridge</p> <p><code>slirp</code></p> <p>Connect using QEMU user networking mode</p> <p><code>sriov</code></p> <p>Pass through a SR-IOV PCI device via <code>vfio</code></p> <p><code>masquerade</code></p> <p>Connect using Iptables rules to nat the traffic</p> <p>Each interface may also have additional configuration fields that modify properties \"seen\" inside guest instances, as listed below:</p> Name Format Default value Description <p><code>model</code></p> <p>One of: <code>e1000</code>, <code>e1000e</code>, <code>ne2k_pci</code>, <code>pcnet</code>, <code>rtl8139</code>, <code>virtio</code></p> <p><code>virtio</code></p> <p>NIC type</p> <p>macAddress</p> <p><code>ff:ff:ff:ff:ff:ff</code> or <code>FF-FF-FF-FF-FF-FF</code></p> <p>MAC address as seen inside the guest system, for example: <code>de:ad:00:00:be:af</code></p> <p>ports</p> <p>empty</p> <p>List of ports to be forwarded to the virtual machine.</p> <p>pciAddress</p> <p><code>0000:81:00.1</code></p> <p>Set network interface PCI address, for example: <code>0000:81:00.1</code></p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          model: e1000 # expose e1000 NIC to the guest\n          masquerade: {} # connect through a masquerade\n          ports:\n           - name: http\n             port: 80\n  networks:\n  - name: default\n    pod: {}\n</code></pre> <p>Note: For secondary interfaces, when a MAC address is specified for a virtual machine interface, it is passed to the underlying CNI plugin which is, in turn, expected to configure the backend to allow for this particular MAC. Not every plugin has native support for custom MAC addresses.</p> <p>Note: For some CNI plugins without native support for custom MAC addresses, there is a workaround, which is to use the <code>tuning</code> CNI plugin to adjust pod interface MAC address. This can be used as follows:</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: ptp-mac\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"ptp-mac\",\n      \"plugins\": [\n        {\n          \"type\": \"ptp\",\n          \"ipam\": {\n            \"type\": \"host-local\",\n            \"subnet\": \"10.1.1.0/24\"\n          }\n        },\n        {\n          \"type\": \"tuning\"\n        }\n      ]\n    }'\n</code></pre> <p>This approach may not work for all plugins. For example, OKD SDN is not compatible with <code>tuning</code> plugin.</p> <ul> <li> <p>Plugins that handle custom MAC addresses natively: <code>ovs</code>, <code>bridge</code>.</p> </li> <li> <p>Plugins that are compatible with <code>tuning</code> plugin: <code>flannel</code>, <code>ptp</code>.</p> </li> <li> <p>Plugins that don't need special MAC address treatment: <code>sriov</code> (in     <code>vfio</code> mode). </p> </li> </ul>"},{"location":"network/interfaces_and_networks/#ports","title":"Ports","text":"<p>Declare ports listen by the virtual machine</p> <p>Note: When using the slirp interface only the configured ports will be forwarded to the virtual machine.</p> Name Format Required Description <p><code>name</code></p> <p>no</p> <p>Name</p> <p><code>port</code></p> <p>1 - 65535</p> <p>yes</p> <p>Port to expose</p> <p><code>protocol</code></p> <p>TCP,UDP</p> <p>no</p> <p>Connection protocol</p> <p>Tip: Use <code>e1000</code> model if your guest image doesn't ship with virtio drivers.</p> <p>If <code>spec.domain.devices.interfaces</code> is omitted, the virtual machine is connected using the default pod network interface of <code>bridge</code> type. If you'd like to have a virtual machine instance without any network connectivity, you can use the <code>autoattachPodInterface</code> field as follows:</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      autoattachPodInterface: false\n</code></pre>"},{"location":"network/interfaces_and_networks/#mtu","title":"MTU","text":"<p>There are two methods for the MTU to be propagated to the guest interface.</p> <ul> <li>Libvirt - for this the guest machine needs new enough virtio network driver that understands the data passed into the guest via a PCI config register in the emulated device.</li> <li>DHCP - for this the guest DHCP client should be able to read the MTU from the DHCP server response.</li> </ul> <p>On Windows guest non virtio interfaces, MTU has to be set manually using <code>netsh</code> or other tool since the Windows DHCP client doesn't request/read the MTU.</p> <p>The table below is summarizing the MTU propagation to the guest.</p> masquerade bridge with CNI IP bridge with no CNI IP Windows virtio DHCP &amp; libvirt DHCP &amp; libvirt libvirt libvirt non-virtio DHCP DHCP X X <ul> <li>bridge with CNI IP - means the CNI gives IP to the pod interface and bridge binding is used to bind the pod interface to the guest.</li> </ul>"},{"location":"network/interfaces_and_networks/#bridge","title":"bridge","text":"<p>In <code>bridge</code> mode, virtual machines are connected to the network backend through a linux \"bridge\". The pod network IPv4 address (if exists) is delegated to the virtual machine via DHCPv4. The virtual machine should be configured to use DHCP to acquire IPv4 addresses.</p> <p>Note: If a specific MAC address is not configured in the virtual machine interface spec the MAC address from the relevant pod interface is delegated to the virtual machine.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          bridge: {} # connect through a bridge\n  networks:\n  - name: red\n    multus:\n      networkName: red\n</code></pre> <p>At this time, <code>bridge</code> mode doesn't support additional configuration fields.</p> <p>Note: due to IPv4 address delegation, in <code>bridge</code> mode the pod doesn't have an IP address configured, which may introduce issues with third-party solutions that may rely on it. For example, Istio may not work in this mode.</p> <p>Note: admin can forbid using <code>bridge</code> interface type for pod networks via a designated configuration flag. To achieve it, the admin should set the following option to <code>false</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  configuration:\n    network:\n      permitBridgeInterfaceOnPodNetwork: false\n</code></pre> <p>Note: binding the pod network using <code>bridge</code> interface type may cause issues. Other than the third-party issue mentioned in the above note, live migration is not allowed with a pod network binding of <code>bridge</code> interface type, and also some CNI plugins might not allow to use a custom MAC address for your VM instances. If you think you may be affected by any of issues mentioned above, consider changing the default interface type to <code>masquerade</code>, and disabling the <code>bridge</code> type for pod network, as shown in the example above.</p>"},{"location":"network/interfaces_and_networks/#slirp","title":"slirp","text":"<p>In <code>slirp</code> mode, virtual machines are connected to the network backend using QEMU user networking mode. In this mode, QEMU allocates internal IP addresses to virtual machines and hides them behind NAT.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          slirp: {} # connect using SLIRP mode\n  networks:\n  - name: red\n    pod: {}\n</code></pre> <p>At this time, <code>slirp</code> mode doesn't support additional configuration fields.</p> <p>Note: in <code>slirp</code> mode, the only supported protocols are TCP and UDP. ICMP is not supported.</p> <p>More information about SLIRP mode can be found in QEMU Wiki.</p> <p>Note: Since v1.1.0, Kubevirt delegates Slirp network configuration to the Slirp network binding plugin by default. In case the binding plugin is not registered, Kubevirt will use the following default image: <code>quay.io/kubevirt/network-slirp-binding:20230830_638c60fc8</code>.</p> <p>Note: In the next release (v1.2.0) no default image will be set by Kubevirt, registering an image will be mandatory.</p> <p>Note: On disconnected clusters it will be necessary to mirror Slirp binding plugin image to the cluster registry.</p>"},{"location":"network/interfaces_and_networks/#masquerade","title":"masquerade","text":"<p>In <code>masquerade</code> mode, KubeVirt allocates internal IP addresses to virtual machines and hides them behind NAT. All the traffic exiting virtual machines is \"source NAT'ed\" using pod IP addresses; thus, cluster workloads should use the pod's IP address to contact the VM over this interface. This IP address is reported in the VMI's <code>spec.status.interface</code>. A guest operating system should be configured to use DHCP to acquire IPv4 addresses.</p> <p>To allow the VM to live-migrate or hard restart (both cause the VM to run on a different pod, with a different IP address) and still be reachable, it should be exposed by a Kubernetes service.</p> <p>To allow traffic of specific ports into virtual machines, the template <code>ports</code> section of the interface should be configured as follows. If the <code>ports</code> section is missing, all ports forwarded into the VM.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          masquerade: {} # connect using masquerade mode\n          ports:\n            - port: 80 # allow incoming traffic on port 80 to get into the virtual machine\n  networks:\n  - name: red\n    pod: {}\n</code></pre> <p>Note: Masquerade is only allowed to connect to the pod network.</p> <p>Note: The network CIDR can be configured in the pod network section using the <code>vmNetworkCIDR</code> attribute.</p>"},{"location":"network/interfaces_and_networks/#masquerade-ipv4-and-ipv6-dual-stack-support","title":"masquerade - IPv4 and IPv6 dual-stack support","text":"<p><code>masquerade</code> mode can be used in IPv4 and IPv6 dual-stack clusters to provide a VM with an IP connectivity over both protocols.</p> <p>As with the IPv4 <code>masquerade</code> mode, the VM can be contacted using the pod's IP address - which will be in this case two IP addresses, one IPv4 and one IPv6. Outgoing traffic is also \"NAT'ed\" to the pod's respective IP address from the given family.</p> <p>Unlike in IPv4, the configuration of the IPv6 address and the default route is not automatic; it should be configured via cloud init, as shown below:</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      disks:\n        - disk:\n          bus: virtio\n          name: cloudinitdisk\n      interfaces:\n        - name: red\n          masquerade: {} # connect using masquerade mode\n          ports:\n            - port: 80 # allow incoming traffic on port 80 to get into the virtual machine\n  networks:\n  - name: red\n    pod: {}\n  volumes:\n  - cloudInitNoCloud:\n      networkData: |\n        version: 2\n        ethernets:\n          eth0:\n            dhcp4: true\n            addresses: [ fd10:0:2::2/120 ]\n            gateway6: fd10:0:2::1\n      userData: |-\n        #!/bin/bash\n        echo \"fedora\" |passwd fedora --stdin\n</code></pre> <p>Note: The IPv6 address for the VM and default gateway must be the ones shown above.</p>"},{"location":"network/interfaces_and_networks/#masquerade-ipv6-single-stack-support","title":"masquerade - IPv6 single-stack support","text":"<p><code>masquerade</code> mode can be used in IPv6 single stack clusters to provide a VM with an IPv6 only connectivity.</p> <p>As with the IPv4 <code>masquerade</code> mode, the VM can be contacted using the pod's IP address - which will be in this case the IPv6 one. Outgoing traffic is also \"NAT'ed\" to the pod's respective IPv6 address.</p> <p>As with the dual-stack cluster, the configuration of the IPv6 address and the default route is not automatic; it should be configured via cloud init, as shown in the dual-stack section.</p> <p>Unlike the dual-stack cluster, which has a DHCP server for IPv4, the IPv6 single stack cluster has no DHCP server at all. Therefore, the VM won't have the search domains information and reaching a destination using its FQDN is not possible. Tracking issue - https://github.com/kubevirt/kubevirt/issues/7184</p>"},{"location":"network/interfaces_and_networks/#passt","title":"passt","text":"<p>Warning: The core binding is being deprecated and targeted for removal in v1.3 . As an alternative, the same functionality is introduced and available as a binding plugin.</p> <p><code>passt</code> is a new approach for user-mode networking which can be used as a simple replacement for Slirp (which is practically dead).</p> <p><code>passt</code> is a universal tool which implements a translation layer between a Layer-2 network interface and native Layer -4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host.</p> <p>Its main benefits are: - doesn't require extra network capabilities as CAP_NET_RAW and CAP_NET_ADMIN. - allows integration with service meshes (which expect applications to run locally) out of the box. - supports IPv6 out of the box (in contrast to the existing bindings which require configuring IPv6 manually).</p> Masquerade Bridge Passt Supports migration Yes No No(will be supported in the future) VM uses Pod IP No Yes Yes(in the future it will be possible to configure the VM IP. Currently the default is the pod IP) Service Mesh out of the box No(only ISTIO is supported, adjustmets on both ISTIO and kubevirt had to be done to make it work) No Yes Doesn\u2019t require extra capabilities on the virt-launcher pod Yes(multiple workarounds had to be added to kuebivrt to make it work) No(Multiple workarounds had to be added to kuebivrt to make it work) Yes Doesn't require extra network devices on the virt-launcher pod No(bridge and tap device are created) No(bridge and tap device are created) Yes Supports IPv6 Yes(requires manual configuration on the VM) No Yes <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: red\n          passt: {} # connect using passt mode\n          ports:\n            - port: 8080 # allow incoming traffic on port 8080 to get into the virtual machine\n  networks:\n    - name: red\n      pod: {}\n</code></pre>"},{"location":"network/interfaces_and_networks/#requirementsrecommendations","title":"Requirements/Recommendations:","text":"<ol> <li>To get better performance the node should be configured with: <pre><code>sysctl -w net.core.rmem_max = 33554432\nsysctl -w net.core.wmem_max = 33554432\n</code></pre></li> <li>To run multiple passt VMs with no explicit ports, the node's <code>fs.file-max</code> should be increased    (for a VM forwards all IPv4 and IPv6 ports, for TCP and UDP, passt needs to create ~2^18 sockets): <pre><code>sysctl -w fs.file-max = 9223372036854775807\n</code></pre></li> </ol> <p>NOTE: To achieve optimal memory consumption with Passt binding, specify ports required for your workload. When no ports are explicitly specified, all ports are forwarded, leading to memory overhead of up to 800 Mi.</p>"},{"location":"network/interfaces_and_networks/#temporary-restrictions","title":"Temporary restrictions:","text":"<ol> <li><code>passt</code> currently only supported as primary network and doesn't allow extra multus networks to be configured on the VM.</li> </ol> <p>passt interfaces are feature gated; to enable the feature, follow these instructions, in order to activate the <code>Passt</code> feature gate (case sensitive).</p> <p>More information about passt mode can be found in passt Wiki.</p>"},{"location":"network/interfaces_and_networks/#virtio-net-multiqueue","title":"virtio-net multiqueue","text":"<p>Setting the <code>networkInterfaceMultiqueue</code> to <code>true</code> will enable the multi-queue functionality, increasing the number of vhost queue, for interfaces configured with a <code>virtio</code> model.</p> <pre><code>kind: VM\nspec:\n  domain:\n    devices:\n      networkInterfaceMultiqueue: true\n</code></pre> <p>Users of a Virtual Machine with multiple vCPUs may benefit of increased network throughput and performance.</p> <p>Currently, the number of queues is being determined by the number of vCPUs of a VM. This is because multi-queue support optimizes RX interrupt affinity and TX queue selection in order to make a specific queue private to a specific vCPU.</p> <p>Without enabling the feature, network performance does not scale as the number of vCPUs increases. Guests cannot transmit or retrieve packets in parallel, as virtio-net has only one TX and RX queue.</p> <p>Virtio interfaces advertise on their status.interfaces.interface entry a field named queueCount. The queueCount field indicates how many queues were assigned to the interface. Queue count value is derived from the domain XML. In case the number of queues can't be determined (i.e interface that is reported by quest-agent only), it will be omitted.</p> <p>NOTE: Although the virtio-net multiqueue feature provides a performance benefit, it has some limitations and therefore should not be unconditionally enabled</p>"},{"location":"network/interfaces_and_networks/#some-known-limitations","title":"Some known limitations","text":"<ul> <li> <p>Guest OS is limited to ~200 MSI vectors. Each NIC queue requires a     MSI vector, as well as any virtio device or assigned PCI device.     Defining an instance with multiple virtio NICs and vCPUs might lead     to a possibility of hitting the guest MSI limit.</p> </li> <li> <p>virtio-net multiqueue works well for incoming traffic, but can     occasionally cause a performance degradation, for outgoing traffic.     Specifically, this may occur when sending packets under 1,500 bytes     over the Transmission Control Protocol (TCP) stream.</p> </li> <li> <p>Enabling virtio-net multiqueue increases the total network     throughput, but in parallel it also increases the CPU consumption.</p> </li> <li> <p>Enabling virtio-net multiqueue in the host QEMU config, does not     enable the functionality in the guest OS. The guest OS administrator     needs to manually turn it on for each guest NIC that requires this     feature, using ethtool.</p> </li> <li> <p>MSI vectors would still be consumed (wasted), if multiqueue was     enabled in the host, but has not been enabled in the guest OS by the     administrator.</p> </li> <li> <p>In case the number of vNICs in a guest instance is proportional to     the number of vCPUs, enabling the multiqueue feature is less     important.</p> </li> <li> <p>Each virtio-net queue consumes 64 KiB of kernel memory for the vhost     driver.</p> </li> </ul> <p>NOTE: Virtio-net multiqueue should be enabled in the guest OS manually, using ethtool. For example: <code>ethtool -L &lt;NIC&gt; combined #num_of_queues</code></p> <p>More information please refer to KVM/QEMU MultiQueue.</p>"},{"location":"network/interfaces_and_networks/#sriov","title":"sriov","text":"<p>In <code>sriov</code> mode, virtual machines are directly exposed to an SR-IOV PCI device, usually allocated by Intel SR-IOV device plugin. The device is passed through into the guest operating system as a host device, using the vfio userspace interface, to maintain high networking performance.</p>"},{"location":"network/interfaces_and_networks/#how-to-expose-sr-iov-vfs-to-kubevirt","title":"How to expose SR-IOV VFs to KubeVirt","text":"<p>To simplify procedure, please use SR-IOV network operator to deploy and configure SR-IOV components in your cluster. On how to use the operator, please refer to their respective documentation.</p> <p>Note: KubeVirt relies on VFIO userspace driver to pass PCI devices into VMI guest. Because of that, when configuring SR-IOV operator policies, make sure you define a pool of VF resources that uses <code>deviceType: vfio-pci</code>.</p> <p>Once the operator is deployed, an SriovNetworkNodePolicy  must be provisioned, in which the list of SR-IOV devices to expose (with respective configurations) is defined.</p> <p>Please refer to the following <code>SriovNetworkNodePolicy</code> for an example:</p> <pre><code>apiVersion: sriovnetwork.openshift.io/v1\nkind: SriovNetworkNodePolicy\nmetadata:\n  name: policy-1\n  namespace: sriov-network-operator\nspec:\n  deviceType: vfio-pci\n  mtu: 9000\n  nicSelector:\n    pfNames:\n    - ens1f0\n  nodeSelector:\n    sriov: \"true\"\n  numVfs: 8\n  priority: 90\n  resourceName: sriov-nic\n</code></pre> <p>The policy above will configure the <code>SR-IOV</code> device plugin, allowing the PF named <code>ens1f0</code> to be exposed in the SRIOV capable nodes as a resource named <code>sriov-nic</code>.</p>"},{"location":"network/interfaces_and_networks/#start-an-sr-iov-vm","title":"Start an SR-IOV VM","text":"<p>Once all the SR-IOV components are deployed, it is needed to indicate how to configure the SR-IOV network. Refer to the following <code>SriovNetwork</code> for an example:</p> <pre><code>apiVersion: sriovnetwork.openshift.io/v1\nkind: SriovNetwork\nmetadata:\n  name: sriov-net\n  namespace: sriov-network-operator\nspec:\n  ipam: |\n    {}\n  networkNamespace: default\n  resourceName: sriov-nic\n  spoofChk: \"off\"\n</code></pre> <p>Finally, to create a VM that will attach to the aforementioned Network, refer to the following VMI spec:</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-perf\n  name: vmi-perf\nspec:\n  domain:\n    cpu:\n      sockets: 2\n      cores: 1\n      threads: 1\n      dedicatedCpuPlacement: true\n    resources:\n      requests:\n        memory: \"4Gi\"\n      limits:\n        memory: \"4Gi\"\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      interfaces:\n      - masquerade: {}\n        name: default\n      - name: sriov-net\n        sriov: {}\n      rng: {}\n    machine:\n      type: \"\"\n  networks:\n  - name: default\n    pod: {}\n  - multus:\n      networkName: default/sriov-net\n    name: sriov-net\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: docker.io/kubevirt/fedora-cloud-container-disk-demo:latest\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |\n        #!/bin/bash\n        echo \"centos\" |passwd centos --stdin\n        dhclient eth1\n    name: cloudinitdisk\n</code></pre> <p>Note: for some NICs (e.g. Mellanox), the kernel module needs to be installed in the guest VM.</p> <p>Note: Placement on dedicated CPUs can only be achieved if the Kubernetes CPU manager is running on the SR-IOV capable workers. For further details please refer to the dedicated cpu resources documentation.</p>"},{"location":"network/interfaces_and_networks/#macvtap","title":"Macvtap","text":"<p>Note: The core binding will be deprecated soon. As an alternative, the same functionality is introduced and available as a binding plugin.</p> <p>In <code>macvtap</code> mode, virtual machines are directly exposed to the Kubernetes nodes L2 network. This is achieved by 'extending' an existing network interface with a virtual device that has its own MAC address.</p> <p>Macvtap interfaces are feature gated; to enable the feature, follow these instructions, in order to activate the <code>Macvtap</code> feature gate (case sensitive).</p> <p>Note: On KinD clusters, the user needs to adjust the cluster configuration, mounting <code>dev</code> of the running host onto the KinD nodes, because of a known issue.</p>"},{"location":"network/interfaces_and_networks/#limitations","title":"Limitations","text":"<ul> <li>Live migration is not seamless, see issue #5912</li> </ul>"},{"location":"network/interfaces_and_networks/#how-to-expose-host-interface-to-the-macvtap-device-plugin","title":"How to expose host interface to the macvtap device plugin","text":"<p>To simplify the procedure, please use the Cluster Network Addons Operator to deploy and configure the macvtap components in your cluster.</p> <p>The aforementioned operator effectively deploys the macvtap-cni cni / device plugin combo.</p> <p>There are two different alternatives to configure which host interfaces get exposed to the user, enabling them to create macvtap interfaces on top of:</p> <ul> <li>select the host interfaces: indicates which host interfaces are exposed.</li> <li>expose all interfaces: all interfaces of all hosts are exposed.</li> </ul> <p>Both options are configured via the <code>macvtap-deviceplugin-config</code> ConfigMap, and more information on how to configure it can be found in the macvtap-cni repo.</p> <p>You can find a minimal example, in which the <code>eth0</code> interface of the Kubernetes nodes is exposed, via the <code>lowerDevice</code> attribute. <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: |\n    [\n        {\n            \"name\"       : \"dataplane\",\n            \"lowerDevice\": \"eth0\",\n            \"mode\"       : \"bridge\",\n            \"capacity\"   : 50\n        }\n    ]\n</code></pre></p> <p>This step can be omitted, since the default configuration of the aforementioned <code>ConfigMap</code> is to expose all host interfaces (which is represented by the following configuration): <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: '[]'\n</code></pre></p>"},{"location":"network/interfaces_and_networks/#start-a-vm-with-macvtap-interfaces","title":"Start a VM with macvtap interfaces","text":"<p>Once the macvtap components are deployed, it is needed to indicate how to configure the macvtap network. Refer to the following <code>NetworkAttachmentDefinition</code> for a simple example:</p> <p><pre><code>---\nkind: NetworkAttachmentDefinition\napiVersion: k8s.cni.cncf.io/v1\nmetadata:\n  name: macvtapnetwork\n  annotations:\n    k8s.v1.cni.cncf.io/resourceName: macvtap.network.kubevirt.io/eth0\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"macvtapnetwork\",\n      \"type\": \"macvtap\",\n      \"mtu\": 1500\n    }'\n</code></pre> The requested <code>k8s.v1.cni.cncf.io/resourceName</code> annotation must point to an exposed host interface (via the <code>lowerDevice</code> attribute, on the <code>macvtap-deviceplugin-config</code> <code>ConfigMap</code>).</p> <p>Finally, to create a VM that will attach to the aforementioned Network, refer to the following VMI spec:</p> <p><pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-host-network\n  name: vmi-host-network\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      interfaces:\n      - macvtap: {}\n        name: hostnetwork\n      rng: {}\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  networks:\n  - multus:\n      networkName: macvtapnetwork\n    name: hostnetwork\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: docker.io/kubevirt/fedora-cloud-container-disk-demo:devel\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #!/bin/bash\n        echo \"fedora\" |passwd fedora --stdin\n    name: cloudinitdisk\n</code></pre> The requested <code>multus</code> <code>networkName</code> - i.e. <code>macvtapnetwork</code> - must match the name of the provisioned <code>NetworkAttachmentDefinition</code>.</p> <p>Note: VMIs with macvtap interfaces can be migrated, but their MAC addresses must be statically set.</p>"},{"location":"network/interfaces_and_networks/#security","title":"Security","text":""},{"location":"network/interfaces_and_networks/#mac-spoof-check","title":"MAC spoof check","text":"<p>MAC spoofing refers to the ability to generate traffic with an arbitrary source MAC address. An attacker may use this option to generate attacks on the network.</p> <p>In order to protect against such scenarios, it is possible to enable the mac-spoof-check support in CNI plugins that support it.</p> <p>The pod primary network which is served by the cluster network provider is not covered by this documentation. Please refer to the relevant provider to check how to enable spoofing check. The following text refers to the secondary networks, served using multus.</p> <p>There are two known CNI plugins that support mac-spoof-check:</p> <ul> <li>sriov-cni:   Through the <code>spoofchk</code> parameter .</li> <li>bridge-cni:   Through the <code>macspoofchk</code> parameter.</li> </ul> <p>The configuration is to be done on the  NetworkAttachmentDefinition by the operator and any interface that refers to it, will have this feature enabled.</p> <p>Below is an example of using the <code>bridge</code> CNI with <code>macspoofchk</code> enabled: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: br-spoof-check\nspec:\n  config: '{\n            \"cniVersion\": \"0.3.1\",\n            \"name\": \"br-spoof-check\",\n            \"type\": \"bridge\",\n            \"bridge\": \"br10\",\n            \"disableContainerInterface\": true,\n            \"macspoofchk\": true\n        }'\n</code></pre></p> <p>On the VMI, the network section should point to this NetworkAttachmentDefinition by name: <pre><code>networks:\n- name: default\n  pod: {}\n- multus:\n    networkName: br-spoof-check\n  name: br10\n</code></pre></p>"},{"location":"network/interfaces_and_networks/#limitations_1","title":"Limitations","text":"<ul> <li>The <code>bridge</code> CNI supports mac-spoof-check through nftables, therefore the node must support nftables and have the <code>nft</code> binary deployed.</li> </ul>"},{"location":"network/istio_service_mesh/","title":"Istio service mesh","text":"<p>Service mesh allows to monitor, visualize and control traffic between pods. Kubevirt supports running VMs as a part of Istio service mesh.</p>"},{"location":"network/istio_service_mesh/#limitations","title":"Limitations","text":"<ul> <li> <p>Istio service mesh is only supported with a pod network masquerade or passt binding.</p> </li> <li> <p>Istio uses a list of ports for its own purposes, these ports must not be explicitly specified in a VMI interface.</p> </li> <li> <p>Istio only supports IPv4.</p> </li> </ul>"},{"location":"network/istio_service_mesh/#prerequisites","title":"Prerequisites","text":"<ul> <li> <p>This guide assumes that Istio is already deployed and uses Istio CNI Plugin. See Istio documentation for more information.</p> </li> <li> <p>Optionally, <code>istioctl</code> binary for troubleshooting. See Istio installation inctructions.</p> </li> <li> <p>The target namespace where the VM is created must be labelled with <code>istio-injection=enabled</code> label.</p> </li> <li> <p>If Multus is used to manage CNI, the following <code>NetworkAttachmentDefinition</code> is required in the application namespace: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: istio-cni\n</code></pre></p> </li> </ul>"},{"location":"network/istio_service_mesh/#create-a-virtualmachineinstance-with-enabled-istio-proxy-injecton","title":"Create a VirtualMachineInstance with enabled Istio proxy injecton","text":"<p>The example below specifies a VMI with masquerade network interface and <code>sidecar.istio.io/inject</code> annotation to register the VM to the service mesh. </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    sidecar.istio.io/inject: \"true\"\n  labels:\n    app: vmi-istio\n  name: vmi-istio\nspec:\n  domain:\n    devices:\n      interfaces:\n        - name: default\n          masquerade: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    resources:\n      requests:\n        memory: 1024M\n  networks:\n    - name: default\n      pod: {}\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: registry:5000/kubevirt/fedora-cloud-container-disk-demo:devel\n</code></pre> <p>Istio expects each application to be associated with at least one Kubernetes service. Create the following Service exposing port 8080:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vmi-istio\nspec:\n  selector:\n    app: vmi-istio\n  ports:\n    - port: 8080\n      name: http\n      protocol: TCP\n</code></pre> <p>Note: Each Istio enabled VMI must feature the <code>sidecar.istio.io/inject</code> annotation instructing KubeVirt to perform necessary network configuration.</p>"},{"location":"network/istio_service_mesh/#verification","title":"Verification","text":"<p>Verify istio-proxy sidecar is deployed and able to synchronize with Istio control plane using <code>istioctl proxy-status</code> command. See Istio Debbuging Envoy and Istiod documentation section for more information about <code>proxy-status</code> subcommand.</p> <pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-ncx7r    3/3     Running   0          7s\n\n$ kubectl get pods virt-launcher-vmi-istio-ncx7r -o jsonpath='{.spec.containers[*].name}'\ncompute volumecontainerdisk istio-proxy\n\n$ istioctl proxy-status\nNAME                                    CDS        LDS        EDS        RDS          ISTIOD                      VERSION\n...\nvirt-launcher-vmi-istio-ncx7r.default   SYNCED     SYNCED     SYNCED     SYNCED       istiod-7c4d8c7757-hshj5     1.10.0\n</code></pre>"},{"location":"network/istio_service_mesh/#troubleshooting","title":"Troubleshooting","text":""},{"location":"network/istio_service_mesh/#istio-sidecar-is-not-deployed","title":"Istio sidecar is not deployed","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-jnw6p    2/2     Running   0          37s\n\n$ kubectl get pods virt-launcher-vmi-istio-jnw6p -o jsonpath='{.spec.containers[*].name}'\ncompute volumecontainerdisk\n</code></pre> <p>Resolution: Make sure the <code>istio-injection=enabled</code> is added to the target namespace. If the issue persists, consult relevant part of Istio documentation.</p>"},{"location":"network/istio_service_mesh/#istio-sidecar-is-not-ready","title":"Istio sidecar is not ready","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS    RESTARTS   AGE\nvirt-launcher-vmi-istio-lg5gp    2/3     Running   0          90s\n\n$ kubectl describe pod virt-launcher-vmi-istio-lg5gp\n  ...\n  Warning  Unhealthy  2d8h (x3 over 2d8h)  kubelet            Readiness probe failed: Get \"http://10.244.186.222:15021/healthz/ready\": dial tcp 10.244.186.222:15021: connect: no route to host\n  Warning  Unhealthy  2d8h (x4 over 2d8h)  kubelet            Readiness probe failed: Get \"http://10.244.186.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n</code></pre> <p>Resolution: Make sure the <code>sidecar.istio.io/inject: \"true\"</code> annotation is defined in the created VMI and that masquerade or passt binding is used for pod network interface.</p>"},{"location":"network/istio_service_mesh/#virt-launcher-pod-for-vmi-is-stuck-at-initialization-phase","title":"Virt-launcher pod for VMI is stuck at initialization phase","text":"<pre><code>$ kubectl get pods\nNAME                             READY   STATUS     RESTARTS   AGE\nvirt-launcher-vmi-istio-44mws    0/3     Init:0/3   0          29s\n\n$ kubectl describe pod virt-launcher-vmi-istio-44mws\n  ...\n  Multus: [default/virt-launcher-vmi-istio-44mws]: error loading k8s delegates k8s args: TryLoadPodDelegates: error in getting k8s network for pod: GetNetworkDelegates: failed getting the delegate: getKubernetesDelegate: cannot find a network-attachment-definition (istio-cni) in namespace (default): network-attachment-definitions.k8s.cni.cncf.io \"istio-cni\" not found\n</code></pre> <p>Resolution: Make sure the <code>istio-cni</code> NetworkAttachmentDefinition (provided in the Prerequisites section) is created in the target namespace.</p>"},{"location":"network/network_binding_plugins/","title":"Network Binding Plugins","text":"<p>[v1.1.0, Alpha feature]</p> <p>A modular plugin which integrates with Kubevirt to implement a network binding.</p>"},{"location":"network/network_binding_plugins/#overview","title":"Overview","text":""},{"location":"network/network_binding_plugins/#network-connectivity","title":"Network Connectivity","text":"<p>In order for a VM to have access to external network(s), several layers need to be defined and configured, depending on the connectivity characteristics needs.</p> <p>These layers include:</p> <ul> <li>Host connectivity: Network provider.</li> <li>Host to Pod connectivity: CNI.</li> <li>Pod to domain connectivity: Network Binding.</li> </ul> <p>This guide focuses on the Network Binding portion.</p>"},{"location":"network/network_binding_plugins/#network-binding","title":"Network Binding","text":"<p>The network binding defines how the domain (VM) network interface is wired in the VM pod through the domain to the guest.</p> <p>The network binding includes:</p> <ul> <li>Domain vNIC configuration.</li> <li>Pod network configuration (optional).</li> <li>Services to deliver network details to the guest (optional).   E.g. DHCP server to pass the IP configuration to the guest.</li> </ul>"},{"location":"network/network_binding_plugins/#plugins","title":"Plugins","text":"<p>The network bindings have been part of Kubevirt core API and codebase. With the increase of the number of network bindings added and frequent requests to tweak and change the existing network bindings, a decision has been made to create a network binding plugin infrastructure.</p> <p>The plugin infrastructure provides means to compose a network binding plugin and integrate it into Kubevirt in a modular manner.</p> <p>Kubevirt is providing several network binding plugins as references. The following plugins are available:</p> <ul> <li>passt [v1.1.0]</li> <li>macvtap [v1.1.1]</li> <li>slirp [v1.1.0]</li> </ul>"},{"location":"network/network_binding_plugins/#definition-flow","title":"Definition &amp; Flow","text":"<p>A network binding plugin configuration consist of the following steps:</p> <ul> <li> <p>Deploy network binding optional components:</p> </li> <li> <p>Binding CNI plugin.</p> </li> <li>Binding NetworkAttachmentDefinition manifest.</li> <li>Access to the sidecar image.</li> <li> <p>Enable <code>NetworkBindingPlugins</code> Feature Gate (FG).</p> </li> <li> <p>Register network binding.</p> </li> <li>Assign VM network interface binding.</li> </ul>"},{"location":"network/network_binding_plugins/#deployment","title":"Deployment","text":"<p>Depending on the plugin, some components need to be deployed in the cluster. Not all network binding plugins require all these components, therefore these steps are optional.</p> <ul> <li>Binding CNI plugin: When it is required to change the pod network stack   (and a core domain-attachment is not a fit), a custom CNI plugin is   composed to serve the network binding plugin.</li> </ul> <p>This binary needs to be deployed on each node of the cluster, like any   other CNI plugin.</p> <p>The binary can be built from source or consumed from an existing artifact.</p> <p>Note: The location of the CNI plugins binaries depends on the platform used and its configuration. A frequently used path for such binaries is <code>/opt/cni/bin/</code>.</p> <ul> <li>Binding NetworkAttachmentDefinition: It references the binding CNI plugin,   with optional configuration settings.   The manifest needs to be deployed on the cluster at a namespace which   is accessible by the VM and its pod.</li> </ul> <p>Example: <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: netbindingpasst\nspec:\n  config: '{\n            \"cniVersion\": \"1.0.0\",\n            \"name\": \"netbindingpasst\",\n            \"plugins\": [\n              {\n                \"type\": \"cni-passt-binding-plugin\"\n              }\n            ]\n  }'\n</code></pre></p> <p>Note: It is possible to deploy the NetworkAttachmentDefinition on the <code>default</code> namespace, where all other namespaces can access it. Nevertheless, it is recommended (for security reasons) to define the NetworkAttachmentDefinition in the same namespace the VM resides.</p> <ul> <li> <p>Multus: In order   for the network binding CNI and the NetworkAttachmentDefinition to operate,   there is a need to have Multus deployed on the cluster.   For more information, check the   Quickstart Intallation Guide.</p> </li> <li> <p>Sidecar image: When a core domain-attachment is not a fit, a sidecar is   used to configure the vNIC domain configuration.   In a more complex scenarios, the sidecar also runs services like DHCP to   deliver IP information to the guest.</p> </li> </ul> <p>The sidecar image is built and usually pushed to an image registry for   consumption. Therefore, the cluster needs to have access to the image.</p> <p>The image can be built from source and pushed to an accessible registry   or used from a given registry that already contains it.</p> <ul> <li>Feature Gate   The network binding plugin is currently (v1.1.0) in Alpha stage, protected   by a feature gate (FG) named <code>NetworkBindingPlugins</code>.</li> </ul> <p>It is therefore necessary to set the FG in the Kubevirt CR.</p> <p>Example (valid when the FG subtree is already defined): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p>"},{"location":"network/network_binding_plugins/#register","title":"Register","text":"<p>In order to use a network binding plugin, the cluster admin needs to register the binding. Registration includes the addition of the binding name with all its parameters to the Kubevirt CR.</p> <p>The following (optional) parameters are currently supported:</p> <ul> <li>networkAttachmentDefinition</li> <li>sidecarImage</li> <li>domainAttachmentType</li> <li>migration</li> </ul>"},{"location":"network/network_binding_plugins/#networkattachmentdefinition","title":"networkAttachmentDefinition","text":"<p>From: v1.1.0</p> <p>Use the  format to specify the NetworkAttachementDefinition that defines the CNI plugin and the configuration the binding plugin uses. Used when the binding plugin needs to change the pod network namespace."},{"location":"network/network_binding_plugins/#sidecarimage","title":"sidecarImage","text":"<p>From: v1.1.0</p> <p>Specify a container image in a registry. Used when the binding plugin needs to modify the domain vNIC configuration or when a service needs to be executed (e.g. DHCP server). </p>"},{"location":"network/network_binding_plugins/#domainattachmenttype","title":"domainAttachmentType","text":"<p>From: v1.1.1</p> <p>The Domain Attachment type is a pre-defined core kubevirt method to attach an interface to the domain.</p> <p>Specify the name of a core domain attachment type. A possible alternative to a sidecar, to configure the domain vNIC.</p> <p>Supported types:</p> <ul> <li><code>tap</code> (from v1.1.1): The domain configuration is set to use an existing   tap device. It also supports existing <code>macvtap</code> devices.</li> </ul> <p>When both the <code>domainAttachmentType</code> and <code>sidecarImage</code> are specified, the domain will first be configured according to the <code>domainAttachmentType</code> and then the <code>sidecarImage</code> may modify it.</p>"},{"location":"network/network_binding_plugins/#migration","title":"migration","text":"<p>From: v1.2.0</p> <p>Specify whether the network binding plugin supports migration. It is possible to specify a migration method. Supported migration method types: - <code>link-refresh</code> (from v1.2.0): after migration, the guest nic will be deactivated and then activated again.    It can be useful to renew the DHCP lease.</p> <p>Note: In some deployments the Kubevirt CR is controlled by an external controller (e.g. HCO). In such cases, make sure to configure the wrapper operator/controller so the changes will get preserved.</p> <p>Example (the <code>passt</code> binding): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\"\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    }\n                }\n            }\n        }}]'\n</code></pre></p>"},{"location":"network/network_binding_plugins/#compute-resource-overhead","title":"Compute Resource Overhead","text":"<p>From: v1.3.0</p> <p>Some plugins may need additional resources to be added to the compute container of the virt-launcher pod.</p> <p>It is possible to specify compute resource overhead that will be added to the <code>compute</code> container of virt-launcher pods derived from virtual machines using the plugin.</p> <p>Note: At the moment, only memory overhead requests are supported.</p> <p>Note: In some deployments the Kubevirt CR is controlled by an external controller (e.g. HCO). In such cases, make sure to configure the wrapper operator/controller so the changes will get preserved.</p> <p>Example (the <code>passt</code> binding): <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\",\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    },\n                    \"computeResourceOverhead\": {\n                      \"requests\": {\n                        \"memory\": \"500Mi\",\n                      }\n                    }\n                }\n            }\n        }}]'\n</code></pre></p> <p>Every compute container in a virt-launcher pod derived from a VM using the passt network binding plugin, will have an additional <code>500Mi</code> memory overhead.</p>"},{"location":"network/network_binding_plugins/#vm-network-interface","title":"VM Network Interface","text":"<p>When configuring the VM/VMI network interface, the binding plugin name can be specified. If it exists in the Kubevirt CR, it will be used to setup the network interface.</p> <p>Example (<code>passt</code> binding): <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-passt\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-passt\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: passtnet\n            binding:\n              name: passt\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: passtnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre></p>"},{"location":"network/networkpolicy/","title":"NetworkPolicy","text":"<p>Before creating NetworkPolicy objects, make sure you are using a networking solution which supports NetworkPolicy. Network isolation is controlled entirely by NetworkPolicy objects. By default, all vmis in a namespace are accessible from other vmis and network endpoints. To isolate one or more vmis in a project, you can create NetworkPolicy objects in that namespace to indicate the allowed incoming connections.</p> <p>Note: vmis and pods are treated equally by network policies, since labels are passed through to the pods which contain the running vmi. With other words, labels on vmis can be matched by <code>spec.podSelector</code> on the policy.</p>"},{"location":"network/networkpolicy/#create-networkpolicy-to-deny-all-traffic","title":"Create NetworkPolicy to Deny All Traffic","text":"<p>To make a project \"deny by default\" add a NetworkPolicy object that matches all vmis but accepts no traffic.</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: deny-by-default\nspec:\n  podSelector: {}\n  ingress: []\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-only-accept-connections-from-vmis-within-namespaces","title":"Create NetworkPolicy to only Accept connections from vmis within namespaces","text":"<p>To make vmis accept connections from other vmis in the same namespace, but reject all other connections from vmis in other namespaces:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: allow-same-namespace\nspec:\n  podSelector: {}\n  ingress:\n  - from:\n    - podSelector: {}\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-only-allow-http-and-https-traffic","title":"Create NetworkPolicy to only allow HTTP and HTTPS traffic","text":"<p>To enable only HTTP and HTTPS access to the vmis, add a NetworkPolicy object similar to:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: allow-http-https\nspec:\n  podSelector: {}\n  ingress:\n  - ports:\n    - protocol: TCP\n      port: 8080\n    - protocol: TCP\n      port: 8443\n</code></pre>"},{"location":"network/networkpolicy/#create-networkpolicy-to-deny-traffic-by-labels","title":"Create NetworkPolicy to deny traffic by labels","text":"<p>To make one specific vmi with a label <code>type: test</code> to reject all traffic from other vmis, create:</p> <pre><code>kind: NetworkPolicy\napiVersion: networking.k8s.io/v1\nmetadata:\n  name: deny-by-label\nspec:\n  podSelector:\n    matchLabels:\n      type: test\n  ingress: []\n</code></pre> <p>Kubernetes NetworkPolicy Documentation can be found here: Kubernetes NetworkPolicy</p>"},{"location":"network/service_objects/","title":"Service objects","text":"<p>Once the VirtualMachineInstance is started, in order to connect to a VirtualMachineInstance, you can create a <code>Service</code> object for a VirtualMachineInstance. Currently, three types of service are supported: <code>ClusterIP</code>, <code>NodePort</code> and <code>LoadBalancer</code>. The default type is <code>ClusterIP</code>.</p> <p>Note: Labels on a VirtualMachineInstance are passed through to the pod, so simply add your labels for service creation to the VirtualMachineInstance. From there on it works like exposing any other k8s resource, by referencing these labels in a service.</p>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-clusterip-service","title":"Expose VirtualMachineInstance as a ClusterIP Service","text":"<p>Give a VirtualMachineInstance with the label <code>special: key</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: vmi-ephemeral\n  labels:\n    special: key\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/cirros-registry-disk-demo:latest\n</code></pre> <p>we can expose its SSH port (22) by creating a <code>ClusterIP</code> service:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vmiservice\nspec:\n  ports:\n  - port: 27017\n    protocol: TCP\n    targetPort: 22\n  selector:\n    special: key\n  type: ClusterIP\n</code></pre> <p>You just need to create this <code>ClusterIP</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl create -f vmiservice.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>Notes: * If <code>--target-port</code> is not set, it will be take the same value as <code>--port</code> * The cluster IP is usually allocated automatically, but it may also be forced into a value using the <code>--cluster-ip</code> flag (assuming value is in the valid range and not taken)</p> <p>Query the service object:</p> <pre><code>$ kubectl get service\nNAME        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE\nvmiservice   ClusterIP   172.30.3.149   &lt;none&gt;        27017/TCP   2m\n</code></pre> <p>You can connect to the VirtualMachineInstance by service IP and service port inside the cluster network:</p> <pre><code>$ ssh cirros@172.30.3.149 -p 27017\n</code></pre>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-nodeport-service","title":"Expose VirtualMachineInstance as a NodePort Service","text":"<p>Expose the SSH port (22) of a VirtualMachineInstance running on KubeVirt by creating a <code>NodePort</code> service:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: nodeport\nspec:\n  externalTrafficPolicy: Cluster\n  ports:\n  - name: nodeport\n    nodePort: 30000\n    port: 27017\n    protocol: TCP\n    targetPort: 22\n  selector:\n    special: key\n  type: NodePort\n</code></pre> <p>You just need to create this <code>NodePort</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl -f nodeport.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name nodeport --type NodePort --port 27017 --target-port 22 --node-port 30000\n</code></pre> <p>Notes: * If <code>--node-port</code> is not set, its value will be allocated dynamically (in the range above 30000) * If the <code>--node-port</code> value is set, it must be unique across all services</p> <p>The service can be listed by querying for the service objects:</p> <pre><code>$ kubectl get service\nNAME           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE\nnodeport       NodePort   172.30.232.73   &lt;none&gt;        27017:30000/TCP   5m\n</code></pre> <p>Connect to the VirtualMachineInstance by using a node IP and node port outside the cluster network:</p> <pre><code>$ ssh cirros@$NODE_IP -p 30000\n</code></pre>"},{"location":"network/service_objects/#expose-virtualmachineinstance-as-a-loadbalancer-service","title":"Expose VirtualMachineInstance as a LoadBalancer Service","text":"<p>Expose the RDP port (3389) of a VirtualMachineInstance running on KubeVirt by creating <code>LoadBalancer</code> service. Here is an example:</p> <pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: lbsvc\nspec:\n  externalTrafficPolicy: Cluster\n  ports:\n  - port: 27017\n    protocol: TCP\n    targetPort: 3389\n  selector:\n    special: key\n  type: LoadBalancer\n</code></pre> <p>You could create this <code>LoadBalancer</code> service by using <code>kubectl</code>:</p> <pre><code>$ kubectl -f lbsvc.yaml\n</code></pre> <p>Alternatively, the VirtualMachineInstance could be exposed using the <code>virtctl</code> command:</p> <pre><code>$ virtctl expose virtualmachineinstance vmi-ephemeral --name lbsvc --type LoadBalancer --port 27017 --target-port 3389\n</code></pre> <p>Note that the external IP of the service could be forced to a value using the <code>--external-ip</code> flag (no validation is performed on this value).</p> <p>The service can be listed by querying for the service objects:</p> <pre><code>$ kubectl get svc\nNAME      TYPE           CLUSTER-IP       EXTERNAL-IP                   PORT(S)           AGE\nlbsvc     LoadBalancer   172.30.27.5      172.29.10.235,172.29.10.235   27017:31829/TCP   5s\n</code></pre> <p>Use <code>vinagre</code> client to connect your VirtualMachineInstance by using the public IP and port.</p> <p>Note that here the external port here (31829) was dynamically allocated.</p>"},{"location":"network/net_binding_plugins/macvtap/","title":"Macvtap binding","text":""},{"location":"network/net_binding_plugins/macvtap/#overview","title":"Overview","text":"<p>With the <code>macvtap</code> binding plugin, virtual machines are directly exposed to the Kubernetes nodes L2 network. This is achieved by 'extending' an existing network interface with a virtual device that has its own MAC address.</p> <p>Its main benefits are:</p> <ul> <li>Direct connection to the node nic with no intermediate bridges.</li> </ul>"},{"location":"network/net_binding_plugins/macvtap/#functionality-support","title":"Functionality support","text":"Functionality Support Run without extra capabilities (on pod) Yes Migration support No IPAM support (on pod) No Primary network (pod network) No Secondary network Yes"},{"location":"network/net_binding_plugins/macvtap/#known-issues","title":"Known Issues","text":"<ul> <li>Live migration is not fully supported, see issue #5912</li> </ul> <p>Warning: On KinD clusters, the user needs to adjust the cluster configuration, mounting <code>dev</code> of the running host onto the KinD nodes, because of a known issue.</p>"},{"location":"network/net_binding_plugins/macvtap/#deployment","title":"Deployment","text":"<p>The <code>macvtap</code> solution consists of a CNI and a DP.</p> <p>In order to use <code>macvtap</code>, the following points need to be covered:</p> <ul> <li>Deploy the CNI plugin binary on the nodes.</li> <li>Deploy the Device Plugin daemon on the nodes.</li> <li>Configure which node interfaces are exposed.</li> <li>Define a NetworkAttachmentDefinition that points to the CNI plugin.</li> </ul>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-cni-and-dp-deployment-on-nodes","title":"Macvtap CNI and DP deployment on nodes","text":"<p>To simplify the procedure, use the Cluster Network Addons Operator to deploy and configure the macvtap components in your cluster.</p> <p>The aforementioned operator effectively deploys the macvtap cni and device plugin.</p>"},{"location":"network/net_binding_plugins/macvtap/#expose-node-interface-to-the-macvtap-device-plugin","title":"Expose node interface to the macvtap device plugin","text":"<p>There are two different alternatives to configure which host interfaces get exposed to the user, enabling them to create macvtap interfaces on top of:</p> <ul> <li>select the host interfaces: indicates which host interfaces are exposed.</li> <li>expose all interfaces: all interfaces of all hosts are exposed.</li> </ul> <p>Both options are configured via the <code>macvtap-deviceplugin-config</code> ConfigMap, and more information on how to configure it can be found in the macvtap-cni repo.</p> <p>This is a minimal example, in which the <code>eth0</code> interface of the Kubernetes nodes is exposed, via the <code>lowerDevice</code> attribute.</p> <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: |\n    [\n        {\n            \"name\"        : \"dataplane\",\n            \"lowerDevice\" : \"eth0\",\n            \"mode\"        : \"bridge\",\n            \"capacity\"    : 50\n        },\n    ]\n</code></pre> <p>This step can be omitted, since the default configuration of the aforementioned <code>ConfigMap</code> is to expose all host interfaces (which is represented by the following configuration):</p> <pre><code>kind: ConfigMap\napiVersion: v1\nmetadata:\n  name: macvtap-deviceplugin-config\ndata:\n  DP_MACVTAP_CONF: '[]'\n</code></pre>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-networkattachmentdefinition","title":"Macvtap NetworkAttachmentDefinition","text":"<p>The configuration needed for a macvtap network attachment can be minimalistic:</p> <pre><code>kind: NetworkAttachmentDefinition\napiVersion: k8s.cni.cncf.io/v1\nmetadata:\n  name: macvtapnetwork\n  annotations:\n    k8s.v1.cni.cncf.io/resourceName: macvtap.network.kubevirt.io/eth0\nspec:\n  config: '{\n      \"cniVersion\": \"0.3.1\",\n      \"name\": \"macvtapnetwork\",\n      \"type\": \"macvtap\",\n      \"mtu\": 1500\n    }'\n</code></pre> <p>The object should be created in a \"default\" namespace where all other namespaces can access, or, in the same namespace the VMs reside in.</p> <p>The requested <code>k8s.v1.cni.cncf.io/resourceName</code> annotation must point to an exposed host interface (via the <code>lowerDevice</code> attribute, on the <code>macvtap-deviceplugin-config</code> <code>ConfigMap</code>).</p>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-network-binding-plugin","title":"Macvtap network binding plugin","text":"<p>[v1.1.1]</p> <p>The binding plugin replaces the experimental core macvtap binding implementation (including its API).</p> <p>Note: The network binding plugin infrastructure and the macvtap plugin specifically are in Alpha stage. Please use them with care, preferably on a non-production deployment.</p> <p>The macvtap binding plugin consists of the following components:</p> <ul> <li>Macvtap CNI plugin.</li> </ul> <p>The plugin needs to:</p> <ul> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>And in detail:</p>"},{"location":"network/net_binding_plugins/macvtap/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\n  \"op\": \"add\",\n  \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",\n  \"value\": \"NetworkBindingPlugins\"\n}]'\n</code></pre></p> <p>Note: The specific macvtap plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster. The macvtap binding is still in evaluation, use it with care.</p>"},{"location":"network/net_binding_plugins/macvtap/#macvtap-registration","title":"Macvtap Registration","text":"<p>The macvtap binding plugin configuration needs to be added to the kubevirt CR in order to be used by VMs.</p> <p>To register the macvtap binding, patch the kubevirt CR as follows:</p> <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"macvtap\": {\n                    \"domainAttachmentType\": \"tap\"\n                }\n            }\n        }}]'\n</code></pre>"},{"location":"network/net_binding_plugins/macvtap/#vm-macvtap-network-interface","title":"VM Macvtap Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-macvtap\n  name: vm-net-binding-macvtap\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-macvtap\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: podnet\n            masquerade: {}\n          - name: hostnetwork\n            binding:\n              name: macvtap\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: podnet\n        pod: {}\n      - name: hostnetwork\n        multus:\n          networkName: macvtapnetwork\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre> <p>The multus <code>networkName</code> value should correspond with the name used in the network attachment definition section.</p> <p>The <code>binding</code> value should correspond with the name used in the registration.</p>"},{"location":"network/net_binding_plugins/passt/","title":"Passt binding","text":""},{"location":"network/net_binding_plugins/passt/#overview","title":"Overview","text":"<p>Plug A Simple Socket Transport is an enhanced alternative to SLIRP, providing user-space network connectivity.</p> <p><code>passt</code> is a universal tool which implements a translation layer between a Layer-2 network interface and native Layer -4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host.</p> <p>Its main benefits are:</p> <ul> <li>Doesn't require extra network capabilities as CAP_NET_RAW and CAP_NET_ADMIN.</li> <li>Allows integration with service meshes (which expect applications to run locally) out of the box.</li> <li>Supports IPv6 out of the box (in contrast to the existing bindings which require configuring IPv6   manually).</li> </ul>"},{"location":"network/net_binding_plugins/passt/#functionality-support","title":"Functionality support","text":"Functionality Support Migration support Yes Service Mesh support Yes Pod IP in guest Yes Custom CIDR in guest No Require extra capabilities (on pod) to operate No Primary network (pod network) Yes Secondary network No"},{"location":"network/net_binding_plugins/passt/#node-optimization-requirementsrecommendations","title":"Node optimization requirements/recommendations:","text":"<ol> <li>To get better performance the node should be configured with: <pre><code>sysctl -w net.core.rmem_max = 33554432\nsysctl -w net.core.wmem_max = 33554432\n</code></pre></li> <li>To run multiple passt VMs with no explicit ports, the node's <code>fs.file-max</code> should be increased    (for a VM forwards all IPv4 and IPv6 ports, for TCP and UDP, passt needs to create ~2^18 sockets): <pre><code>sysctl -w fs.file-max = 9223372036854775807\n</code></pre></li> </ol> <p>NOTE: To achieve optimal memory consumption with Passt binding, specify ports required for your workload. When no ports are explicitly specified, all ports are forwarded, leading to memory overhead of up to 800 Mi.</p>"},{"location":"network/net_binding_plugins/passt/#passt-network-binding-plugin","title":"Passt network binding plugin","text":"<p>[v1.1.0]</p> <p>The binding plugin replaces the experimental core passt binding implementation (including its API).</p> <p>Note: The network binding plugin infrastructure and the passt plugin specifically are in Alpha stage. Please use them with care, preferably on a non-production deployment.</p> <p>The passt binding plugin consists of the following components:</p> <ul> <li>Passt CNI plugin.</li> <li>Sidecar image.</li> </ul> <p>As described in the definition &amp; flow section, the passt plugin needs to:</p> <ul> <li>Deploy the CNI plugin binary on the nodes.</li> <li>Define a NetworkAttachmentDefinition that points to the CNI plugin.</li> <li>Assure access to the sidecar image.</li> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>And in detail:</p>"},{"location":"network/net_binding_plugins/passt/#passt-cni-deployment-on-nodes","title":"Passt CNI deployment on nodes","text":"<p>The CNI plugin binary can be retrieved directly from the kubevirt release assets (on GitHub) or to be built from its sources.</p> <p>Note: The kubevirt project uses Bazel to build the binaries and container images. For more information in how to build the whole project, visit the developer getting started guide.</p> <p>Once the binary is ready, you may rename it to a meaningful name (e.g. <code>kubevirt-passt-binding</code>). This name is used in the NetworkAttachmentDefinition configuration.</p> <p>Copy the binary to each node in your cluster. The location of the CNI plugins may vary between platforms and versions. One common path is <code>/opt/cni/bin/</code>.</p>"},{"location":"network/net_binding_plugins/passt/#passt-networkattachmentdefinition","title":"Passt NetworkAttachmentDefinition","text":"<p>The configuration needed for passt is minimalistic:</p> <pre><code>apiVersion: \"k8s.cni.cncf.io/v1\"\nkind: NetworkAttachmentDefinition\nmetadata:\n  name: netbindingpasst\nspec:\n  config: '{\n            \"cniVersion\": \"1.0.0\",\n            \"name\": \"netbindingpasst\",\n            \"plugins\": [\n              {\n                \"type\": \"kubevirt-passt-binding\"\n              }\n            ]\n  }'\n</code></pre> <p>The object should be created in a \"default\" namespace where all other namespaces can access, or, in the same namespace the VMs reside in.</p>"},{"location":"network/net_binding_plugins/passt/#passt-sidecar-image","title":"Passt sidecar image","text":"<p>Passt sidecar image is built and pushed to kubevirt quay repository.</p> <p>The sidecar sources can be found here.</p> <p>The relevant sidecar image needs to be accessible by the cluster and specified in the Kubevirt CR when registering the network binding plugin.</p>"},{"location":"network/net_binding_plugins/passt/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p> <p>Note: The specific passt plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster. The passt binding is still in evaluation, use it with care.</p>"},{"location":"network/net_binding_plugins/passt/#passt-registration","title":"Passt Registration","text":"<p>As described in the registration section, passt binding plugin configuration needs to be added to the kubevirt CR.</p> <p>To register the passt binding, patch the kubevirt CR as follows: <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"passt\": {\n                    \"networkAttachmentDefinition\": \"default/netbindingpasst\",\n                    \"sidecarImage\": \"quay.io/kubevirt/network-passt-binding:20231205_29a16d5c9\",\n                    \"migration\": {\n                        \"method\": \"link-refresh\"\n                    },\n                    \"computeResourceOverhead\": {\n                      \"requests\": {\n                        \"memory\": \"500Mi\",\n                      }\n                    }\n                }\n            }\n        }}]'\n</code></pre></p> <p>The NetworkAttachmentDefinition and sidecarImage values should correspond with the names used in the previous sections, here and here.</p> <p>When using the plugin, additional memory overhead of <code>500Mi</code> will be requested for the compute container in the virt-launcher pod.</p>"},{"location":"network/net_binding_plugins/passt/#vm-passt-network-interface","title":"VM Passt Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-passt\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-passt\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: passtnet\n            binding:\n              name: passt\n            ports:\n            - name: http\n              port: 80\n              protocol: TCP\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: passtnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre>"},{"location":"network/net_binding_plugins/slirp/","title":"Slirp","text":""},{"location":"network/net_binding_plugins/slirp/#overview","title":"Overview","text":"<p>SLIRP provides user-space network connectivity.</p> <p>Note: in <code>slirp</code> mode, the only supported protocols are TCP and UDP. ICMP is not supported.</p>"},{"location":"network/net_binding_plugins/slirp/#slirp-network-binding-plugin","title":"Slirp network binding plugin","text":"<p>[v1.1.0]</p> <p>The binding plugin replaces the core <code>slirp</code> binding API.</p> <p>Note: The network binding plugin infrastructure is in Alpha stage. Please use them with care.</p> <p>The slirp binding plugin consists of the following components:</p> <ul> <li>Sidecar image.</li> </ul> <p>As described in the definition &amp; flow section, the slirp plugin needs to:</p> <ul> <li>Assure access to the sidecar image.</li> <li>Enable the network binding plugin framework FG.</li> <li>Register the binding plugin on the Kubevirt CR.</li> <li>Reference the network binding by name from the VM spec interface.</li> </ul> <p>Note: In order for the core slirp binding to use the network binding plugin the registered name for this binding should be <code>slirp</code>.</p>"},{"location":"network/net_binding_plugins/slirp/#feature-gate","title":"Feature Gate","text":"<p>If not already set, add the <code>NetworkBindingPlugins</code> FG. <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/developerConfiguration/featureGates/-\",   \"value\": \"NetworkBindingPlugins\"}]'\n</code></pre></p> <p>Note: The specific slirp plugin has no FG by its own. It is up to the cluster admin to decide if the plugin is to be available in the cluster.</p>"},{"location":"network/net_binding_plugins/slirp/#slirp-registration","title":"Slirp Registration","text":"<p>As described in the registration section, slirp binding plugin configuration needs to be added to the kubevirt CR.</p> <p>To register the slirp binding, patch the kubevirt CR as follows: <pre><code>kubectl patch kubevirts -n kubevirt kubevirt --type=json -p='[{\"op\": \"add\", \"path\": \"/spec/configuration/network\",   \"value\": {\n            \"binding\": {\n                \"slirp\": {\n                    \"sidecarImage\": \"quay.io/kubevirt/network-slirp-binding:v1.1.0\"\n                }\n            }\n        }}]'\n</code></pre></p>"},{"location":"network/net_binding_plugins/slirp/#vm-slirp-network-interface","title":"VM Slirp Network Interface","text":"<p>Set the VM network interface binding name to reference the one defined in the kubevirt CR.</p> <pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-net-binding-slirp\n  name: vm-net-binding-passt\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-net-binding-slirp\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          interfaces:\n          - name: slirpnet\n            binding:\n              name: slirp\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      networks:\n      - name: slirpnet\n        pod: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:v1.1.0\n        name: containerdisk\n      - cloudInitNoCloud:\n          networkData: |\n            version: 2\n            ethernets:\n              eth0:\n                dhcp4: true\n        name: cloudinitdisk\n</code></pre>"},{"location":"storage/clone_api/","title":"Clone API","text":"<p>The <code>clone.kubevirt.io</code> API Group defines resources for cloning KubeVirt objects. Currently, the only supported cloning type is <code>VirtualMachine</code>, but more types are planned to be supported in the future (see future roadmap below).</p> <p>Please bear in mind that the clone API is in version <code>v1alpha1</code>. This means that this API is not fully stable yet and that APIs may change in the future.</p>"},{"location":"storage/clone_api/#prerequesites","title":"Prerequesites","text":""},{"location":"storage/clone_api/#snapshot-restore","title":"Snapshot / Restore","text":"<p>Under the hood, the clone API relies upon Snapshot &amp; Restore APIs. Therefore, in order to be able to use the clone API, please see Snapshot &amp; Restore prerequesites.</p>"},{"location":"storage/clone_api/#snapshot-feature-gate","title":"Snapshot Feature Gate","text":"<p>Currently, clone API is guarded by Snapshot feature gate. The feature gates field in the KubeVirt CR must be expanded by adding the <code>Snapshot</code> to it.</p>"},{"location":"storage/clone_api/#the-clone-object","title":"The clone object","text":"<p>Firstly, as written above, the clone API relies upon Snapshot &amp; Restore APIs under the hood. Therefore, it might be helpful to look at Snapshot &amp; Restore user-guide page for more info.</p>"},{"location":"storage/clone_api/#virtualmachineclone-object-overview","title":"VirtualMachineClone object overview","text":"<p>In order to initiate cloning, a <code>VirtualMachineClone</code> object (CRD) needs to be created on the cluster. An example for such an object is: <pre><code>kind: VirtualMachineClone\napiVersion: \"clone.kubevirt.io/v1alpha1\"\nmetadata:\n  name: testclone\n\nspec:\n  # source &amp; target definitions\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: vm-cirros\n  target:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: vm-clone-target\n\n  # labels &amp; annotations definitions\n  labelFilters:\n    - \"*\"\n    - \"!someKey/*\"\n  annotationFilters:\n    - \"anotherKey/*\"\n\n  # template labels &amp; annotations definitions\n  template:\n    labelFilters:\n      - \"*\"\n      - \"!someKey/*\"\n    annotationFilters:\n      - \"anotherKey/*\"\n\n  # other identity stripping specs:\n  newMacAddresses:\n    interfaceName: \"00-11-22\"\n  newSMBiosSerial: \"new-serial\"\n</code></pre></p> <p>In the next section I will go through the different settings to elaborate them.</p>"},{"location":"storage/clone_api/#source-target","title":"Source &amp; Target","text":"<p>The source and target indicate the source/target API group, kind and name. A few important notes:</p> <ul> <li> <p>Currently, the only supported kinds are <code>VirtualMachine</code> (of <code>kubevirt.io</code> api group) and <code>VirtualMachineSnapshot</code> ( of <code>snapshot.kubevirt.io</code> api group), but more types are expected to be supported in the future. See \"future roadmap\" below for more info.</p> </li> <li> <p>The target name is optional. If unspecified, the clone controller will generate a name for the target automatically.</p> </li> <li> <p>The target and source must reside in the same namespace.</p> </li> </ul>"},{"location":"storage/clone_api/#label-annotation-filters","title":"Label &amp; Annotation filters","text":"<p>These spec fields are intended to determine which labels / annotations are being copied to the target or stripped away.</p> <p>The filters are a list of strings. Each string represents a key that may exist at the source. Every source key that matches to one of these values is being copied to the cloned target. In addition, special regular-expression-like characters can be used:</p> <ul> <li>Wildcard character (*) can be used to match anything. Wildcard can be only used at the end of the filter.</li> <li>These filters are valid:<ul> <li>\"*\"</li> <li>\"some/key*\"</li> </ul> </li> <li>These filters are invalid:<ul> <li>\"some/*/key\"</li> <li>\"*/key\"</li> </ul> </li> <li>Negation character (!) can be used to avoid matching certain keys. Negation can be only used at the beginning of a filter.   Note that a Negation and Wildcard can be used together.</li> <li>These filters are valid:<ul> <li>\"!some/key\"</li> <li>\"!some/*\"</li> </ul> </li> <li>These filters are invalid:<ul> <li>\"key!\"</li> <li>\"some/!key\"</li> </ul> </li> </ul> <p>Setting label / annotation filters is optional. If unset, all labels / annotations will be copied as a default.</p>"},{"location":"storage/clone_api/#template-label-template-annotation-filters","title":"Template Label &amp; Template Annotation filters","text":"<p>Some network CNIs such as Kube-OVN or OVN-Kubernetes inject network information into the annotations of a VM. When cloning a VM from a target VM the cloned VM will use the same network. To avoid this you can use template labels and annotation filters.</p>"},{"location":"storage/clone_api/#newmacaddresses","title":"newMacAddresses","text":"<p>This field is used to explicitly replace MAC addresses for certain interfaces. The field is a string to string map; the keys represent interface names and the values represent the new MAC address for the clone target.</p> <p>This field is optional. By default, all mac addresses are stripped out. This suits situations when kube-mac-pool is deployed in the cluster which would automatically assign the target with a fresh valid MAC address.</p>"},{"location":"storage/clone_api/#newsmbiosserial","title":"newSMBiosSerial","text":"<p>This field is used to explicitly set an SMBios serial for the target.</p> <p>This field is optional. By default, the target would have an auto-generated serial that's based on the VM name.</p>"},{"location":"storage/clone_api/#creating-a-virtualmachineclone-object","title":"Creating a VirtualMachineClone object","text":"<p>After the clone manifest is ready, we can create it: <pre><code>kubectl create -f clone.yaml\n</code></pre></p> <p>To wait for a clone to complete, execute: <pre><code>kubectl wait vmclone testclone --for condition=Ready\n</code></pre></p> <p>You can check the clone's phase in the clone's status. It can be one of the following:</p> <ul> <li> <p>SnapshotInProgress</p> </li> <li> <p>CreatingTargetVM</p> </li> <li> <p>RestoreInProgress</p> </li> <li> <p>Succeeded</p> </li> <li> <p>Failed</p> </li> <li> <p>Unknown</p> </li> </ul> <p>After the clone is finished, the target can be inspected: <pre><code>kubectl get vm vm-clone-target -o yaml\n</code></pre></p>"},{"location":"storage/clone_api/#future-roadmap","title":"Future roadmap","text":"<p>The clone API is in an early alpha version and may change dramatically. There are many improvements and features that are expected to be added, the most significant goals are:</p> <ul> <li>Add more supported source types like <code>VirtualMachineInstace</code> in the future.</li> <li>Add a cross-namespace clone support. This needs to be supported for snapshots / restores first.</li> </ul>"},{"location":"storage/clone_api/#using-clones-as-a-golden-vm-image","title":"Using clones as a \"golden VM image\"","text":"<p>One of the great things that could be accomplished with the clone API when the source is of kind <code>VirtualMachineSnapshot</code> is to create \"golden VM images\" (a.k.a. Templates / Bookmark VMs / etc). In other words, the following workflow would be available:</p> <p>Create a golden image</p> <ul> <li> <p>Create a VM</p> </li> <li> <p>Prepare a \"golden VM\" environment</p> </li> <li> <p>This can mean different things in different contexts. For example, write files, install applications, apply configurations,     or anything else.</p> </li> <li> <p>Snapshot the VM</p> </li> <li> <p>Delete the VM</p> </li> </ul> <p>Then, this \"golden image\" can be duplicated as many times as needed. To instantiate a VM from the snapshot:</p> <ul> <li>Create a Clone object where the source would point to the previously taken snapshot</li> <li>Create as many VMs you need</li> </ul> <p>This feature is still under discussions and may be implemented differently then explained here.</p>"},{"location":"storage/containerized_data_importer/","title":"Containerized Data Importer","text":"<p>The Containerized Data Importer (CDI) project provides facilities for enabling Persistent Volume Claims (PVCs) to be used as disks for KubeVirt VMs by way of DataVolumes. The three main CDI use cases are:</p> <ul> <li>Import a disk image from a web server or container registry to a DataVolume</li> <li>Clone an existing PVC to a DataVolume</li> <li>Upload a local disk image to a DataVolume</li> </ul> <p>This document deals with the third use case. So you should have CDI installed in your cluster, a VM disk that you'd like to upload, and virtctl in your path.</p>"},{"location":"storage/containerized_data_importer/#install-cdi","title":"Install CDI","text":"<p>Install the latest CDI release here</p> <pre><code>export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)\nexport VERSION=$(echo ${TAG##*/})\nkubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml\nkubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml\n</code></pre>"},{"location":"storage/containerized_data_importer/#expose-cdi-uploadproxy-service","title":"Expose cdi-uploadproxy service","text":"<p>The <code>cdi-uploadproxy</code> service must be accessible from outside the cluster. Here are some ways to do that:</p> <ul> <li> <p>NodePort     Service</p> </li> <li> <p>Ingress</p> </li> <li> <p>Route</p> </li> <li> <p>kubectl     port-forward     (not recommended for production clusters)</p> </li> </ul> <p>Look here for example manifests.</p>"},{"location":"storage/containerized_data_importer/#supported-image-formats","title":"Supported image formats","text":"<p>CDI supports the <code>raw</code> and <code>qcow2</code> image formats which are supported by qemu. See the qemu documentation for more details.  Bootable ISO images can also be used and are treated like <code>raw</code> images.  Images may be compressed with either the <code>gz</code> or <code>xz</code> format.</p> <p>The example in this document uses this CirrOS image</p>"},{"location":"storage/containerized_data_importer/#virtctl-image-upload","title":"virtctl image-upload","text":"<p>virtctl has an image-upload command with the following options:</p> <pre><code>virtctl image-upload --help\nUpload a VM image to a DataVolume/PersistentVolumeClaim.\n\nUsage:\n  virtctl image-upload [flags]\n\nExamples:\n  # Upload a local disk image to a newly created DataVolume:\n  virtctl image-upload dv dv-name --size=10Gi --image-path=/images/fedora30.qcow2\n\n  # Upload a local disk image to an existing DataVolume\n  virtctl image-upload dv dv-name --no-create --image-path=/images/fedora30.qcow2\n\n  # Upload a local disk image to an existing PersistentVolumeClaim\n  virtctl image-upload pvc pvc-name --image-path=/images/fedora30.qcow2\n\n  # Upload to a DataVolume with explicit URL to CDI Upload Proxy\n  virtctl image-upload dv dv-name --uploadproxy-url=https://cdi-uploadproxy.mycluster.com --image-path=/images/fedora30.qcow2\n\nFlags:\n      --access-mode string       The access mode for the PVC. (default \"ReadWriteOnce\")\n      --block-volume             Create a PVC with VolumeMode=Block (default Filesystem).\n  -h, --help                     help for image-upload\n      --image-path string        Path to the local VM image.\n      --insecure                 Allow insecure server connections when using HTTPS.\n      --no-create                Don't attempt to create a new DataVolume/PVC.\n      --pvc-name string          DEPRECATED - The destination DataVolume/PVC name.\n      --pvc-size string          DEPRECATED - The size of the PVC to create (ex. 10Gi, 500Mi).\n      --size string              The size of the DataVolume to create (ex. 10Gi, 500Mi).\n      --storage-class string     The storage class for the PVC.\n      --uploadproxy-url string   The URL of the cdi-upload proxy service.\n      --wait-secs uint           Seconds to wait for upload pod to start. (default 60)\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre> <p><code>virtctl image-upload</code> works by creating a DataVolume of the requested size, sending an <code>UploadTokenRequest</code> to the <code>cdi-apiserver</code>, and uploading the file to the <code>cdi-uploadproxy</code>.</p> <pre><code>virtctl image-upload dv cirros-vm-disk --size=500Mi --image-path=/home/mhenriks/images/cirros-0.4.0-x86_64-disk.img --uploadproxy-url=&lt;url to upload proxy service&gt;\n</code></pre>"},{"location":"storage/containerized_data_importer/#addressing-certificate-issues-when-uploading-images","title":"Addressing Certificate Issues when Uploading Images","text":"<p>Issues with the certificates can be circumvented by using the <code>--insecure</code> flag to prevent the virtctl command from verifying the remote host. It is better to resolve certificate issues that prevent uploading images using the <code>virtctl image-upload</code> command and not use the <code>--insecure</code> flag.</p> <p>The following are some common issues with certificates and some easy ways to fix them.</p>"},{"location":"storage/containerized_data_importer/#does-not-contain-any-ip-sans","title":"Does not contain any IP SANs","text":"<p>This issue happens when trying to upload images using an IP address instead of a resolvable name. For example, trying to upload to the IP address 192.168.39.32 at port 31001 would produce the following error.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://192.168.39.32:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://192.168.39.32:31001\n\n 0 B / 193.89 MiB [-------------------------------------------------------]   0.00% 0s\n\nPost https://192.168.39.32:31001/v1beta1/upload: x509: cannot validate certificate for 192.168.39.32 because it doesn't contain any IP SANs\n</code></pre> <p>It is easily fixed by adding an entry it your local name resolution service. This could be a DNS server or the local hosts file. The URL used to upload the proxy should be changed to reflect the resolvable name.</p> <p>The <code>Subject</code> and the <code>Subject Alternative Name</code> in the certificate contain valid names that can be used for resolution. Only one of these names needs to be resolvable. Use the <code>openssl</code> command to view the names of the cdi-uploadproxy service.</p> <pre><code>echo | openssl s_client -showcerts -connect 192.168.39.32:31001 2&gt;/dev/null \\\n     | openssl x509 -inform pem -noout -text \\\n     | sed -n -e '/Subject.*CN/p' -e '/Subject Alternative/{N;p}'\n\n    Subject: CN = cdi-uploadproxy\n        X509v3 Subject Alternative Name: \n            DNS:cdi-uploadproxy, DNS:cdi-uploadproxy.cdi, DNS:cdi-uploadproxy.cdi.svc\n</code></pre> <p>Adding the following entry to the /etc/hosts file, if it provides name resolution, should fix this issue. Any service that provides name resolution for the system could be used.</p> <pre><code>echo \"192.168.39.32  cdi-uploadproxy\" &gt;&gt; /etc/hosts\n</code></pre> <p>The upload should now work.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 193.89 MiB / 193.89 MiB [=============================================] 100.00% 1m38s\n\nUploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress\nProcessing completed successfully\nUploading Fedora-Cloud-Base-33-1.2.x86_64.raw.xz completed successfully\n</code></pre>"},{"location":"storage/containerized_data_importer/#certificate-signed-by-unknown-authority","title":"Certificate Signed by Unknown Authority","text":"<p>This happens because the cdi-uploadproxy certificate is self signed and the system does not trust the cdi-uploadproxy as a Certificate Authority.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 0 B / 193.89 MiB [-------------------------------------------------------]   0.00% 0s\n\nPost https://cdi-uploadproxy:31001/v1beta1/upload: x509: certificate signed by unknown authority\n</code></pre> <p>This can be fixed by adding the certificate to the systems trust store. Download the cdi-uploadproxy-server-cert.</p> <pre><code>kubectl get secret -n cdi cdi-uploadproxy-server-cert \\\n  -o jsonpath=\"{.data['tls\\.crt']}\" \\\n  | base64 -d &gt; cdi-uploadproxy-server-cert.crt\n</code></pre> <p>Add this certificate to the systems trust store. On Fedora, this can be done as follows.</p> <pre><code>sudo cp cdi-uploadproxy-server-cert.crt /etc/pki/ca-trust/source/anchors\n\nsudo update-ca-trust\n</code></pre> <p>The upload should now work.</p> <pre><code>virtctl image-upload dv f33 \\\n  --size 5Gi \\\n  --image-path Fedora-Cloud-Base-33-1.2.x86_64.raw.xz \\\n  --uploadproxy-url https://cdi-uploadproxy:31001\n\nPVC default/f33 not found \nDataVolume default/f33 created\nWaiting for PVC f33 upload pod to be ready...\nPod now ready\nUploading data to https://cdi-uploadproxy:31001\n\n 193.89 MiB / 193.89 MiB [=============================================] 100.00% 1m36s\n\nUploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress\nProcessing completed successfully\nUploading Fedora-Cloud-Base-33-1.2.x86_64.raw.xz completed successfully\n</code></pre>"},{"location":"storage/containerized_data_importer/#setting-the-url-of-the-cdi-upload-proxy-service","title":"Setting the URL of the cdi-upload Proxy Service","text":"<p>Setting the URL for the cdi-upload proxy service allows the <code>virtctl image-upload</code> command to upload the images without specifying the <code>--uploadproxy-url</code> flag. Permanently setting the URL is done by patching the CDI configuration.</p> <p>The following will set the default upload proxy to use port 31001 of cdi-uploadproxy. An IP address could also be used instead of the dns name.</p> <p>See the section Addressing Certificate Issues when Uploading for why cdi-uploadproxy was chosen and issues that can be encountered when using an IP address.</p> <pre><code>kubectl patch cdi cdi \\\n  --type merge \\\n  --patch '{\"spec\":{\"config\":{\"uploadProxyURLOverride\":\"https://cdi-uploadproxy:31001\"}}}'\n</code></pre>"},{"location":"storage/containerized_data_importer/#create-a-virtualmachineinstance","title":"Create a VirtualMachineInstance","text":"<p>To create a <code>VirtualMachineInstance</code> from a DataVolume, you can execute the following:</p> <pre><code>cat &lt;&lt;EOF | kubectl apply -f -\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: cirros-vm\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: dvdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: dvdisk\n    dataVolume:\n      name: cirros-vm-disk\nstatus: {}\nEOF\n</code></pre>"},{"location":"storage/containerized_data_importer/#connect-to-virtualmachineinstance-console","title":"Connect to VirtualMachineInstance console","text":"<p>Use <code>virtctl</code> to connect to the newly create <code>VirtualMachineInstance</code>.</p> <pre><code>virtctl console cirros-vm\n</code></pre>"},{"location":"storage/disks_and_volumes/","title":"Filesystems, Disks and Volumes","text":"<p>Making persistent storage in the cluster (volumes) accessible to VMs consists of three parts. First, volumes are specified in <code>spec.volumes</code>. Second, disks are added to the VM by specifying them in <code>spec.domain.devices.disks</code>. Finally, a reference to the specified volume is added to the disk specification by name.</p>"},{"location":"storage/disks_and_volumes/#disks","title":"Disks","text":"<p>Like all other vmi devices a <code>spec.domain.devices.disks</code> element has a mandatory <code>name</code>, and furthermore, the disk's <code>name</code> must reference the <code>name</code> of a volume inside <code>spec.volumes</code>.</p> <p>A disk can be made accessible via four different types:</p> <ul> <li> <p>lun</p> </li> <li> <p>disk</p> </li> <li> <p>cdrom</p> </li> <li> <p>fileystems</p> </li> </ul> <p>All possible configuration options are available in the Disk API Reference.</p> <p>All types allow you to specify the <code>bus</code> attribute. The <code>bus</code> attribute determines how the disk will be presented to the guest operating system.</p>"},{"location":"storage/disks_and_volumes/#lun","title":"lun","text":"<p>A <code>lun</code> disk will expose the volume as a LUN device to the VM. This allows the VM to execute arbitrary iSCSI command passthrough.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>lun</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-lun\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a lun device\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#persistent-reservation","title":"persistent reservation","text":"<p>It is possible to reserve a LUN through the the SCSI Persistent Reserve commands. In order to issue privileged SCSI ioctls, the VM requires activation of the persistent resevation flag:</p> <pre><code>devices:\n  disks:\n  - name: mypvcdisk\n    lun:\n      reservation: true\n</code></pre> <p>This feature is enabled by the feature gate <code>PersistentReservation</code>:</p> <pre><code>configuration:\n  developerConfiguration:\n    featureGates:\n    -  PersistentReservation\n</code></pre> <p>Note: The persistent reservation feature enables an additional privileged component to be deployed together with virt-handler. Because this feature allows for sensitive security procedures, it is disabled by default and requires cluster administrator configuration.</p>"},{"location":"storage/disks_and_volumes/#disk","title":"disk","text":"<p>A <code>disk</code> disk will expose the volume as an ordinary disk to the VM.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>disk</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a disk\n        disk: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>You can set the disk <code>bus</code> type, overriding the defaults, which in turn depends on the chipset the VM is configured to use:</p> <pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a disk\n        disk:\n          # This makes it exposed as /dev/vda, being the only and thus first\n          # disk attached to the VM\n          bus: virtio\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#cdrom","title":"cdrom","text":"<p>A <code>cdrom</code> disk will expose the volume as a cdrom drive to the VM. It is read-only by default.</p> <p>A minimal example which attaches a <code>PersistentVolumeClaim</code> named <code>mypvc</code> as a <code>cdrom</code> device to the VM:</p> <pre><code>metadata:\n  name: testvmi-cdrom\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        # This makes it a cdrom\n        cdrom:\n          # This makes the cdrom writeable\n          readOnly: false\n          # This makes the cdrom be exposed as SATA device\n          bus: sata\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#filesystems","title":"filesystems","text":"<p>A <code>filesystem</code> device will expose the volume as a filesystem to the VM. <code>filesystems</code> rely on <code>virtiofs</code> to make visible external filesystems to <code>KubeVirt</code> VMs.  Further information about <code>virtiofs</code> can be found at the Official Virtiofs Site.</p> <p>Compared with <code>disk</code>, <code>filesystems</code> allow changes in the source to be dynamically reflected in the volumes inside the VM. For instance, if a given <code>configMap</code> is shared with <code>filesystems</code> any change made on it will be reflected in the VMs. However, it is important to note that <code>filesystems</code> do not allow live migration.</p> <p>Additionally, <code>filesystem</code> devices must be mounted inside the VM. This can be done through cloudInitNoCloud or manually connecting to the VM shell and targeting the same command. The main challenge is to understand how the device tag used to identify the new filesystem and mount it with the  <code>mount -t virtiofs [device tag] [path]</code> command. For that purpose, the tag is assigned to the filesystem in the VM spec <code>spec.domain.devices.filesystems.name</code>. For instance, if in a given VM spec is <code>spec.domain.devices.filesystems.name: foo</code>, the required command inside the VM to mount the filesystem in the <code>/tmp/foo</code> path will be <code>mount -t virtiofs foo /tmp/foo</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-filesystems\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: foo\n          virtiofs: {}\n      disks:\n        - name: containerdisk\n          disk:\n            bus: virtio\n        - name: cloudinitdisk\n          disk:\n            bus: virtio\n    volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk \n      - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: fedora\n              user: fedora\n              bootcmd:\n                - \"sudo mkdir /tmp/foo\"\n                - \"sudo mount -t virtiofs foo /tmp/foo\"\n      - persistentVolumeClaim:\n          claimName: mypvc\n        name: foo\n</code></pre> <p>Note: As stated, <code>filesystems</code> rely on <code>virtiofs</code>. Moreover, <code>virtiofs</code> requires kernel linux support to work in  the VM. To check if the linux image of the VM has the required support, you can address the following command: <code>modprobe virtiofs</code>. If the command output is <code>modprobe: FATAL: Module virtiofs not found</code>, the linux image of the VM does not support virtiofs. Also, you can check if the kernel version is up to 5.4 in any linux distribution or up to 4.18 in centos/rhel.  To check this, you can target the following command: <code>uname -r</code>.</p> <p>Refer to section Sharing Directories with VMs for usage examples of <code>filesystems</code>.</p>"},{"location":"storage/disks_and_volumes/#error-policy","title":"error policy","text":"<p>The error policy controls how the hypervisor should behave when an IO error occurs on a disk read or write. The default behaviour is to stop the guest and a Kubernetes event is generated. However, it is possible to change the value to either:</p> <ul> <li><code>report</code>: the error is reported in the guest</li> <li><code>ignore</code>: the error is ignored, but the read/write failure goes undetected</li> <li><code>enospace</code>: error when there isn't enough space on the disk</li> </ul> <p>The error policy can be specified per disk or lun.</p> <p>Example: <pre><code>spec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n        errorPolicy: \"report\"\n      - lun:\n          bus: scsi\n        name: scsi-disk\n        errorPolicy: \"report\"\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#volumes","title":"Volumes","text":"<p>Supported volume sources are</p> <ul> <li> <p>cloudInitNoCloud</p> </li> <li> <p>cloudInitConfigDrive</p> </li> <li> <p>persistentVolumeClaim</p> </li> <li> <p>dataVolume</p> </li> <li> <p>ephemeral</p> </li> <li> <p>containerDisk</p> </li> <li> <p>emptyDisk</p> </li> <li> <p>hostDisk</p> </li> <li> <p>configMap</p> </li> <li> <p>secret</p> </li> <li> <p>serviceAccount</p> </li> <li> <p>downwardMetrics</p> </li> </ul> <p>All possible configuration options are available in the Volume API Reference.</p>"},{"location":"storage/disks_and_volumes/#cloudinitnocloud","title":"cloudInitNoCloud","text":"<p>Allows attaching <code>cloudInitNoCloud</code> data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.</p> <p>A simple example which attaches a <code>Secret</code> as a cloud-init <code>disk</code> datasource may look like this:</p> <pre><code>metadata:\n  name: testvmi-cloudinitnocloud\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mybootdisk\n        lun: {}\n      - name: mynoclouddisk\n        disk: {}\n  volumes:\n    - name: mybootdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n    - name: mynoclouddisk\n      cloudInitNoCloud:\n        secretRef:\n          name: testsecret\n</code></pre>"},{"location":"storage/disks_and_volumes/#cloudinitconfigdrive","title":"cloudInitConfigDrive","text":"<p>Allows attaching <code>cloudInitConfigDrive</code> data-sources to the VM. If the VM contains a proper cloud-init setup, it will pick up the disk as a user-data source.</p> <p>A simple example which attaches a <code>Secret</code> as a cloud-init <code>disk</code> datasource may look like this:</p> <pre><code>metadata:\n  name: testvmi-cloudinitconfigdrive\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mybootdisk\n        lun: {}\n      - name: myconfigdrivedisk\n        disk: {}\n  volumes:\n    - name: mybootdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n    - name: myconfigdrivedisk\n      cloudInitConfigDrive:\n        secretRef:\n          name: testsecret\n</code></pre> <p>The <code>cloudInitConfigDrive</code> can also be used to configure VMs with Ignition. You just need to replace the cloud-init data by the Ignition data.</p>"},{"location":"storage/disks_and_volumes/#persistentvolumeclaim","title":"persistentVolumeClaim","text":"<p>Allows connecting a <code>PersistentVolumeClaim</code> to a VM disk.</p> <p>Use a PersistentVolumeClaim when the VirtualMachineInstance's disk needs to persist after the VM terminates. This allows for the VM's data to remain persistent between restarts.</p> <p>A <code>PersistentVolume</code> can be in \"filesystem\" or \"block\" mode:</p> <ul> <li> <p>Filesystem: For KubeVirt to be able to consume the disk present on a     PersistentVolume's filesystem, the disk must be named <code>disk.img</code> and     be placed in the root path of the filesystem. Currently the disk is     also required to be in raw format. &gt; Important: The     <code>disk.img</code> image file needs to be owned by the user-id <code>107</code> in     order to avoid permission issues.</p> <p>Note: If the <code>disk.img</code> image file has not been created manually before starting a VM then it will be created automatically with the <code>PersistentVolumeClaim</code> size. Since not every storage provisioner provides volumes with the exact usable amount of space as requested (e.g. due to filesystem overhead), KubeVirt tolerates up to 10% less available space. This can be configured with the <code>developerConfiguration.pvcTolerateLessSpaceUpToPercent</code> value in the KubeVirt CR (<code>kubectl edit kubevirt kubevirt -n kubevirt</code>).</p> </li> <li> <p>Block: Use a block volume for consuming raw block devices. Note: you     need to enable the <code>BlockVolume</code> feature gate.</p> </li> </ul> <p>A simple example which attaches a <code>PersistentVolumeClaim</code> as a <code>disk</code> may look like this:</p> <pre><code>metadata:\n  name: testvmi-pvc\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#thick-and-thin-volume-provisioning","title":"Thick and thin volume provisioning","text":"<p>Sparsification can make a disk thin-provisioned, in other words it allows to convert the freed space within the disk image into free space back on the host. The fstrim utility can be used on a mounted filesystem to discard the blocks not used by the filesystem. In order to be able to sparsify a disk inside the guest, the disk needs to be configured in the libvirt xml with the option <code>discard=unmap</code>. In KubeVirt, every disk is passed as default with this option enabled. It is possible to check if the trim configuration is supported in the guest by running<code>lsblk -D</code>, and check the discard options supported on every disk.</p> <p>Example: <pre><code>$ lsblk -D\nNAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO\nloop0         0        4K       4G         0\nloop1         0       64K       4M         0\nsr0           0        0B       0B         0\nrbd0          0       64K       4M         0\nvda         512      512B       2G         0\n\u2514\u2500vda1        0      512B       2G         0\n</code></pre></p> <p>However, in certain cases like preallocaton or when the disk is thick provisioned, the option needs to be disabled. The disk's PVC has to be marked with an annotation that contains <code>/storage.preallocation</code> or <code>/storage.thick-provisioned</code>, and set to true. If the volume is preprovisioned using CDI and the preallocation is enabled, then the PVC is automatically annotated with: <code>cdi.kubevirt.io/storage.preallocation: true</code> and the discard passthrough option is disabled.</p> <p>Example of a PVC definition with the annotation to disable discard passthrough: <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: pvc\n  annotations:\n    user.custom.annotation/storage.thick-provisioned: \"true\"\nspec:\n  storageClassName: local\n  accessModes:\n    - ReadWriteOnce\n  volumeMode: Filesystem\n  resources:\n    requests:\n      storage: 1Gi\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#disk-expansion","title":"disk expansion","text":"<p>For some storage methods, Kubernetes may support expanding storage in-use (allowVolumeExpansion feature). KubeVirt can respond to it by making the additional storage available for the virtual machines. This feature is currently off by default, and requires enabling a feature gate. To enable it, add the ExpandDisks feature gate in the kubevirt object:</p> <pre><code>spec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n      - ExpandDisks\n</code></pre> <p>Enabling this feature does two things: - Notify the virtual machine about size changes - If the disk is a Filesystem PVC, the matching file is expanded   to the remaining size (while reserving some space for file system overhead).</p>"},{"location":"storage/disks_and_volumes/#statically-provisioned-block-pvcs","title":"Statically provisioned block PVCs","text":"<p>To use an externally managed local block device from a host ( e.g. /dev/sdb , zvol, LVM, etc... ) in a VM directly, you would need a provisioner that supports block devices, such as OpenEBS LocalPV.</p> <p>Alternatively, local volumes can be provisioned by hand. I.e. the following PVC:</p> <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: myblock\nspec:\n  storageClassName: local-device\n  volumeMode: Block\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 100Gi\n</code></pre> <p>can claim a PersistentVolume pre-created by a cluster admin like so:</p> <pre><code>apiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n  name: local-device\nprovisioner: kubernetes.io/no-provisioner\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: myblock\nspec:\n  volumeMode: Block\n  storageClassName: local-device\n  nodeAffinity:\n    required:\n      nodeSelectorTerms:\n      - matchExpressions:\n        - key: kubernetes.io/hostname\n          operator: In\n          values:\n          - my-node\n  accessModes:\n    - ReadWriteOnce\n  capacity:\n    storage: 100Gi\n  local:\n    path: /dev/sdb\n</code></pre>"},{"location":"storage/disks_and_volumes/#datavolume","title":"dataVolume","text":"<p>DataVolumes are a way to automate importing virtual machine disks onto PVCs during the virtual machine's launch flow. Without using a DataVolume, users have to prepare a PVC with a disk image before assigning it to a VM or VMI manifest. With a DataVolume, both the PVC creation and import is automated on behalf of the user.</p>"},{"location":"storage/disks_and_volumes/#datavolume-vm-behavior","title":"DataVolume VM Behavior","text":"<p>DataVolumes can be defined in the VM spec directly by adding the DataVolumes to the <code>dataVolumeTemplates</code> list. Below is an example.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-alpine-datavolume\n  name: vm-alpine-datavolume\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-alpine-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 64M\n      volumes:\n      - dataVolume:\n          name: alpine-dv\n        name: datavolumedisk1\n  dataVolumeTemplates:\n  - metadata:\n      name: alpine-dv\n    spec:\n      storage:\n        resources:\n          requests:\n            storage: 2Gi\n      source:\n        http:\n          url: http://cdi-http-import-server.kubevirt/images/alpine.iso\n</code></pre> <p>You can see the DataVolume defined in the dataVolumeTemplates section has two parts. The source and pvc</p> <p>The source part declares that there is a disk image living on an http server that we want to use as a volume for this VM. The pvc part declares the spec that should be used to create the PVC that hosts the source data.</p> <p>When this VM manifest is posted to the cluster, as part of the launch flow a PVC will be created using the spec provided and the source data will be automatically imported into that PVC before the VM starts. When the VM is deleted, the storage provisioned by the DataVolume will automatically be deleted as well.</p>"},{"location":"storage/disks_and_volumes/#datavolume-vmi-behavior","title":"DataVolume VMI Behavior","text":"<p>For a VMI object, DataVolumes can be referenced as a volume source for the VMI. When this is done, it is expected that the referenced DataVolume exists in the cluster. The VMI will consume the DataVolume, but the DataVolume's life-cycle will not be tied to the VMI.</p> <p>Below is an example of a DataVolume being referenced by a VMI. It is expected that the DataVolume alpine-datavolume was created prior to posting the VMI manifest to the cluster. It is okay to post the VMI manifest to the cluster while the DataVolume is still having data imported. KubeVirt knows not to start the VMI until all referenced DataVolumes have finished their clone and import phases.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-alpine-datavolume\n  name: vmi-alpine-datavolume\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: disk1\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: disk1\n    dataVolume:\n      name: alpine-datavolume\n</code></pre>"},{"location":"storage/disks_and_volumes/#enabling-datavolume-support","title":"Enabling DataVolume support.","text":"<p>A DataVolume is a custom resource provided by the Containerized Data Importer (CDI) project. KubeVirt integrates with CDI in order to provide users a workflow for dynamically creating PVCs and importing data into those PVCs.</p> <p>In order to take advantage of the DataVolume volume source on a VM or VMI, CDI must be installed.</p> <p>Installing CDI</p> <p>Go to the CDI release page</p> <p>Pick the latest stable release and post the corresponding cdi-controller-deployment.yaml manifest to your cluster.</p>"},{"location":"storage/disks_and_volumes/#ephemeral","title":"ephemeral","text":"<p>An ephemeral volume is a local COW (copy on write) image that uses a network volume as a read-only backing store. With an ephemeral volume, the network backing store is never mutated. Instead all writes are stored on the ephemeral image which exists on local storage. KubeVirt dynamically generates the ephemeral images associated with a VM when the VM starts, and discards the ephemeral images when the VM stops.</p> <p>Ephemeral volumes are useful in any scenario where disk persistence is not desired. The COW image is discarded when VM reaches a final state (e.g., succeeded, failed).</p> <p>Currently, only <code>PersistentVolumeClaim</code> may be used as a backing store of the ephemeral volume.</p> <p>Up-to-date information on supported backing stores can be found in the KubeVirt API.</p> <pre><code>metadata:\n  name: testvmi-ephemeral-pvc\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: mypvcdisk\n        lun: {}\n  volumes:\n    - name: mypvcdisk\n      ephemeral:\n        persistentVolumeClaim:\n          claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#containerdisk","title":"containerDisk","text":"<p>containerDisk was originally registryDisk, please update your code when needed.</p> <p>The <code>containerDisk</code> feature provides the ability to store and distribute VM disks in the container image registry. <code>containerDisks</code> can be assigned to VMs in the disks section of the VirtualMachineInstance spec.</p> <p>No network shared storage devices are utilized by <code>containerDisks</code>. The disks are pulled from the container registry and reside on the local node hosting the VMs that consume the disks.</p>"},{"location":"storage/disks_and_volumes/#when-to-use-a-containerdisk","title":"When to use a containerDisk","text":"<p><code>containerDisks</code> are ephemeral storage devices that can be assigned to any number of active VirtualMachineInstances. This makes them an ideal tool for users who want to replicate a large number of VM workloads that do not require persistent data. <code>containerDisks</code> are commonly used in conjunction with VirtualMachineInstanceReplicaSets.</p>"},{"location":"storage/disks_and_volumes/#when-not-to-use-a-containerdisk","title":"When Not to use a containerDisk","text":"<p><code>containerDisks</code> are not a good solution for any workload that requires persistent root disks across VM restarts.</p>"},{"location":"storage/disks_and_volumes/#containerdisk-workflow-example","title":"containerDisk Workflow Example","text":"<p>Users can inject a VirtualMachineInstance disk into a container image in a way that is consumable by the KubeVirt runtime. Disks must be placed into the <code>/disk</code> directory inside the container. Raw and qcow2 formats are supported. Qcow2 is recommended in order to reduce the container image's size. <code>containerdisks</code> can and should be based on <code>scratch</code>. No content except the image is required.</p> <p>Note: Prior to kubevirt 0.20, the containerDisk image needed to have kubevirt/container-disk-v1alpha as base image.</p> <p>Note: The containerDisk needs to be readable for the user with the UID 107 (qemu).</p> <p>Example: Inject a local VirtualMachineInstance disk into a container image.</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD --chown=107:107 fedora25.qcow2 /disk/\nEND\n\ndocker build -t vmidisks/fedora25:latest .\n</code></pre> <p>Example: Inject a remote VirtualMachineInstance disk into a container image.</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD --chown=107:107 https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 /disk/\nEND\n</code></pre> <p>Example: Upload the ContainerDisk container image to a registry.</p> <pre><code>docker push vmidisks/fedora25:latest\n</code></pre> <p>Example: Attach the ContainerDisk as an ephemeral disk to a VM.</p> <pre><code>metadata:\n  name: testvmi-containerdisk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk: {}\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: vmidisks/fedora25:latest\n</code></pre> <p>Note that a <code>containerDisk</code> is file-based and therefore cannot be attached as a <code>lun</code> device to the VM.</p>"},{"location":"storage/disks_and_volumes/#custom-disk-image-path","title":"Custom disk image path","text":"<p>ContainerDisk also allows to store disk images in any folder, when required. The process is the same as previous. The main difference is, that in custom location, kubevirt does not scan for any image. It is your responsibility to provide full path for the disk image. Providing image <code>path</code> is optional. When no <code>path</code> is provided, kubevirt searches for disk images in default location: <code>/disk</code>.</p> <p>Example: Build container disk image:</p> <pre><code>cat &lt;&lt; END &gt; Dockerfile\nFROM scratch\nADD fedora25.qcow2 /custom-disk-path/fedora25.qcow2\nEND\n\ndocker build -t vmidisks/fedora25:latest .\ndocker push vmidisks/fedora25:latest\n</code></pre> <p>Create VMI with container disk pointing to the custom location:</p> <pre><code>metadata:\n  name: testvmi-containerdisk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk: {}\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: vmidisks/fedora25:latest\n        path: /custom-disk-path/fedora25.qcow2\n</code></pre>"},{"location":"storage/disks_and_volumes/#emptydisk","title":"emptyDisk","text":"<p>An <code>emptyDisk</code> works similar to an <code>emptyDir</code> in Kubernetes. An extra sparse <code>qcow2</code> disk will be allocated and it will live as long as the VM. Thus it will survive guest side VM reboots, but not a VM re-creation. The disk <code>capacity</code> needs to be specified.</p> <p>Example: Boot cirros with an extra <code>emptyDisk</code> with a size of <code>2GiB</code>:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: emptydisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: emptydisk\n      emptyDisk:\n        capacity: 2Gi\n</code></pre>"},{"location":"storage/disks_and_volumes/#when-to-use-an-emptydisk","title":"When to use an emptyDisk","text":"<p>Ephemeral VMs very often come with read-only root images and limited tmpfs space. In many cases this is not enough to install application dependencies and provide enough disk space for the application data. While this data is not critical and thus can be lost, it is still needed for the application to function properly during its lifetime. This is where an <code>emptyDisk</code> can be useful. An emptyDisk is often used and mounted somewhere in <code>/var/lib</code> or <code>/var/run</code>.</p>"},{"location":"storage/disks_and_volumes/#hostdisk","title":"hostDisk","text":"<p>A <code>hostDisk</code> volume type provides the ability to create or use a disk image located somewhere on a node. It works similar to a <code>hostPath</code> in Kubernetes and provides two usage types:</p> <ul> <li> <p><code>DiskOrCreate</code> if a disk image does not exist at a given location     then create one</p> </li> <li> <p><code>Disk</code> a disk image must exist at a given location</p> </li> </ul> <p>Note: you need to enable the HostDisk feature gate.</p> <p>Example: Create a 1Gi disk image located at /data/disk.img and attach it to a VM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-host-disk\n  name: vmi-host-disk\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: host-disk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - hostDisk:\n      capacity: 1Gi\n      path: /data/disk.img\n      type: DiskOrCreate\n    name: host-disk\nstatus: {}\n</code></pre> <p>Note: This does not always work as expected. Instead you may want to consider creating a PersistentVolume</p>"},{"location":"storage/disks_and_volumes/#configmap","title":"configMap","text":"<p>A <code>configMap</code> is a reference to a ConfigMap in Kubernetes.  A <code>configMap</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk","title":"As a disk","text":"<p>By using disk, an extra <code>iso</code> disk will be allocated which has to be mounted on a VM. To mount the <code>configMap</code> users can use <code>cloudInit</code> and the disk's serial number. The <code>name</code> needs to be set for a reference to the created kubernetes <code>ConfigMap</code>.</p> <p>Note: Currently, ConfigMap update is not propagate into the VMI. If a ConfigMap is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where ConfigMap keys are projected.</p> <p>Example: Attach the <code>configMap</code> to a VM and use <code>cloudInit</code> to mount the <code>iso</code> disk:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      - disk:\n        name: app-config-disk\n        # set serial\n        serial: CVLY623300HK240D\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          # mount the ConfigMap\n          - \"sudo mkdir /mnt/app-config\"\n          - \"sudo mount /dev/$(lsblk --nodeps -no name,serial | grep CVLY623300HK240D | cut -f1 -d' ') /mnt/app-config\"\n    name: cloudinitdisk\n  - configMap:\n      name: app-config\n    name: app-config-disk\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#as-a-filesystem","title":"As a filesystem","text":"<p>By using filesystem, <code>configMaps</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>configMaps</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>configMaps</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>configMap</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: config-fs\n          virtiofs: {}\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        chpasswd:\n          expire: false\n        password: fedora\n        user: fedora\n        bootcmd:\n          # mount the ConfigMap\n          - \"sudo mkdir /mnt/app-config\"\n          - \"sudo mount -t virtiofs config-fs /mnt/app-config\"\n    name: cloudinitdisk      \n  - configMap:\n      name: app-config\n    name: config-fs\n</code></pre>"},{"location":"storage/disks_and_volumes/#secret","title":"secret","text":"<p>A <code>secret</code> is a reference to a Secret in Kubernetes. A <code>secret</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk_1","title":"As a disk","text":"<p>By using disk, an extra <code>iso</code> disk will be allocated which has to be mounted on a VM. To mount the <code>secret</code> users can use <code>cloudInit</code> and the disks serial number. The <code>secretName</code> needs to be set for a reference to the created kubernetes <code>Secret</code>.</p> <p>Note: Currently, Secret update propagation is not supported. If a Secret is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Note: Due to a Kubernetes CRD issue, you cannot control the paths within the volume where Secret keys are projected.</p> <p>Example: Attach the <code>secret</code> to a VM and use <code>cloudInit</code> to mount the <code>iso</code> disk:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      - disk:\n        name: app-secret-disk\n        # set serial\n        serial: D23YZ9W6WA5DJ487\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          # mount the Secret\n          - \"sudo mkdir /mnt/app-secret\"\n          - \"sudo mount /dev/$(lsblk --nodeps -no name,serial | grep D23YZ9W6WA5DJ487 | cut -f1 -d' ') /mnt/app-secret\"\n    name: cloudinitdisk\n  - secret:\n      secretName: app-secret\n    name: app-secret-disk\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#as-a-filesystem_1","title":"As a filesystem","text":"<p>By using filesystem, <code>secrets</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>secrets</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>secrets</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>secret</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: app-secret-fs\n          virtiofs: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          chpasswd:\n            expire: false\n          password: fedora\n          user: fedora\n          bootcmd:\n            # mount the Secret\n            - \"sudo mkdir /mnt/app-secret\"\n            - \"sudo mount -t virtiofs app-secret-fs /mnt/app-secret\"\n      name: cloudinitdisk\n    - secret:\n        secretName: app-secret\n      name: app-secret-fs\n</code></pre>"},{"location":"storage/disks_and_volumes/#serviceaccount","title":"serviceAccount","text":"<p>A <code>serviceAccount</code> volume references a Kubernetes <code>ServiceAccount</code>. A <code>serviceAccount</code> can be presented to the VM as disks or as a filesystem. Each method is described in the following sections and both have some advantages and disadvantages, e.g. <code>disk</code> does not support dynamic change propagation and <code>filesystem</code> does not support live migration. Therefore, depending on the use-case, one or the other may be more suitable.</p>"},{"location":"storage/disks_and_volumes/#as-a-disk_2","title":"As a disk","text":"<p>By using disk, a new <code>iso</code> disk will be allocated with the content of the service account (<code>namespace</code>, <code>token</code> and <code>ca.crt</code>), which needs to be mounted in the VM. For automatic mounting, see the <code>configMap</code> and <code>secret</code> examples above.</p> <p>Note: Currently, ServiceAccount update propagation is not supported. If a ServiceAccount is updated, only a pod will be aware of changes, not running VMIs.</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n        name: containerdisk\n      - disk:\n        name: serviceaccountdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: serviceaccountdisk\n    serviceAccount:\n      serviceAccountName: default\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#as-a-filesystem_2","title":"As a filesystem","text":"<p>By using filesystem, <code>serviceAccounts</code> are shared through <code>virtiofs</code>. In contrast with using disk for sharing <code>serviceAccounts</code>, <code>filesystem</code> allows you to dynamically propagate changes on <code>serviceAccounts</code> to VMIs (i.e. the VM does not need to be rebooted).</p> <p>Note: Currently, VMIs can not be live migrated since <code>virtiofs</code> does not support live migration.</p> <p>To share a given <code>serviceAccount</code>, the following VM definition could be used:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      filesystems:\n        - name: serviceaccount-fs\n          virtiofs: {}\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          chpasswd:\n            expire: false\n          password: fedora\n          user: fedora\n          bootcmd:\n            # mount the ConfigMap\n            - \"sudo mkdir /mnt/serviceaccount\"\n            - \"sudo mount -t virtiofs serviceaccount-fs /mnt/serviceaccount\"\n      name: cloudinitdisk\n    - name: serviceaccount-fs\n      serviceAccount:\n        serviceAccountName: default\n</code></pre>"},{"location":"storage/disks_and_volumes/#downwardmetrics","title":"downwardMetrics","text":"<p><code>downwardMetrics</code> expose a limited set of VM and host metrics to the guest. The format is compatible with vhostmd.</p> <p>Getting a limited set of host and VM metrics is in some cases required to allow third-parties diagnosing performance issues on their appliances. One prominent example is SAP HANA.</p> <p>In order to expose <code>downwardMetrics</code> to VMs, the methods <code>disk</code> and <code>virtio-serial port</code> are supported.</p> <p>Note: The DownwardMetrics feature gate must be enabled to use the metrics. Available starting with KubeVirt v0.42.0.</p>"},{"location":"storage/disks_and_volumes/#disk_1","title":"Disk","text":"<p>A volume is created, and it is exposed to the guest as a raw block volume.  KubeVirt will update it periodically (by default, every 5 seconds).</p> <p>Example: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: metrics\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - name: metrics\n    downwardMetrics: {}\n</code></pre></p>"},{"location":"storage/disks_and_volumes/#virtio-serial-port","title":"Virtio-serial port","text":"<p>This method uses a virtio-serial port to expose the metrics data to the VM. KubeVirt creates a port named <code>/dev/virtio-ports/org.github.vhostmd.1</code> inside the VM, in which the Virtio Transport protocol is supported. <code>downwardMetrics</code> can be retrieved from this port. See vhostmd documentation under <code>Virtio Transport</code> for further information.</p> <p>To expose the metrics using a virtio-serial port, a <code>downwardMetrics</code> device must be added (i.e., <code>spec.domain.devices.downwardMetrics: {}</code>).</p> <p>Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-fedora\n  name: vmi-fedora\nspec:\n  domain:\n    devices:\n      downwardMetrics: {}\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n</code></pre>"},{"location":"storage/disks_and_volumes/#accessing-metrics-data","title":"Accessing Metrics Data","text":"<p>To access the DownwardMetrics shared with a disk or a virtio-serial port, the <code>vm-dump-metrics</code> tool can be used:</p> <pre><code>$ sudo dnf install -y vm-dump-metrics\n$ sudo vm-dump-metrics\n&lt;metrics&gt;\n  &lt;metric type=\"string\" context=\"host\"&gt;\n    &lt;name&gt;HostName&lt;/name&gt;\n    &lt;value&gt;node01&lt;/value&gt;\n[...]\n  &lt;metric type=\"int64\" context=\"host\" unit=\"s\"&gt;\n    &lt;name&gt;Time&lt;/name&gt;\n    &lt;value&gt;1619008605&lt;/value&gt;\n  &lt;/metric&gt;\n  &lt;metric type=\"string\" context=\"host\"&gt;\n    &lt;name&gt;VirtualizationVendor&lt;/name&gt;\n    &lt;value&gt;kubevirt.io&lt;/value&gt;\n  &lt;/metric&gt;\n&lt;/metrics&gt;\n</code></pre> <p><code>vm-dump-metrics</code> is useful as a standalone tool to verify the serial port is working and to inspect the metrics.  However, applications that consume metrics will usually connect to the virtio-serial port themselves.</p> <p>Note: The tool <code>vm-dump-metrics</code> provides the option <code>--virtio</code> in case the virtio-serial port is used. Please, refer to <code>vm-dump-metrics --help</code> for further information.</p>"},{"location":"storage/disks_and_volumes/#high-performance-features","title":"High Performance Features","text":""},{"location":"storage/disks_and_volumes/#iothreads","title":"IOThreads","text":"<p>Libvirt has the ability to use IOThreads for dedicated disk access (for supported devices). These are dedicated event loop threads that perform block I/O requests and improve scalability on SMP systems. KubeVirt exposes this libvirt feature through the <code>ioThreadsPolicy</code> setting. Additionally, each <code>Disk</code> device exposes a <code>dedicatedIOThread</code> setting. This is a boolean that indicates the specified disk should be allocated an exclusive IOThread that will never be shared with other disks.</p> <p>Currently valid policies are <code>shared</code> and <code>auto</code>. If <code>ioThreadsPolicy</code> is omitted entirely, use of IOThreads will be disabled. However, if any disk requests a dedicated IOThread, <code>ioThreadsPolicy</code> will be enabled and default to <code>shared</code>.</p>"},{"location":"storage/disks_and_volumes/#shared","title":"Shared","text":"<p>An <code>ioThreadsPolicy</code> of <code>shared</code> indicates that KubeVirt should use one thread that will be shared by all disk devices. This policy stems from the fact that large numbers of IOThreads is generally not useful as additional context switching is incurred for each thread.</p> <p>Disks with <code>dedicatedIOThread</code> set to <code>true</code> will not use the shared thread, but will instead be allocated an exclusive thread. This is generally useful if a specific Disk is expected to have heavy I/O traffic, e.g. a database spindle.</p>"},{"location":"storage/disks_and_volumes/#auto","title":"Auto","text":"<p><code>auto</code> IOThreads indicates that KubeVirt should use a pool of IOThreads and allocate disks to IOThreads in a round-robin fashion. The pool size is generally limited to twice the number of VCPU's allocated to the VM. This essentially attempts to dedicate disks to separate IOThreads, but only up to a reasonable limit. This would come in to play for systems with a large number of disks and a smaller number of CPU's for instance.</p> <p>As a caveat to the size of the IOThread pool, disks with <code>dedicatedIOThread</code> will always be guaranteed their own thread. This effectively diminishes the upper limit of the number of threads allocated to the rest of the disks. For example, a VM with 2 CPUs would normally use 4 IOThreads for all disks. However if one disk had <code>dedicatedIOThread</code> set to true, then KubeVirt would only use 3 IOThreads for the shared pool.</p> <p>There is always guaranteed to be at least one thread for disks that will use the shared IOThreads pool. Thus if a sufficiently large number of disks have dedicated IOThreads assigned, <code>auto</code> and <code>shared</code> policies would essentially result in the same layout.</p>"},{"location":"storage/disks_and_volumes/#iothreads-with-dedicated-pinned-cpus","title":"IOThreads with Dedicated (pinned) CPUs","text":"<p>When guest's vCPUs are pinned to a host's physical CPUs, it is also best to pin the IOThreads to specific CPUs to prevent these from floating between the CPUs. KubeVirt will automatically calculate and pin each IOThread to a CPU or a set of CPUs, depending on the ration between them. In case there are more IOThreads than CPUs, each IOThread will be pinned to a CPU, in a round-robin fashion. Otherwise, when there are fewer IOThreads than CPU, each IOThread will be pinned to a set of CPUs.</p>"},{"location":"storage/disks_and_volumes/#iothreads-with-qemu-emulator-thread-and-dedicated-pinned-cpus","title":"IOThreads with QEMU Emulator thread and Dedicated (pinned) CPUs","text":"<p>To further improve the vCPUs latency, KubeVirt can allocate an additional dedicated physical CPU<sup>1</sup>, exclusively for the emulator thread, to which it will be pinned. This will effectively \"isolate\" the emulator thread from the vCPUs of the VMI. When <code>ioThreadsPolicy</code> is set to <code>auto</code> IOThreads will also be \"isolated\" from the vCPUs and placed on the same physical CPU as the QEMU emulator thread.</p>"},{"location":"storage/disks_and_volumes/#examples","title":"Examples","text":""},{"location":"storage/disks_and_volumes/#shared-iothreads","title":"Shared IOThreads","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-shared\n  name: vmi-shared\nspec:\n  domain:\n    ioThreadsPolicy: shared\n    cpu:\n      cores: 2\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: vmi-shared_disk\n      - disk:\n          bus: virtio\n        name: emptydisk\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk2\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk3\n      - disk:\n          bus: virtio\n        name: emptydisk4\n      - disk:\n          bus: virtio\n        name: emptydisk5\n      - disk:\n          bus: virtio\n        name: emptydisk6\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: vmi-shared_disk\n    persistentVolumeClaim:\n      claimName: vmi-shared_pvc\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk2\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk3\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk4\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk5\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk6\n</code></pre> <p>In this example, emptydisk and emptydisk2 both request a dedicated IOThread. vmi-shared_disk, and emptydisk 3 through 6 will all shared one IOThread.</p> <pre><code>mypvc:        1\nemptydisk:    2\nemptydisk2:   3\nemptydisk3:   1\nemptydisk4:   1\nemptydisk5:   1\nemptydisk6:   1\n</code></pre>"},{"location":"storage/disks_and_volumes/#auto-iothreads","title":"Auto IOThreads","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-shared\n  name: vmi-shared\nspec:\n  domain:\n    ioThreadsPolicy: auto\n    cpu:\n      cores: 2\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: mydisk\n      - disk:\n          bus: virtio\n        name: emptydisk\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk2\n        dedicatedIOThread: true\n      - disk:\n          bus: virtio\n        name: emptydisk3\n      - disk:\n          bus: virtio\n        name: emptydisk4\n      - disk:\n          bus: virtio\n        name: emptydisk5\n      - disk:\n          bus: virtio\n        name: emptydisk6\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  volumes:\n  - name: mydisk\n    persistentVolumeClaim:\n      claimName: mypvc\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk2\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk3\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk4\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk5\n  - emptyDisk:\n      capacity: 1Gi\n    name: emptydisk6\n</code></pre> <p>This VM is identical to the first, except it requests auto IOThreads. <code>emptydisk</code> and <code>emptydisk2</code> will still be allocated individual IOThreads, but the rest of the disks will be split across 2 separate iothreads (twice the number of CPU cores is 4).</p> <p>Disks will be assigned to IOThreads like this:</p> <pre><code>mypvc:        1\nemptydisk:    3\nemptydisk2:   4\nemptydisk3:   2\nemptydisk4:   1\nemptydisk5:   2\nemptydisk6:   1\n</code></pre>"},{"location":"storage/disks_and_volumes/#virtio-block-multi-queue","title":"Virtio Block Multi-Queue","text":"<p>Block Multi-Queue is a framework for the Linux block layer that maps Device I/O queries to multiple queues. This splits I/O processing up across multiple threads, and therefor multiple CPUs. libvirt recommends that the number of queues used should match the number of CPUs allocated for optimal performance.</p> <p>This feature is enabled by the <code>BlockMultiQueue</code> setting under <code>Devices</code>:</p> <pre><code>spec:\n  domain:\n    devices:\n      blockMultiQueue: true\n      disks:\n      - disk:\n          bus: virtio\n        name: mydisk\n</code></pre> <p>Note: Due to the way KubeVirt implements CPU allocation, blockMultiQueue can only be used if a specific CPU allocation is requested. If a specific number of CPUs hasn't been allocated to a VirtualMachine, KubeVirt will use all CPU's on the node on a best effort basis. In that case the amount of CPU allocation to a VM at the host level could change over time. If blockMultiQueue were to request a number of queues to match all the CPUs on a node, that could lead to over-allocation scenarios. To avoid this, KubeVirt enforces that a specific slice of CPU resources is requested in order to take advantage of this feature.</p>"},{"location":"storage/disks_and_volumes/#example","title":"Example","text":"<pre><code>metadata:\n  name: testvmi-disk\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nspec:\n  domain:\n    resources:\n      requests:\n        memory: 64M\n        cpu: 4\n    devices:\n      blockMultiQueue: true\n      disks:\n      - name: mypvcdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: mypvcdisk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre> <p>This example will enable Block Multi-Queue for the disk <code>mypvcdisk</code> and allocate 4 queues (to match the number of CPUs requested).</p>"},{"location":"storage/disks_and_volumes/#disk-device-cache","title":"Disk device cache","text":"<p>KubeVirt supports <code>none</code>, <code>writeback</code>, and <code>writethrough</code> KVM/QEMU cache modes.</p> <ul> <li> <p><code>none</code> I/O from the guest is not cached on the host. Use this option     for guests with large I/O requirements. This option is generally the     best choice.</p> </li> <li> <p><code>writeback</code> I/O from the guest is cached on the host and written through     to the physical media when the guest OS issues a flush.</p> </li> <li> <p><code>writethrough</code> I/O from the guest is cached on the host but must be written     through to the physical medium before the write operation completes.</p> </li> </ul> <p>Important: <code>none</code> cache mode is set as default if the file system supports direct I/O, otherwise, <code>writethrough</code> is used.</p> <p>Note: It is possible to force a specific cache mode, although if <code>none</code> mode has been chosen and the file system does not support direct I/O then started VMI will return an error.</p> <p>Example: force <code>writethrough</code> cache mode</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-pvc\n  name: vmi-pvc\nspec:\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: pvcdisk\n        cache: writethrough\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 64M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: pvcdisk\n    persistentVolumeClaim:\n      claimName: disk-alpine\nstatus: {}\n</code></pre>"},{"location":"storage/disks_and_volumes/#disk-sharing","title":"Disk sharing","text":"<p>Shareable disks allow multiple VMs to share the same underlying storage. In order to use this feature, special care is required because this could lead to data corruption and the loss of important data. Shareable disks demand either data synchronization at the application level or the use of clustered filesystems. These advanced configurations are not within the scope of this documentation and are use-case specific.</p> <p>If the <code>shareable</code> option is set, it indicates to libvirt/QEMU that the disk is going to be accessed by multiple VMs and not to create a lock for the writes.</p> <p>In this example, we use Rook Ceph in order to dynamically provisioning the PVC. <pre><code>apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: block-pvc\nspec:\n  accessModes:\n    - ReadWriteMany\n  volumeMode: Block\n  resources:\n    requests:\n      storage: 1Gi\n  storageClassName: rook-ceph-block\n</code></pre> <pre><code>$ kubectl get pvc\nNAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE\nblock-pvc   Bound    pvc-0a161bb2-57c7-4d97-be96-0a20ff0222e2   1Gi        RWO            rook-ceph-block   51s\n</code></pre> Then, we can declare 2 VMs and set the <code>shareable</code> option to true for the shared disk. <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-block-1\n  name: vm-block-1\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-block-1\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          - disk:\n              bus: virtio\n            shareable: true\n            name: block-disk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 2G\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\n      - name: block-disk\n        persistentVolumeClaim:\n          claimName: block-pvc\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-block-2\n  name: vm-block-2\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-block-2\n    spec:\n      affinity:\n        podAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n          - labelSelector:\n              matchExpressions:\n              - key: kubevirt.io/vm\n                operator: In\n                values:\n                - vm-block-1\n            topologyKey: \"kubernetes.io/hostname\"\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          - disk:\n              bus: virtio\n            shareable: true\n            name: block-disk\n        machine:\n          type: \"\"\n        resources:\n          requests:\n            memory: 2G\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/fedora-with-test-tooling-container-disk:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\n      - name: block-disk\n        persistentVolumeClaim:\n          claimName: block-pvc                                        \n</code></pre> We can now attempt to write a string from the first guest and then read the string from the second guest to test that the sharing is working. <pre><code>$ virtctl console vm-block-1\n$ printf \"Test awesome shareable disks\" | sudo dd  of=/dev/vdc bs=1 count=150 conv=notrunc\n28+0 records in\n28+0 records out\n28 bytes copied, 0.0264182 s, 1.1 kB/s\n# Log into the second guest\n$ virtctl console vm-block-2\n$ sudo dd  if=/dev/vdc bs=1 count=150 conv=notrunc\nTest awesome shareable disks150+0 records in\n150+0 records out\n150 bytes copied, 0.136753 s, 1.1 kB/s\n</code></pre></p> <p>If you are using local devices or RWO PVCs, setting the affinity on the VMs that share the storage guarantees they will be scheduled on the same node. In the example, we set the affinity on the second VM using the label used on the first VM. If you are using shared storage with RWX PVCs, then the affinity rule is not necessary as the storage can be attached simultaneously on multiple nodes.</p>"},{"location":"storage/disks_and_volumes/#sharing-directories-with-vms","title":"Sharing Directories with VMs","text":"<p><code>Virtiofs</code> allows to make visible external filesystems to <code>KubeVirt</code> VMs. <code>Virtiofs</code> is a shared file system that lets VMs access a directory tree on the host. Further details can be found at Official Virtiofs Site.</p>"},{"location":"storage/disks_and_volumes/#non-privileged-and-privileged-sharing-modes","title":"Non-Privileged and Privileged Sharing Modes","text":"<p>KubeVirt supports two PVC sharing modes: non-privileged and privileged.</p> <p>The non-privileged mode is enabled by default. This mode has the advantage of not requiring any administrative privileges for creating the VM. However, it has some limitations:</p> <ul> <li>The virtiofsd daemon (the daemon in charge of sharing the PVC with the VM) will run with the QEMU UID/GID (107),   and cannot switch between different UIDs/GIDs.   Therefore, it will only have access to directories and files that UID/GID 107 has permission to.   Additionally, when creating new files they will always be created with QEMU's UID/GID regardless of the UID/GID of the   process within the guest.</li> <li>Extended attributes are not supported. </li> </ul> <p>To switch to the privileged mode, the feature gate ExperimentalVirtiofsSupport has to be enabled. Take into account that this mode requires privileges to run rootful containers. </p>"},{"location":"storage/disks_and_volumes/#sharing-persistent-volume-claims","title":"Sharing Persistent Volume Claims","text":""},{"location":"storage/disks_and_volumes/#cluster-configuration","title":"Cluster Configuration","text":"<p>We need to create a new VM definition including the <code>spec.devices.disk.filesystems.virtiofs</code> and a PVC. Example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-fs\nspec:\n  domain:\n    devices:\n      disks:\n        - disk:\n            bus: virtio\n          name: containerdisk\n        - disk:\n            bus: virtio\n          name: cloudinitdisk\n      filesystems:\n        - name: virtiofs-disk\n          virtiofs: {}\n    resources:\n      requests:\n        memory: 1024Mi\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: quay.io/containerdisks/fedora:latest\n    - cloudInitNoCloud:\n        userData: |-\n          #cloud-config\n          password: fedora\n          chpasswd: { expire: False }\n      name: cloudinitdisk\n    - name: virtiofs-disk\n      persistentVolumeClaim:\n        claimName: mypvc\n</code></pre>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-vm","title":"Configuration Inside the VM","text":"<p>The following configuration can be done in using startup script. See cloudInitNoCloud section for  more details.  However, we can do it manually by logging in to the VM and mounting it.  Here are examples of how to mount it in a linux and windows VMs:</p> <ul> <li>Linux Example</li> </ul> <pre><code>$ sudo mkdir -p /mnt/disks/virtio\n$ sudo mount -t virtiofs virtiofs-disk /mnt/disks/virtio\n</code></pre> <ul> <li>Windows Example</li> </ul> <p>See this guide for details on startup steps needed for Windows VMs.</p>"},{"location":"storage/disks_and_volumes/#sharing-node-directories","title":"Sharing Node Directories","text":"<p>It is allowed using hostpaths. The following configuration example is shown for illustrative purposes. However, the PVCs method is preferred since using hostpath is generally discouraged for security reasons.</p>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-node","title":"Configuration Inside the Node","text":"<p>To share the directory with the VMs, we need to log in to the node, create the shared directory (if it does not already exist), and set the proper SELinux context label <code>container_file_t</code> to the shared directory. In this example we are going to share a new directory <code>/mnt/data</code> (if the desired directory is an existing one, you can skip the <code>mkdir</code> command):</p> <pre><code>$ mkdir /tmp/data\n$ sudo chcon -t container_file_t /tmp/data\n</code></pre> <p>Note: If you are attempting to share an existing directory, you must first check the SELinux context label with the command <code>ls -Z &lt;directory&gt;</code>. In the case that the label is not present or is not <code>container_file_t</code> you need to label  it with the <code>chcon</code>  command.</p>"},{"location":"storage/disks_and_volumes/#cluster-configuration_1","title":"Cluster Configuration","text":"<p>We need a <code>StorageClass</code> which uses the provider <code>no-provisioner</code>:</p> <pre><code>apiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n   name: no-provisioner-storage-class\nprovisioner: kubernetes.io/no-provisioner\nreclaimPolicy: Delete\nvolumeBindingMode: WaitForFirstConsumer\n</code></pre> <p>To make the shared directory available for VMs, we need to create a PV and a PVC that could be consumed by the VMs:</p> <pre><code>kind: PersistentVolume\napiVersion: v1\nmetadata:\n  name: hostpath\nspec:\n  capacity:\n    storage: 10Gi\n  accessModes:\n    - ReadWriteMany\n  hostPath:\n    path: \"/tmp/data\"\n  storageClassName: \"no-provisioner-storage-class\"\n  nodeAffinity:\n    required:\n      nodeSelectorTerms:\n      - matchExpressions:\n        - key: kubernetes.io/hostname\n          operator: In\n          values:\n          - node01\n---  \napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: hostpath-claim\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: \"no-provisioner-storage-class\"\n  resources:\n    requests:\n      storage: 10Gi\n</code></pre> <p>Note: Change the <code>node01</code> value for the node name where you want the shared directory will be located.</p> <p>The VM definitions have to request the PVC <code>hostpath-claim</code> and attach it as a virtiofs filesystem:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: hostpath-vm\n  name: hostpath\nspec:\n  running: true\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: hostpath\n        kubevirt.io/vm: hostpath\n    spec:\n      domain:\n        cpu:\n          cores: 1\n          sockets: 1\n          threads: 1\n        devices:\n          filesystems:\n            - name: vm-hostpath\n              virtiofs: {}\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          interfaces:\n            - name: default\n              masquerade: {}\n          rng: {}\n        resources:\n          requests:\n            memory: 1Gi\n      networks:\n        - name: default\n          pod: {}\n      terminationGracePeriodSeconds: 180\n      volumes:\n        - containerDisk:\n            image: quay.io/containerdisks/fedora:latest\n          name: containerdisk\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              chpasswd:\n                expire: false\n              password: password\n              user: fedora\n          name: cloudinitdisk\n        - name: vm-hostpath\n          persistentVolumeClaim:\n            claimName: hostpath-claim\n</code></pre>"},{"location":"storage/disks_and_volumes/#configuration-inside-the-vm_1","title":"Configuration Inside the VM","text":"<p>We need to log in to the VM and mount the shared directory:</p> <pre><code>$ sudo mount -t virtiofs vm-hostpath /mnt\n</code></pre>"},{"location":"storage/disks_and_volumes/#update-volumes-strategy","title":"Update volumes strategy","text":"<p>The <code>updateVolumesStrategy</code> field is used to specify the strategy for updating the volumes of a running VM. The following strategies are supported:   * <code>Replacement</code>: the update volumes will be replaced upon the VM restart.   * <code>Migration</code>: the update of the volumes will trigger a storage migration of     the old volumes to the new ones. More details about volume migration can be     found in the volume migration documentation.</p> <p>The update volume migration depends on the feature gate <code>VolumesUpdateStrategy</code> which depends on the VMLiveUpdateFeatures feature gate and configuration.</p> <p>KubeVirt CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n        - VolumesUpdateStrategy\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre></p>"},{"location":"storage/export_api/","title":"Export API","text":"<p>It can be desirable to export a Virtual Machine and its related disks out of a cluster so you can import that Virtual Machine into another system or cluster. The Virtual Machine disks are the most prominent things you will want to export. The export API makes it possible to declaratively export Virtual Machine disks. It is also possible to export individual PVCs and their contents, for instance when you have created a memory dump from a VM or are using virtio-fs to have a Virtual Machine populate a PVC.</p> <p>In order not to overload the kubernetes API server the data is transferred through a dedicated export proxy server. The proxy server can then be exposed to the outside world through a service associated with an Ingress/Route or NodePort. As an alternative, the <code>port-forward</code> flag can be used with the virtctl integration to bypass the need of an Ingress/Route.</p>"},{"location":"storage/export_api/#export-feature-gate","title":"Export Feature Gate","text":"<p>VMExport support must be enabled in the feature gates to be available. The feature gates field in the KubeVirt CR must be expanded by adding the <code>VMExport</code> to it.</p>"},{"location":"storage/export_api/#export-token","title":"Export token","text":"<p>In order to securely export a Virtual Machine Disk, you must create a token that is used to authorize users accessing the export endpoint. This token must be in the same namespace as the Virtual Machine. The contents of the secret can be passed as a token header or parameter to the export URL. The name of the header or argument is <code>x-kubevirt-export-token</code> with a value that matches the content of the secret. The secret can be named any valid secret in the namespace. We recommend you generate an alpha numeric token of at least 12 characters. The data key should be <code>token</code>. For example:</p> <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: example-token\nstringData:\n  token: 1234567890ab\n</code></pre>"},{"location":"storage/export_api/#export-virtual-machine-volumes","title":"Export Virtual Machine volumes","text":"<p>After you have created the token you can now create a VMExport CR that identifies the Virtual Machine you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\n</code></pre> <p>The following volumes present in the VM will be exported:</p> <ul> <li>PersistentVolumeClaims</li> <li>DataVolumes</li> <li>MemoryDump</li> </ul> <p>All other volume types are not exported. To avoid the export of inconsistent data, a Virtual Machine can only be exported while it is powered off. Any active VM exports will be terminated if the Virtual Machine is started. To export data from a running Virtual Machine you must first create a Virtual Machine Snapshot (see below).</p> <p>If the VM contains multiple volumes that can be exported, each volume will get its own URL links. If the VM contains no volumes that can be exported, the VMExport will go into a <code>Skipped</code> phase, and no export server is started.</p>"},{"location":"storage/export_api/#export-virtual-machine-snapshot-volumes","title":"Export Virtual Machine Snapshot volumes","text":"<p>You can create a VMExport CR that identifies the Virtual Machine Snapshot you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"snapshot.kubevirt.io\"\n    kind: VirtualMachineSnapshot\n    name: example-vmsnapshot\n</code></pre> <p>When you create a VMExport based on a Virtual Machine Snapshot, the controller will attempt to create PVCs from the volume snapshots contained in Virtual Machine Snapshot. Once all the PVCs are ready, the export server will start and you can begin the export. If the Virtual Machine Snapshot contains multiple volumes that can be exported, each volume will get its own URL links. If the Virtual Machine snapshot contains no volumes that can be exported, the VMExport will go into a <code>skipped</code> phase, and no export server is started.</p>"},{"location":"storage/export_api/#export-persistent-volume-claim","title":"Export Persistent Volume Claim","text":"<p>You can create a VMExport CR that identifies the Persistent Volume Claim (PVC) you want to export. You can create a VMExport that looks like this:</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n</code></pre> <p>In this example the PVC name is <code>example-pvc</code>. Note the PVC doesn't need to contain a Virtual Machine Disk, it can contain any content, but the main use case is exporting Virtual Machine Disks. After you post this yaml to the cluster, a new export server is created in the same namespace as the PVC. If the source PVC is in use by another pod (such as the virt-launcher pod) then the export will remain pending until the PVC is no longer in use. If the exporter server is active and another pod starts using the PVC, the exporter server will be terminated until the PVC is not in use anymore.</p>"},{"location":"storage/export_api/#export-status-links","title":"Export status links","text":"<p>The VirtualMachineExport CR will contain a status with internal and external links to the export service. The internal links are only valid inside the cluster, and the external links are valid for external access through an Ingress or Route. The <code>cert</code> field will contain the CA that signed the certificate of the export server for internal links, or the CA that signed the Route or Ingress.</p>"},{"location":"storage/export_api/#kubevirt-content-type","title":"KubeVirt content-type","text":"<p>The following is an example of exporting a PVC that contains a KubeVirt disk image. The controller determines if the PVC contains a kubevirt disk by checking if there is a special annotation on the PVC, or if there is a DataVolume ownerReference on the PVC, or if the PVC has a volumeMode of block.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img\n        - format: gzip\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-disk/disk.img.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img\n        - format: gzip\n          url: https://virt-export-example-export.example.svc/volumes/example-disk/disk.img.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#archive-content-type","title":"Archive content-type","text":"<p>Archive content-type is automatically selected if we are unable to determine the PVC contains a KubeVirt disk. The archive will contain all the files that are in the PVC.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: dir\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example/dir\n        - format: tar.gz\n          url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example/disk.tar.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: dir\n          url: https://virt-export-example-export.example.svc/volumes/example/dir\n        - format: tar.gz\n          url: https://virt-export-example-export.example.svc/volumes/example/disk.tar.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#manifests","title":"Manifests","text":"<p>The VirtualMachine manifests can be retrieved by accessing the <code>manifests</code> in the VirtualMachineExport status. The <code>all</code> type will return the VirtualMachine manifest, any DataVolumes, and a configMap that contains the public CA certificate of the Ingress/Route of the external URL, or the CA of the export server of the internal URL. The <code>auth-header-secret</code> will be a secret that contains a Containerized Data Importer (CDI) compatible header. This header contains a text version of the export token.</p> <p>Both internal and external links will contain a <code>manifests</code> field. If there are no external links, then there will not be any external manifests either. The virtualMachine <code>manifests</code> field is only available if the source is a <code>VirtualMachine</code> or <code>VirtualMachineSnapshot</code>. Exporting a <code>PersistentVolumeClaim</code> will not generate a Virtual Machine manifest.</p> <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  source:\n    apiGroup: \"\"\n    kind: PersistentVolumeClaim\n    name: example-pvc\n  tokenSecretRef: example-token\nstatus:\n  conditions:\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:10:09Z\"\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    lastTransitionTime: \"2022-06-21T14:09:02Z\"\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      ...\n      manifests:\n      - type: all\n        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/all\n      - type: auth-header-secret\n        url: https://vmexport-proxy.test.net/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/external/manifests/secret\n    internal:\n      ...\n      manifests:\n      - type: all\n        url: https://virt-export-export-pvc.default.svc/internal/manifests/all\n      - type: auth-header-secret\n        url: https://virt-export-export-pvc.default.svc/internal/manifests/secret\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre>"},{"location":"storage/export_api/#format-types","title":"Format types","text":"<p>There are 4 format types that are possible:</p> <ul> <li>Raw. The unaltered raw KubeVirt disk image.</li> <li>Gzip. The raw KubeVirt disk image but gzipped to help with transferring efficiency.</li> <li>Dir. A directory listing, allowing you to find the files contained in the PVC.</li> <li>Tar.gz The contents of the PVC tarred and gzipped in a single file.</li> </ul> <p>Raw and Gzip will be selected if the PVC is determined to be a KubeVirt disk. KubeVirt disks contain a single disk.img file (or are a block device). Dir will return a list of the files in the PVC, to download a specific file you can replace <code>/dir</code> in the URL with the path and file name. For instance if the PVC contains the file <code>/example/data.txt</code> you can replace <code>/dir</code> with <code>/example/data.txt</code> to download just data.txt file. Or you can use the tar.gz URL to get all the contents of the PVC in a tar file.</p>"},{"location":"storage/export_api/#internal-link-certificates","title":"Internal link certificates","text":"<p>The export server certificate is valid for 7 days after which it is rotated by deleting the export server pod and associated secret and generating a new one. If for whatever reason the export server pod dies, the associated secret is also automatically deleted and a new pod and secret are generated. The VirtualMachineExport object status will be automatically updated to reflect the new certificate.</p>"},{"location":"storage/export_api/#external-link-certificates","title":"External link certificates","text":"<p>The external link certificates are associated with the Ingress/Route that points to the service created by the KubeVirt operator. The CA that signed the Ingress/Route will part of the certificates.</p>"},{"location":"storage/export_api/#ttl-time-to-live-for-an-export","title":"TTL (Time to live) for an Export","text":"<p>For various reasons (security being one), users should be able to specify a TTL for the VMExport objects that limits the lifetime of an export. This is done via the <code>ttlDuration</code> field which accepts a k8s duration, which defaults to 2 hours when not specified. <pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n    name: example-export\nspec:\n    source:\n        apiGroup: \"kubevirt.io\"\n        kind: VirtualMachine\n        name: example-vm\n    tokenSecretRef: example-token\n    ttlDuration: 1h\n</code></pre></p>"},{"location":"storage/export_api/#virtctl-integration-vmexport","title":"virtctl integration: vmexport","text":"<p>The virtctl <code>vmexport</code> command allows users to interact with the export API in an easy-to-use way.</p> <p><code>vmexport</code> uses two mandatory arguments:</p> <ul> <li>The vmexport functions (create|delete|download).</li> <li>The VirtualMachineExport name.</li> </ul> <p>These three functions are:</p>"},{"location":"storage/export_api/#create","title":"Create","text":"<pre><code># Creates a VMExport object according to the specified flag.\n\n# The flag should either be:\n\n# --pvc, to specify the name of the pvc to export.\n# --snapshot, to specify the name of the VM snapshot to export.\n# --vm, to specify the name of the Virtual Machine to export.\n\n$ virtctl vmexport create name [flags]\n</code></pre>"},{"location":"storage/export_api/#delete","title":"Delete","text":"<pre><code># Deletes the specified VMExport object.\n\n$ virtctl vmexport delete name\n</code></pre>"},{"location":"storage/export_api/#download","title":"Download","text":"<pre><code># Downloads a volume from the defined VMExport object.\n\n# The main available flags are:\n\n# --output, mandatory flag to specify the output file.\n# --volume, optional flag to specify the name of the downloadable volume.\n# --vm|--snapshot|--pvc, if specified, are used to create the VMExport object assuming it doesn't exist. The name of the object to export has to be specified.\n# --format, optional flag to specify wether to download the file in compressed (default) or raw format.\n# --port-forward, optional flag to easily download the volume without the need of an ingress or route. Also, the local port can be optionally specified with the --local-port flag.\n\n$ virtctl vmexport download name [flags]\n</code></pre> <p>By default, the volume will be downloaded in compressed format. Users can specify the desired format (gzip or raw) by using the <code>format</code> flag, as shown below:</p> <pre><code># Downloads a volume from the defined VMExport object and, if necessary, decompresses it.\n$ virtctl vmexport download name --format=raw [flags]\n</code></pre>"},{"location":"storage/export_api/#ttl-time-to-live","title":"TTL (Time to live)","text":"<p>TTL can also be added when creating a VMExport via virtctl <pre><code>$ virtctl vmexport create name --ttl=1h\n</code></pre></p> <p>For more information about usage and examples:</p> <pre><code>$ virtctl vmexport --help\n\nExport a VM volume.\n\nUsage:\n  virtctl vmexport [flags]\n\nExamples:\n  # Create a VirtualMachineExport to export a volume from a virtual machine:\n    virtctl vmexport create vm1-export --vm=vm1\n\n    # Create a VirtualMachineExport to export a volume from a virtual machine snapshot\n    virtctl vmexport create snap1-export --snapshot=snap1\n\n    # Create a VirtualMachineExport to export a volume from a PVC\n    virtctl vmexport create pvc1-export --pvc=pvc1\n\n    # Delete a VirtualMachineExport resource\n    virtctl vmexport delete snap1-export\n\n    # Download a volume from an already existing VirtualMachineExport (--volume is optional when only one volume is available)\n    virtctl vmexport download vm1-export --volume=volume1 --output=disk.img.gz\n\n    # Create a VirtualMachineExport and download the requested volume from it\n    virtctl vmexport download vm1-export --vm=vm1 --volume=volume1 --output=disk.img.gz\n\nFlags:\n  -h, --help              help for vmexport\n      --insecure          When used with the 'download' option, specifies that the http request should be insecure.\n      --keep-vme          When used with the 'download' option, specifies that the vmexport object should not be deleted after the download finishes.\n      --output string     Specifies the output path of the volume to be downloaded.\n      --pvc string        Sets PersistentVolumeClaim as vmexport kind and specifies the PVC name.\n      --snapshot string   Sets VirtualMachineSnapshot as vmexport kind and specifies the snapshot name.\n      --vm string         Sets VirtualMachine as vmexport kind and specifies the vm name.\n      --volume string     Specifies the volume to be downloaded.\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre>"},{"location":"storage/export_api/#use-cases","title":"Use cases","text":""},{"location":"storage/export_api/#clone-vm-from-one-cluster-to-another-cluster","title":"Clone VM from one cluster to another cluster","text":"<p>If you want to transfer KubeVirt disk images from a source cluster to another target cluster, you can use the VMExport in the source to expose the disks and use Containerized Data Importer (CDI) in the target cluster to import the image into the target cluster. Let's assume we have an Ingress or Route in the source cluster that exposes the export proxy with the following example domain <code>virt-exportproxy-example.example.com</code> and we have a Virtual Machine in the source cluster with one disk, which looks like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-example-datavolume\n  name: example-vm\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: example-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        resources:\n          requests:\n            storage: 20Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://quay.io/containerdisks/centos-stream:9\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-example-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 2Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: example-dv\n        name: datavolumedisk1\n</code></pre> <p>This is a VM that has a DataVolume (DV) <code>example-dv</code> that is populated from a container disk and we want to export that disk to the target cluster. To export this VM we have to create a token that we can use in the target cluster to get access to the export, or we can let the export controller generate one for us. For example</p> <p><pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: example-token\nstringData:\n  token: 1234567890ab\n</code></pre> The value of the token is <code>1234567890ab</code> hardly a secure token, but it is an example. We can now create a VMExport that looks like this:</p> <p><pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\nspec:\n  tokenSecretRef: example-token #optional, if omitted the export controller will generate a token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\n</code></pre> If the VM is not running the status of the VMExport object will get updated once the export-server pod is running to look something like this:</p> <p><pre><code>apiVersion: export.kubevirt.io/v1beta1\nkind: VirtualMachineExport\nmetadata:\n  name: example-export\n  namespace: example\nspec:\n  tokenSecretRef: example-token\n  source:\n    apiGroup: \"kubevirt.io\"\n    kind: VirtualMachine\n    name: example-vm\nstatus:\n  conditions:\n  - lastProbeTime: null\n    reason: podReady\n    status: \"True\"\n    type: Ready\n  - lastProbeTime: null\n    reason: pvcBound\n    status: \"True\"\n    type: PVCReady\n  links:\n    external:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img\n        - format: gzip\n          url: https://virt-exportproxy-example.example.com/api/export.kubevirt.io/v1beta1/namespaces/example/virtualmachineexports/example-export/volumes/example-dv/disk.img.gz\n        name: example-disk\n    internal:\n      cert: |-\n        -----BEGIN CERTIFICATE-----\n        ...\n        -----END CERTIFICATE-----\n      volumes:\n      - formats:\n        - format: raw\n          url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img\n        - format: gzip\n          url: https://virt-export-example-export.example.svc/volumes/example-dv/disk.img.gz\n        name: example-disk\n  phase: Ready\n  serviceName: virt-export-example-export\n</code></pre> Note in this example we are in the <code>example</code> namespace in the source cluster, which is why the internal links domain ends with <code>.example.svc</code>. The external links are what will be visible to outside of the source cluster, so we can use that for when we import into the target cluster.</p> <p>Now we are ready to import this disk into the target cluster. In order for CDI to import, we will need to provide appropriate yaml that contains the following: - CA cert (as config map) - The token needed to access the disk images in a CDI compatible format - The VM yaml - DataVolume yaml (optional if not part of the VM definition)</p> <p>virtctl provides an additional argument to the download command called <code>--manifest</code> that will retrieve the appropriate information from the export server, and either save it to a file with the <code>--output</code> argument or write to standard out. By default this output will not contain the header secret as it contains the token in plaintext. To get the header secret you specify the <code>--include-secret</code> argument. The default output format is <code>yaml</code> but it is possible to get <code>json</code> output as well.</p> <p>Assuming there is a running VirtualMachineExport called <code>example-export</code> and the same namespace exists in the target cluster. The name of the kubeconfig of the target cluster is named <code>kubeconfig-target</code>, to clone the vm into the target cluster run the following commands:</p> <pre><code>$ virtctl vmexport download example-export --manifest --include-secret --output=import.yaml\n$ kubectl apply -f import.yaml --kubeconfig=kubeconfig-target\n</code></pre> <p>The first command generates the yaml and writes it to <code>import.yaml</code>. The second command applies the generated yaml to the target cluster. It is possible to combine the two commands writing to standard <code>out</code> with the first command, and piping it into the second command. Use this option if the export token should not be written to a file anywhere. This will create the VM in the target cluster, and provides CDI in the target cluster with everything required to import the disk images.</p> <p>After the import completes you should be able to start the VM in the target cluster.</p>"},{"location":"storage/export_api/#download-a-vm-volume-locally-using-virtctl-vmexport","title":"Download a VM volume locally using virtctl vmexport","text":"<p>Several steps from the previous section can be simplified considerably by using the <code>vmexport</code> command.</p> <p>Again, let's assume we have an Ingress or Route in our cluster that exposes the export proxy, and that we have a Virtual Machine in the cluster with one disk like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-example-datavolume\n  name: example-vm\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: example-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        resources:\n          requests:\n            storage: 20Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://quay.io/containerdisks/centos-stream:9\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-example-datavolume\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n        resources:\n          requests:\n            memory: 2Gi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: example-dv\n        name: datavolumedisk1\n</code></pre> <p>Once we meet these requirements, the process of downloading the volume locally can be accomplished by different means:</p>"},{"location":"storage/export_api/#performing-each-step-separately","title":"Performing each step separately","text":"<p>We can download the volume by performing every single step in a different command. We start by creating the export object:</p> <pre><code># We use an arbitrary name for the VMExport object, but specify our VM name in the flag.\n\n$ virtctl vmexport create vmexportname --vm=example-vm\n</code></pre> <p>Then, we download the volume in the specified output:</p> <pre><code># Since our virtual machine only has one volume, there's no need to specify the volume name with the --volume flag.\n\n# After the download, the VMExport object is deleted by default, so we are using the optional --keep-vme flag to delete it manually.\n\n$ virtctl vmexport download vmexportname --output=/tmp/disk.img --keep-vme\n</code></pre> <p>Lastly, we delete the VMExport object:</p> <pre><code>$ virtctl vmexport delete vmexportname\n</code></pre>"},{"location":"storage/export_api/#performing-one-single-step","title":"Performing one single step","text":"<p>All the previous steps can be simplified in one, single command:</p> <pre><code># Since we are using a create flag (--vm) with download, the command creates the object assuming the VMExport doesn't exist.\n\n# Also, since we are not using --keep-vme, the VMExport object is deleted after the download.\n\n$ virtctl vmexport download vmexportname --vm=example-vm --output=/tmp/disk.img\n</code></pre> <p>After the download finishes, we can find our disk in <code>/tmp/disk.img</code>.</p>"},{"location":"storage/guestfs/","title":"Usage of libguestfs-tools and virtctl guestfs","text":"<p>Libguestfs tools are a set of utilities for accessing and modifying VM disk images. The command <code>virtctl guestfs</code> helps to deploy an interactive container with the libguestfs-tools and the PVC attached to it. This command is particularly useful if the users need to modify, inspect or debug VM disks on a PVC. <pre><code>$ virtctl guestfs -h\nCreate a pod with libguestfs-tools, mount the pvc and attach a shell to it. The pvc is mounted under the /disks directory inside the pod for filesystem-based pvcs, or as /dev/vda for block-based pvcs\n\nUsage:\n  virtctl guestfs [flags]\n\nExamples:\n  # Create a pod with libguestfs-tools, mount the pvc and attach a shell to it:\n  virtctl guestfs &lt;pvc-name&gt;\n\nFlags:\n  -h, --help                 help for guestfs\n      --image string         libguestfs-tools container image\n      --kvm                  Use kvm for the libguestfs-tools container (default true)\n      --pull-policy string   pull policy for the libguestfs image (default \"IfNotPresent\")\n\nUse \"virtctl options\" for a list of global command-line options (applies to all commands).\n</code></pre></p> <p>By default <code>virtctl guestfs</code> sets up <code>kvm</code> for the interactive container. This considerably speeds up the execution of the libguestfs-tools since they use QEMU. If the cluster doesn't have any kvm supporting nodes, the user must disable kvm by setting the option <code>--kvm=false</code>. If not set, the libguestfs-tools pod will remain pending because it cannot be scheduled on any node.</p> <p>The command automatically uses the image exposed by KubeVirt under the http endpoint <code>/apis/subresources.kubevirt.io/&lt;kubevirt-version&gt;/guestfs</code>, but it can be configured to use a custom image by using the option <code>--image</code>. Users can also overwrite the pull policy of the image by setting the option <code>pull-policy</code>.</p> <p>The command checks if a PVC is used by another pod in which case it will fail. However, once libguestfs-tools has started, the setup doesn't prevent a new pod starting and using the same PVC. The user needs to verify that there are no active virtctl guestfs pods before starting the VM which accesses the same PVC.</p> <p>Currently, <code>virtctl guestfs</code> supports only a single PVC. Future versions might support multiple PVCs attached to the interactive pod.</p>"},{"location":"storage/guestfs/#examples-and-use-cases","title":"Examples and use-cases","text":"<p>Generally, the user can take advantage of the <code>virtctl guestfs</code> command for all typical usage of libguestfs-tools. It is strongly recommended to consult the official documentation. This command simply aims to help in configuring the correct containerized environment in the Kubernetes cluster where KubeVirt is installed.</p> <p>For all the examples, the user has to start the interactive container by referencing the PVC in the <code>virtctl guestfs</code> command. This will deploy the interactive pod and attach the stdin and stdout. </p> <p>Example:</p> <p><pre><code>$ virtctl guestfs pvc-test\nUse image: registry:5000/kubevirt/libguestfs-tools@sha256:6644792751b2ba9442e06475a809448b37d02d1937dbd15ad8da4d424b5c87dd \nThe PVC has been mounted at /disk \nWaiting for container libguestfs still in pending, reason: ContainerCreating, message:  \nWaiting for container libguestfs still in pending, reason: ContainerCreating, message:  \nbash-5.0#\n</code></pre> Once the libguestfs-tools pod has been deployed, the user can access the disk and execute the desired commands. Later, once the user has completed the operations on the disk, simply <code>exit</code> the container and the pod be will automatically terminated.</p> <ol> <li>Inspect the disk filesystem to retrive the version of the OS on the disk: <pre><code>bash-5.0# virt-cat -a disk.img /etc/os-release \nNAME=Fedora\nVERSION=\"34 (Cloud Edition)\"\nID=fedora\nVERSION_ID=34\nVERSION_CODENAME=\"\"\nPLATFORM_ID=\"platform:f34\"\nPRETTY_NAME=\"Fedora 34 (Cloud Edition)\"\nANSI_COLOR=\"0;38;2;60;110;180\"\nLOGO=fedora-logo-icon\nCPE_NAME=\"cpe:/o:fedoraproject:fedora:34\"\nHOME_URL=\"https://fedoraproject.org/\"\nDOCUMENTATION_URL=\"https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/\"\nSUPPORT_URL=\"https://fedoraproject.org/wiki/Communicating_and_getting_help\"\nBUG_REPORT_URL=\"https://bugzilla.redhat.com/\"\nREDHAT_BUGZILLA_PRODUCT=\"Fedora\"\nREDHAT_BUGZILLA_PRODUCT_VERSION=34\nREDHAT_SUPPORT_PRODUCT=\"Fedora\"\nREDHAT_SUPPORT_PRODUCT_VERSION=34\nPRIVACY_POLICY_URL=\"https://fedoraproject.org/wiki/Legal:PrivacyPolicy\"\nVARIANT=\"Cloud Edition\"\nVARIANT_ID=cloud\n</code></pre></li> <li>Add users (for example after the disk has been imported using CDI) <pre><code>bash-5.0# virt-customize -a disk.img --run-command 'useradd -m test-user -s /bin/bash' --password  'test-user:password:test-password'\n[   0.0] Examining the guest ...\n[   4.1] Setting a random seed\n[   4.2] Setting the machine ID in /etc/machine-id\n[   4.2] Running: useradd -m test-user -s /bin/bash\n[   4.3] Setting passwords\n[   5.3] Finishing off\n</code></pre></li> <li> <p>Run virt-rescue and repair a broken partition or initrd (for example by running dracut) <pre><code>bash-5.0# virt-rescue -a disk.img\n[...]\nThe virt-rescue escape key is \u2018^]\u2019.  Type \u2018^] h\u2019 for help.\n\n------------------------------------------------------------\n\nWelcome to virt-rescue, the libguestfs rescue shell.\n\nNote: The contents of / (root) are the rescue appliance.\nYou have to mount the guest\u2019s partitions under /sysroot\nbefore you can examine them.\n&gt;&lt;rescue&gt; fdisk -l\nDisk /dev/sda: 6 GiB, 6442450944 bytes, 12582912 sectors\nDisk model: QEMU HARDDISK   \nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\nDisklabel type: gpt\nDisk identifier: F8DC0844-9194-4B34-B432-13FA4B70F278\n\nDevice       Start      End  Sectors Size Type\n/dev/sda1     2048     4095     2048   1M BIOS boot\n/dev/sda2     4096  2101247  2097152   1G Linux filesystem\n/dev/sda3  2101248 12580863 10479616   5G Linux filesystem\n\n\nDisk /dev/sdb: 4 GiB, 4294967296 bytes, 8388608 sectors\nDisk model: QEMU HARDDISK   \nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical/physical): 512 bytes / 512 bytes\nI/O size (minimum/optimal): 512 bytes / 512 bytes\n&gt;&lt;rescue&gt; mount /dev/sda3 sysroot/\n&gt;&lt;rescue&gt; mount /dev/sda2 sysroot/boot\n&gt;&lt;rescue&gt; chroot sysroot/\n&gt;&lt;rescue&gt; ls boot/\nSystem.map-5.11.12-300.fc34.x86_64\nconfig-5.11.12-300.fc34.x86_64\nefi\ngrub2\ninitramfs-0-rescue-8afb5b540fab48728e48e4196a3a48ee.img\ninitramfs-5.11.12-300.fc34.x86_64.img\nloader\nvmlinuz-0-rescue-8afb5b540fab48728e48e4196a3a48ee\n&gt;&lt;rescue&gt; dracut -f boot/initramfs-5.11.12-300.fc34.x86_64.img 5.11.12-300.fc34.x86_64\n[...]\n&gt;&lt;rescue&gt; exit # &lt;- exit from chroot\n&gt;&lt;rescue&gt; umount sysroot/boot\n&gt;&lt;rescue&gt; umount sysroot\n&gt;&lt;rescue&gt; exit\n</code></pre></p> </li> <li> <p>Install an OS from scratch <pre><code>bash-5.0# virt-builder centos-8.2 -o disk.img --root-password password:password-test\n[   1.5] Downloading: http://builder.libguestfs.org/centos-8.2.xz\n######################################################################## 100.0%#=#=#                                                    ######################################################################## 100.0%\n[  58.3] Planning how to build this image\n[  58.3] Uncompressing\n[  65.7] Opening the new disk\n[  70.8] Setting a random seed\n[  70.8] Setting passwords\n[  72.0] Finishing off\n                   Output file: disk.img\n                   Output size: 6.0G\n                 Output format: raw\n            Total usable space: 5.3G\n                    Free space: 4.0G (74%)\n</code></pre></p> </li> <li>Identify the partition and filesystem on the disk <pre><code>bash-5.0# virt-filesystems -a disk.img --partitions --filesystem --long\nName       Type        VFS   Label  MBR  Size        Parent\n/dev/sda2  filesystem  ext4  -      -    1023303680  -\n/dev/sda4  filesystem  xfs   -      -    4710203392  -\n/dev/sda1  partition   -     -      -    1048576     /dev/sda\n/dev/sda2  partition   -     -      -    1073741824  /dev/sda\n/dev/sda3  partition   -     -      -    644874240   /dev/sda\n/dev/sda4  partition   -     -      -    4720689152  /dev/sda\n</code></pre></li> </ol>"},{"location":"storage/guestfs/#limitations","title":"Limitations","text":"<p>Currently, it is not possible to resize the xfs filesystem.</p>"},{"location":"storage/hotplug_volumes/","title":"Hotplug Volumes","text":"<p>KubeVirt now supports hotplugging volumes into a running Virtual Machine Instance (VMI). The volume must be either a block volume or contain a disk image. When a VM that has hotplugged volumes is rebooted, the hotplugged volumes will be attached to the restarted VM. If the volumes are persisted they will become part of the VM spec, and will not be considered hotplugged. If they are not persisted, the volumes will be reattached as hotplugged volumes</p>"},{"location":"storage/hotplug_volumes/#enabling-hotplug-volume-support","title":"Enabling hotplug volume support","text":"<p>Hotplug volume support must be enabled in the feature gates to be supported. The feature gates field in the KubeVirt CR must be expanded by adding the <code>HotplugVolumes</code> to it.</p>"},{"location":"storage/hotplug_volumes/#virtctl-support","title":"Virtctl support","text":"<p>In order to hotplug a volume, you must first prepare a volume. This can be done by using a DataVolume (DV). In the example we will use a blank DV in order to add some extra storage to a running VMI</p> <p><pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: example-volume-hotplug\nspec:\n  source:\n    blank: {}\n  storage:\n    resources:\n      requests:\n        storage: 5Gi\n</code></pre> In this example we are using <code>ReadWriteOnce</code> accessMode, and the default FileSystem volume mode. Volume hotplugging supports all combinations of block volume mode and <code>ReadWriteMany</code>/<code>ReadWriteOnce</code>/<code>ReadOnlyMany</code> accessModes, if your storage supports the combination.</p>"},{"location":"storage/hotplug_volumes/#addvolume","title":"Addvolume","text":"<p>Now lets assume we have started a VMI like the Fedora VMI in examples and the name of the VMI is 'vmi-fedora'. We can add the above blank volume to this running VMI by using the 'addvolume' command  available with virtctl</p> <pre><code>$ virtctl addvolume vmi-fedora --volume-name=example-volume-hotplug\n</code></pre> <p>This will hotplug the volume into the running VMI, and set the serial of the disk to the volume name. In this example it is set to example-hotplug-volume.</p>"},{"location":"storage/hotplug_volumes/#why-virtio-scsi","title":"Why virtio-scsi","text":"<p>The bus of hotplug disk is specified as a <code>scsi</code> disk. Why is it not specified as <code>virtio</code> instead, like regular disks? The reason is a limitation of <code>virtio</code> disks that each disk uses a pcie slot in the virtual machine and there is a maximum of 32 slots. This means there is a low limit on the maximum number of disks you can hotplug especially given that other things will also need pcie slots. Another issue is these slots need to be reserved ahead of time. So if the number of hotplugged disks is not known ahead of time, it is impossible to properly reserve the required number of slots. To work around this issue, each VM has a virtio-scsi controller, which allows the use of a <code>scsi</code> bus for hotplugged disks. This controller allows for hotplugging of over 4 million disks. <code>virtio-scsi</code> is very close in performance to <code>virtio</code></p>"},{"location":"storage/hotplug_volumes/#serial","title":"Serial","text":"<p>You can change the serial of the disk by specifying the --serial parameter, for example: <pre><code>$ virtctl addvolume vmi-fedora --volume-name=example-volume-hotplug --serial=1234567890\n</code></pre></p> <p>The serial will be used in the guest so you can identify the disk inside the guest by the serial. For instance in Fedora the disk by id will contain the serial. <pre><code>$ virtctl console vmi-fedora\n\nFedora 32 (Cloud Edition)\nKernel 5.6.6-300.fc32.x86_64 on an x86_64 (ttyS0)\n\nSSH host key: SHA256:c8ik1A9F4E7AxVrd6eE3vMNOcMcp6qBxsf8K30oC/C8 (ECDSA)\nSSH host key: SHA256:fOAKptNAH2NWGo2XhkaEtFHvOMfypv2t6KIPANev090 (ED25519)\neth0: 10.244.196.144 fe80::d8b7:51ff:fec4:7099\nvmi-fedora login:fedora\nPassword:fedora\n[fedora@vmi-fedora ~]$ ls /dev/disk/by-id\nscsi-0QEMU_QEMU_HARDDISK_1234567890\n[fedora@vmi-fedora ~]$ \n</code></pre> As you can see the serial is part of the disk name, so you can uniquely identify it.</p> <p>The format and length of serials are specified according to the libvirt documentation: <pre><code>    If present, this specify serial number of virtual hard drive. For example, it may look like &lt;serial&gt;WD-WMAP9A966149&lt;/serial&gt;. Not supported for scsi-block devices, that is those using disk type 'block' using device 'lun' on bus 'scsi'. Since 0.7.1\n\n    Note that depending on hypervisor and device type the serial number may be truncated silently. IDE/SATA devices are commonly limited to 20 characters. SCSI devices depending on hypervisor version are limited to 20, 36 or 247 characters.\n\n    Hypervisors may also start rejecting overly long serials instead of truncating them in the future so it's advised to avoid the implicit truncation by testing the desired serial length range with the desired device and hypervisor combination.\n</code></pre></p>"},{"location":"storage/hotplug_volumes/#supported-disk-types","title":"Supported Disk types","text":"<p>Kubevirt supports hotplugging disk devices of type disk and lun. As with other volumes, using type <code>disk</code> will expose the hotplugged volume as a regular disk, while using <code>lun</code> allows additional functionalities like the execution of iSCSI commands.</p> <p>You can specify the desired type by using the --disk-type parameter, for example:</p> <pre><code># Allowed values are lun and disk. If no option is specified, we use disk by default.\n$ virtctl addvolume vmi-fedora --volume-name=example-lun-hotplug --disk-type=lun\n</code></pre>"},{"location":"storage/hotplug_volumes/#retain-hotplugged-volumes-after-restart","title":"Retain hotplugged volumes after restart","text":"<p>In many cases it is desirable to keep hotplugged volumes after a VM restart. It may also be desirable to be able to unplug these volumes after the restart. The <code>persist</code> option makes it impossible to unplug the disks after a restart. If you don't specify <code>persist</code> the default behaviour is to retain hotplugged volumes as hotplugged volumes after a VM restart. This makes the <code>persist</code> flag mostly obsolete unless you want to make a volume permanent on restart.</p>"},{"location":"storage/hotplug_volumes/#persist","title":"Persist","text":"<p>In some cases you want a hotplugged volume to become part of the standard disks after a restart of the VM. For instance if you added some permanent storage to the VM. We also assume that the running VMI has a matching VM that defines it specification. You can call the addvolume command with the --persist flag. This will update the VM domain disks section in addition to updating the VMI domain disks. This means that when you restart the VM, the disk is already defined in the VM, and thus in the new VMI.</p> <pre><code>$ virtctl addvolume vm-fedora --volume-name=example-volume-hotplug --persist\n</code></pre> <p>In the VM spec this will now show as a new disk <pre><code>spec:\ndomain:\n    devices:\n        disks:\n        - disk:\n            bus: virtio\n            name: containerdisk\n        - disk:\n            bus: virtio\n            name: cloudinitdisk\n        - disk:\n            bus: scsi\n            name: example-volume-hotplug\n    machine:\n      type: \"\"\n</code></pre></p>"},{"location":"storage/hotplug_volumes/#removevolume","title":"Removevolume","text":"<p>In addition to hotplug plugging the volume, you can also unplug it by using the 'removevolume' command available with virtctl <pre><code>$ virtctl removevolume vmi-fedora --volume-name=example-volume-hotplug\n</code></pre></p> <p>NOTE You can only unplug volumes that were dynamically added with addvolume, or using the API.</p>"},{"location":"storage/hotplug_volumes/#volumestatus","title":"VolumeStatus","text":"<p>VMI objects have a new <code>status.VolumeStatus</code> field. This is an array containing each disk, hotplugged or not. For example, after hotplugging the volume in the addvolume example, the VMI status will contain this: <pre><code>volumeStatus:\n- name: cloudinitdisk\n  target: vdb\n- name: containerdisk\n  target: vda\n- hotplugVolume:\n    attachPodName: hp-volume-7fmz4\n    attachPodUID: 62a7f6bf-474c-4e25-8db5-1db9725f0ed2\n  message: Successfully attach hotplugged volume volume-hotplug to VM\n  name: example-volume-hotplug\n  phase: Ready\n  reason: VolumeReady\n  target: sda\n</code></pre> Vda is the container disk that contains the Fedora OS, vdb is the cloudinit disk. As you can see those just contain the name and target used when assigning them to the VM. The target is the value passed to QEMU when specifying the disks. The value is unique for the VM and does NOT represent the naming inside the guest. For instance for a Windows Guest OS the target has no meaning. The same will be true for hotplugged volumes. The target is just a unique identifier meant for QEMU, inside the guest the disk can be assigned a different name.</p> <p>The hotplugVolume has some extra information that regular volume statuses do not have. The attachPodName is the name of the pod that was used to attach the volume to the node the VMI is running on. If this pod is deleted it will also stop the VMI as we cannot guarantee the volume will remain attached to the node. The other fields are similar to conditions and indicate the status of the hot plug process. Once a Volume is ready it can be used by the VM.</p>"},{"location":"storage/hotplug_volumes/#live-migration","title":"Live Migration","text":"<p>Currently Live Migration is enabled for any VMI that has volumes hotplugged into it. </p> <p>NOTE However there is a known issue that the migration may fail for VMIs with hotplugged block volumes if the target node uses CPU manager with static policy and <code>runc</code> prior to version <code>v1.0.0</code>.</p>"},{"location":"storage/snapshot_restore_api/","title":"Snapshot Restore API","text":"<p>The <code>snapshot.kubevirt.io</code> API Group defines resources for snapshotting and restoring KubeVirt <code>VirtualMachines</code></p>"},{"location":"storage/snapshot_restore_api/#prerequesites","title":"Prerequesites","text":""},{"location":"storage/snapshot_restore_api/#volumesnapshotclass","title":"VolumeSnapshotClass","text":"<p>KubeVirt leverages the <code>VolumeSnapshot</code> functionality of Kubernetes CSI drivers for capturing persistent <code>VirtualMachine</code> state.  So, you should make sure that your <code>VirtualMachine</code> uses <code>DataVolumes</code> or <code>PersistentVolumeClaims</code> backed by a <code>StorageClass</code> that supports <code>VolumeSnapshots</code> and a <code>VolumeSnapshotClass</code> is properly configured for that <code>StorageClass</code>.</p> <p>KubeVirt looks for Kubernetes Volume Snapshot related APIs/resources in the <code>v1</code> version. To make sure that KubeVirt's snapshot controller is able to snapshot the VirtualMachine and referenced volumes as expected, Kubernetes Volume Snapshot APIs must be served from <code>v1</code> version.</p> <p>To list <code>VolumeSnapshotClasses</code>:</p> <pre><code>kubectl get volumesnapshotclass\n</code></pre> <p>Make sure the <code>provisioner</code> property of your <code>StorageClass</code> matches the <code>driver</code> property of the <code>VolumeSnapshotClass</code></p> <p>Even if you have no <code>VolumeSnapshotClasses</code> in your cluster, <code>VirtualMachineSnapshots</code> are not totally useless.  They will still backup your <code>VirtualMachine</code> configuration.</p>"},{"location":"storage/snapshot_restore_api/#snapshot-feature-gate","title":"Snapshot Feature Gate","text":"<p>Snapshot/Restore support must be enabled in the feature gates to be supported. The feature gates field in the KubeVirt CR must be expanded by adding the <code>Snapshot</code> to it.</p>"},{"location":"storage/snapshot_restore_api/#snapshot-a-virtualmachine","title":"Snapshot a VirtualMachine","text":"<p>Snapshotting a virtualMachine is supported for online and offline vms.</p> <p>When snapshotting a running vm the controller will check for qemu guest agent in the vm. If the agent exists it will freeze the vm filesystems before taking the snapshot and unfreeze after the snapshot. It is recommended to take online snapshots with the guest agent for a better snapshot, if not present a best effort snapshot will be taken.</p> <p>Note To check if your vm has a qemu-guest-agent check for 'AgentConnected' in the vm status.</p> <p>There will be an indication in the vmSnapshot status if the snapshot was taken online and with or without guest agent participation.</p> <p>Note Online snapshot with hotplugged disks is supported, only persistent hotplugged disks will be included in the snapshot.</p> <p>To snapshot a <code>VirtualMachine</code> named <code>larry</code>, apply the following yaml.</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineSnapshot\nmetadata:\n  name: snap-larry\nspec:\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n</code></pre> <p>To wait for a snapshot to complete, execute:</p> <pre><code>kubectl wait vmsnapshot snap-larry --for condition=Ready\n</code></pre> <p>You can check the vmSnapshot phase in the vmSnapshot status. It can be one of the following:</p> <ul> <li>InProgress</li> <li>Succeeded</li> <li>Failed.</li> </ul> <p>The vmSnapshot has a default deadline of 5 minutes. If the vmSnapshot has not succeessfully completed before the deadline, it will be marked as Failed. The VM will be unfrozen and the created snapshot content will be cleaned up if necessary. The vmSnapshot object will remain in Failed state until deleted by the user. To change the default deadline add 'FailureDeadline' to the VirtualMachineSnapshot spec with a new value. The allowed format is a duration string which is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as \"300ms\", \"-1.5h\" or \"2h45m\"</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineSnapshot\nmetadata:\n  name: snap-larry\nspec:\n  source:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n  failureDeadline: 1m\n</code></pre> <p>In order to set an infinite deadline you can set it to 0 (not recommended).</p>"},{"location":"storage/snapshot_restore_api/#restoring-a-virtualmachine","title":"Restoring a VirtualMachine","text":"<p>To restore the <code>VirtualMachine</code> <code>larry</code> from <code>VirtualMachineSnapshot</code> <code>snap-larry</code>, Stop the VM, wait for it to be stopped and then apply the following yaml.</p> <pre><code>apiVersion: snapshot.kubevirt.io/v1beta1\nkind: VirtualMachineRestore\nmetadata:\n  name: restore-larry\nspec:\n  target:\n    apiGroup: kubevirt.io\n    kind: VirtualMachine\n    name: larry\n  virtualMachineSnapshotName: snap-larry\n</code></pre> <p>To wait for a restore to complete, execute:</p> <pre><code>kubectl wait vmrestore restore-larry --for condition=Ready\n</code></pre>"},{"location":"storage/snapshot_restore_api/#cleanup","title":"Cleanup","text":"<p>Keep <code>VirtualMachineSnapshots</code> (and their corresponding <code>VirtualMachineSnapshotContents</code>) around as long as you may want to restore from them again.</p> <p>Feel free to delete <code>restore-larry</code> as it is not needed once the restore is complete.</p>"},{"location":"storage/volume_migration/","title":"Migration update volume strategy and volume migration","text":"<p>Storage migration is possible while the VM is running by using the update volume strategy. Storage migration can be useful in the cases where the users need to change the underlying storage, for example, if the storage class has been deprecated, or there is a new more performant driver available.</p> <p>This feature doesn't handle the volume creation or cover migration between storage classes, but rather implements a basic API which can be used by overlaying tools to perform more advanced migration planning.</p> <p>If <code>Migration</code> is specified as <code>updateVolumesStrategy</code>, KubeVirt will try to migrate the storage from the old volume set to the new one when the VirtualMachine spec is updated. The migration considers the changed volumes present into a single update. A single update may contain modifications to more than one volume, but sequential changes to the volume set will be handled as separate migrations.</p> <p>Updates are declarative and GitOps compatible. For example, a new version of the VM specification with the new volume set and the migration volume update strategy can be directly applied using <code>kubectl apply</code> or interactively editing the VM with <code>kubectl edit</code></p> <p>Example: Original VM with a datavolume and datavolume template: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-dv\n  name: vm-dv\nspec:\n  dataVolumeTemplates:\n  - metadata:\n      creationTimestamp: null\n      name: src-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        volumeMode: Filesystem\n        resources:\n          requests:\n            storage: 2Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://registry:5000/kubevirt/alpine-container-disk-demo:devel\n  runStrategy: \"Always\"\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-dv\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n          interfaces:\n          - masquerade: {}\n            name: default\n        resources:\n          requests:\n            memory: 128Mi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: src-dv\n        name: datavolumedisk1\n      networks:\n      - name: default\n        pod: {}\n</code></pre></p> <p>The datavolume <code>src-dv</code> is migrated to the pvc <code>dest-pvc</code>: <pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: dst-dv\nspec:\n  source:\n      blank: {}\n  storage:\n    accessModes:\n      - ReadWriteMany\n    volumeMode: Block\n    resources:\n      requests:\n        storage: 5Gi\n    storageClassName: rook-ceph\n</code></pre></p> <p>by updating the VM: <pre><code> apiVersion: kubevirt.io/v1\n kind: VirtualMachine\n     kubevirt.io/vm: vm-dv\n   name: vm-dv\n spec:\n+  updateVolumesStrategy: Migration\n   dataVolumeTemplates:\n   - metadata:\n-      name: src-pvc\n+      name: dst-dv\n\n       volumes:\n       - dataVolume:\n-          name: src-pvc\n+          name: dst-dv\n         name: datavolumedisk1\n</code></pre></p> <p>The destination volume may be of a different type or size than the source. It is possible to migrate from and to a block volume as well as a filesystem volume. The destination volume should be equal to or larger than the source volume. However, the additional difference in the size of the destination volume is not instantly visible within the VM and must be manually resized because the guest is unaware of the migration.</p> <p>The volume migration depends on the <code>VolumeMigration</code> and <code>VolumesUpdateStrategy</code> feature gates and the <code>LiveMigrate</code> workloadUpdateStrategy. To fully enable the feature, add the following to the KubeVirt CR: <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n        - VolumesUpdateStrategy\n        - VolumeMigration\n    vmRolloutStrategy: LiveUpdate\n  workloadUpdateStrategy:\n    workloadUpdateMethods:\n    - LiveMigrate\n</code></pre></p> <p>The volume migration progress can be monitored by watching the corresponding VirtualMachineInstanceMigration object using the label <code>kubevirt.io/volume-update-in-progress: &lt;vm-name&gt;</code>. Example: <pre><code>$ kubectl get virtualmachineinstancemigrations -l kubevirt.io/volume-update-in-progress: vmi` --watch=true\nNAME                           PHASE       VMI\nkubevirt-workload-update-abcd  Running     vmi\n</code></pre></p>"},{"location":"storage/volume_migration/#datavolume-template","title":"Datavolume template","text":"<p>Updating a datavolume that is referenced by a datavolume template requires special caution. The volumes section must include a reference to the name of the datavolume template. This means that the datavolume templates must either be entirely deleted or updated as well.</p> <p>Example of updating the datavolume for the original VM in the first example: <pre><code>apiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: dst-dv\nspec:\n  source:\n      blank: {}\n  storage:\n    accessModes:\n      - ReadWriteMany\n    volumeMode: Block\n    resources:\n      requests:\n        storage: 10Gi\n    storageClassName: rook-ceph-block\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    kubevirt.io/vm: vm-dv\n  name: vm-dv\nspec:\n  updateVolumesStrategy: Migration\n  dataVolumeTemplates:\n  - metadata:\n      name: dst-dv\n    spec:\n      storage:\n        accessModes:\n        - ReadWriteOnce\n        volumeMode: Filesystem\n        resources:\n          requests:\n            storage: 2Gi\n        storageClassName: local\n      source:\n        registry:\n          url: docker://registry:5000/kubevirt/alpine-container-disk-demo:devel\n  runStrategy: \"Always\"\n  template:\n    metadata:\n      labels:\n        kubevirt.io/vm: vm-dv\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: datavolumedisk1\n          interfaces:\n          - masquerade: {}\n            name: default\n        resources:\n          requests:\n            memory: 128Mi\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - dataVolume:\n          name: dst-dv\n        name: datavolumedisk1\n      networks:\n      - name: default\n        pod: {}\n</code></pre></p>"},{"location":"storage/volume_migration/#limitations","title":"Limitations","text":"<p>Only certain types of disks and volumes are supported to be migrated. For an invalid type of volume the RestartRequired condition is set and volumes will be replaced upon VM restart. Currently, the volume migration is supported between PersistentVolumeClaims and Datavolumes. Additionally, volume migration is forbidden if the disk is: * shareable, since it cannot guarantee the data consistency with multiple writers * hotpluggable, this case isn't currently supported * filesystem, since virtiofs doesn't currently support live-migration * lun, originally the disk might support SCSI protocol but the destination PVC class does not. This case isn't currently supported.</p> <p>Currently, KubeVirt only enables live migration between separate nodes. Volume migration relies on live migration; hence, live migrating storage on the same node is also not possible. Volume migration is possible between local storage, like between 2 PVCs with RWO access mode, but they need to be located on two different host.</p>"},{"location":"user_workloads/accessing_virtual_machines/","title":"Accessing Virtual Machines","text":""},{"location":"user_workloads/accessing_virtual_machines/#graphical-and-serial-console-access","title":"Graphical and Serial Console Access","text":"<p>Once a virtual machine is started you are able to connect to the consoles it exposes. Usually there are two types of consoles:</p> <ul> <li>Serial Console</li> <li>Graphical Console (VNC)</li> </ul> <p>Note: You need to have <code>virtctl</code> installed to gain access to the VirtualMachineInstance.</p>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-serial-console","title":"Accessing the Serial Console","text":"<p>The serial console of a virtual machine can be accessed by using the <code>console</code> command:</p> <pre><code>virtctl console testvm\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-graphical-console-vnc","title":"Accessing the Graphical Console (VNC)","text":"<p>To access the graphical console of a virtual machine the VNC protocol is typically used. This requires <code>remote-viewer</code> to be installed. Once the tool is installed, you can access the graphical console using:</p> <pre><code>virtctl vnc testvm\n</code></pre> <p>If you only want to open a vnc-proxy without executing the <code>remote-viewer</code> command, it can be accomplished with:</p> <pre><code>virtctl vnc --proxy-only testvm\n</code></pre> <p>This would print the port number on your machine where you can manually connect using any VNC viewer.</p>"},{"location":"user_workloads/accessing_virtual_machines/#debugging-console-access","title":"Debugging console access","text":"<p>If the connection fails, you can use the <code>-v</code> flag to get more verbose output from both <code>virtctl</code> and the <code>remote-viewer</code> tool to troubleshoot the problem.</p> <pre><code>virtctl vnc testvm -v 4\n</code></pre> <p>Note: If you are using virtctl via SSH on a remote machine, you need to forward the X session to your machine. Look up the -X and -Y flags of <code>ssh</code> if you are not familiar with that. As an alternative you can proxy the API server port with SSH to your machine (either direct or in combination with <code>kubectl proxy</code>).</p>"},{"location":"user_workloads/accessing_virtual_machines/#ssh-access","title":"SSH Access","text":"<p>A common operational pattern used when managing virtual machines is to inject SSH public keys into the virtual machines at boot. This allows automation tools (like Ansible) to provision the virtual machine. It also gives operators a way of gaining secure and passwordless access to a virtual machine.</p> <p>KubeVirt provides multiple ways to inject SSH public keys into a virtual machine.</p> <p>In general, these methods fall into two categories:  - Static key injection, which places keys on the virtual machine the first time it is booted.  - Dynamic key injection, which allows keys to be dynamically updated both at boot and during runtime.</p> <p>Once a SSH public key is injected into the virtual machine, it can be accessed via <code>virtctl</code>.</p>"},{"location":"user_workloads/accessing_virtual_machines/#static-ssh-public-key-injection-via-cloud-init","title":"Static SSH public key injection via cloud-init","text":"<p>Users creating virtual machines can provide startup scripts to their virtual machines, allowing multiple customization operations.</p> <p>One option for injecting public SSH keys into a VM is via cloud-init startup script. However, there are more flexible options available.</p> <p>The virtual machine's access credential API allows statically injecting SSH public keys at startup time independently of the cloud-init user data by placing the SSH public key into a Kubernetes <code>Secret</code>. This allows keeping the application data in the cloud-init user data separate from the credentials used to access the virtual machine.</p> <p>A Kubernetes <code>Secret</code> can be created from an SSH public key like this:</p> <pre><code># Place SSH public key into a Secret\nkubectl create secret generic my-pub-key --from-file=key1=id_rsa.pub\n</code></pre> <p>The <code>Secret</code> containing the public key is then assigned to a virtual machine using the access credentials API with the <code>noCloud</code> propagation method.</p> <p>KubeVirt injects the SSH public key into the virtual machine by using the generated cloud-init metadata instead of the user data. This separates the application user data and user credentials.</p> <p>Note: The cloud-init <code>userData</code> is not touched.</p> <pre><code># Create a VM referencing the Secret using propagation method noCloud\nkubectl create -f - &lt;&lt;EOF\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      terminationGracePeriodSeconds: 0\n      accessCredentials:\n      - sshPublicKey:\n          source:\n            secret:\n              secretName: my-pub-key\n          propagationMethod:\n            noCloud: {}\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n        name: cloudinitdisk\nEOF\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#dynamic-ssh-public-key-injection-via-qemu-guest-agent","title":"Dynamic SSH public key injection via qemu-guest-agent","text":"<p>KubeVirt allows the dynamic injection of SSH public keys into a VirtualMachine with the access credentials API.</p> <p>Utilizing the <code>qemuGuestAgent</code> propagation method, configured Secrets are attached to a VirtualMachine when the VM is started.  This allows for dynamic injection of SSH public keys at runtime by updating the attached Secrets.</p> <p>Please note that new Secrets cannot be attached to a running VM: You must restart the VM to attach the new Secret.</p> <p>Note: This requires the qemu-guest-agent to be installed within the guest.</p> <p>Note: When using qemuGuestAgent propagation, the <code>/home/$USER/.ssh/authorized_keys</code> file will be owned by the guest agent. Changes to the file not made by the guest agent will be lost.</p> <p>Note: More information about the motivation behind the access credentials API can be found in the pull request description that introduced the API.</p> <p>In the example below the <code>Secret</code> containing the SSH public key is attached to the virtual machine via the access credentials API with the <code>qemuGuestAgent</code> propagation method. This allows updating the contents of the <code>Secret</code> at any time, which will result in the changes getting applied to the running virtual machine immediately. The <code>Secret</code> may also contain multiple SSH public keys.</p> <pre><code># Place SSH public key into a secret\nkubectl create secret generic my-pub-key --from-file=key1=id_rsa.pub\n</code></pre> <p>Now reference this secret in the <code>VirtualMachine</code> spec with the access credentials API using <code>qemuGuestAgent</code> propagation.</p> <pre><code># Create a VM referencing the Secret using propagation method qemuGuestAgent\nkubectl create -f - &lt;&lt;EOF\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: testvm\nspec:\n  running: true\n  template:\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n              bus: virtio\n            name: containerdisk\n          - disk:\n              bus: virtio\n            name: cloudinitdisk\n          rng: {}\n        resources:\n          requests:\n            memory: 1024M\n      terminationGracePeriodSeconds: 0\n      accessCredentials:\n      - sshPublicKey:\n          source:\n            secret:\n              secretName: my-pub-key\n          propagationMethod:\n            qemuGuestAgent:\n              users:\n              - fedora\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            password: fedora\n            chpasswd: { expire: False }\n            # Disable SELinux for now, so qemu-guest-agent can write the authorized_keys file\n            # The selinux-policy is too restrictive currently, see open bugs:\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=1917024\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=2028762\n            #   - https://bugzilla.redhat.com/show_bug.cgi?id=2057310\n            bootcmd:\n              - setenforce 0\n        name: cloudinitdisk\nEOF\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#accessing-the-vmi-using-virtctl","title":"Accessing the VMI using virtctl","text":"<p>The user can create a websocket backed network tunnel to a port inside the instance by using the <code>virtualmachineinstances/portforward</code> subresource of the <code>VirtualMachineInstance</code>.</p> <p>One use-case for this subresource is to forward SSH traffic into the <code>VirtualMachineInstance</code> either from the CLI or a web-UI.</p> <p>To connect to a <code>VirtualMachineInstance</code> from your local machine, <code>virtctl</code> provides a lightweight SSH client with the <code>ssh</code> command, that uses port forwarding. Refer to the command's help for more details.</p> <pre><code>virtctl ssh\n</code></pre> <p>To transfer files from or to a <code>VirtualMachineInstance</code> <code>virtctl</code> also provides a lightweight SCP client with the <code>scp</code> command. Its usage is similar to the <code>ssh</code> command. Refer to the command's help for more details.</p> <pre><code>virtctl scp\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#using-virtctl-as-proxy","title":"Using virtctl as proxy","text":"<p>If you prefer to use your local OpenSSH client, there are two ways of doing that in combination with virtctl.</p> <p>Note: Most of this applies to the <code>virtctl scp</code> command too.</p> <ol> <li>The <code>virtctl ssh</code> command has a <code>--local-ssh</code> option. With this option    <code>virtctl</code> wraps the local OpenSSH client transparently to the user. The    executed SSH command can be viewed by increasing the verbosity (<code>-v 3</code>).</li> </ol> <pre><code>virtctl ssh --local-ssh -v 3 testvm\n</code></pre> <ol> <li>The <code>virtctl port-forward</code> command provides an option to tunnel a single    port to your local stdout/stdin. This allows the command to be used in    combination with the OpenSSH client's <code>ProxyCommand</code> option.</li> </ol> <pre><code>ssh -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' fedora@testvm.mynamespace\n</code></pre> <p>To provide easier access to arbitrary virtual machines you can add the following lines to your SSH <code>config</code>:</p> <pre><code>Host vmi/*\n   ProxyCommand virtctl port-forward --stdio=true %h %p\nHost vm/*\n   ProxyCommand virtctl port-forward --stdio=true %h %p\n</code></pre> <p>This allows you to simply call <code>ssh user@vmi/testvmi.mynamespace</code> and your SSH config and virtctl will do the rest. Using this method it becomes easy to set up different identities for different namespaces inside your SSH <code>config</code>.</p> <p>This feature can also be used with Ansible to automate configuration of virtual machines running on KubeVirt. You can put the snippet above into its own file (e.g. <code>~/.ssh/virtctl-proxy-config</code>) and add the following lines to your <code>.ansible.cfg</code>:</p> <pre><code>[ssh_connection]\nssh_args = -F ~/.ssh/virtctl-proxy-config\n</code></pre> <p>Note that all port forwarding traffic will be sent over the Kubernetes control plane. A high amount of connections and traffic can increase pressure on the API server. If you regularly need a high amount of connections and traffic consider using a dedicated Kubernetes <code>Service</code> instead.</p>"},{"location":"user_workloads/accessing_virtual_machines/#example","title":"Example","text":"<ol> <li> <p>Create virtual machine and inject SSH public key as explained above</p> </li> <li> <p>SSH into virtual machine</p> </li> </ol> <pre><code># Add --local-ssh to transparently use local OpenSSH client\nvirtctl ssh -i id_rsa fedora@testvm\n</code></pre> <p>or</p> <pre><code>ssh -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' -i id_rsa fedora@vmi/testvm.mynamespace\n</code></pre> <ol> <li>SCP file to the virtual machine</li> </ol> <pre><code># Add --local-ssh to transparently use local OpenSSH client\nvirtctl scp -i id_rsa testfile fedora@testvm:/tmp\n</code></pre> <p>or</p> <pre><code>scp -o 'ProxyCommand=virtctl port-forward --stdio=true vmi/testvm.mynamespace 22' -i id_rsa testfile fedora@testvm.mynamespace:/tmp\n</code></pre>"},{"location":"user_workloads/accessing_virtual_machines/#rbac-permissions-for-consolevncssh-access","title":"RBAC permissions for Console/VNC/SSH access","text":""},{"location":"user_workloads/accessing_virtual_machines/#using-default-rbac-cluster-roles","title":"Using default RBAC cluster roles","text":"<p>Every KubeVirt installation starting with version v0.5.1 ships a set of default RBAC cluster roles that can be used to grant users access to VirtualMachineInstances.</p> <p>The <code>kubevirt.io:admin</code> and <code>kubevirt.io:edit</code> cluster roles have console, VNC and SSH respectively port-forwarding access permissions built into them. By binding either of these roles to a user, they will have the ability to use virtctl to access the console, VNC and SSH.</p>"},{"location":"user_workloads/accessing_virtual_machines/#using-custom-rbac-cluster-role","title":"Using custom RBAC cluster role","text":"<p>The default KubeVirt cluster roles grant access to more than just the console, VNC and port-forwarding. The <code>ClusterRole</code> below demonstrates how to craft a custom role, that only allows access to the console, VNC and port-forwarding.</p> <pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1\nkind: ClusterRole\nmetadata:\n  name: allow-console-vnc-port-forward-access\nrules:\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/console\n      - virtualmachineinstances/vnc\n    verbs:\n      - get\n  - apiGroups:\n      - subresources.kubevirt.io\n    resources:\n      - virtualmachineinstances/portforward\n    verbs:\n      - update\n</code></pre> <p>When bound with a <code>ClusterRoleBinding</code> the <code>ClusterRole</code> above grants access to virtual machines across all namespaces.</p> <p>In order to reduce the scope to a single namespace, bind this <code>ClusterRole</code> using a <code>RoleBinding</code> that targets a single namespace.</p>"},{"location":"user_workloads/basic_use/","title":"Basic use","text":"<p>Using KubeVirt should be fairly natural if you are used to working with Kubernetes.</p> <p>The primary way of using KubeVirt is by working with the KubeVirt kinds in the Kubernetes API:</p> <pre><code>$ kubectl create -f vmi.yaml\n$ kubectl wait --for=condition=Ready vmis/my-vmi\n$ kubectl get vmis\n$ kubectl delete vmis testvmi\n</code></pre> <p>The following pages describe how to use and discover the API, manage, and access virtual machines.</p>"},{"location":"user_workloads/basic_use/#user-interface","title":"User Interface","text":"<p>KubeVirt does not come with a UI, it is only extending the Kubernetes API with virtualization functionality.</p>"},{"location":"user_workloads/boot_from_external_source/","title":"Booting From External Source","text":"<p>When installing a new guest virtual machine OS, it is often useful to boot directly from a kernel and initrd stored in the host physical machine OS, allowing command line arguments to be passed directly to the installer.</p> <p>Booting from an external source is supported in Kubevirt starting from version v0.42.0-rc.0. This enables the capability to define a Virtual Machine that will use a custom kernel / initrd binary, with possible custom arguments, during its boot process.</p> <p>The binaries are provided though a container image. The container is pulled from the container registry and resides on the local node hosting the VMs.</p>"},{"location":"user_workloads/boot_from_external_source/#use-cases","title":"Use cases","text":"<p>Some use cases for this may be: - For a kernel developer it may be very convenient to launch VMs that are defined to boot from the latest kernel binary that is often being changed. - Initrd can be set with files that need to reside on-memory during all the VM's life-cycle.</p>"},{"location":"user_workloads/boot_from_external_source/#workflow","title":"Workflow","text":"<p>Defining an external boot source can be done in the following way: <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: ext-kernel-boot-vm\nspec:\n  runStrategy: Manual\n  template:\n    spec:\n      domain:\n        devices: {}\n        firmware:\n          kernelBoot:\n            container:\n              image: vmi_ext_boot/kernel_initrd_binaries_container:latest\n              initrdPath: /boot/initramfs-virt\n              kernelPath: /boot/vmlinuz-virt\n              imagePullPolicy: Always\n              imagePullSecret: IfNotPresent\n            kernelArgs: console=ttyS0\n        resources:\n          requests:\n            memory: 1Gi\n</code></pre></p> <p>Notes:</p> <ul> <li> <p><code>initrdPath</code> and <code>kernelPath</code> define the path for the binaries inside the container.</p> </li> <li> <p>Kernel and Initrd binaries must be owned by <code>qemu</code> user &amp; group.</p> </li> <li> <p>To change ownership: <code>chown qemu:qemu &lt;binary&gt;</code> when <code>&lt;binary&gt;</code> is the binary file.</p> </li> <li> <p><code>kernelArgs</code> can only be provided if a kernel binary is provided (i.e. <code>kernelPath</code> not defined). These arguments will be passed to the default kernel the VM boots from.</p> </li> <li> <p><code>imagePullSecret</code> and <code>imagePullPolicy</code> are optional</p> </li> <li> <p>if <code>imagePullPolicy</code> is <code>Always</code> and the container image is updated then the VM will be booted   into the new kernel when VM restarts</p> </li> </ul>"},{"location":"user_workloads/component_monitoring/","title":"Component monitoring","text":"<p>All KubeVirt system-components expose Prometheus metrics at their <code>/metrics</code> REST endpoint.</p> <p>You can consult the complete and up-to-date metric list at kubevirt/monitoring.</p>"},{"location":"user_workloads/component_monitoring/#custom-service-discovery","title":"Custom Service Discovery","text":"<p>Prometheus supports service discovery based on Pods and Endpoints out of the box. Both can be used to discover KubeVirt services.</p> <p>All Pods which expose metrics are labeled with <code>prometheus.kubevirt.io</code> and contain a port-definition which is called <code>metrics</code>. In the KubeVirt release-manifests, the default <code>metrics</code> port is <code>8443</code>.</p> <p>The above labels and port informations are collected by a <code>Service</code> called <code>kubevirt-prometheus-metrics</code>. Kubernetes automatically creates a corresponding <code>Endpoint</code> with an equal name:</p> <pre><code>$ kubectl get endpoints -n kubevirt kubevirt-prometheus-metrics -o yaml\napiVersion: v1\nkind: Endpoints\nmetadata:\n  labels:\n    kubevirt.io: \"\"\n    prometheus.kubevirt.io: \"\"\n  name: kubevirt-prometheus-metrics\n  namespace: kubevirt\nsubsets:\n- addresses:\n  - ip: 10.244.0.5\n    nodeName: node01\n    targetRef:\n      kind: Pod\n      name: virt-handler-cjzg6\n      namespace: kubevirt\n      resourceVersion: \"4891\"\n      uid: c67331f9-bfcf-11e8-bc54-525500d15501\n  - ip: 10.244.0.6\n  [...]\n  ports:\n  - name: metrics\n    port: 8443\n    protocol: TCP\n</code></pre> <p>By watching this endpoint for added and removed IPs to <code>subsets.addresses</code> and appending the <code>metrics</code> port from <code>subsets.ports</code>, it is possible to always get a complete list of ready-to-be-scraped Prometheus targets.</p>"},{"location":"user_workloads/component_monitoring/#integrating-with-the-prometheus-operator","title":"Integrating with the prometheus-operator","text":"<p>The prometheus-operator can make use of the <code>kubevirt-prometheus-metrics</code> service to automatically create the appropriate Prometheus config.</p> <p>KubeVirt's <code>virt-operator</code> checks if the <code>ServiceMonitor</code> custom resource exists when creating an install strategy for deployment. KubeVirt will automatically create a <code>ServiceMonitor</code> resource in the <code>monitorNamespace</code>, as well as an appropriate role and rolebinding in KubeVirt's namespace.</p> <p>Three settings are exposed in the <code>KubeVirt</code> custom resource to direct KubeVirt to create these resources correctly:</p> <ul> <li> <p><code>monitorNamespace</code>: The namespace that prometheus-operator runs in.     Defaults to <code>openshift-monitoring</code>.</p> </li> <li> <p><code>monitorAccount</code>: The serviceAccount that prometheus-operator runs     with. Defaults to <code>prometheus-k8s</code>.</p> </li> <li> <p><code>serviceMonitorNamespace</code>: The namespace that the serviceMonitor runs in.     Defaults to be <code>monitorNamespace</code> </p> </li> </ul> <p>Please note that if you decide to set <code>serviceMonitorNamespace</code> than this  namespace must be included in <code>serviceMonitorNamespaceSelector</code> field of  Prometheus spec.</p> <p>If the prometheus-operator for a given deployment uses these defaults, then these values can be omitted.</p> <p>An example of the KubeVirt resource depicting these default values:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nmetadata:\n  name: kubevirt\nspec:\n  monitorNamespace: openshift-monitoring\n  monitorAccount: prometheus-k8s\n</code></pre>"},{"location":"user_workloads/component_monitoring/#integrating-with-the-okd-cluster-monitoring-operator","title":"Integrating with the OKD cluster-monitoring-operator","text":"<p>After the cluster-monitoring-operator is up and running, KubeVirt will detect the existence of the <code>ServiceMonitor</code> resource. Because the definition contains the <code>openshift.io/cluster-monitoring</code> label, it will automatically be picked up by the cluster monitor.</p>"},{"location":"user_workloads/component_monitoring/#metrics-about-virtual-machines","title":"Metrics about Virtual Machines","text":"<p>The endpoints report metrics related to the runtime behaviour of the Virtual Machines. All the relevant metrics are prefixed with <code>kubevirt_vmi</code>.</p> <p>The metrics have labels that allow to connect to the VMI objects they refer to. At minimum, the labels will expose <code>node</code>, <code>name</code> and <code>namespace</code> of the related VMI object.</p> <p>For example, reported metrics could look like</p> <pre><code>kubevirt_vmi_memory_resident_bytes{domain=\"default_vm-test-01\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\"} 2.5595904e+07\nkubevirt_vmi_network_traffic_bytes_total{domain=\"default_vm-test-01\",interface=\"vnet0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",type=\"rx\"} 8431\nkubevirt_vmi_network_traffic_bytes_total{domain=\"default_vm-test-01\",interface=\"vnet0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",type=\"tx\"} 1835\nkubevirt_vmi_vcpu_seconds_total{domain=\"default_vm-test-01\",id=\"0\",name=\"vm-test-01\",namespace=\"default\",node=\"node01\",state=\"1\"} 19\n</code></pre> <p>Please note the <code>domain</code> label in the above example. This label is deprecated and it will be removed in a future release. You should identify the VMI using the <code>node</code>, <code>namespace</code>, <code>name</code> labels instead.</p>"},{"location":"user_workloads/component_monitoring/#important-queries","title":"Important Queries","text":""},{"location":"user_workloads/component_monitoring/#detecting-connection-issues-for-the-rest-client","title":"Detecting connection issues for the REST client","text":"<p>Use the following query to get a counter for all REST call which indicate connection issues:</p> <pre><code>rest_client_requests_total{code=\"&lt;error&gt;\"}\n</code></pre> <p>If this counter is continuously increasing, it is an indicator that the corresponding KubeVirt component has general issues to connect to the apiserver</p>"},{"location":"user_workloads/creating_vms/","title":"Creating VirtualMachines","text":"<p>The virtctl sub command <code>create vm</code> allows easy creation of VirtualMachine manifests from the command line. It leverages instance types and preferences and inference by default (see Specifying or inferring instance types and preferences) and provides several flags to control details of the created virtual machine.</p> <p>For example there are flags to specify the name or run strategy of a virtual machine or flags to add volumes to a virtual machine. Instance types and preferences can either be specified directly or it is possible to let KubeVirt infer those from the volume used to boot the virtual machine.</p> <p>For a full set of flags and their description use the following command:</p> <pre><code>virtctl create vm -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#creating-virtualmachines-on-a-cluster","title":"Creating VirtualMachines on a cluster","text":"<p>The output of virtctl <code>create vm</code> can be piped into <code>kubectl</code> to directly create a VirtualMachine on a cluster, e.g.:</p> <pre><code># Create a VM with name my-vm on the cluster\nvirtctl create vm --name my-vm | kubectl create -f -\nvirtualmachine.kubevirt.io/my-vm created\n</code></pre>"},{"location":"user_workloads/creating_vms/#creating-instance-types","title":"Creating Instance Types","text":"<p>The virtctl subcommand <code>create instancetype</code> allows easy creation of an instance type manifest from the command line. The command also provides several flags that can be used to create your desired manifest.</p> <p>There are two required flags that need to be specified: the number of vCPUs and the amount of memory to be requested. Additionally, there are several optional flags that can be used, such as specifying a list of GPUs for passthrough, choosing the desired IOThreadsPolicy, or simply providing the name of our instance type.</p> <p>By default, the command creates the cluster-wide resource. If the user wants to create the namespaced version, they need to provide the namespaced flag. The namespace name can be specified by using the namespace flag.</p> <p>For a complete list of flags and their descriptions, use the following command:</p> <pre><code>virtctl create instancetype -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples","title":"Examples","text":"<p>Create a manifest for a VirtualMachineClusterInstancetype with the required --cpu and --memory flags <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi\n</code></pre></p> <p>Create a manifest for a VirtualMachineInstancetype with a specified namespace <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi --namespace my-namespace\n</code></pre></p> <p>Create a manifest for a VirtualMachineInstancetype without a specified namespace name <pre><code>virtctl create instancetype --cpu 2 --memory 256Mi --namespaced\n</code></pre></p>"},{"location":"user_workloads/creating_vms/#creating-preferences","title":"Creating Preferences","text":"<p>The virtctl subcommand <code>create preference</code> allows easy creation of a preference manifest from the command line. This command serves as a starting point to create the basic structure of a manifest, as it does not allow specifying all of the options that are supported in preferences.</p> <p>The current set of flags allows us, for example, to specify the preferred CPU topology, machine type or a storage class.</p> <p>By default, the command creates the cluster-wide resource. If the user wants to create the namespaced version, they need to provide the namespaced flag. The namespace name can be specified by using the namespace flag.</p> <p>For a complete list of flags and their descriptions, use the following command:</p> <pre><code>virtctl create preference -h\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples_1","title":"Examples","text":"<p>Create a manifest for a VirtualMachineClusterPreference with a preferred cpu topology <pre><code>virtctl create preference --cpu-topology preferSockets\n</code></pre></p> <p>Create a manifest for a VirtualMachinePreference with a specified namespace <pre><code>virtctl create preference --namespace my-namespace\n</code></pre></p> <p>Create a manifest for a VirtualMachinePreference with the preferred storage class <pre><code>virtctl create preference --namespaced --volume-storage-class my-storage\n</code></pre></p>"},{"location":"user_workloads/creating_vms/#specifying-or-inferring-instance-types-and-preferences","title":"Specifying or inferring instance types and preferences","text":"<p>Instance types and preference can be specified with the appropriate flags, e.g.:</p> <pre><code>virtctl create vm --instancetype my-instancetype --preference my-preference\n</code></pre> <p>The type of the instance type or preference (namespaced or cluster scope) can be controlled by prefixing the instance type or preference name with the corresponding CRD name, e.g.:</p> <pre><code># Using a cluster scoped instancetype and a namespaced preference\nvirtctl create vm \\\n  --instancetype virtualmachineclusterinstancetype/my-instancetype \\\n  --preference virtualmachinepreference/my-preference\n</code></pre> <p>If a prefix was not supplied the cluster scoped resources will be used by default.</p> <p>To explicitly infer instance types and/or preferences from the volume used to boot the virtual machine add the following flags:</p> <pre><code>virtctl create vm --infer-instancetype --infer-preference\n</code></pre> <p>The implicit default is to always try inferring an instancetype and preference from the boot volume. This feature makes use of the <code>IgnoreInferFromVolumeFailure</code> policy, which suppresses failures on inference of instancetypes and preferences. If one of the above switches was provided explicitly, then the <code>RejectInferFromVolumeFailure</code> policy is used instead. This way users are made aware of potential issues during the virtual machine creation.</p>"},{"location":"user_workloads/creating_vms/#boot-order-of-added-volumes","title":"Boot order of added volumes","text":"<p>Please note that volumes of different kinds currently have the following fixed boot order regardless of the order their flags were specified on the command line:</p> <ol> <li>ContainerDisk</li> <li>DataSource</li> <li>Cloned PVC</li> <li>Directly used PVC</li> </ol> <p>If multiple volumes of the same kind were specified their order is determined by the order in which their flags were specified.</p>"},{"location":"user_workloads/creating_vms/#specifying-cloud-init-user-data","title":"Specifying cloud-init user data","text":"<p>To pass cloud-init user data to virtctl it needs to be encoded into a base64 string. Here is an example how to do it:</p> <pre><code># Put your cloud-init user data into a file.\n# This will add an authorized key to the default user.\n# To get the default username read the documentation for the cloud image\n$ cat cloud-init.txt\n#cloud-config\nssh_authorized_keys:\n  - ssh-rsa AAAA...\n\n# Base64 encode the contents of the file without line wraps and store it in a variable\n$ CLOUD_INIT_USERDATA=$(base64 -w 0 cloud-init.txt)\n\n# Show the contents of the variable\n$ echo $CLOUD_INIT_USERDATA I2Nsb3VkLWNvbmZpZwpzc2hfYXV0aG9yaXplZF9rZXlzOgogIC0gc3NoLXJzYSBBQUFBLi4uCg==\n</code></pre> <p>You can now use this variable as an argument to the <code>--cloud-init-user-data</code> flag:</p> <pre><code>virtctl create vm --cloud-init-user-data $CLOUD_INIT_USERDATA\n</code></pre>"},{"location":"user_workloads/creating_vms/#examples_2","title":"Examples","text":"<p>Create a manifest for a VirtualMachine with a random name:</p> <pre><code>virtctl create vm\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified name and RunStrategy Always</p> <pre><code>virtctl create vm --name=my-vm --run-strategy=Always\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineClusterInstancetype</p> <pre><code>virtctl create vm --instancetype=my-instancetype\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineInstancetype (namespaced)</p> <pre><code>virtctl create vm --instancetype=virtualmachineinstancetype/my-instancetype\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineClusterPreference</p> <pre><code>virtctl create vm --preference=my-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachinePreference (namespaced)</p> <pre><code>virtctl create vm --preference=virtualmachinepreference/my-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with an ephemeral containerdisk volume</p> <pre><code>virtctl create vm --volume-containerdisk=src:my.registry/my-image:my-tag\n</code></pre> <p>Create a manifest for a VirtualMachine with a cloned DataSource in namespace and specified size</p> <pre><code>virtctl create vm --volume-datasource=src:my-ns/my-ds,size:50Gi\n</code></pre> <p>Create a manifest for a VirtualMachine with a cloned DataSource and inferred instancetype and preference</p> <pre><code>virtctl create vm --volume-datasource=src:my-annotated-ds --infer-instancetype --infer-preference\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and cloned PVC</p> <pre><code>virtctl create vm --volume-clone-pvc=my-ns/my-pvc\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and directly used PVC</p> <pre><code>virtctl create vm --volume-pvc=my-pvc\n</code></pre> <p>Create a manifest for a VirtualMachine with a clone DataSource and a blank volume</p> <pre><code>virtctl create vm --volume-datasource=src:my-ns/my-ds --volume-blank=size:50Gi\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and cloned DataSource</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-datasource=src:my-ds\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and two cloned DataSources (flag can be provided multiple times)</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-datasource=src:my-ds1 --volume-datasource=src:my-ds2\n</code></pre> <p>Create a manifest for a VirtualMachine with a specified VirtualMachineCluster{Instancetype,Preference} and directly used PVC</p> <pre><code>virtctl create vm --instancetype=my-instancetype --preference=my-preference --volume-pvc=my-pvc\n</code></pre>"},{"location":"user_workloads/deploy_common_instancetypes/","title":"Deploy common-instancetypes","text":"<p>The <code>kubevirt/common-instancetypes</code> provide a set of instancetypes and preferences to help create KubeVirt <code>VirtualMachines</code>.</p> <p>Beginning with the 1.1 release of KubeVirt, cluster wide resources can be deployed directly through KubeVirt, without another operator. This allows deployment of a set of default instancetypes and preferences along side KubeVirt.</p>"},{"location":"user_workloads/deploy_common_instancetypes/#enable-automatic-deployment-of-common-instancetypes","title":"Enable automatic deployment of common-instancetypes","text":"<p>To enable the deployment of cluster-wide common-instancetypes through the KubeVirt <code>virt-operator</code>, the <code>CommonInstancetypesDeploymentGate</code> feature gate needs to be enabled.</p> <p>See Activating feature gates on how to enable it.</p>"},{"location":"user_workloads/deploy_common_instancetypes/#deploy-common-instancetypes-manually","title":"Deploy common-instancetypes manually","text":"<p>For customization purposes or to install namespaced resources, common-instancetypes can also be deployed by hand.</p> <p>To install all resources provided by the <code>kubevirt/common-instancetypes</code> project without further customizations, simply apply with <code>kustomize</code> enabled (-k flag):</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git\n</code></pre> <p>Alternatively, targets for each of the available custom resource types (e.g. namespaced instancetypes) are available.</p> <p>For example, to deploy <code>VirtualMachineInstancetypes</code> run the following command:</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git/VirtualMachineInstancetypes\n</code></pre>"},{"location":"user_workloads/guest_agent_information/","title":"Guest Agent information","text":"<p>Guest Agent (GA) is an optional component that can run inside of Virtual Machines. The GA provides plenty of additional runtime information about the running operating system (OS). More technical detail about available GA commands is available here.</p>"},{"location":"user_workloads/guest_agent_information/#guest-agent-info-in-virtual-machine-status","title":"Guest Agent info in Virtual Machine status","text":"<p>GA presence in the Virtual Machine is signaled with a condition in the VirtualMachineInstance status. The condition tells that the GA is connected and can be used.</p> <p>GA condition on VirtualMachineInstance</p> <pre><code>status:\n  conditions:\n  - lastProbeTime: \"2020-02-28T10:22:59Z\"\n    lastTransitionTime: null\n    status: \"True\"\n    type: AgentConnected\n</code></pre> <p>When the GA is connected, additional OS information is shown in the status. This information comprises:</p> <ul> <li>guest info, which contains OS runtime data</li> <li>interfaces info, which shows QEMU interfaces merged with GA interfaces info.</li> </ul> <p>Below is the example of the information shown in the VirtualMachineInstance status.</p> <p>GA info with merged into status</p> <pre><code>status:\n  guestOSInfo:\n    id: fedora\n    kernelRelease: 4.18.16-300.fc29.x86_64\n    kernelVersion: '#1 SMP Sat Oct 20 23:24:08 UTC 2018'\n    name: Fedora\n    prettyName: Fedora 29 (Cloud Edition)\n    version: \"29\"\n    versionId: \"29\"\n  interfaces:\n  - infoSource: domain, guest-agent\n    interfaceName: eth0\n    ipAddress: 10.244.0.23/24\n    ipAddresses:\n    - 10.244.0.23/24\n    - fe80::858:aff:fef4:17/64\n    mac: 0a:58:0a:f4:00:17\n    name: default\n</code></pre> <p>When the Guest Agent is not present in the Virtual Machine, the Guest Agent information is not shown. No error is reported because the Guest Agent is an optional component.</p> <p>The infoSource field indicates where the info is gathered from. Valid values:</p> <ul> <li>domain: the info is based on the domain spec</li> <li>guest-agent: the info is based on Guest Agent report</li> <li>domain, guest-agent: the info is based on both the domain spec and the Guest Agent report</li> </ul>"},{"location":"user_workloads/guest_agent_information/#guest-agent-info-available-through-the-api","title":"Guest Agent info available through the API","text":"<p>The data shown in the VirtualMachineInstance status are a subset of the information available. The rest of the data is available via the REST API exposed in the Kubernetes <code>kube-api</code> server.</p> <p>There are three new subresources added to the VirtualMachineInstance object:</p> <pre><code>- guestosinfo\n- userlist\n- filesystemlist\n</code></pre> <p>The whole GA data is returned via <code>guestosinfo</code> subresource available behind the API endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/guestosinfo\n</code></pre> <p>GuestOSInfo sample data:</p> <pre><code>{\n    \"fsInfo\": {\n        \"disks\": [\n            {\n                \"diskName\": \"vda1\",\n                \"fileSystemType\": \"ext4\",\n                \"mountPoint\": \"/\",\n                \"totalBytes\": 0,\n                \"usedBytes\": 0\n            }\n        ]\n    },\n    \"guestAgentVersion\": \"2.11.2\",\n    \"hostname\": \"testvmi6m5krnhdlggc9mxfsrnhlxqckgv5kqrwcwpgr5mdpv76grrk\",\n    \"metadata\": {\n        \"creationTimestamp\": null\n    },\n    \"os\": {\n        \"id\": \"fedora\",\n        \"kernelRelease\": \"4.18.16-300.fc29.x86_64\",\n        \"kernelVersion\": \"#1 SMP Sat Oct 20 23:24:08 UTC 2018\",\n        \"machine\": \"x86_64\",\n        \"name\": \"Fedora\",\n        \"prettyName\": \"Fedora 29 (Cloud Edition)\",\n        \"version\": \"29 (Cloud Edition)\",\n        \"versionId\": \"29\"\n    },\n    \"timezone\": \"UTC, 0\"\n}\n</code></pre> <p>Items FSInfo and UserList are capped to the max capacity of 10 items, as a precaution for VMs with thousands of users.</p> <p>Full list of Filesystems is available through the subresource <code>filesystemlist</code> which is available as endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/filesystemlist\n</code></pre> <p>Filesystem sample data:</p> <pre><code>{\n    \"items\": [\n        {\n            \"diskName\": \"vda1\",\n            \"fileSystemType\": \"ext4\",\n            \"mountPoint\": \"/\",\n            \"totalBytes\": 3927900160,\n            \"usedBytes\": 1029201920\n        }\n    ],\n    \"metadata\": {}\n}\n</code></pre> <p>Full list of the Users is available through the subresource <code>userlist</code> which is available as endpoint.</p> <pre><code>/apis/subresources.kubevirt.io/v1/namespaces/{namespace}/virtualmachineinstances/{name}/userlist\n</code></pre> <p>Userlist sample data:</p> <pre><code>{\n    \"items\": [\n        {\n            \"loginTime\": 1580467675.876078,\n            \"userName\": \"fedora\"\n        }\n    ],\n    \"metadata\": {}\n}\n</code></pre> <p>User LoginTime is in fractional seconds since epoch time. It is left for the consumer to convert to the desired format.</p>"},{"location":"user_workloads/guest_operating_system_information/","title":"Guest Operating System Information","text":"<p>Guest operating system identity for the VirtualMachineInstance will be provided by the label <code>kubevirt.io/os</code> :</p> <pre><code>metadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win2k12r2\n</code></pre> <p>The <code>kubevirt.io/os</code> label is based on the short OS identifier from libosinfo database. The following Short IDs are currently supported:</p> Short ID Name Version Family ID <p>win2k12r2</p> <p>Microsoft Windows Server 2012 R2</p> <p>6.3</p> <p>winnt</p> <p>https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2012-r2</p>"},{"location":"user_workloads/guest_operating_system_information/#use-with-presets","title":"Use with presets","text":"<p>A VirtualMachineInstancePreset representing an operating system with a <code>kubevirt.io/os</code> label could be applied on any given VirtualMachineInstance that have and match the <code>kubevirt.io/os</code> label.</p> <p>Default presets for the OS identifiers above are included in the current release.</p>"},{"location":"user_workloads/guest_operating_system_information/#windows-server-2012r2-virtualmachineinstancepreset-example","title":"Windows Server 2012R2 <code>VirtualMachineInstancePreset</code> Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: windows-server-2012r2\n  selector:\n    matchLabels:\n      kubevirt.io/os: win2k12r2\nspec:\n  domain:\n    cpu:\n      cores: 2\n    resources:\n      requests:\n        memory: 2G\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    clock:\n      utc: {}\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/os: win2k12r2\n  name: windows2012r2\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    firmware:\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n    devices:\n      disks:\n      - name: server2012r2\n        disk:\n          dev: vda\n  volumes:\n    - name: server2012r2\n      persistentVolumeClaim:\n        claimName: my-windows-image\n\nOnce the `VirtualMachineInstancePreset` is applied to the\n`VirtualMachineInstance`, the resulting resource would look like this:\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/windows-server-2012r2: kubevirt.io/v1\n  labels:\n    kubevirt.io/os: win2k12r2\n  name: windows2012r2\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    cpu:\n      cores: 2\n    resources:\n      requests:\n        memory: 2G\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n    clock:\n      utc: {}\n      timer:\n        hpet:\n          present: false\n        pit:\n          tickPolicy: delay\n        rtc:\n          tickPolicy: catchup\n        hyperv: {}\n    firmware:\n      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223\n    devices:\n      disks:\n      - name: server2012r2\n        disk:\n          dev: vda\n  volumes:\n    - name: server2012r2\n      persistentVolumeClaim:\n        claimName: my-windows-image\n</code></pre> <p>For more information see VirtualMachineInstancePresets</p>"},{"location":"user_workloads/guest_operating_system_information/#hyperv-optimizations","title":"HyperV optimizations","text":"<p>KubeVirt supports quite a lot of so-called \"HyperV enlightenments\", which are optimizations for Windows Guests. Some of these optimization may require an up to date host kernel support to work properly, or to deliver the maximum performance gains.</p> <p>KubeVirt can perform extra checks on the hosts before to run Hyper-V enabled VMs, to make sure the host has no known issues with Hyper-V support, properly expose all the required features and thus we can expect optimal performance. These checks are disabled by default for backward compatibility and because they depend on the node-feature-discovery and on extra configuration.</p> <p>To enable strict host checking, the user may expand the <code>featureGates</code> field in the KubeVirt CR by adding the <code>HypervStrictCheck</code> to it.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: Kubevirt\nmetadata:\n  name: kubevirt\n  namespace: kubevirt\nspec:\n  ...\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"HypervStrictCheck\"\n</code></pre> <p>Alternatively, users can edit an existing kubevirt CR:</p> <p><code>kubectl edit kubevirt kubevirt -n kubevirt</code></p> <pre><code>...\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - \"HypervStrictCheck\"\n        - \"CPUManager\"\n</code></pre>"},{"location":"user_workloads/hook-sidecar/","title":"Hook Sidecar Container","text":""},{"location":"user_workloads/hook-sidecar/#introduction","title":"Introduction","text":"<p>In KubeVirt, a Hook Sidecar container is a sidecar container (a secondary container that runs along with the main application container within the same Pod) used to apply customizations before the Virtual Machine is initialized. This ability is provided since configurable elements in the VMI specification do not cover all of the libvirt domain XML elements.</p> <p>The sidecar containers communicate with the main container over a socket with a gRPC protocol. There are two main sidecar hooks:</p> <ol> <li><code>onDefineDomain</code>: This hook helps to customize libvirt's XML and return the new XML over gRPC for the VM creation.</li> <li><code>preCloudInitIso</code>: This hook helps to customize the cloud-init configuration. It operates on and returns JSON    formatted cloud-init data.</li> </ol>"},{"location":"user_workloads/hook-sidecar/#enabling-sidecar-feature-gate","title":"Enabling <code>Sidecar</code> feature gate","text":"<p><code>Sidecar</code> feature gate can be enabled by following the steps mentioned in Activating feature gates.</p> <p>In case of a development cluster created using kubevirtci, follow the steps mentioned in the  developer doc to enable  the feature gate.</p>"},{"location":"user_workloads/hook-sidecar/#sidecar-shim-container-image","title":"Sidecar-shim container image","text":"<p>To run a VM with custom modifications, the sidecar-shim-image  takes care of implementing the communication with the main container.</p> <p>The image contains the <code>sidecar-shim</code> binary built using <code>sidecar_shim.go</code> which should be kept as the entrypoint of the container. This binary will search in <code>$PATH</code> for binaries named after the hook names (e.g <code>onDefineDomain</code> and <code>preCloudInitIso</code>) and run them. Users must provide the necessary arguments as command line options (flags).</p> <p>In the case of <code>onDefineDomain</code>, the arguments will be the VMI information as JSON string, (e.g <code>--vmi vmiJSON</code>) and the current domain XML (e.g <code>--domain domainXML</code>). It outputs the modified domain XML on the standard output.</p> <p>In the case of <code>preCloudInitIso</code>, the arguments will be the VMI information as JSON string, (e.g <code>--vmi vmiJSON</code>) and the CloudInitData (e.g <code>--cloud-init cloudInitJSON</code>). It outputs the modified CloudInitData (as JSON) on the standard ouput.</p> <p>Shell or python scripts can be used as alternatives to the binary, by making them available at the expected location  (<code>/usr/bin/onDefineDomain</code> or <code>/usr/bin/preCloudInitIso</code> depending upon the hook).</p> <p>A prebuilt image named <code>sidecar-shim</code> capable of running Shell or Python scripts is shipped as part of KubeVirt releases.</p>"},{"location":"user_workloads/hook-sidecar/#go-python-shell-pick-any-one","title":"Go, Python, Shell - pick any one","text":"<p>Although a binary doesn't strictly need to be generated from Go code, and a script doesn't strictly need to be one among Shell or Python, for the purpose of this guide, we will use those as examples.</p>"},{"location":"user_workloads/hook-sidecar/#go-binary","title":"Go binary","text":"<p>Example Go code modifiying the SMBIOS system information can be found in the KubeVirt repo. Binary generated from this code, when available under <code>/usr/bin/ondefinedomain</code> in the sidecar-shim-image, is run right before VMI creation and the baseboard manufacturer value is modified to reflect what's provided in the <code>smbios.vm.kubevirt.io/baseBoardManufacturer</code> annotation in VMI spec.</p>"},{"location":"user_workloads/hook-sidecar/#shell-or-python-script","title":"Shell or Python script","text":"<p>If you pefer writing a shell or python script instead of a Go program, create a Kubernetes ConfigMap and use annotations to make sure the script is run before the VMI creation. The flow would be as below:</p> <ol> <li>Create a ConfigMap containing the shell or python script you want to run</li> <li>Create a VMI containing the annotation <code>hooks.kubevirt.io/hookSidecars</code> and mention the ConfigMap information in it.</li> <li>In this case a predefined image can be used to handle the communication with the main container.</li> </ol>"},{"location":"user_workloads/hook-sidecar/#configmap-with-shell-script","title":"ConfigMap with shell script","text":"<pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/bin/sh\n    tempFile=`mktemp --dry-run`\n    echo $4 &gt; $tempFile\n    sed -i \"s|&lt;baseBoard&gt;&lt;/baseBoard&gt;|&lt;baseBoard&gt;&lt;entry name='manufacturer'&gt;Radical Edward&lt;/entry&gt;&lt;/baseBoard&gt;|\" $tempFile\n    cat $tempFile\n</code></pre>"},{"location":"user_workloads/hook-sidecar/#configmap-with-python-script","title":"ConfigMap with python script","text":"<pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: my-config-map\ndata:\n  my_script.sh: |\n    #!/usr/bin/env python3\n\n    import xml.etree.ElementTree as ET\n    import sys\n\n    def main(s):\n        # write to a temporary file\n        f = open(\"/tmp/orig.xml\", \"w\")\n        f.write(s)\n        f.close()\n\n        # parse xml from file\n        xml = ET.parse(\"/tmp/orig.xml\")\n        # get the root element\n        root = xml.getroot()\n        # find the baseBoard element\n        baseBoard = root.find(\"sysinfo\").find(\"baseBoard\")\n\n        # prepare new element to be inserted into the xml definition\n        element = ET.Element(\"entry\", {\"name\": \"manufacturer\"})\n        element.text = \"Radical Edward\"\n        # insert the element\n        baseBoard.insert(0, element)\n\n        # write to a new file\n        xml.write(\"/tmp/new.xml\")\n        # print file contents to stdout\n        f = open(\"/tmp/new.xml\")\n        print(f.read())\n        f.close()\n\n    if __name__ == \"__main__\":\n        main(sys.argv[4])\n</code></pre> <p>After creating one of the above ConfigMap, create the VMI using the manifest in this example. Of importance here is the ConfigMap information stored in the annotations:</p> <pre><code>annotations:\n  hooks.kubevirt.io/hookSidecars: &gt;\n    [\n        {\n            \"args\": [\"--version\", \"v1alpha2\"],\n            \"configMap\": {\"name\": \"my-config-map\", \"key\": \"my_script.sh\", \"hookPath\": \"/usr/bin/onDefineDomain\"}\n        }\n    ]\n</code></pre> <p>The <code>name</code> field indicates the name of the ConfigMap on the cluster which contains the script you want to execute. The <code>key</code> field indicates the key in the ConfigMap which contains the script to be executed. Finally, <code>hookPath</code> indicates the path where you want the script to be mounted. It could be either of <code>/usr/bin/onDefineDomain</code> or <code>/usr/bin/preCloudInitIso</code> depending upon the hook you want to execute. An optional value can be specified with the <code>\"image\"</code> key if a custom image is needed, if omitted the default Sidecar-shim image built together with the other KubeVirt images will be used. The default Sidecar-shim image, if not override with a custom value, will also be updated as other images as for Updating KubeVirt Workloads.</p>"},{"location":"user_workloads/hook-sidecar/#verify-everything-works","title":"Verify everything works","text":"<p>Whether you used the Go binary or a Shell/Python script from the above examples, you would be able to see the newly  created VMI have the modified baseboard manufacturer information. After creating the VMI, verify that it is in the  <code>Running</code> state, and connect to its console and see if the desired changes to baseboard manufacturer get reflected:</p> <pre><code># Once the VM is ready, connect to its display and login using name and password \"fedora\"\ncluster/virtctl.sh vnc vmi-with-sidecar-hook-configmap\n\n# Check whether the base board manufacturer value was successfully overwritten\nsudo dmidecode -s baseboard-manufacturer\n</code></pre>"},{"location":"user_workloads/instancetypes/","title":"Instance types and preferences","text":"<p>FEATURE STATE: </p> <ul> <li><code>instancetype.kubevirt.io/v1alpha1</code> (Experimental) as of the <code>v0.56.0</code> KubeVirt release</li> <li><code>instancetype.kubevirt.io/v1alpha2</code> (Experimental) as of the <code>v0.58.0</code> KubeVirt release</li> <li><code>instancetype.kubevirt.io/v1beta1</code> as of the <code>v1.0.0</code> KubeVirt release</li> </ul> <p>See the Version History section for more details.</p>"},{"location":"user_workloads/instancetypes/#introduction","title":"Introduction","text":"<p>KubeVirt's <code>VirtualMachine</code> API contains many advanced options for tuning the performance of a VM that goes beyond what typical users need to be aware of. Users have previously been unable to simply define the storage/network they want assigned to their VM and then declare in broad terms what quality of resources and kind of performance characteristics they need for their VM.</p> <p>Instance types and preferences provide a way to define a set of resource, performance and other runtime characteristics, allowing users to reuse these definitions across multiple <code>VirtualMachines</code>.</p>"},{"location":"user_workloads/instancetypes/#virtualmachineinstancetype","title":"VirtualMachineInstancetype","text":"<pre><code>---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachineInstancetype\nmetadata:\n  name: example-instancetype\nspec:\n  cpu:\n    guest: 1\n  memory:\n    guest: 128Mi\n</code></pre> <p>KubeVirt provides two <code>CRDs</code> for instance types, a cluster wide <code>VirtualMachineClusterInstancetype</code> and a namespaced <code>VirtualMachineInstancetype</code>. These <code>CRDs</code> encapsulate the following resource related characteristics of a <code>VirtualMachine</code> through a shared <code>VirtualMachineInstancetypeSpec</code>:</p> <ul> <li><code>CPU</code> : Required number of vCPUs presented to the guest</li> <li><code>Memory</code> : Required amount of memory presented to the guest</li> <li><code>GPUs</code> : Optional list of vGPUs to passthrough</li> <li><code>HostDevices</code> : Optional list of <code>HostDevices</code> to passthrough</li> <li><code>IOThreadsPolicy</code> : Optional <code>IOThreadsPolicy</code> to be used</li> <li><code>LaunchSecurity</code>: Optional <code>LaunchSecurity</code> to be used</li> </ul> <p>Anything provided within an instance type cannot be overridden within the <code>VirtualMachine</code>. For example, as <code>CPU</code> and <code>Memory</code> are both required attributes of an instance type, if a user makes any requests for <code>CPU</code> or <code>Memory</code> resources within the underlying <code>VirtualMachine</code>, the instance type will conflict and the request will be rejected during creation.</p>"},{"location":"user_workloads/instancetypes/#virtualmachinepreference","title":"VirtualMachinePreference","text":"<pre><code>---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: example-preference\nspec:\n  devices:\n    preferredDiskBus: virtio\n    preferredInterfaceModel: virtio\n</code></pre> <p>KubeVirt also provides two further preference based <code>CRDs</code>, again a cluster wide <code>VirtualMachineClusterPreference</code> and namespaced <code>VirtualMachinePreference</code>. These <code>CRDs</code>encapsulate the preferred value of any remaining attributes of a <code>VirtualMachine</code> required to run a given workload, again this is through a shared <code>VirtualMachinePreferenceSpec</code>.</p> <p>Unlike instance types, preferences only represent the preferred values and as such, they can be overridden by values in the <code>VirtualMachine</code> provided by the user.</p> <p>In the example shown below, a user has provided a <code>VirtualMachine</code> with a disk bus already defined within a <code>DiskTarget</code> and has also selected a set of preferences with <code>DevicePreference</code> and <code>preferredDiskBus</code> , so the user's original choice within the <code>VirtualMachine</code> and <code>DiskTarget</code> are used:</p> <pre><code>$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: example-preference-disk-virtio\nspec:\n  devices:\n    preferredDiskBus: virtio\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: example-preference-user-override\nspec:\n  preference:\n    kind: VirtualMachinePreference\n    name: example-preference-disk-virtio\n  running: false\n  template:\n    spec:\n      domain:\n        memory:\n          guest: 128Mi\n        devices:\n          disks:\n          - disk:\n              bus: sata\n            name: containerdisk\n          - disk: {}\n            name: cloudinitdisk\n        resources: {}\n      terminationGracePeriodSeconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |\n            #!/bin/sh\n\n            echo 'printed from cloud-init userdata'\n        name: cloudinitdisk\nEOF\nvirtualmachinepreference.instancetype.kubevirt.io/example-preference-disk-virtio created\nvirtualmachine.kubevirt.io/example-preference-user-override configured\n\n\n$ virtctl start example-preference-user-override\nVM example-preference-user-override was scheduled to start\n\n# We can see the original request from the user within the VirtualMachine lists `containerdisk` with a `SATA` bus\n$ kubectl get vms/example-preference-user-override -o json | jq .spec.template.spec.domain.devices.disks\n[\n  {\n    \"disk\": {\n      \"bus\": \"sata\"\n    },\n    \"name\": \"containerdisk\"\n  },\n  {\n    \"disk\": {},\n    \"name\": \"cloudinitdisk\"\n  }\n]\n\n# This is still the case in the VirtualMachineInstance with the remaining disk using the `preferredDiskBus` from the preference of `virtio`\n$ kubectl get vmis/example-preference-user-override -o json | jq .spec.domain.devices.disks\n[\n  {\n    \"disk\": {\n      \"bus\": \"sata\"\n    },\n    \"name\": \"containerdisk\"\n  },\n  {\n    \"disk\": {\n      \"bus\": \"virtio\"\n    },\n    \"name\": \"cloudinitdisk\"\n  }\n]\n</code></pre>"},{"location":"user_workloads/instancetypes/#virtualmachine","title":"VirtualMachine","text":"<pre><code>---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: example-vm\nspec:\n  instancetype:\n    kind: VirtualMachineInstancetype\n    name: example-instancetype\n  preference:\n    kind: VirtualMachinePreference\n    name: example-preference\n</code></pre> <p>The previous instance type and preference CRDs are matched to a given <code>VirtualMachine</code> through the use of a matcher. Each matcher consists of the following:</p> <ul> <li><code>Name</code> (string): Name of the resource being referenced</li> <li><code>Kind</code> (string):  Optional, defaults to the cluster wide CRD kinds of <code>VirtualMachineClusterInstancetype</code> or <code>VirtualMachineClusterPreference</code> if not provided</li> <li><code>RevisionName</code> (string) : Optional, name of a <code>ControllerRevision</code> containing a copy of the <code>VirtualMachineInstancetypeSpec</code> or <code>VirtualMachinePreferenceSpec</code> taken when the <code>VirtualMachine</code> is first created. See the Versioning section below for more details on how and why this is captured.</li> <li><code>InferFromVolume</code> (string): Optional, see the Inferring defaults from a Volume section below for more details.</li> </ul>"},{"location":"user_workloads/instancetypes/#creating-instancetypes-preferences-and-virtualmachines","title":"Creating InstanceTypes, Preferences and VirtualMachines","text":"<p>It is possible to streamline the creation of instance types, preferences, and virtual machines with the usage of the virtctl command-line tool. To read more about it, please see the Creating VirtualMachines.</p>"},{"location":"user_workloads/instancetypes/#versioning","title":"Versioning","text":"<p>Versioning of these resources is required to ensure the eventual <code>VirtualMachineInstance</code> created when starting a <code>VirtualMachine</code> does not change between restarts if any referenced instance type or set of preferences are updated during the lifetime of the <code>VirtualMachine</code>.</p> <p>This is currently achieved by using <code>ControllerRevision</code> to retain a copy of the <code>VirtualMachineInstancetype</code> or <code>VirtualMachinePreference</code> at the time the <code>VirtualMachine</code> is created. A reference to these <code>ControllerRevisions</code> are then retained in the <code>InstancetypeMatcher</code> and <code>PreferenceMatcher</code> within the <code>VirtualMachine</code> for future use.</p> <pre><code>$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall created\nvirtualmachine.kubevirt.io/vm-cirros-csmall created\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\"\n}\n\n$ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .\n{\n  \"apiVersion\": \"apps/v1\",\n  \"data\": {\n    \"apiVersion\": \"instancetype.kubevirt.io/v1beta1\",\n    \"kind\": \"VirtualMachineInstancetype\",\n    \"metadata\": {\n      \"creationTimestamp\": \"2022-09-30T12:20:19Z\",\n      \"generation\": 1,\n      \"name\": \"csmall\",\n      \"namespace\": \"default\",\n      \"resourceVersion\": \"10303\",\n      \"uid\": \"72c3a35b-6e18-487d-bebf-f73c7d4f4a40\"\n    },\n    \"spec\": {\n      \"cpu\": {\n        \"guest\": 1\n      },\n      \"memory\": {\n        \"guest\": \"128Mi\"\n      }\n    }\n  },\n  \"kind\": \"ControllerRevision\",\n  \"metadata\": {\n    \"creationTimestamp\": \"2022-09-30T12:20:19Z\",\n    \"name\": \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\",\n    \"namespace\": \"default\",\n    \"ownerReferences\": [\n      {\n        \"apiVersion\": \"kubevirt.io/v1\",\n        \"blockOwnerDeletion\": true,\n        \"controller\": true,\n        \"kind\": \"VirtualMachine\",\n        \"name\": \"vm-cirros-csmall\",\n        \"uid\": \"5216527a-1d31-4637-ad3a-b640cb9949a2\"\n      }\n    ],\n    \"resourceVersion\": \"10307\",\n    \"uid\": \"a7bc784b-4cea-45d7-8432-15418e1dd7d3\"\n  },\n  \"revision\": 0\n}\n\n\n$ kubectl delete vm/vm-cirros-csmall\nvirtualmachine.kubevirt.io \"vm-cirros-csmall\" deleted\n\n$ kubectl get controllerrevision/controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\nError from server (NotFound): controllerrevisions.apps \"vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1\" not found\n</code></pre> <p>Users can opt in to moving to a newer generation of an instance type or preference by removing the referenced <code>revisionName</code> from the appropriate matcher within the <code>VirtualMachine</code> object. This will result in fresh <code>ControllerRevisions</code> being captured and used.</p> <p>The following example creates a <code>VirtualMachine</code> using an initial version of the csmall instance type before increasing the number of vCPUs provided by the instance type:</p> <pre><code>$ kubectl apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall created\nvirtualmachine.kubevirt.io/vm-cirros-csmall created\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-1\"\n}\n\n$ virtctl start vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to start\n\n$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu\n{\n  \"cores\": 1,\n  \"model\": \"host-model\",\n  \"sockets\": 1,\n  \"threads\": 1\n}\n\n$ kubectl patch VirtualMachineInstancetype/csmall --type merge -p '{\"spec\":{\"cpu\":{\"guest\":2}}}'\nvirtualmachineinstancetype.instancetype.kubevirt.io/csmall patched\n</code></pre> <p>In order for this change to be picked up within the <code>VirtualMachine</code>, we need to stop the running <code>VirtualMachine</code> and clear the <code>revisionName</code> referenced by the <code>InstancetypeMatcher</code>:</p> <pre><code>$ virtctl stop vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to stop\n\n$ kubectl patch vm/vm-cirros-csmall --type merge -p '{\"spec\":{\"instancetype\":{\"revisionName\":\"\"}}}'\nvirtualmachine.kubevirt.io/vm-cirros-csmall patched\n\n$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype\n{\n  \"kind\": \"VirtualMachineInstancetype\",\n  \"name\": \"csmall\",\n  \"revisionName\": \"vm-cirros-csmall-csmall-3e86e367-9cd7-4426-9507-b14c27a08671-2\"\n}\n</code></pre> <p>As you can see above, the <code>InstancetypeMatcher</code> now references a new <code>ControllerRevision</code> containing generation 2 of the instance type. We can now start the <code>VirtualMachine</code> again and see the new number of vCPUs being used by the <code>VirtualMachineInstance</code>:</p> <pre><code>$ virtctl start vm-cirros-csmall\nVM vm-cirros-csmall was scheduled to start\n\n$ kubectl get vmi/vm-cirros-csmall -o json | jq .spec.domain.cpu\n{\n  \"cores\": 1,\n  \"model\": \"host-model\",\n  \"sockets\": 2,\n  \"threads\": 1\n}\n</code></pre>"},{"location":"user_workloads/instancetypes/#inferfromvolume","title":"inferFromVolume","text":"<p>The <code>inferFromVolume</code> attribute of both the <code>InstancetypeMatcher</code> and <code>PreferenceMatcher</code> allows a user to request that defaults are inferred from a volume. When requested, KubeVirt will look for the following labels on the underlying <code>PVC</code>, <code>DataSource</code> or <code>DataVolume</code> to determine the default name and kind:</p> <ul> <li><code>instancetype.kubevirt.io/default-instancetype</code></li> <li><code>instancetype.kubevirt.io/default-instancetype-kind</code> (optional, defaults to <code>VirtualMachineClusterInstancetype</code>)</li> <li><code>instancetype.kubevirt.io/default-preference</code></li> <li><code>instancetype.kubevirt.io/default-preference-kind</code> (optional, defaults to <code>VirtualMachineClusterPreference</code>)</li> </ul> <p>These values are then written into the appropriate matcher by the mutation webhook and used during validation before the <code>VirtualMachine</code> is formally accepted.</p> <p>The validation can be controlled by the value provided to <code>inferFromVolumeFailurePolicy</code> in either the <code>InstancetypeMatcher</code> or <code>PreferenceMatcher</code> of a <code>VirtualMachine</code>.</p> <p>The default value of <code>Reject</code> will cause the request to be rejected on failure to find the referenced <code>Volume</code> or labels on an underlying resource.</p> <p>If <code>Ignore</code> was provided, the respective <code>InstancetypeMatcher</code> or <code>PreferenceMatcher</code> will be cleared on a failure instead.</p> <p>Example with implicit default value of <code>Reject</code>:</p> <pre><code>$ kubectl apply -k https://github.com/kubevirt/common-instancetypes.git\n[..]\n$ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img\n[..]\n$ kubectl label pvc/cirros-pvc \\\n  instancetype.kubevirt.io/default-instancetype=server.tiny \\\n  instancetype.kubevirt.io/default-preference=cirros\n[..]\n$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataSource\nmetadata:\n  name: cirros-datasource\nspec:\n  source:\n    pvc:\n      name: cirros-pvc\n      namespace: default\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: cirros\nspec:\n  instancetype:\n    inferFromVolume: cirros-volume\n  preference:\n    inferFromVolume: cirros-volume\n  running: false\n  dataVolumeTemplates:\n    - metadata:\n        name: cirros-datavolume\n      spec:\n        storage:\n          resources:\n            requests:\n              storage: 1Gi\n          storageClassName: local\n        sourceRef:\n          kind: DataSource\n          name: cirros-datasource\n          namespace: default\n  template:\n    spec:\n      domain:\n        devices: {}\n      volumes:\n        - dataVolume:\n            name: cirros-datavolume\n          name: cirros-volume\nEOF\n[..]\nkubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'\n{\n  \"kind\": \"virtualmachineclusterinstancetype\",\n  \"name\": \"server.tiny\",\n  \"revisionName\": \"cirros-server.tiny-76454433-3d82-43df-a7e5-586e48c71f68-1\"\n}\n{\n  \"kind\": \"virtualmachineclusterpreference\",\n  \"name\": \"cirros\",\n  \"revisionName\": \"cirros-cirros-85823ddc-9e8c-4d23-a94c-143571b5489c-1\"\n}\n</code></pre> <p>Example with explicit value of <code>Ignore</code>:</p> <pre><code>$ virtctl image-upload pvc cirros-pvc --size=1Gi --image-path=./cirros-0.5.2-x86_64-disk.img\n$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataSource\nmetadata:\n  name: cirros-datasource\nspec:\n  source:\n    pvc:\n      name: cirros-pvc\n      namespace: default\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  name: cirros\nspec:\n  instancetype:\n    inferFromVolume: cirros-volume\n    inferFromVolumeFailurePolicy: Ignore\n  preference:\n    inferFromVolume: cirros-volume\n    inferFromVolumeFailurePolicy: Ignore\n  running: false\n  dataVolumeTemplates:\n    - metadata:\n        name: cirros-datavolume\n      spec:\n        storage:\n          accessModes:\n            - ReadWriteOnce\n          resources:\n            requests:\n              storage: 1Gi\n          storageClassName: local\n        sourceRef:\n          kind: DataSource\n          name: cirros-datasource\n          namespace: default\n  template:\n    spec:\n      domain:\n        devices: {}\n      volumes:\n        - dataVolume:\n            name: cirros-datavolume\n          name: cirros-volume\nEOF\n[..]\nkubectl get vms/cirros -o json | jq '.spec.instancetype, .spec.preference'\nnull\nnull\n</code></pre>"},{"location":"user_workloads/instancetypes/#common-instancetypes","title":"common-instancetypes","text":"<p>The <code>kubevirt/common-instancetypes</code> provide a set of instancetypes and preferences to help create KubeVirt <code>VirtualMachines</code>.</p> <p>See Deploy common-instancetypes on how to deploy them.</p>"},{"location":"user_workloads/instancetypes/#examples","title":"Examples","text":"<p>Various examples are available within the <code>kubevirt</code> repo under <code>/examples</code>. The following uses an example <code>VirtualMachine</code> provided by the <code>containerdisk/fedora</code> repo and replaces much of the <code>DomainSpec</code> with the equivalent instance type and preferences:</p> <pre><code>$ kubectl apply -f - &lt;&lt; EOF\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachineInstancetype\nmetadata:\n  name: cmedium\nspec:\n  cpu:\n    guest: 1\n  memory:\n    guest: 1Gi\n---\napiVersion: instancetype.kubevirt.io/v1beta1\nkind: VirtualMachinePreference\nmetadata:\n  name: fedora\nspec:\n  devices:\n    preferredDiskBus: virtio\n    preferredInterfaceModel: virtio\n    preferredRng: {}\n  features:\n    preferredAcpi: {}\n    preferredSmm: {}\n  firmware:\n    preferredUseEfi: true\n    preferredUseSecureBoot: true    \n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  creationTimestamp: null\n  name: fedora\nspec:\n  instancetype:\n    name: cmedium\n    kind: virtualMachineInstancetype\n  preference:\n    name: fedora\n    kind: virtualMachinePreference\n  runStrategy: Always\n  template:\n    metadata:\n      creationTimestamp: null\n    spec:\n      domain:\n        devices: {}\n      volumes:\n      - containerDisk:\n          image: quay.io/containerdisks/fedora:latest\n        name: containerdisk\n      - cloudInitNoCloud:\n          userData: |-\n            #cloud-config\n            ssh_authorized_keys:\n              - ssh-rsa AAAA...\n        name: cloudinit\nEOF\n</code></pre>"},{"location":"user_workloads/instancetypes/#version-history","title":"Version History","text":""},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1alpha1-experimental","title":"<code>instancetype.kubevirt.io/v1alpha1</code> (Experimental)","text":"<ul> <li>Initial development version.</li> </ul>"},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1alpha2-experimental","title":"<code>instancetype.kubevirt.io/v1alpha2</code> (Experimental)","text":"<ul> <li> <p>This version captured complete <code>VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}</code> objects within the created <code>ControllerRevisions</code></p> </li> <li> <p>This version is backwardly compatible with <code>instancetype.kubevirt.io/v1alpha1</code>.</p> </li> </ul>"},{"location":"user_workloads/instancetypes/#instancetypekubevirtiov1beta1","title":"<code>instancetype.kubevirt.io/v1beta1</code>","text":"<ul> <li>The following instance type attribute has been added:</li> <li> <p><code>Spec.Memory.OvercommitPercent</code></p> </li> <li> <p>The following preference attributes have been added:</p> </li> <li><code>Spec.CPU.PreferredCPUFeatures</code></li> <li><code>Spec.Devices.PreferredInterfaceMasquerade</code></li> <li><code>Spec.PreferredSubdomain</code></li> <li><code>Spec.PreferredTerminationGracePeriodSeconds</code></li> <li> <p><code>Spec.Requirements</code></p> </li> <li> <p>This version is backwardly compatible with <code>instancetype.kubevirt.io/v1alpha1</code> and <code>instancetype.kubevirt.io/v1alpha2</code> objects, no modifications are required to existing  <code>VirtualMachine{Instancetype,ClusterInstancetype,Preference,ClusterPreference}</code> or <code>ControllerRevisions</code>.</p> </li> <li> <p>As with the migration to <code>kubevirt.io/v1</code> it is recommend previous users of <code>instancetype.kubevirt.io/v1alpha1</code> or <code>instancetype.kubevirt.io/v1alpha2</code> use <code>kube-storage-version-migrator</code> to upgrade any stored objects to <code>instancetype.kubevirt.io/v1beta1</code>.</p> </li> </ul>"},{"location":"user_workloads/lifecycle/","title":"Lifecycle","text":"<p>Every <code>VirtualMachineInstance</code> represents a single virtual machine instance. In general, the management of VirtualMachineInstances is kept similar to how <code>Pods</code> are managed: Every VM that is defined in the cluster is expected to be running, just like Pods. Deleting a VirtualMachineInstance is equivalent to shutting it down, this is also equivalent to how Pods behave.</p>"},{"location":"user_workloads/lifecycle/#launching-a-virtual-machine","title":"Launching a virtual machine","text":"<p>In order to start a VirtualMachineInstance, you just need to create a <code>VirtualMachineInstance</code> object using <code>kubectl</code>:</p> <pre><code>$ kubectl create -f vmi.yaml\n</code></pre>"},{"location":"user_workloads/lifecycle/#listing-virtual-machines","title":"Listing virtual machines","text":"<p>VirtualMachineInstances can be listed by querying for VirtualMachineInstance objects:</p> <pre><code>$ kubectl get vmis\n</code></pre>"},{"location":"user_workloads/lifecycle/#retrieving-a-virtual-machine-instance-definition","title":"Retrieving a virtual machine instance definition","text":"<p>A single VirtualMachineInstance definition can be retrieved by getting the specific VirtualMachineInstance object:</p> <pre><code>$ kubectl get vmis testvmi\n</code></pre>"},{"location":"user_workloads/lifecycle/#stopping-a-virtual-machine-instance","title":"Stopping a virtual machine instance","text":"<p>To stop the VirtualMachineInstance, you just need to delete the corresponding <code>VirtualMachineInstance</code> object using <code>kubectl</code>.</p> <pre><code>$ kubectl delete -f vmi.yaml\n# OR\n$ kubectl delete vmis testvmi\n</code></pre> <p>Note: Stopping a VirtualMachineInstance implies that it will be deleted from the cluster. You will not be able to start this VirtualMachineInstance object again.</p>"},{"location":"user_workloads/lifecycle/#starting-and-stopping-a-virtual-machine","title":"Starting and stopping a virtual machine","text":"<p>Virtual machines, in contrast to VirtualMachineInstances, have a running state. Thus on VM you can define if it should be running, or not. VirtualMachineInstances are, if they are defined in the cluster, always running and consuming resources.</p> <p><code>virtctl</code> is used in order to start and stop a VirtualMachine:</p> <pre><code>$ virtctl start my-vm\n$ virtctl stop my-vm\n</code></pre> <p>Note: You can force stop a VM (which is like pulling the power cord, with all its implications like data inconsistencies or [in the worst case] data loss) by</p> <pre><code>$ virtctl stop my-vm --grace-period 0 --force\n</code></pre>"},{"location":"user_workloads/lifecycle/#pausing-and-unpausing-a-virtual-machine","title":"Pausing and unpausing a virtual machine","text":"<p>Note: Pausing in this context refers to libvirt's <code>virDomainSuspend</code> command: \"The process is frozen without further access to CPU resources and I/O but the memory used by the domain at the hypervisor level will stay allocated\"</p> <p>To pause a virtual machine, you need the <code>virtctl</code> command line tool. Its <code>pause</code> command works on either <code>VirtualMachine</code> s or <code>VirtualMachinesInstance</code> s:</p> <pre><code>$ virtctl pause vm testvm\n# OR\n$ virtctl pause vmi testvm\n</code></pre> <p>Paused VMIs have a <code>Paused</code> condition in their status:</p> <pre><code>$ kubectl get vmi testvm -o=jsonpath='{.status.conditions[?(@.type==\"Paused\")].message}'\nVMI was paused by user\n</code></pre> <p>Unpausing works similar to pausing:</p> <pre><code>$ virtctl unpause vm testvm\n# OR\n$ virtctl unpause vmi testvm\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/","title":"Liveness and Readiness Probes","text":"<p>It is possible to configure Liveness and Readiness Probes in a similar fashion like it is possible to configure Liveness and Readiness Probes on Containers.</p> <p>Liveness Probes will effectively stop the VirtualMachineInstance if they fail, which will allow higher level controllers, like VirtualMachine or VirtualMachineInstanceReplicaSet to spawn new instances, which will hopefully be responsive again.</p> <p>Readiness Probes are an indicator for Services and Endpoints if the VirtualMachineInstance is ready to receive traffic from Services. If Readiness Probes fail, the VirtualMachineInstance will be removed from the Endpoints which back services until the probe recovers.</p> <p>Watchdogs focus on ensuring that an Operating System is still responsive. They complement the probes which are more workload centric. Watchdogs require kernel support from the guest and additional tooling like the commonly used <code>watchdog</code> binary.</p> <p>Exec probes are Liveness or Readiness probes specifically intended for VMs. These probes run a command inside the VM and determine the VM ready/live state based on its success. For running commands inside the VMs, the qemu-guest-agent package is used. A command supplied to an exec probe will be wrapped by <code>virt-probe</code> in the  operator and forwarded to the guest.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-a-http-liveness-probe","title":"Define a HTTP Liveness Probe","text":"<p>The following VirtualMachineInstance configures a HTTP Liveness Probe via <code>spec.livenessProbe.httpGet</code>, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  livenessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    httpGet:\n      port: 1500\n    timeoutSeconds: 10\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-a-tcp-liveness-probe","title":"Define a TCP Liveness Probe","text":"<p>The following VirtualMachineInstance configures a TCP Liveness Probe via <code>spec.livenessProbe.tcpSocket</code>, which will query port 1500 of the VirtualMachineInstance, after an initial delay of 120 seconds. The VirtualMachineInstance itself installs and runs a minimal HTTP server on port 1500 via cloud-init.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  livenessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    tcpSocket:\n      port: 1500\n    timeoutSeconds: 10\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre>"},{"location":"user_workloads/liveness_and_readiness_probes/#define-readiness-probes","title":"Define Readiness Probes","text":"<p>Readiness Probes are configured in a similar way like liveness probes. Instead of <code>spec.livenessProbe</code>, <code>spec.readinessProbe</code> needs to be filled:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-fedora-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-fedora\n        kubevirt.io/vm: vmi-fedora\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  readinessProbe:\n    initialDelaySeconds: 120\n    periodSeconds: 20\n    timeoutSeconds: 10\n    failureThreshold: 3\n    successThreshold: 3\n    httpGet:\n      port: 1500\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"nmap-ncat\"]\n          - [\"sudo\", \"systemd-run\", \"--unit=httpserver\", \"nc\", \"-klp\", \"1500\", \"-e\", '/usr/bin/echo -e HTTP/1.1 200 OK\\\\nContent-Length: 12\\\\n\\\\nHello World!']\n    name: cloudinitdisk\n</code></pre> <p>Note that in the case of Readiness Probes, it is also possible to set a <code>failureThreshold</code> and a <code>successThreashold</code> to only flip between ready and non-ready state if the probe succeeded or failed multiple times.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#dual-stack-considerations","title":"Dual-stack considerations","text":"<p>Some context is needed to understand the limitations imposed by a dual-stack network configuration on readiness - or liveness - probes. Users must be fully aware that a dual-stack configuration is currently only available when using a masquerade binding type. Furthermore, it must be recalled that accessing a VM using masquerade binding type is performed via the pod IP address; in dual-stack mode, both IPv4 and IPv6 addresses can be used to reach the VM.</p> <p>Dual-stack networking configurations have a limitation when using HTTP / TCP probes - you cannot probe the VMI by its IPv6 address. The reason for this is the <code>host</code> field for both the HTTP and TCP probe actions default to the pod's IP address, which is currently always the IPv4 address.</p> <p>Since the pod's IP address is not known before creating the VMI, it is not possible to pre-provision the probe's host field.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#defining-a-watchdog","title":"Defining a Watchdog","text":"<p>A watchdog is a more VM centric approach where the responsiveness of the Operating System is focused on. One can configure the <code>i6300esb</code> watchdog device:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    special: vmi-with-watchdog\n  name: vmi-with-watchdog\nspec:\n  domain:\n    devices:\n      watchdog:\n        name: mywatchdog\n        i6300esb:\n          action: \"poweroff\"\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n    machine:\n      type: \"\"\n    resources:\n      requests:\n        memory: 1024M\n  terminationGracePeriodSeconds: 0\n  volumes:\n  - containerDisk:\n      image: quay.io/containerdisks/fedora:latest\n    name: containerdisk\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        bootcmd:\n          - [\"sudo\", \"dnf\", \"install\", \"-y\", \"busybox\"]\n    name: cloudinitdisk\n</code></pre> <p>The example above configures it with the <code>poweroff</code> action. It defines what will happen if the OS can't respond anymore. Other possible actions are <code>reset</code> and <code>shutdown</code>. The VM in this example will have the device exposed as <code>/dev/watchdog</code>. This device can then be used by the <code>watchdog</code> binary. For example, if root executes this command inside the VM:</p> <pre><code>sudo busybox watchdog -t 2000ms -T 4000ms /dev/watchdog\n</code></pre> <p>the watchdog will send a heartbeat every two seconds to <code>/dev/watchdog</code> and after four seconds without a heartbeat the defined action will be executed. In this case a hard <code>poweroff</code>.</p>"},{"location":"user_workloads/liveness_and_readiness_probes/#defining-guest-agent-ping-probes","title":"Defining Guest-Agent Ping Probes","text":"<p>Guest-Agent probes are based on qemu-guest-agent <code>guest-ping</code>.  This will ping the guest and return an error if the guest is not up and running.  To easily define this on VM spec, specify <code>guestAgentPing: {}</code> in VM's  <code>spec.template.spec.readinessProbe</code>.  <code>virt-controller</code> will translate this  into a corresponding command wrapped by <code>virt-probe</code>.</p> <p>Note: You can only define one of the type of probe, i.e. guest-agent exec  or ping probes.</p> <p>Important: If the qemu-guest-agent is not installed and enabled inside the VM, the probe will fail.  Many images don't enable the agent by default so make sure you either run one that does or enable it. </p> <p>Make sure to provide enough delay and failureThreshold for the VM and the agent to be online.</p> <p>In the following example the Fedora image does have qemu-guest-agent available by default. Nevertheless, in case qemu-guest-agent is not installed, it will be installed and enabled via cloud-init as shown in the example below.  Also, cloud-init assigns the proper SELinux context, i.e. virt_qemu_ga_exec_t, to the <code>/tmp/healthy.txt</code> file.  Otherwise, SELinux will deny the attempts to open the <code>/tmp/healthy.txt</code> file causing the probe to fail.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  labels:\n    kubevirt.io/vm: vmi-guest-probe-vmi\n  name: vmi-fedora\nspec:\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: vmi-guest-probe\n        kubevirt.io/vm: vmi-guest-probe\n  domain:\n    devices:\n      disks:\n      - disk:\n          bus: virtio\n        name: containerdisk\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n      rng: {}\n    resources:\n      requests:\n        memory: 1024M\n  readinessProbe:\n    exec:\n      command: [\"cat\", \"/tmp/healthy.txt\"]\n    failureThreshold: 10\n    initialDelaySeconds: 20\n    periodSeconds: 10\n    timeoutSeconds: 5\n  terminationGracePeriodSeconds: 180\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: quay.io/containerdisks/fedora\n  - cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        user: fedora\n        chpasswd: { expire: False }\n        packages:\n          qemu-guest-agent\n        runcmd:\n          - [\"touch\", \"/tmp/healthy.txt\"]\n          - [\"sudo\", \"chcon\", \"-t\", \"virt_qemu_ga_exec_t\", \"/tmp/healthy.txt\"]\n          - [\"sudo\", \"systemctl\", \"enable\", \"--now\", \"qemu-guest-agent\"]\n    name: cloudinitdisk\n</code></pre> <p>Note that, in the above example if SELinux is not installed in your container disk image, the command <code>chcon</code> should be removed from the VM manifest shown below. Otherwise, the <code>chcon</code>  command will fail.</p> <p>The <code>.status.ready</code> field will switch to <code>true</code> indicating that probes are returning successfully:</p> <pre><code>kubectl wait vmis/vmi-guest-probe --for=condition=Ready --timeout=5m\n</code></pre> <p>Additionally, the following command can be used inside the VM to watch the incoming qemu-ga commands:</p> <pre><code>journalctl _COMM=qemu-ga --follow \n</code></pre>"},{"location":"user_workloads/pool/","title":"VirtualMachinePool","text":"<p>A VirtualMachinePool tries to ensure that a specified number of VirtualMachine replicas and their respective VirtualMachineInstances are in the ready state at any time. In other words, a VirtualMachinePool makes sure that a VirtualMachine or a set of VirtualMachines is always up and ready. </p> <p>No state is kept and no guarantees are made about the maximum number of VirtualMachineInstance replicas running at any time. For example, the VirtualMachinePool may decide to create new replicas if possibly still running VMs are entering an unknown state.</p>"},{"location":"user_workloads/pool/#using-virtualmachinepool","title":"Using VirtualMachinePool","text":"<p>The VirtualMachinePool allows us to specify a VirtualMachineTemplate in <code>spec.virtualMachineTemplate</code>. It consists of <code>ObjectMetadata</code> in <code>spec.virtualMachineTemplate.metadata</code>, and a <code>VirtualMachineSpec</code> in <code>spec.virtualMachineTemplate.spec</code>. The specification of the virtual machine is equal to the specification of the virtual machine in the <code>VirtualMachine</code> workload.</p> <p><code>spec.replicas</code> can be used to specify how many replicas are wanted. If unspecified, the default value is 1. This value can be updated anytime. The controller will react to the changes.</p> <p><code>spec.selector</code> is used by the controller to keep track of managed virtual machines. The selector specified there must be able to match the virtual machine labels as specified in <code>spec.virtualMachineTemplate.metadata.labels</code>. If the selector does not match these labels, or they are empty, the controller will simply do nothing except log an error. The user is responsible for avoiding the creation of other virtual machines or VirtualMachinePools which may conflict with the selector and the template labels.</p>"},{"location":"user_workloads/pool/#creating-a-virtualmachinepool","title":"Creating a VirtualMachinePool","text":"<p>VirtualMachinePool is part of the Kubevirt API <code>pool.kubevirt.io/v1alpha1</code>.</p> <p>The example below shows how to create a simple <code>VirtualMachinePool</code>:</p>"},{"location":"user_workloads/pool/#example","title":"Example","text":"<pre><code>apiVersion: pool.kubevirt.io/v1alpha1\nkind: VirtualMachinePool\nmetadata:\n  name: vm-pool-cirros\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      kubevirt.io/vmpool: vm-pool-cirros\n  virtualMachineTemplate:\n    metadata:\n      creationTimestamp: null\n      labels:\n        kubevirt.io/vmpool: vm-pool-cirros\n    spec:\n      running: true\n      template:\n        metadata:\n          creationTimestamp: null\n          labels:\n            kubevirt.io/vmpool: vm-pool-cirros\n        spec:\n          domain:\n            devices:\n              disks:\n              - disk:\n                  bus: virtio\n                name: containerdisk\n            resources:\n              requests:\n                memory: 128Mi\n          terminationGracePeriodSeconds: 0\n          volumes:\n          - containerDisk:\n              image: kubevirt/cirros-container-disk-demo:latest\n            name: containerdisk \n</code></pre> <p>Saving this manifest into <code>vm-pool-cirros.yaml</code> and submitting it to Kubernetes will create three virtual machines based on the template.</p> <pre><code>$ kubectl create -f vm-pool-cirros.yaml\nvirtualmachinepool.pool.kubevirt.io/vm-pool-cirros created\n$ kubectl describe vmpool vm-pool-cirros\nName:         vm-pool-cirros\nNamespace:    default\nLabels:       &lt;none&gt;\nAnnotations:  &lt;none&gt;\nAPI Version:  pool.kubevirt.io/v1alpha1\nKind:         VirtualMachinePool\nMetadata:\n  Creation Timestamp:  2023-02-09T18:30:08Z\n  Generation:          1\n    Manager:      kubectl-create\n    Operation:    Update\n    Time:         2023-02-09T18:30:08Z\n    API Version:  pool.kubevirt.io/v1alpha1\n    Fields Type:  FieldsV1\n    fieldsV1:\n      f:status:\n        .:\n        f:labelSelector:\n        f:readyReplicas:\n        f:replicas:\n    Manager:         virt-controller\n    Operation:       Update\n    Subresource:     status\n    Time:            2023-02-09T18:30:44Z\n  Resource Version:  6606\n  UID:               ba51daf4-f99f-433c-89e5-93f39bc9989d\nSpec:\n  Replicas:  3\n  Selector:\n    Match Labels:\n      kubevirt.io/vmpool:  vm-pool-cirros\n  Virtual Machine Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        kubevirt.io/vmpool:  vm-pool-cirros\n    Spec:\n      Running:  true\n      Template:\n        Metadata:\n          Creation Timestamp:  &lt;nil&gt;\n          Labels:\n            kubevirt.io/vmpool:  vm-pool-cirros\n        Spec:\n          Domain:\n            Devices:\n              Disks:\n                Disk:\n                  Bus:  virtio\n                Name:   containerdisk\n            Resources:\n              Requests:\n                Memory:                      128Mi\n          Termination Grace Period Seconds:  0\n          Volumes:\n            Container Disk:\n              Image:  kubevirt/cirros-container-disk-demo:latest\n            Name:     containerdisk\nStatus:\n  Label Selector:  kubevirt.io/vmpool=vm-pool-cirros\n  Ready Replicas:  2\n  Replicas:        3\nEvents:\n  Type    Reason            Age   From                           Message\n  ----    ------            ----  ----                           -------\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-0\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-2\n  Normal  SuccessfulCreate  17s   virtualmachinepool-controller  Created VM default/vm-pool-cirros-1\n</code></pre> <p><code>Replicas</code> is <code>3</code> and <code>Ready Replicas</code> is <code>2</code>. This means that at the moment when showing the status, three Virtual Machines were already created, but only two are running and ready.</p>"},{"location":"user_workloads/pool/#scaling-via-the-scale-subresource","title":"Scaling via the Scale Subresource","text":"<p>Note: This requires KubeVirt 0.59 or newer.</p> <p>The <code>VirtualMachinePool</code> supports the <code>scale</code> subresource. As a consequence it is possible to scale it via <code>kubectl</code>:</p> <pre><code>$ kubectl scale vmpool vm-pool-cirros --replicas 5\n</code></pre>"},{"location":"user_workloads/pool/#removing-a-virtualmachine-from-virtualmachinepool","title":"Removing a VirtualMachine from VirtualMachinePool","text":"<p>It is also possible to remove a <code>VirtualMachine</code> from its <code>VirtualMachinePool</code>.</p> <p>In this scenario, the <code>ownerReferences</code> needs to be removed from the <code>VirtualMachine</code>. This can be achieved either by using <code>kubectl edit</code> or <code>kubectl patch</code>. Using <code>kubectl patch</code> it would look like:</p> <pre><code>kubectl patch vm vm-pool-cirros-0 --type merge --patch '{\"metadata\":{\"ownerReferences\":null}}'\n</code></pre> <p>Note: You may want to update your VirtualMachine labels as well to avoid impact on selectors.</p>"},{"location":"user_workloads/pool/#using-the-horizontal-pod-autoscaler","title":"Using the Horizontal Pod Autoscaler","text":"<p>Note: This requires KubeVirt 0.59 or newer.</p> <p>The HorizontalPodAutoscaler (HPA) can be used with a <code>VirtualMachinePool</code>. Simply reference it in the spec of the autoscaler:</p> <pre><code>apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n  creationTimestamp: null\n  name: vm-pool-cirros\nspec:\n  maxReplicas: 10\n  minReplicas: 3\n  scaleTargetRef:\n    apiVersion: pool.kubevirt.io/v1alpha1\n    kind: VirtualMachinePool\n    name: vm-pool-cirros\n  targetCPUUtilizationPercentage: 50\n</code></pre> <p>or use <code>kubectl autoscale</code> to define the HPA via the commandline:</p> <pre><code>$ kubectl autoscale vmpool vm-pool-cirros --min=3 --max=10 --cpu-percent=50\n</code></pre>"},{"location":"user_workloads/pool/#exposing-a-virtualmachinepool-as-a-service","title":"Exposing a VirtualMachinePool as a Service","text":"<p>A VirtualMachinePool may be exposed as a service. When this is done, one of the VirtualMachine replicas will be picked for the actual delivery of the service.</p> <p>For example, exposing SSH port (22) as a ClusterIP service:</p> <p><pre><code>apiVersion: v1\nkind: Service\nmetadata:\n  name: vm-pool-cirros-ssh\nspec:\n  type: ClusterIP\n  selector:\n    kubevirt.io/vmpool: vm-pool-cirros\n  ports:\n    - protocol: TCP\n      port: 2222\n      targetPort: 22\n</code></pre> Saving this manifest into <code>vm-pool-cirros-ssh.yaml</code> and submitting it to Kubernetes will create the <code>ClusterIP</code> service listening on port 2222 and forwarding to port 22.</p> <p>See Service Objects for more details.</p>"},{"location":"user_workloads/pool/#using-persistent-storage","title":"Using Persistent Storage","text":"<p>Note: DataVolumes are part of CDI</p> <p>Usage of a <code>DataVolumeTemplates</code> within a <code>spec.virtualMachineTemplate.spec</code> will result in the creation of unique persistent storage for each VM within a VMPool. The <code>DataVolumeTemplate</code> name will have the VM's sequential postfix appended to it when the VM is created from the <code>spec.virtualMachineTemplate.spec.dataVolumeTemplates</code>. This makes each VM a completely unique stateful workload.</p>"},{"location":"user_workloads/pool/#using-unique-cloudinit-and-configmap-volumes-with-virtualmachinepools","title":"Using Unique CloudInit and ConfigMap Volumes with VirtualMachinePools","text":"<p>By default, any secrets or configMaps references in a <code>spec.virtualMachineTemplate.spec.template</code> Volume section will be used directly as is, without any modification to the naming. This means if you specify a secret in a <code>CloudInitNoCloud</code> volume, that every VM instance spawned from the VirtualMachinePool with this volume will get the exact same secret used for their cloud-init user data.</p> <p>This default behavior can be modified by setting the <code>AppendPostfixToSecretReferences</code> and <code>AppendPostfixToConfigMapReferences</code> booleans to true on the VMPool spec. When these booleans are enabled, references to secret and configMap names will have the VM's sequential postfix appended to the secret and configmap name. This allows someone to pre-generate unique per VM <code>secret</code> and <code>configMap</code> data for a VirtualMachinePool ahead of time in a way that will be predictably assigned to VMs within the VirtualMachinePool.</p>"},{"location":"user_workloads/presets/","title":"Presets","text":"<p>FEATURE STATE: </p> <ul> <li><code>VirtualMachineInstancePresets</code> are deprecated as of the <code>v0.57.0</code> release and will be removed in a future release. </li> <li>Users should instead look to use Instancetypes and preferences as a replacement.</li> </ul> <p><code>VirtualMachineInstancePresets</code> are an extension to general <code>VirtualMachineInstance</code> configuration behaving much like <code>PodPresets</code> from Kubernetes. When a <code>VirtualMachineInstance</code> is created, any applicable <code>VirtualMachineInstancePresets</code> will be applied to the existing spec for the <code>VirtualMachineInstance</code>. This allows for re-use of common settings that should apply to multiple <code>VirtualMachineInstances</code>.</p>"},{"location":"user_workloads/presets/#create-a-virtualmachineinstancepreset","title":"Create a VirtualMachineInstancePreset","text":"<p>You can describe a <code>VirtualMachineInstancePreset</code> in a YAML file. For example, the <code>vmi-preset.yaml</code> file below describes a <code>VirtualMachineInstancePreset</code> that requests a <code>VirtualMachineInstance</code> be created with a resource request for 64M of RAM.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: small-qemu\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/size: small\n  domain:\n    resources:\n      requests:\n        memory: 64M\n</code></pre> <ul> <li>Create a <code>VirtualMachineInstancePreset</code> based on that YAML file:</li> </ul> <pre><code>    kubectl create -f vmipreset.yaml\n</code></pre>"},{"location":"user_workloads/presets/#required-fields","title":"Required Fields","text":"<p>As with most Kubernetes resources, a <code>VirtualMachineInstancePreset</code> requires <code>apiVersion</code>, <code>kind</code> and <code>metadata</code> fields.</p> <p>Additionally <code>VirtualMachineInstancePresets</code> also need a <code>spec</code> section. While not technically required to satisfy syntax, it is strongly recommended to include a <code>Selector</code> in the <code>spec</code> section, otherwise a <code>VirtualMachineInstancePreset</code> will match all <code>VirtualMachineInstances</code> in a namespace.</p>"},{"location":"user_workloads/presets/#virtualmachine-selector","title":"VirtualMachine Selector","text":"<p>KubeVirt uses Kubernetes <code>Labels</code> and <code>Selectors</code> to determine which <code>VirtualMachineInstancePresets</code> apply to a given <code>VirtualMachineInstance</code>, similarly to how <code>PodPresets</code> work in Kubernetes. If a setting from a <code>VirtualMachineInstancePreset</code> is applied to a <code>VirtualMachineInstance</code>, the <code>VirtualMachineInstance</code> will be marked with an Annotation upon completion.</p> <p>Any domain structure can be listed in the <code>spec</code> of a <code>VirtualMachineInstancePreset</code>, e.g. Clock, Features, Memory, CPU, or Devices such as network interfaces. All elements of the <code>spec</code> section of a <code>VirtualMachineInstancePreset</code> will be applied to the <code>VirtualMachineInstance</code>.</p> <p>Once a <code>VirtualMachineInstancePreset</code> is successfully applied to a <code>VirtualMachineInstance</code>, the <code>VirtualMachineInstance</code> will be marked with an annotation to indicate that it was applied. If a conflict occurs while a <code>VirtualMachineInstancePreset</code> is being applied, that portion of the <code>VirtualMachineInstancePreset</code> will be skipped.</p> <p>Any valid <code>Label</code> can be matched against, but it is suggested that a general rule of thumb is to use os/shortname, e.g. <code>kubevirt.io/os: rhel7</code>.</p>"},{"location":"user_workloads/presets/#updating-a-virtualmachineinstancepreset","title":"Updating a VirtualMachineInstancePreset","text":"<p>If a <code>VirtualMachineInstancePreset</code> is modified, changes will not be applied to existing <code>VirtualMachineInstances</code>. This applies to both the <code>Selector</code> indicating which <code>VirtualMachineInstances</code> should be matched, and also the <code>Domain</code> section which lists the settings that should be applied to a <code>VirtualMachine</code>.</p>"},{"location":"user_workloads/presets/#overrides","title":"Overrides","text":"<p><code>VirtualMachineInstancePresets</code> use a similar conflict resolution strategy to Kubernetes <code>PodPresets</code>. If a portion of the domain spec is present in both a <code>VirtualMachineInstance</code> and a <code>VirtualMachineInstancePreset</code> and both resources have the identical information, then creation of the <code>VirtualMachineInstance</code> will continue normally. If however there is a difference between the resources, an Event will be created indicating which <code>DomainSpec</code> element of which <code>VirtualMachineInstancePreset</code> was overridden. For example: If both the <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code> define a <code>CPU</code>, but use a different number of <code>Cores</code>, KubeVirt will note the difference.</p> <p>If any settings from the <code>VirtualMachineInstancePreset</code> were successfully applied, the <code>VirtualMachineInstance</code> will be annotated.</p> <p>In the event that there is a difference between the <code>Domains</code> of a <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code>, KubeVirt will create an <code>Event</code>. <code>kubectl get events</code> can be used to show all <code>Events</code>. For example:</p> <pre><code>    $ kubectl get events\n    ....\n    Events:\n      FirstSeen                         LastSeen                        Count From                              SubobjectPath                Reason    Message\n      2m          2m           1         myvmi.1515bbb8d397f258                       VirtualMachineInstance                                     Warning   Conflict                  virtualmachineinstance-preset-controller   Unable to apply VirtualMachineInstancePreset 'example-preset': spec.cpu: &amp;{6} != &amp;{4}\n</code></pre>"},{"location":"user_workloads/presets/#usage","title":"Usage","text":"<p><code>VirtualMachineInstancePresets</code> are namespaced resources, so should be created in the same namespace as the <code>VirtualMachineInstances</code> that will use them:</p> <p><code>kubectl create -f &lt;preset&gt;.yaml [--namespace &lt;namespace&gt;]</code></p> <p>KubeVirt will determine which <code>VirtualMachineInstancePresets</code> apply to a Particular <code>VirtualMachineInstance</code> by matching <code>Labels</code>. For example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: example-preset\n  selector:\n    matchLabels:\n      kubevirt.io/os: win10\n  ...\n</code></pre> <p>would match any <code>VirtualMachineInstance</code> in the same namespace with a <code>Label</code> of <code>flavor: foo</code>. For example:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win10\n  ...\n</code></pre>"},{"location":"user_workloads/presets/#conflicts","title":"Conflicts","text":"<p>When multiple <code>VirtualMachineInstancePresets</code> match a particular <code>VirtualMachineInstance</code>, if they specify the same settings within a Domain, those settings must match. If two <code>VirtualMachineInstancePresets</code> have conflicting settings (e.g. for the number of CPU cores requested), an error will occur, and the <code>VirtualMachineInstance</code> will enter the <code>Failed</code> state, and a <code>Warning</code> event will be emitted explaining which settings of which <code>VirtualMachineInstancePresets</code> were problematic.</p>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances","title":"Matching Multiple <code>VirtualMachineInstances</code>","text":"<p>The main use case for <code>VirtualMachineInstancePresets</code> is to create re-usable settings that can be applied across various machines. Multiple methods are available to match the labels of a <code>VirtualMachineInstance</code> using selectors.</p> <ul> <li> <p>matchLabels: Each <code>VirtualMachineInstance</code> can use a specific label     shared by all</p> <p>instances. * matchExpressions: Logical operators for sets can be used to match multiple</p> <p>labels.</p> </li> </ul> <p>Using matchLabels, the label used in the <code>VirtualMachineInstancePreset</code> must match one of the labels of the <code>VirtualMachineInstance</code>:</p> <pre><code>selector:\n  matchLabels:\n    kubevirt.io/memory: large\n</code></pre> <p>would match</p> <pre><code>metadata:\n  labels:\n    kubevirt.io/memory: large\n    kubevirt.io/os: win10\n</code></pre> <p>or</p> <pre><code>metadata:\n  labels:\n    kubevirt.io/memory: large\n    kubevirt.io/os: fedora27\n</code></pre> <p>Using matchExpressions allows for matching multiple labels of <code>VirtualMachineInstances</code> without needing to explicity list a label.</p> <pre><code>selector:\n  matchExpressions:\n    - {key: kubevirt.io/os, operator: In, values: [fedora27, fedora26]}\n</code></pre> <p>would match both: <pre><code>metadata:\n  labels:\n    kubevirt.io/os: fedora26\n\nmetadata:\n  labels:\n    kubevirt.io/os: fedora27\n</code></pre></p> <p>The Kubernetes documentation has a detailed explanation. Examples are provided below.</p>"},{"location":"user_workloads/presets/#exclusions","title":"Exclusions","text":"<p>Since <code>VirtualMachineInstancePresets</code> use <code>Selectors</code> that indicate which <code>VirtualMachineInstances</code> their settings should apply to, there needs to exist a mechanism by which <code>VirtualMachineInstances</code> can opt out of <code>VirtualMachineInstancePresets</code> altogether. This is done using an annotation:</p> <pre><code>kind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  annotations:\n    virtualmachineinstancepresets.admission.kubevirt.io/exclude: \"true\"\n  ...\n</code></pre>"},{"location":"user_workloads/presets/#examples","title":"Examples","text":""},{"location":"user_workloads/presets/#simple-virtualmachineinstancepreset-example","title":"Simple <code>VirtualMachineInstancePreset</code> Example","text":"<pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nversion: v1\nmetadata:\n  name: example-preset\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/os: win10\n  domain:\n    features:\n      acpi: {}\n      apic: {}\n      hyperv:\n        relaxed: {}\n        vapic: {}\n        spinlocks:\n          spinlocks: 8191\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/os: win10\nspec:\n  domain:\n    firmware:\n      uuid: c8f99fc8-20f5-46c4-85e5-2b841c547cef\n</code></pre> <p>Once the <code>VirtualMachineInstancePreset</code> is applied to the <code>VirtualMachineInstance</code>, the resulting resource would look like this:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/example-preset: kubevirt.io/v1\n  labels:\n    kubevirt.io/os: win10\n    kubevirt.io/nodeName: master\n  name: myvmi\n  namespace: default\nspec:\n  domain:\n    devices: {}\n    features:\n      acpi:\n        enabled: true\n      apic:\n        enabled: true\n      hyperv:\n        relaxed:\n          enabled: true\n        spinlocks:\n          enabled: true\n          spinlocks: 8191\n        vapic:\n          enabled: true\n    firmware:\n      uuid: c8f99fc8-20f5-46c4-85e5-2b841c547cef\n    machine:\n      type: q35\n    resources:\n      requests:\n        memory: 8Mi\n</code></pre>"},{"location":"user_workloads/presets/#conflict-example","title":"Conflict Example","text":"<p>This is an example of a merge conflict. In this case both the <code>VirtualMachineInstance</code> and <code>VirtualMachineInstancePreset</code> request different number of CPU's.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nversion: v1\nmetadata:\n  name: example-preset\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/flavor: default-features\n  domain:\n    cpu:\n      cores: 4\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nversion: v1\nmetadata:\n  name: myvmi\n  labels:\n    kubevirt.io/flavor: default-features\nspec:\n  domain:\n    cpu:\n      cores: 6\n</code></pre> <p>In this case the <code>VirtualMachineInstance</code> Spec will remain unmodified. Use <code>kubectl get events</code> to show events.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n  generation: 0\n  labels:\n    kubevirt.io/flavor: default-features\n  name: myvmi\n  namespace: default\nspec:\n  domain:\n    cpu:\n      cores: 6\n    devices: {}\n    machine:\n      type: \"\"\n    resources: {}\nstatus: {}\n</code></pre> <p>Calling <code>kubectl get events</code> would have a line like:</p> <pre><code>2m 2m 1 myvmi.1515bbb8d397f258 VirtualMachineInstance Warning Conflict virtualmachineinstance-preset-controller Unable to apply VirtualMachineInstancePreset example-preset: spec.cpu: &amp;{6} != &amp;{4}\n</code></pre>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances-using-matchlabels","title":"Matching Multiple VirtualMachineInstances Using MatchLabels","text":"<p>These <code>VirtualMachineInstances</code> have multiple labels, one that is unique and one that is shared.</p> <p>Note: This example breaks from the convention of using os-shortname as a <code>Label</code> for demonstration purposes.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: twelve-cores\nspec:\n  selector:\n    matchLabels:\n      kubevirt.io/cpu: dodecacore\n  domain:\n    cpu:\n      cores: 12\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: windows-10\n  labels:\n    kubevirt.io/os: win10\n    kubevirt.io/cpu: dodecacore\nspec:\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: windows-7\n  labels:\n    kubevirt.io/os: win7\n    kubevirt.io/cpu: dodecacore\nspec:\n  terminationGracePeriodSeconds: 0\n</code></pre> <p>Adding this <code>VirtualMachineInstancePreset</code> and these <code>VirtualMachineInstances</code> will result in:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/twelve-cores: kubevirt.io/v1\n  labels:\n    kubevirt.io/cpu: dodecacore\n    kubevirt.io/os: win10\n  name: windows-10\nspec:\n  domain:\n    cpu:\n      cores: 12\n    devices: {}\n    resources:\n      requests:\n        memory: 4Gi\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  annotations:\n    presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n    virtualmachineinstancepreset.kubevirt.io/twelve-cores: kubevirt.io/v1\n  labels:\n    kubevirt.io/cpu: dodecacore\n    kubevirt.io/os: win7\n  name: windows-7\nspec:\n  domain:\n    cpu:\n      cores: 12\n    devices: {}\n    resources:\n      requests:\n        memory: 4Gi\n  terminationGracePeriodSeconds: 0\n</code></pre>"},{"location":"user_workloads/presets/#matching-multiple-virtualmachineinstances-using-matchexpressions","title":"Matching Multiple VirtualMachineInstances Using MatchExpressions","text":"<p>This <code>VirtualMachineInstancePreset</code> has a matchExpression that will match two labels: <code>kubevirt.io/os: win10</code> and <code>kubevirt.io/os: win7</code>.</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstancePreset\nmetadata:\n  name: windows-vmis\nspec:\n  selector:\n    matchExpressions:\n      - {key: kubevirt.io/os, operator: In, values: [win10, win7]}\n  domain:\n    resources:\n      requests:\n        memory: 128M\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: smallvmi\n  labels:\n    kubevirt.io/os: win10\nspec:\n  terminationGracePeriodSeconds: 60\n---\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: largevmi\n  labels:\n    kubevirt.io/os: win7\nspec:\n  terminationGracePeriodSeconds: 120\n</code></pre> <p>Applying the preset to both VM's will result in:</p> <pre><code>apiVersion: v1\nitems:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachineInstance\n  metadata:\n    annotations:\n      presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n      virtualmachineinstancepreset.kubevirt.io/windows-vmis: kubevirt.io/v1\n    labels:\n      kubevirt.io/os: win7\n    name: largevmi\n  spec:\n    domain:\n      resources:\n        requests:\n          memory: 128M\n    terminationGracePeriodSeconds: 120\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachineInstance\n  metadata:\n    annotations:\n      presets.virtualmachineinstances.kubevirt.io/presets-applied: kubevirt.io/v1\n      virtualmachineinstancepreset.kubevirt.io/windows-vmis: kubevirt.io/v1\n    labels:\n      kubevirt.io/os: win10\n    name: smallvmi\n  spec:\n    domain:\n      resources:\n        requests:\n          memory: 128M\n    terminationGracePeriodSeconds: 60\n</code></pre>"},{"location":"user_workloads/replicaset/","title":"VirtualMachineInstanceReplicaSet","text":"<p>A VirtualMachineInstanceReplicaSet tries to ensures that a specified number of VirtualMachineInstance replicas are running at any time. In other words, a VirtualMachineInstanceReplicaSet makes sure that a VirtualMachineInstance or a homogeneous set of VirtualMachineInstances is always up and ready. It is very similar to a Kubernetes ReplicaSet.</p> <p>No state is kept and no guarantees about the maximum number of VirtualMachineInstance replicas which are up are given. For example, the VirtualMachineInstanceReplicaSet may decide to create new replicas if possibly still running VMs are entering an unknown state.</p>"},{"location":"user_workloads/replicaset/#using-virtualmachineinstancereplicaset","title":"Using VirtualMachineInstanceReplicaSet","text":"<p>The VirtualMachineInstanceReplicaSet allows us to specify a VirtualMachineInstanceTemplate in <code>spec.template</code>. It consists of <code>ObjectMetadata</code> in <code>spec.template.metadata</code>, and a <code>VirtualMachineInstanceSpec</code> in <code>spec.template.spec</code>. The specification of the virtual machine is equal to the specification of the virtual machine in the <code>VirtualMachineInstance</code> workload.</p> <p><code>spec.replicas</code> can be used to specify how many replicas are wanted. If unspecified, the default value is 1. This value can be updated anytime. The controller will react to the changes.</p> <p><code>spec.selector</code> is used by the controller to keep track of managed virtual machines. The selector specified there must be able to match the virtual machine labels as specified in <code>spec.template.metadata.labels</code>. If the selector does not match these labels, or they are empty, the controller will simply do nothing except from logging an error. The user is responsible for not creating other virtual machines or VirtualMachineInstanceReplicaSets which conflict with the selector and the template labels.</p>"},{"location":"user_workloads/replicaset/#exposing-a-virtualmachineinstancereplicaset-as-a-service","title":"Exposing a VirtualMachineInstanceReplicaSet as a Service","text":"<p>A VirtualMachineInstanceReplicaSet could be exposed as a service. When this is done, one of the VirtualMachineInstances replicas will be picked for the actual delivery of the service.</p> <p>For example, exposing SSH port (22) as a ClusterIP service using virtctl on a VirtualMachineInstanceReplicaSet:</p> <pre><code>$ virtctl expose vmirs vmi-ephemeral --name vmiservice --port 27017 --target-port 22\n</code></pre> <p>All service exposure options that apply to a VirtualMachineInstance apply to a VirtualMachineInstanceReplicaSet. See Exposing VirtualMachineInstance for more details.</p>"},{"location":"user_workloads/replicaset/#when-to-use-a-virtualmachineinstancereplicaset","title":"When to use a VirtualMachineInstanceReplicaSet","text":"<p>Note: The base assumption is that referenced disks are read-only or that the VMIs are writing internally to a tmpfs. The most obvious volume sources for VirtualMachineInstanceReplicaSets which KubeVirt supports are referenced below. If other types are used data corruption is possible.</p> <p>Using VirtualMachineInstanceReplicaSet is the right choice when one wants many identical VMs and does not care about maintaining any disk state after the VMs are terminated.</p> <p>Volume types which work well in combination with a VirtualMachineInstanceReplicaSet are:</p> <ul> <li>cloudInitNoCloud</li> <li>ephemeral</li> <li>containerDisk</li> <li>emptyDisk</li> <li>configMap</li> <li>secret</li> <li>any other type, if the VMI writes internally to a tmpfs</li> </ul>"},{"location":"user_workloads/replicaset/#fast-starting-ephemeral-virtual-machines","title":"Fast starting ephemeral Virtual Machines","text":"<p>This use-case involves small and fast booting VMs with little provisioning performed during initialization.</p> <p>In this scenario, migrations are not important. Redistributing VM workloads between Nodes can be achieved simply by deleting managed VirtualMachineInstances which are running on an overloaded Node. The <code>eviction</code> of such a VirtualMachineInstance can happen by directly deleting the VirtualMachineInstance instance (KubeVirt aware workload redistribution) or by deleting the corresponding Pod where the Virtual Machine runs in (Only Kubernetes aware workload redistribution).</p>"},{"location":"user_workloads/replicaset/#slow-starting-ephemeral-virtual-machines","title":"Slow starting ephemeral Virtual Machines","text":"<p>In this use-case one has big and slow booting VMs, and complex or resource intensive provisioning is done during boot. More specifically, the timespan between the creation of a new VM and it entering the ready state is long.</p> <p>In this scenario, one still does not care about the state, but since re-provisioning VMs is expensive, migrations are important. Workload redistribution between Nodes can be achieved by migrating VirtualMachineInstances to different Nodes. A workload redistributor needs to be aware of KubeVirt and create migrations, instead of <code>evicting</code> VirtualMachineInstances by deletion.</p> <p>Note: The simplest form of having a migratable ephemeral VirtualMachineInstance, will be to use local storage based on <code>ContainerDisks</code> in combination with a file based backing store. However, migratable backing store support has not officially landed yet in KubeVirt and is untested.</p>"},{"location":"user_workloads/replicaset/#example","title":"Example","text":"<p><pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstanceReplicaSet\nmetadata:\n  name: testreplicaset\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      myvmi: myvmi\n  template:\n    metadata:\n      name: test\n      labels:\n        myvmi: myvmi\n    spec:\n      domain:\n        devices:\n          disks:\n          - disk:\n            name: containerdisk\n        resources:\n          requests:\n            memory: 64M\n      volumes:\n      - name: containerdisk\n        containerDisk:\n          image: kubevirt/cirros-container-disk-demo:latest\n</code></pre> Saving this manifest into <code>testreplicaset.yaml</code> and submitting it to Kubernetes will create three virtual machines based on the template.</p> <pre><code>$ kubectl create -f testreplicaset.yaml\nvirtualmachineinstancereplicaset \"testreplicaset\" created\n$ kubectl describe vmirs testreplicaset\nName:         testreplicaset\nNamespace:    default\nLabels:       &lt;none&gt;\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachineInstanceReplicaSet\nMetadata:\n  Cluster Name:\n  Creation Timestamp:  2018-01-03T12:42:30Z\n  Generation:          0\n  Resource Version:    6380\n  Self Link:           /apis/kubevirt.io/v1/namespaces/default/virtualmachineinstancereplicasets/testreplicaset\n  UID:                 903a9ea0-f083-11e7-9094-525400ee45b0\nSpec:\n  Replicas:  3\n  Selector:\n    Match Labels:\n      Myvmi:  myvmi\n  Template:\n    Metadata:\n      Creation Timestamp:  &lt;nil&gt;\n      Labels:\n        Myvmi:  myvmi\n      Name:    test\n    Spec:\n      Domain:\n        Devices:\n          Disks:\n            Disk:\n            Name:         containerdisk\n            Volume Name:  containerdisk\n        Resources:\n          Requests:\n            Memory:  64M\n      Volumes:\n        Name:  containerdisk\n        Container Disk:\n          Image:  kubevirt/cirros-container-disk-demo:latest\nStatus:\n  Conditions:      &lt;nil&gt;\n  Ready Replicas:  2\n  Replicas:        3\nEvents:\n  Type    Reason            Age   From                                 Message\n  ----    ------            ----  ----                                 -------\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: testh8998\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: testf474w\n  Normal  SuccessfulCreate  13s   virtualmachineinstancereplicaset-controller  Created virtual machine: test5lvkd\n</code></pre> <p><code>Replicas</code> is <code>3</code> and <code>Ready Replicas</code> is <code>2</code>. This means that at the moment when showing the status, three Virtual Machines were already created, but only two are running and ready.</p>"},{"location":"user_workloads/replicaset/#scaling-via-the-scale-subresource","title":"Scaling via the Scale Subresource","text":"<p>Note: This requires the <code>CustomResourceSubresources</code> feature gate to be enabled for clusters prior to 1.11.</p> <p>The <code>VirtualMachineInstanceReplicaSet</code> supports the <code>scale</code> subresource. As a consequence it is possible to scale it via <code>kubectl</code>:</p> <pre><code>$ kubectl scale vmirs myvmirs --replicas 5\n</code></pre>"},{"location":"user_workloads/replicaset/#using-the-horizontal-pod-autoscaler","title":"Using the Horizontal Pod Autoscaler","text":"<p>Note: This requires at cluster newer or equal to 1.11.</p> <p>The HorizontalPodAutoscaler (HPA) can be used with a <code>VirtualMachineInstanceReplicaSet</code>. Simply reference it in the spec of the autoscaler:</p> <pre><code>apiVersion: autoscaling/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: myhpa\nspec:\n  scaleTargetRef:\n    kind: VirtualMachineInstanceReplicaSet\n    name: vmi-replicaset-cirros\n    apiVersion: kubevirt.io/v1\n  minReplicas: 3\n  maxReplicas: 10\n  targetCPUUtilizationPercentage: 50\n</code></pre> <p>or use <code>kubectl autoscale</code> to define the HPA via the commandline:</p> <pre><code>$ kubectl autoscale vmirs vmi-replicaset-cirros --min=3 --max=10\n</code></pre>"},{"location":"user_workloads/startup_scripts/","title":"Startup Scripts","text":"<p>KubeVirt supports the ability to assign a startup script to a VirtualMachineInstance instance which is executed automatically when the VM initializes.</p> <p>These scripts are commonly used to automate injection of users and SSH keys into VMs in order to provide remote access to the machine. For example, a startup script can be used to inject credentials into a VM that allows an Ansible job running on a remote host to access and provision the VM.</p> <p>Startup scripts are not limited to any specific use case though. They can be used to run any arbitrary script in a VM on boot.</p>"},{"location":"user_workloads/startup_scripts/#cloud-init","title":"Cloud-init","text":"<p>cloud-init is a widely adopted project used for early initialization of a VM. Used by cloud providers such as AWS and GCP, cloud-init has established itself as the defacto method of providing startup scripts to VMs.</p> <p>Cloud-init documentation can be found here: Cloud-init Documentation.</p> <p>KubeVirt supports cloud-init's NoCloud  and ConfigDrive datasources which involve injecting startup scripts into a VM instance through the use of an ephemeral disk. VMs with the cloud-init package installed will detect the ephemeral disk and execute custom userdata scripts at boot.</p>"},{"location":"user_workloads/startup_scripts/#ignition","title":"Ignition","text":"<p>Ignition is an alternative to cloud-init which allows for configuring the VM disk on first boot. You can find the Ignition documentation here.  You can also find a comparison between cloud-init and Ignition here.</p> <p>Ignition can be used with Kubevirt by using the <code>cloudInitConfigDrive</code> volume.</p>"},{"location":"user_workloads/startup_scripts/#sysprep","title":"Sysprep","text":"<p>Sysprep is an automation tool for Windows that automates Windows installation, setup, and custom software provisioning.</p> <p>The general flow is:</p> <ol> <li> <p>Seal the vm image with the Sysprep tool, for example by running:</p> <pre><code>%WINDIR%\\system32\\sysprep\\sysprep.exe /generalize /shutdown /oobe /mode:vm\n</code></pre> <p>Note</p> <p>We need to make sure the base vm does not restart, which can be done by setting the vm run strategy as <code>RerunOnFailure</code>.</p> <p>VM runStrategy:</p> <pre><code>spec:\n  runStrategy: RerunOnFailure\n</code></pre> <p>More information can be found here:</p> <ul> <li>Sysprep Process Overview</li> <li>Sysprep (Generalize) a Windows installation</li> </ul> <p>Note</p> <p>It is important that there is no answer file detected when the Sysprep Tool is triggered, because Windows Setup searches for answer files at the beginning of each configuration pass and caches it. If that happens, when the OS will start - it will just use the cached answer file, ignoring the one we provide through the Sysprep API. More information can be found here.</p> </li> <li> <p>Providing an Answer file named <code>autounattend.xml</code> in an attached media. The answer file can be provided in a ConfigMap or a Secret with the key <code>autounattend.xml</code></p> <p>The configuration file can be generated with Windows SIM or it can be specified manually according to the information found here:</p> <ul> <li>Answer files (unattend.xml)</li> <li>Answer File Reference</li> <li>Answer File Components Reference</li> </ul> <p>Note</p> <p>There are also many easy to find online tools available for creating an answer file.</p> </li> </ol>"},{"location":"user_workloads/startup_scripts/#cloud-init-examples","title":"Cloud-init Examples","text":""},{"location":"user_workloads/startup_scripts/#user-data","title":"User Data","text":"<p>KubeVirt supports the cloud-init NoCloud and ConfigDrive data sources which involve injecting startup scripts through the use of a disk attached to the VM.</p> <p>In order to assign a custom userdata script to a VirtualMachineInstance using this method, users must define a disk and a volume for the NoCloud or ConfigDrive datasource in the VirtualMachineInstance's spec.</p>"},{"location":"user_workloads/startup_scripts/#data-sources","title":"Data Sources","text":"<p>Under most circumstances users should stick to the NoCloud data source as it is the simplest cloud-init data source. Only if NoCloud is not supported by the cloud-init implementation (e.g. coreos-cloudinit) users should switch the data source to ConfigDrive.</p> <p>Switching the cloud-init data source to ConfigDrive is as easy as changing the volume type in the VirtualMachineInstance's spec from <code>cloudInitNoCloud</code> to <code>cloudInitConfigDrive</code>.</p> <p>NoCloud data source:</p> <pre><code>volumes:\n  - name: cloudinitvolume\n    cloudInitNoCloud:\n      userData: \"#cloud-config\"\n</code></pre> <p>ConfigDrive data source:</p> <pre><code>volumes:\n  - name: cloudinitvolume\n    cloudInitConfigDrive:\n      userData: \"#cloud-config\"\n</code></pre> <p>When using the ConfigDrive datasource, the <code>networkData</code> part has to be in the OpenStack Metadata Service Network format: <pre><code>  spec:\n    domain:\n      interfaces: \n        - name: secondary-net\n          bridge: {}\n          macAddress: '02:26:19:00:00:30'\n          model: virtio\n    networks: \n        - multus:\n            networkName: my-ns/my-net\n          name: secondary-net\n    volumes:\n      - name: cloudinitvolume\n        cloudInitConfigDrive:\n          networkData: |\n            {\"links\":[{\"id\":\"enp2s0\",\"type\":\"phy\",\"ethernet_mac_address\":\"02:26:19:00:00:30\"}],\"networks\":[{\"id\":\"NAD1\",\"type\":\"ipv4\",\"link\":\"enp2s0\",\"ip_address\":\"10.184.0.244\",\"netmask\":\"255.255.240.0\",\"routes\":[{\"network\":\"0.0.0.0\",\"netmask\":\"0.0.0.0\",\"gateway\":\"23.253.157.1\"}],\"network_id\":\"\"}],\"services\":[]}\n          userData: \"#cloud-config\"\n</code></pre></p> <p>Note The MAC address of the secondary interface should be predefined and identical in  the network interface and the cloud-init networkData. </p> <p>See the examples below for more complete cloud-init examples.</p>"},{"location":"user_workloads/startup_scripts/#cloud-init-user-data-as-clear-text","title":"Cloud-init user-data as clear text","text":"<p>In the example below, a SSH key is stored in the cloudInitNoCloud Volume's userData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <pre><code># Create a VM manifest with the startup script\n# a cloudInitNoCloud volume's userData field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userData: |\n          #cloud-config\n          ssh_authorized_keys:\n            - ssh-rsa AAAAB3NzaK8L93bWxnyp test@test.com\n\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-user-data-as-base64-string","title":"Cloud-init user-data as base64 string","text":"<p>In the example below, a simple bash script is base64 encoded and stored in the cloudInitNoCloud Volume's userDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <p>Users also have the option of storing the startup script in a Kubernetes Secret and referencing the Secret in the VM's spec. Examples further down in the document illustrate how that is done.</p> <pre><code># Create a simple startup script\n\ncat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\necho \"Hi from startup script!\"\nEND\n\n# Create a VM manifest with the startup script base64 encoded into\n# a cloudInitNoCloud volume's userDataBase64 field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script.sh | base64 -w0)\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-userdata-as-k8s-secret","title":"Cloud-init UserData as k8s Secret","text":"<p>Users who wish to not store the cloud-init userdata directly in the VirtualMachineInstance spec have the option to store the userdata into a Kubernetes Secret and reference that Secret in the spec.</p> <p>Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init userdata.</p> <p>Below is an example of how to create a Kubernetes Secret containing a startup script and reference that Secret in the VM's spec.</p> <pre><code># Create a simple startup script\n\ncat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\necho \"Hi from startup script!\"\nEND\n\n# Store the startup script in a Kubernetes Secret\nkubectl create secret generic my-vmi-secret --from-file=userdata=startup-script.sh\n\n# Create a VM manifest and reference the Secret's name in the cloudInitNoCloud\n# Volume's secretRef field\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        disk:\n          bus: virtio\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        secretRef:\n          name: my-vmi-secret\nEND\n\n# Post the VM\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#injecting-ssh-keys-with-cloud-inits-cloud-config","title":"Injecting SSH keys with Cloud-init's Cloud-config","text":"<p>In the examples so far, the cloud-init userdata script has been a bash script. Cloud-init has it's own configuration that can handle some common tasks such as user creation and SSH key injection.</p> <p>More cloud-config examples can be found here: Cloud-init Examples</p> <p>Below is an example of using cloud-config to inject an SSH key for the default user (fedora in this case) of a Fedora Atomic disk image.</p> <pre><code># Create the cloud-init cloud-config userdata.\ncat &lt;&lt; END &gt; startup-script\n#cloud-config\npassword: atomic\nchpasswd: { expire: False }\nssh_pwauth: False\nssh_authorized_keys:\n    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain\nEND\n\n# Create the VM spec\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: sshvmi\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          dev: vda\n      - name: cloudinitdisk\n        disk:\n          dev: vdb\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/fedora-atomic-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script | base64 -w0)\nEND\n\n# Post the VirtualMachineInstance spec to KubeVirt.\nkubectl create -f my-vmi.yaml\n\n# Connect to VM with passwordless SSH key\nssh -i &lt;insert private key here&gt; fedora@&lt;insert ip here&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#inject-ssh-key-using-a-custom-shell-script","title":"Inject SSH key using a Custom Shell Script","text":"<p>Depending on the boot image in use, users may have a mixed experience using cloud-init's cloud-config to create users and inject SSH keys.</p> <p>Below is an example of creating a user and injecting SSH keys for that user using a script instead of cloud-config.</p> <pre><code>cat &lt;&lt; END &gt; startup-script.sh\n#!/bin/bash\nexport NEW_USER=\"foo\"\nexport SSH_PUB_KEY=\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J $NEW_USER@localhost.localdomain\"\n\nsudo adduser -U -m $NEW_USER\necho \"$NEW_USER:atomic\" | chpasswd\nsudo mkdir /home/$NEW_USER/.ssh\nsudo echo \"$SSH_PUB_KEY\" &gt; /home/$NEW_USER/.ssh/authorized_keys\nsudo chown -R ${NEW_USER}: /home/$NEW_USER/.ssh\nEND\n\n# Create the VM spec\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: sshvmi\nspec:\n  terminationGracePeriodSeconds: 0\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          dev: vda\n      - name: cloudinitdisk\n        disk:\n          dev: vdb\n  volumes:\n    - name: containerdisk\n      containerDisk:\n        image: kubevirt/fedora-atomic-registry-disk-demo:latest\n    - name: cloudinitdisk\n      cloudInitNoCloud:\n        userDataBase64: $(cat startup-script.sh | base64 -w0)\nEND\n\n# Post the VirtualMachineInstance spec to KubeVirt.\nkubectl create -f my-vmi.yaml\n\n# Connect to VM with passwordless SSH key\nssh -i &lt;insert private key here&gt; foo@&lt;insert ip here&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#network-config","title":"Network Config","text":"<p>A cloud-init network version 1 configuration can be set to configure the network at boot.</p> <p>Cloud-init user-data must be set for cloud-init to parse network-config even if it is just the user-data config header:</p> <pre><code>#cloud-config\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-clear-text","title":"Cloud-init network-config as clear text","text":"<p>In the example below, a simple cloud-init network-config is stored in the cloudInitNoCloud Volume's networkData field as clean text. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <pre><code># Create a VM manifest with the network-config in\n# a cloudInitNoCloud volume's networkData field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkData: |\n          network:\n            version: 1\n            config:\n            - type: physical\n            name: eth0\n            subnets:\n              - type: dhcp\n\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-base64-string","title":"Cloud-init network-config as base64 string","text":"<p>In the example below, a simple network-config is base64 encoded and stored in the cloudInitNoCloud Volume's networkDataBase64 field. There is a corresponding disks entry that references the cloud-init volume and assigns it to the VM's device.</p> <p>Users also have the option of storing the network-config in a Kubernetes Secret and referencing the Secret in the VM's spec. Examples further down in the document illustrate how that is done.</p> <pre><code># Create a simple network-config\n\ncat &lt;&lt; END &gt; network-config\nnetwork:\n  version: 1\n  config:\n  - type: physical\n  name: eth0\n  subnets:\n    - type: dhcp\nEND\n\n# Create a VM manifest with the networkData base64 encoded into\n# a cloudInitNoCloud volume's networkDataBase64 field.\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-container-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkDataBase64: $(cat network-config | base64 -w0)\nEND\n\n# Post the Virtual Machine spec to KubeVirt.\n\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#cloud-init-network-config-as-k8s-secret","title":"Cloud-init network-config as k8s Secret","text":"<p>Users who wish to not store the cloud-init network-config directly in the VirtualMachineInstance spec have the option to store the network-config into a Kubernetes Secret and reference that Secret in the spec.</p> <p>Multiple VirtualMachineInstance specs can reference the same Kubernetes Secret containing cloud-init network-config.</p> <p>Below is an example of how to create a Kubernetes Secret containing a network-config and reference that Secret in the VM's spec.</p> <pre><code># Create a simple network-config\n\ncat &lt;&lt; END &gt; network-config\nnetwork:\n  version: 1\n  config:\n  - type: physical\n  name: eth0\n  subnets:\n    - type: dhcp\nEND\n\n# Store the network-config in a Kubernetes Secret\nkubectl create secret generic my-vmi-secret --from-file=networkdata=network-config\n\n# Create a VM manifest and reference the Secret's name in the cloudInitNoCloud\n# Volume's secretRef field\n\ncat &lt;&lt; END &gt; my-vmi.yaml\napiVersion: kubevirt.io/v1alpha2\nkind: VirtualMachineInstance\nmetadata:\n  name: myvmi\nspec:\n  terminationGracePeriodSeconds: 5\n  domain:\n    resources:\n      requests:\n        memory: 64M\n    devices:\n      disks:\n      - name: containerdisk\n        volumeName: registryvolume\n        disk:\n          bus: virtio\n      - name: cloudinitdisk\n        volumeName: cloudinitvolume\n        disk:\n          bus: virtio\n  volumes:\n    - name: registryvolume\n      containerDisk:\n        image: kubevirt/cirros-registry-disk-demo:latest\n    - name: cloudinitvolume\n      cloudInitNoCloud:\n        userData: \"#cloud-config\"\n        networkDataSecretRef:\n          name: my-vmi-secret\nEND\n\n# Post the VM\nkubectl create -f my-vmi.yaml\n</code></pre>"},{"location":"user_workloads/startup_scripts/#debugging","title":"Debugging","text":"<p>Depending on the operating system distribution in use, cloud-init output is often printed to the console output on boot up. When developing userdata scripts, users can connect to the VM's console during boot up to debug.</p> <p>Example of connecting to console using virtctl:</p> <pre><code>virtctl console &lt;name of vmi&gt;\n</code></pre>"},{"location":"user_workloads/startup_scripts/#device-role-tagging","title":"Device Role Tagging","text":"<p>KubeVirt provides a mechanism for users to tag devices such as Network Interfaces with a specific role. The tag will be matched to the hardware address of the device and this mapping exposed to the guest OS via cloud-init.</p> <p>This additional metadata will help the guest OS users with multiple networks interfaces to identify the devices that may have a specific role, such as a network device dedicated to a specific service or a disk intended to be used by a specific application (database, webcache, etc.)</p> <p>This functionality already exists in platforms such as OpenStack. KubeVirt will provide the data in a similar format, known to users and services like cloud-init.</p> <p>For example: <pre><code>kind: VirtualMachineInstance\nspec:\n  domain:\n    devices:\n      interfaces:\n      - masquerade: {}\n        name: default\n      - bridge: {}\n        name: ptp\n        tag: ptp\n      - name: sriov-net\n        sriov: {}\n        tag: nfvfunc\n  networks:\n  - name: default\n    pod: {}\n  - multus:\n      networkName: ptp-conf\n    name: ptp\n  - multus:\n      networkName: sriov/sriov-network\n    name: sriov-net\n\nThe metadata will be available in the guests config drive `openstack/latest/meta_data.json`\n\n{\n  \"devices\": [\n    {\n        \"type\": \"nic\",\n        \"bus\": \"pci\",\n        \"address\": \"0000:00:02.0\",\n        \"mac\": \"01:22:22:42:22:21\",\n        \"tags\": [\"ptp\"]\n    },\n    {\n        \"type\": \"nic\",\n        \"bus\": \"pci\",\n        \"address\": \"0000:81:10.1\",\n        \"mac\": \"01:22:22:42:22:22\",\n        \"tags\": [\"nfvfunc\"]\n    },\n  ]\n}\n</code></pre></p>"},{"location":"user_workloads/startup_scripts/#ignition-examples","title":"Ignition Examples","text":"<p>Ignition data can be passed into a <code>cloudInitConfigDrive</code> source using  either clear text, a base64 string or a k8s Secret.</p> <p>Some examples of Ignition configurations can be found  in the examples  given by the Ignition documentation.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-clear-text","title":"Ignition as clear text","text":"<p>Here is a complete example of a Kubevirt VM using Ignition to add an ssh key to the  <code>coreos</code> user at first boot : </p> <pre><code>apiVersion: kubevirt.io/v1alpha3\nkind: VirtualMachine\nmetadata:\n  name: ign-demo\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/size: small\n        kubevirt.io/domain: ign-demo\n    spec:\n      domain:\n        devices:\n          disks:\n            - name: containerdisk\n              disk:\n                bus: virtio\n            - name: cloudinitdisk\n              disk:\n                bus: virtio\n          interfaces:\n            - name: default\n              masquerade: {}\n        resources:\n          requests:\n            memory: 2G\n      networks:\n        - name: default\n          pod: {}\n      volumes:\n        - name: containerdisk\n          containerDisk:\n            image: quay.io/containerdisks/rhcos:4.9\n        - name: cloudinitdisk\n          cloudInitConfigDrive:\n            userData: |\n              {\n                \"ignition\": {\n                  \"config\": {},\n                  \"proxy\": {},\n                  \"security\": {},\n                  \"timeouts\": {},\n                  \"version\": \"3.2.0\"\n                },\n                \"passwd\": {\n                  \"users\": [\n                    {\n                      \"name\": \"coreos\",\n                      \"sshAuthorizedKeys\": [\n                        \"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPL3axFGHI3db9iJWkPXVbYzD7OaWTtHuqmxLvj+DztB user@example\"\n                      ]\n                    }\n                  ]\n                },\n                \"storage\": {},\n                \"systemd\": {}\n              }\n</code></pre> <p>See that the Ignition config is simply passed to the <code>userData</code> annotation of the  <code>cloudInitConfigDrive</code> volume.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-base64","title":"Ignition as base64","text":"<p>You can also pass the Ignition config as a base64 string by using the <code>userDatabase64</code> annotation : </p> <pre><code>...\ncloudInitConfigDrive:\n  userDataBase64: eyJpZ25pdGlvbiI6eyJjb25maWciOnt9LCJwcm94eSI6e30sInNlY3VyaXR5Ijp7fSwidGltZW91dHMiOnt9LCJ2ZXJzaW9uIjoiMy4yLjAifSwicGFzc3dkIjp7InVzZXJzIjpbeyJuYW1lIjoiY29yZW9zIiwic3NoQXV0aG9yaXplZEtleXMiOlsic3NoLWVkMjU1MTlBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJUEwzYXhGR0hJM2RiOWlKV2tQWFZiWXpEN09hV1R0SHVxbXhMdmorRHp0QiB1c2VyQGV4YW1wbGUiXX1dfSwic3RvcmFnZSI6e30sInN5c3RlbWQiOnt9fQ==\n</code></pre> <p>You can obtain the base64 string by doing <code>cat ignition.json | base64 -w0</code> in your terminal.</p>"},{"location":"user_workloads/startup_scripts/#ignition-as-k8s-secret","title":"Ignition as k8s Secret","text":"<p>If you do not want to store the Ignition config into the VM configuration, you can use a k8s Secret.</p> <p>First, create the secret with the ignition data in it :</p> <pre><code>kubectl create secret generic my-ign-secret --from-file=ignition=ignition.json\n</code></pre> <p>Then specify this secret into your VM configuration :</p> <pre><code>...\ncloudInitConfigDrive:\n  secretRef:\n    name: my-ign-secret\n</code></pre>"},{"location":"user_workloads/startup_scripts/#sysprep-examples","title":"Sysprep Examples","text":""},{"location":"user_workloads/startup_scripts/#sysprep-in-a-configmap","title":"Sysprep in a ConfigMap","text":"<p>The answer file can be provided in a ConfigMap:</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: sysprep-config\ndata:\n  autounattend.xml: |\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n    ...\n    &lt;/unattend&gt;\n</code></pre> <p>And attached to the VM like so:</p> <pre><code>kind: VirtualMachine\nmetadata:\n  name: windows-with-sysprep\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: windows-with-sysprep\n    spec:\n      domain:\n        cpu:\n          cores: 3\n        devices:\n          disks:\n          - bootOrder: 1\n            disk:\n              bus: virtio\n            name: harddrive\n          - name: sysprep\n            cdrom:\n              bus: sata\n        machine:\n          type: q35\n        resources:\n          requests:\n            memory: 6G\n      volumes:\n      - name: harddrive\n        persistentVolumeClaim:\n          claimName: windows_pvc\n      - name: sysprep\n        sysprep:\n          configMap:\n            name: sysprep-config\n</code></pre>"},{"location":"user_workloads/startup_scripts/#sysprep-in-a-secret","title":"Sysprep in a Secret","text":"<p>The answer file can be provided in a Secret:</p> <pre><code>apiVersion: v1\nkind: Secret\nmetadata:\n  name: sysprep-config\nstringData:\ndata:\n  autounattend.xml: |\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n    ...\n    &lt;/unattend&gt;\n</code></pre> <p>And attached to the VM like so:</p> <pre><code>kind: VirtualMachine\nmetadata:\n  name: windows-with-sysprep\nspec:\n  running: false\n  template:\n    metadata:\n      labels:\n        kubevirt.io/domain: windows-with-sysprep\n    spec:\n      domain:\n        cpu:\n          cores: 3\n        devices:\n          disks:\n          - bootOrder: 1\n            disk:\n              bus: virtio\n            name: harddrive\n          - name: sysprep\n            cdrom:\n              bus: sata\n        machine:\n          type: q35\n        resources:\n          requests:\n            memory: 6G\n      volumes:\n      - name: harddrive\n        persistentVolumeClaim:\n          claimName: windows_pvc\n      - name: sysprep\n        sysprep:\n          secret:\n            name: sysprep-secret\n</code></pre>"},{"location":"user_workloads/startup_scripts/#base-sysprep-vm","title":"Base Sysprep VM","text":"<p>In the example below, a configMap with <code>autounattend.xml</code> file is used to modify the Windows iso image which is downloaded from Microsoft and creates a base installed Windows machine with virtio drivers installed and all the commands executed in <code>post-install.ps1</code> For the below manifests to work it needs to have <code>win10-iso</code> DataVolume.</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: win10-template-configmap\ndata:\n  autounattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n      &lt;settings pass=\"windowsPE\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-International-Core-WinPE\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;SetupUILanguage&gt;\n            &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;/SetupUILanguage&gt;\n          &lt;InputLocale&gt;0409:00000409&lt;/InputLocale&gt;\n          &lt;SystemLocale&gt;en-US&lt;/SystemLocale&gt;\n          &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;UILanguageFallback&gt;en-US&lt;/UILanguageFallback&gt;\n          &lt;UserLocale&gt;en-US&lt;/UserLocale&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-PnpCustomizationsWinPE\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;DriverPaths&gt;\n            &lt;PathAndCredentials wcm:keyValue=\"4b29ba63\" wcm:action=\"add\"&gt;\n              &lt;Path&gt;E:\\amd64\\2k19&lt;/Path&gt;\n            &lt;/PathAndCredentials&gt;\n            &lt;PathAndCredentials wcm:keyValue=\"25fe51ea\" wcm:action=\"add\"&gt;\n              &lt;Path&gt;E:\\NetKVM\\2k19\\amd64&lt;/Path&gt;\n            &lt;/PathAndCredentials&gt;\n          &lt;/DriverPaths&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;DiskConfiguration&gt;\n            &lt;Disk wcm:action=\"add\"&gt;\n              &lt;CreatePartitions&gt;\n                &lt;CreatePartition wcm:action=\"add\"&gt;\n                  &lt;Order&gt;1&lt;/Order&gt;\n                  &lt;Type&gt;Primary&lt;/Type&gt;\n                  &lt;Size&gt;100&lt;/Size&gt;\n                &lt;/CreatePartition&gt;\n                &lt;CreatePartition wcm:action=\"add\"&gt;\n                  &lt;Extend&gt;true&lt;/Extend&gt;\n                  &lt;Order&gt;2&lt;/Order&gt;\n                  &lt;Type&gt;Primary&lt;/Type&gt;\n                &lt;/CreatePartition&gt;\n              &lt;/CreatePartitions&gt;\n              &lt;ModifyPartitions&gt;\n                &lt;ModifyPartition wcm:action=\"add\"&gt;\n                  &lt;Format&gt;NTFS&lt;/Format&gt;\n                  &lt;Label&gt;System Reserved&lt;/Label&gt;\n                  &lt;Order&gt;1&lt;/Order&gt;\n                  &lt;PartitionID&gt;1&lt;/PartitionID&gt;\n                  &lt;TypeID&gt;0x27&lt;/TypeID&gt;\n                &lt;/ModifyPartition&gt;\n                &lt;ModifyPartition wcm:action=\"add\"&gt;\n                  &lt;Format&gt;NTFS&lt;/Format&gt;\n                  &lt;Label&gt;OS&lt;/Label&gt;\n                  &lt;Letter&gt;C&lt;/Letter&gt;\n                  &lt;Order&gt;2&lt;/Order&gt;\n                  &lt;PartitionID&gt;2&lt;/PartitionID&gt;\n                &lt;/ModifyPartition&gt;\n              &lt;/ModifyPartitions&gt;\n              &lt;DiskID&gt;0&lt;/DiskID&gt;\n              &lt;WillWipeDisk&gt;true&lt;/WillWipeDisk&gt;\n            &lt;/Disk&gt;\n          &lt;/DiskConfiguration&gt;\n          &lt;ImageInstall&gt;\n            &lt;OSImage&gt;\n              &lt;InstallFrom&gt;\n                &lt;MetaData wcm:action=\"add\"&gt;\n                  &lt;Key&gt;/Image/Description&lt;/Key&gt;\n                  &lt;Value&gt;Windows 10 Pro&lt;/Value&gt;\n                &lt;/MetaData&gt;\n              &lt;/InstallFrom&gt;\n              &lt;InstallTo&gt;\n                &lt;DiskID&gt;0&lt;/DiskID&gt;\n                &lt;PartitionID&gt;2&lt;/PartitionID&gt;\n              &lt;/InstallTo&gt;\n            &lt;/OSImage&gt;\n          &lt;/ImageInstall&gt;\n          &lt;UserData&gt;\n            &lt;AcceptEula&gt;true&lt;/AcceptEula&gt;\n            &lt;FullName/&gt;\n            &lt;Organization/&gt;\n            &lt;ProductKey&gt;\n              &lt;Key/&gt;\n            &lt;/ProductKey&gt;\n          &lt;/UserData&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"offlineServicing\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-LUA-Settings\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;EnableLUA&gt;false&lt;/EnableLUA&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"specialize\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-International-Core\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;InputLocale&gt;0409:00000409&lt;/InputLocale&gt;\n          &lt;SystemLocale&gt;en-US&lt;/SystemLocale&gt;\n          &lt;UILanguage&gt;en-US&lt;/UILanguage&gt;\n          &lt;UILanguageFallback&gt;en-US&lt;/UILanguageFallback&gt;\n          &lt;UserLocale&gt;en-US&lt;/UserLocale&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Security-SPP-UX\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;SkipAutoActivation&gt;true&lt;/SkipAutoActivation&gt;\n        &lt;/component&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-SQMApi\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;CEIPEnabled&gt;0&lt;/CEIPEnabled&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n      &lt;settings pass=\"oobeSystem\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Shell-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;OOBE&gt;\n            &lt;HideEULAPage&gt;true&lt;/HideEULAPage&gt;\n            &lt;HideOEMRegistrationScreen&gt;true&lt;/HideOEMRegistrationScreen&gt;\n            &lt;HideOnlineAccountScreens&gt;true&lt;/HideOnlineAccountScreens&gt;\n            &lt;HideWirelessSetupInOOBE&gt;true&lt;/HideWirelessSetupInOOBE&gt;\n            &lt;NetworkLocation&gt;Work&lt;/NetworkLocation&gt;\n            &lt;SkipUserOOBE&gt;true&lt;/SkipUserOOBE&gt;\n            &lt;SkipMachineOOBE&gt;true&lt;/SkipMachineOOBE&gt;\n            &lt;ProtectYourPC&gt;3&lt;/ProtectYourPC&gt;\n          &lt;/OOBE&gt;\n          &lt;AutoLogon&gt;\n            &lt;Password&gt;\n              &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/Password&gt;\n            &lt;Enabled&gt;true&lt;/Enabled&gt;\n            &lt;Username&gt;Administrator&lt;/Username&gt;\n          &lt;/AutoLogon&gt;\n          &lt;UserAccounts&gt;\n            &lt;AdministratorPassword&gt;\n              &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/AdministratorPassword&gt;\n          &lt;/UserAccounts&gt;\n          &lt;RegisteredOrganization/&gt;\n          &lt;RegisteredOwner/&gt;\n          &lt;TimeZone&gt;Eastern Standard Time&lt;/TimeZone&gt;\n          &lt;FirstLogonCommands&gt;\n            &lt;SynchronousCommand wcm:action=\"add\"&gt;\n              &lt;CommandLine&gt;powershell -ExecutionPolicy Bypass -NoExit -NoProfile f:\\post-install.ps1&lt;/CommandLine&gt;\n              &lt;RequiresUserInput&gt;false&lt;/RequiresUserInput&gt;\n              &lt;Order&gt;1&lt;/Order&gt;\n              &lt;Description&gt;Post Installation Script&lt;/Description&gt;\n            &lt;/SynchronousCommand&gt;\n          &lt;/FirstLogonCommands&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n    &lt;/unattend&gt;\n\n\n  post-install.ps1: |-\n    # Remove AutoLogin\n    # https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unattend/microsoft-windows-shell-setup-autologon-logoncount#logoncount-known-issue\n    reg add \"HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon\" /v AutoAdminLogon /t REG_SZ /d 0 /f\n\n    # install Qemu Tools (Drivers)\n    Start-Process msiexec -Wait -ArgumentList '/i e:\\virtio-win-gt-x64.msi /qn /passive /norestart'\n\n    # install Guest Agent\n    Start-Process msiexec -Wait -ArgumentList '/i e:\\guest-agent\\qemu-ga-x86_64.msi /qn /passive /norestart'\n\n    # Rename cached unattend.xml to avoid it is picked up by sysprep\n    mv C:\\Windows\\Panther\\unattend.xml C:\\Windows\\Panther\\unattend.install.xml\n\n    # Eject CD, to avoid that the autounattend.xml on the CD is picked up by sysprep\n    (new-object -COM Shell.Application).NameSpace(17).ParseName('F:').InvokeVerb('Eject')\n\n    # Run Sysprep and Shutdown\n    C:\\Windows\\System32\\Sysprep\\sysprep.exe /generalize /oobe /shutdown /mode:vm\n\n---\n\napiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  annotations:\n    name.os.template.kubevirt.io/win10: Microsoft Windows 10\n    vm.kubevirt.io/validations: |\n      [\n        {\n          \"name\": \"minimal-required-memory\",\n          \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n          \"rule\": \"integer\",\n          \"message\": \"This VM requires more memory.\",\n          \"min\": 2147483648\n        }, {\n          \"name\": \"windows-virtio-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"virto disk bus type has better performance, install virtio drivers in VM and change bus type\",\n          \"values\": [\"virtio\"],\n          \"justWarning\": true\n        }, {\n          \"name\": \"windows-disk-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].disk.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"disk bus has to be either virtio or sata or scsi\",\n          \"values\": [\"virtio\", \"sata\", \"scsi\"]\n        }, {\n          \"name\": \"windows-cd-bus\",\n          \"path\": \"jsonpath::.spec.domain.devices.disks[*].cdrom.bus\",\n          \"valid\": \"jsonpath::.spec.domain.devices.disks[*].cdrom.bus\",\n          \"rule\": \"enum\",\n          \"message\": \"cd bus has to be sata\",\n          \"values\": [\"sata\"]\n        }\n      ]\n  name: win10-template\n  namespace: default\n  labels:\n    app: win10-template\n    flavor.template.kubevirt.io/medium: 'true'\n    os.template.kubevirt.io/win10: 'true'\n    vm.kubevirt.io/template: windows10-desktop-medium\n    vm.kubevirt.io/template.namespace: openshift\n    vm.kubevirt.io/template.revision: '1'\n    vm.kubevirt.io/template.version: v0.14.0\n    workload.template.kubevirt.io/desktop: 'true'\nspec:\n  runStrategy: RerunOnFailure\n  dataVolumeTemplates:\n    - metadata:\n        name: win10-template-windows-iso\n      spec:\n        storage: {}\n        source:\n          pvc:\n            name: windows10-iso\n            namespace: default\n    - metadata:\n        name: win10-template\n      spec:\n        storage:\n          resources:\n            requests:\n              storage: 25Gi\n          volumeMode: Filesystem\n        source:\n          blank: {}\n  template:\n    metadata:\n      annotations:\n        vm.kubevirt.io/flavor: medium\n        vm.kubevirt.io/os: windows10\n        vm.kubevirt.io/workload: desktop\n      labels:\n        flavor.template.kubevirt.io/medium: 'true'\n        kubevirt.io/domain: win10-template\n        kubevirt.io/size: medium\n        os.template.kubevirt.io/win10: 'true'\n        vm.kubevirt.io/name: win10-template\n        workload.template.kubevirt.io/desktop: 'true'\n    spec:\n      domain:\n        clock:\n          timer:\n            hpet:\n              present: false\n            hyperv: {}\n            pit:\n              tickPolicy: delay\n            rtc:\n              tickPolicy: catchup\n          utc: {}\n        cpu:\n          cores: 1\n          sockets: 1\n          threads: 1\n        devices:\n          disks:\n            - bootOrder: 1\n              disk:\n                bus: virtio\n              name: win10-template\n            - bootOrder: 2\n              cdrom:\n                bus: sata\n              name: windows-iso\n            - cdrom:\n                bus: sata\n              name: windows-guest-tools\n            - name: sysprep\n              cdrom:\n                bus: sata\n          inputs:\n            - bus: usb\n              name: tablet\n              type: tablet\n          interfaces:\n            - masquerade: {}\n              model: virtio\n              name: default\n        features:\n          acpi: {}\n          apic: {}\n          hyperv:\n            reenlightenment: {}\n            ipi: {}\n            synic: {}\n            synictimer:\n              direct: {}\n            spinlocks:\n              spinlocks: 8191\n            reset: {}\n            relaxed: {}\n            vpindex: {}\n            runtime: {}\n            tlbflush: {}\n            frequencies: {}\n            vapic: {}\n        machine:\n          type: pc-q35-rhel8.4.0\n        resources:\n          requests:\n            memory: 4Gi\n      hostname: win10-template\n      networks:\n        - name: default\n          pod: {}\n      volumes:\n        - dataVolume:\n            name: win10-iso\n          name: windows-iso\n        - dataVolume:\n            name: win10-template-windows-iso\n          name: win10-template\n        - containerDisk:\n            image: quay.io/kubevirt/virtio-container-disk\n          name: windows-guest-tools\n        - name: sysprep\n          sysprep:\n            configMap:\n              name: win10-template-configmap\n</code></pre>"},{"location":"user_workloads/startup_scripts/#launching-a-vm-from-template","title":"Launching a VM from template","text":"<p>From the above example after the sysprep command is executed in the <code>post-install.ps1</code> and the vm is in shutdown state, A new VM can be launched from the base <code>win10-template</code> with additional changes mentioned from the below <code>unattend.xml</code> in <code>sysprep-config</code>. The new VM can take upto 5 minutes to be in running state since Windows goes through oobe setup in the background with the customizations specified in the below <code>unattend.xml</code> file.</p> <pre><code>apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: sysprep-config\ndata:\n  autounattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;!-- responsible for installing windows, ignored on sysprepped images --&gt;\n  unattend.xml: |-\n    &lt;?xml version=\"1.0\" encoding=\"utf-8\"?&gt;\n    &lt;unattend xmlns=\"urn:schemas-microsoft-com:unattend\"&gt;\n      &lt;settings pass=\"oobeSystem\"&gt;\n        &lt;component xmlns:wcm=\"http://schemas.microsoft.com/WMIConfig/2002/State\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" name=\"Microsoft-Windows-Shell-Setup\" processorArchitecture=\"amd64\" publicKeyToken=\"31bf3856ad364e35\" language=\"neutral\" versionScope=\"nonSxS\"&gt;\n          &lt;OOBE&gt;\n            &lt;HideEULAPage&gt;true&lt;/HideEULAPage&gt;\n            &lt;HideOEMRegistrationScreen&gt;true&lt;/HideOEMRegistrationScreen&gt;\n            &lt;HideOnlineAccountScreens&gt;true&lt;/HideOnlineAccountScreens&gt;\n            &lt;HideWirelessSetupInOOBE&gt;true&lt;/HideWirelessSetupInOOBE&gt;\n            &lt;NetworkLocation&gt;Work&lt;/NetworkLocation&gt;\n            &lt;SkipUserOOBE&gt;true&lt;/SkipUserOOBE&gt;\n            &lt;SkipMachineOOBE&gt;true&lt;/SkipMachineOOBE&gt;\n            &lt;ProtectYourPC&gt;3&lt;/ProtectYourPC&gt;\n          &lt;/OOBE&gt;\n          &lt;AutoLogon&gt;\n            &lt;Password&gt;\n            &lt;Value&gt;123456&lt;/Value&gt;\n              &lt;PlainText&gt;true&lt;/PlainText&gt;\n            &lt;/Password&gt;\n            &lt;Enabled&gt;true&lt;/Enabled&gt;\n        &lt;Username&gt;Administrator&lt;/Username&gt;\n    &lt;/AutoLogon&gt;\n    &lt;UserAccounts&gt;\n         &lt;AdministratorPassword&gt;\n                &lt;Value&gt;123456&lt;/Value&gt;\n                &lt;PlainText&gt;true&lt;/PlainText&gt;\n        &lt;/AdministratorPassword&gt;\n          &lt;/UserAccounts&gt;\n          &lt;RegisteredOrganization&gt;Kuebvirt&lt;/RegisteredOrganization&gt;\n          &lt;RegisteredOwner&gt;Kubevirt&lt;/RegisteredOwner&gt;\n          &lt;TimeZone&gt;Eastern Standard Time&lt;/TimeZone&gt;\n                &lt;FirstLogonCommands&gt;\n                    &lt;SynchronousCommand wcm:action=\"add\"&gt;\n                    &lt;CommandLine&gt;powershell -ExecutionPolicy Bypass -NoExit -WindowStyle Hidden -NoProfile d:\\customize.ps1&lt;/CommandLine&gt;\n                        &lt;RequiresUserInput&gt;false&lt;/RequiresUserInput&gt;\n                        &lt;Order&gt;1&lt;/Order&gt;\n                &lt;Description&gt;Customize Script&lt;/Description&gt;\n            &lt;/SynchronousCommand&gt;\n                &lt;/FirstLogonCommands&gt;\n        &lt;/component&gt;\n      &lt;/settings&gt;\n    &lt;/unattend&gt;\n  customize.ps1: |-\n    # Enable RDP\n    Set-ItemProperty -Path 'HKLM:\\System\\CurrentControlSet\\Control\\Terminal Server' -name \"fDenyTSConnections\" -value 0\n    Enable-NetFirewallRule -DisplayGroup \"Remote Desktop\"\n\n\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse\n    # Install the OpenSSH Server\n    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0\n    # Start the sshd service\n    Start-Service sshd\n\n    Set-Service -Name sshd -StartupType 'Automatic'\n\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration\n    # use powershell as default shell for ssh\n    New-ItemProperty -Path \"HKLM:\\SOFTWARE\\OpenSSH\" -Name DefaultShell -Value \"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\" -PropertyType String -Force\n\n\n    # Add ssh authorized_key for administrator\n    # https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement\n    $MyDir = $MyInvocation.MyCommand.Path | Split-Path -Parent\n    $PublicKey = Get-Content -Path $MyDir\\id_rsa.pub\n    $authrized_keys_path = $env:ProgramData + \"\\ssh\\administrators_authorized_keys\" \n    Add-Content -Path $authrized_keys_path -Value $PublicKey\n    icacls.exe $authrized_keys_path /inheritance:r /grant \"Administrators:F\" /grant \"SYSTEM:F\"\n\n\n    # install application via exe file installer from url\n    function Install-Exe {\n      $dlurl = $args[0]\n      $installerPath = Join-Path $env:TEMP (Split-Path $dlurl -Leaf)\n      Invoke-WebRequest -UseBasicParsing $dlurl -OutFile $installerPath\n      Start-Process -FilePath $installerPath -Args \"/S\" -Verb RunAs -Wait\n      Remove-Item $installerPath\n\n    }\n\n    # Wait for networking before running a task at startup\n    do {\n      $ping = test-connection -comp kubevirt.io -count 1 -Quiet\n    } until ($ping)\n\n    # Installing the Latest Notepad++ with PowerShell\n    $BaseUri = \"https://notepad-plus-plus.org\"\n    $BasePage = Invoke-WebRequest -Uri $BaseUri -UseBasicParsing\n    $ChildPath = $BasePage.Links | Where-Object { $_.outerHTML -like '*Current Version*' } | Select-Object -ExpandProperty href\n    $DownloadPageUri = $BaseUri + $ChildPath\n    $DownloadPage = Invoke-WebRequest -Uri $DownloadPageUri -UseBasicParsing\n    $DownloadUrl = $DownloadPage.Links | Where-Object { $_.outerHTML -like '*npp.*.Installer.x64.exe\"*' } | Select-Object -ExpandProperty href\n    Install-Exe $DownloadUrl\n  id_rsa.pub: |-\n    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6zdgFiLr1uAK7PdcchDd+LseA5fEOcxCCt7TLlr7Mx6h8jUg+G+8L9JBNZuDzTZSF0dR7qwzdBBQjorAnZTmY3BhsKcFr8Gt4KMGrS6r3DNmGruP8GORvegdWZuXgASKVpXeI7nCIjRJwAaK1x+eGHwAWO9Z8ohcboHbLyffOoSZDSIuk2kRIc47+ENRjg0T6x2VRsqX27g6j4DfPKQZGk0zvXkZaYtr1e2tZgqTBWqZUloMJK8miQq6MktCKAS4VtPk0k7teQX57OGwD6D7uo4b+Cl8aYAAwhn0hc0C2USfbuVHgq88ESo2/+NwV4SQcl3sxCW21yGIjAGt4Hy7J fedora@localhost.localdomain\n</code></pre>"},{"location":"user_workloads/templates/","title":"Templates","text":"<p>Note</p> <p>By deploying KubeVirt on top of OpenShift the user can benefit from the OpenShift Template functionality.</p>"},{"location":"user_workloads/templates/#virtual-machine-templates","title":"Virtual machine templates","text":""},{"location":"user_workloads/templates/#what-is-a-virtual-machine-template","title":"What is a virtual machine template?","text":"<p>The KubeVirt projects provides a set of templates to create VMs to handle common usage scenarios. These templates provide a combination of some key factors that could be further customized and processed to have a Virtual Machine object. The key factors which define a template are</p> <ul> <li> <p>Workload Most Virtual Machine should be server or desktop to have maximum     flexibility; the highperformance workload trades some of this     flexibility to provide better performances.</p> </li> <li> <p>Guest Operating System (OS) This allow to ensure that the emulated     hardware is compatible with the guest OS. Furthermore, it allows to     maximize the stability of the VM, and allows performance     optimizations.</p> </li> <li> <p>Size (flavor) Defines the amount of resources (CPU, memory) to     allocate to the VM.</p> </li> </ul> <p>More documentation is available in the common templates subproject</p>"},{"location":"user_workloads/templates/#accessing-the-virtual-machine-templates","title":"Accessing the virtual machine templates","text":"<p>If you installed KubeVirt using a supported method you should find the common templates preinstalled in the cluster. Should you want to upgrade the templates, or install them from scratch, you can use one of the supported releases</p> <p>To install the templates:</p> <pre><code>    $ export VERSION=$(curl -s https://api.github.com/repos/kubevirt/common-templates/releases | grep tag_name | grep -v -- '-rc' | head -1 | awk -F': ' '{print $2}' | sed 's/,//' | xargs)\n    $ oc create -f https://github.com/kubevirt/common-templates/releases/download/$VERSION/common-templates-$VERSION.yaml\n</code></pre>"},{"location":"user_workloads/templates/#editable-fields","title":"Editable fields","text":"<p>You can edit the fields of the templates which define the amount of resources which the VMs will receive.</p> <p>Each template can list a different set of fields that are to be considered editable. The fields are used as hints for the user interface, and also for other components in the cluster.</p> <p>The editable fields are taken from annotations in the template. Here is a snippet presenting a couple of most commonly found editable fields:</p> <pre><code>    metadata:\n      annotations:\n        template.kubevirt.io/editable: |\n          /objects[0].spec.template.spec.domain.cpu.sockets\n          /objects[0].spec.template.spec.domain.cpu.cores\n          /objects[0].spec.template.spec.domain.cpu.threads\n          /objects[0].spec.template.spec.domain.resources.requests.memory\n</code></pre> <p>Each entry in the editable field list must be a jsonpath. The jsonpath root is the objects: element of the template. The actually editable field is the last entry (the \"leaf\") of the path. For example, the following minimal snippet highlights the fields which you can edit:</p> <pre><code>    objects:\n      spec:\n        template:\n          spec:\n            domain:\n              cpu:\n                sockets:\n                  VALUE # this is editable\n                cores:\n                  VALUE # this is editable\n                threads:\n                  VALUE # this is editable\n              resources:\n                requests:\n                  memory:\n                    VALUE # this is editable\n</code></pre>"},{"location":"user_workloads/templates/#relationship-between-templates-and-vms","title":"Relationship between templates and VMs","text":"<p>Once processed the templates produce VM objects to be used in the cluster. The VMs produced from templates will have a <code>vm.kubevirt.io/template</code> label, whose value will be the name of the parent template, for example <code>fedora-desktop-medium</code>:</p> <pre><code>      metadata:\n        labels:\n          vm.kubevirt.io/template: fedora-desktop-medium\n</code></pre> <p>In addition, these VMs can include an optional label <code>vm.kubevirt.io/template-namespace</code>, whose value will be the namespace of the parent template, for example:</p> <pre><code>      metadata:\n        labels:\n          vm.kubevirt.io/template-namespace: openshift\n</code></pre> <p>If this label is not defined, the template is expected to belong to the same namespace as the VM.</p> <p>This make it possible to query for all the VMs built from any template.</p> <p>Example:</p> <pre><code>    oc process -o yaml -f dist/templates/rhel8-server-tiny.yaml NAME=rheltinyvm SRC_PVC_NAME=rhel SRC_PVC_NAMESPACE=kubevirt\n</code></pre> <p>And the output:</p> <pre><code>apiVersion: v1\nitems:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachine\n  metadata:\n    annotations:\n      vm.kubevirt.io/flavor: tiny\n      vm.kubevirt.io/os: rhel8\n      vm.kubevirt.io/validations: |\n        [\n          {\n            \"name\": \"minimal-required-memory\",\n            \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n            \"rule\": \"integer\",\n            \"message\": \"This VM requires more memory.\",\n            \"min\": 1610612736\n          }\n        ]\n      vm.kubevirt.io/workload: server\n    labels:\n      app: rheltinyvm\n      vm.kubevirt.io/template: rhel8-server-tiny\n      vm.kubevirt.io/template.revision: \"45\"\n      vm.kubevirt.io/template.version: 0.11.3\n    name: rheltinyvm\n  spec:\n    dataVolumeTemplates:\n    - apiVersion: cdi.kubevirt.io/v1beta1\n      kind: DataVolume\n      metadata:\n        name: rheltinyvm\n      spec:\n        storage:\n          accessModes:\n          - ReadWriteMany\n        source:\n          pvc:\n            name: rhel\n            namespace: kubevirt\n    running: false\n    template:\n      metadata:\n        labels:\n          kubevirt.io/domain: rheltinyvm\n          kubevirt.io/size: tiny\n      spec:\n        domain:\n          cpu:\n            cores: 1\n            sockets: 1\n            threads: 1\n          devices:\n            disks:\n            - disk:\n                bus: virtio\n              name: rheltinyvm\n            - disk:\n                bus: virtio\n              name: cloudinitdisk\n            interfaces:\n            - masquerade: {}\n              name: default\n            networkInterfaceMultiqueue: true\n            rng: {}\n          resources:\n            requests:\n              memory: 1.5Gi\n        networks:\n        - name: default\n          pod: {}\n        terminationGracePeriodSeconds: 180\n        volumes:\n        - dataVolume:\n            name: rheltinyvm\n          name: rheltinyvm\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              user: cloud-user\n              password: lymp-fda4-m1cv\n              chpasswd: { expire: False }\n          name: cloudinitdisk\nkind: List\nmetadata: {}\n</code></pre> <p>You can add the VM from the template to the cluster in one go</p> <pre><code>    oc process rhel8-server-tiny NAME=rheltinyvm SRC_PVC_NAME=rhel SRC_PVC_NAMESPACE=kubevirt | oc apply -f -\n</code></pre> <p>Please note that after the generation step VM and template objects have no relationship with each other besides the aforementioned label.  Changes in templates do not automatically affect VMs or vice versa.</p>"},{"location":"user_workloads/templates/#common-template-customization","title":"common template customization","text":"<p>The templates provided by the kubevirt project provide a set of conventions and annotations that augment the basic feature of the openshift templates. You can customize your kubevirt-provided templates editing these annotations, or you can add them to your existing templates to make them consumable by the kubevirt services.</p> <p>Here's a description of the kubevirt annotations. Unless otherwise specified, the following keys are meant to be top-level entries of the template metadata, like</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: windows-10\n  annotations:\n    openshift.io/display-name: \"Generic demo template\"\n</code></pre> <p>All the following annotations are prefixed with <code>defaults.template.kubevirt.io</code>, which is omitted below for brevity. So the actual annotations you should use will look like</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: windows-10\n  annotations:\n    defaults.template.kubevirt.io/disk: default-disk\n    defaults.template.kubevirt.io/volume: default-volume\n    defaults.template.kubevirt.io/nic: default-nic\n    defaults.template.kubevirt.io/network: default-network\n</code></pre> <p>Unless otherwise specified, all annotations are meant to be safe defaults, both for performance and compatibility, and hints for the CNV-aware UI and tooling.</p>"},{"location":"user_workloads/templates/#disk","title":"disk","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/disk: rhel-disk\n</code></pre>"},{"location":"user_workloads/templates/#nic","title":"nic","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Windows\n  annotations:\n    defaults.template.kubevirt.io/nic: my-nic\n</code></pre>"},{"location":"user_workloads/templates/#volume","title":"volume","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/volume: custom-volume\n</code></pre>"},{"location":"user_workloads/templates/#network","title":"network","text":"<p>See the section <code>references</code> below.</p> <p>Example:</p> <pre><code>apiVersion: v1\nkind: Template\nmetadata:\n  name: Linux\n  annotations:\n    defaults.template.kubevirt.io/network: fast-net\n</code></pre>"},{"location":"user_workloads/templates/#references","title":"references","text":"<p>The default values for network, nic, volume, disk are meant to be the name of a section later in the document that the UI will find and consume to find the default values for the corresponding types. For example, considering the annotation <code>defaults.template.kubevirt.io/disk: my-disk</code>: we assume that later in the document it exists an element called <code>my-disk</code> that the UI can use to find the data it needs. The names actually don't matter as long as they are legal for kubernetes and consistent with the content of the document.</p>"},{"location":"user_workloads/templates/#complete-example","title":"complete example","text":"<p><code>demo-template.yaml</code></p> <pre><code>apiversion: v1\nitems:\n- apiversion: kubevirt.io/v1\n  kind: virtualmachine\n  metadata:\n    labels:\n      vm.kubevirt.io/template: rhel7-generic-tiny\n    name: rheltinyvm\n    osinfoname: rhel7.0\n    defaults.template.kubevirt.io/disk: rhel-default-disk\n    defaults.template.kubevirt.io/nic: rhel-default-net\n  spec:\n    running: false\n    template:\n      spec:\n        domain:\n          cpu:\n            sockets: 1\n            cores: 1\n            threads: 1\n          devices:\n            rng: {}\n          resources:\n            requests:\n              memory: 1g\n        terminationgraceperiodseconds: 0\n        volumes:\n        - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n          name: rhel-default-disk\n        networks:\n        - genie:\n          networkName: flannel\n          name: rhel-default-net\nkind: list\nmetadata: {}\n</code></pre> <p>once processed becomes: <code>demo-vm.yaml</code></p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachine\nmetadata:\n  labels:\n    vm.kubevirt.io/template: rhel7-generic-tiny\n  name: rheltinyvm\n  osinfoname: rhel7.0\nspec:\n  running: false\n  template:\n    spec:\n      domain:\n        cpu:\n          sockets: 1\n          cores: 1\n          threads: 1\n        resources:\n          requests:\n            memory: 1g\n        devices:\n          rng: {}\n          disks:\n          - disk:\n            name: rhel-default-disk\n        interfaces:\n        - bridge: {}\n          name: rhel-default-nic\n      terminationgraceperiodseconds: 0\n      volumes:\n      - containerDisk:\n          image: registry:5000/kubevirt/cirros-container-disk-demo:devel\n        name: containerdisk\n      networks:\n      - genie:\n          networkName: flannel\n        name: rhel-default-nic\n</code></pre>"},{"location":"user_workloads/templates/#virtual-machine-creation","title":"Virtual machine creation","text":""},{"location":"user_workloads/templates/#overview","title":"Overview","text":"<p>The KubeVirt projects provides a set of templates to create VMs to handle common usage scenarios. These templates provide a combination of some key factors that could be further customized and processed to have a Virtual Machine object.</p> <p>The key factors which define a template are - Workload Most Virtual Machine should be server or desktop to have maximum flexibility; the highperformance workload trades some of this flexibility to provide better performances. - Guest Operating System (OS) This allow to ensure that the emulated hardware is compatible with the guest OS. Furthermore, it allows to maximize the stability of the VM, and allows performance optimizations. - Size (flavor) Defines the amount of resources (CPU, memory) to allocate to the VM.</p>"},{"location":"user_workloads/templates/#openshift-console","title":"Openshift Console","text":"<p>VMs can be created through OpenShift Cluster Console UI . This UI supports creation VM using templates and templates features - flavors and workload profiles. To create VM from template, choose WorkLoads in the left panel &gt;&gt; choose Virtualization &gt;&gt; press to the \"Create Virtual Machine\" blue button &gt;&gt; choose \"Create from wizard\". Next, you have to see \"Create Virtual Machine\" window</p>"},{"location":"user_workloads/templates/#common-templates","title":"Common-templates","text":"<p>There is the common-templates subproject. It provides official prepared and useful templates. You can also create templates by hand. You can find an example below, in the \"Example template\" section.</p>"},{"location":"user_workloads/templates/#example-template","title":"Example template","text":"<p>In order to create a virtual machine via OpenShift CLI, you need to provide a template defining the corresponding object and its metadata.</p> <p>NOTE Only <code>VirtualMachine</code> object is currently supported.</p> <p>Here is an example template that defines an instance of the <code>VirtualMachine</code> object:</p> <pre><code>apiVersion: template.openshift.io/v1\nkind: Template\nmetadata:\n  name: fedora-desktop-large\n  annotations:\n    openshift.io/display-name: \"Fedora 32+ VM\"\n    description: &gt;-\n      Template for Fedora 32 VM or newer.\n      A PVC with the Fedora disk image must be available.\n      Recommended disk image:\n      https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.qcow2\n    tags: \"hidden,kubevirt,virtualmachine,fedora\"\n    iconClass: \"icon-fedora\"\n    openshift.io/provider-display-name: \"KubeVirt\"\n    openshift.io/documentation-url: \"https://github.com/kubevirt/common-templates\"\n    openshift.io/support-url: \"https://github.com/kubevirt/common-templates/issues\"\n    template.openshift.io/bindable: \"false\"\n    template.kubevirt.io/version: v1alpha1\n    defaults.template.kubevirt.io/disk: rootdisk\n    template.kubevirt.io/editable: |\n      /objects[0].spec.template.spec.domain.cpu.sockets\n      /objects[0].spec.template.spec.domain.cpu.cores\n      /objects[0].spec.template.spec.domain.cpu.threads\n      /objects[0].spec.template.spec.domain.resources.requests.memory\n      /objects[0].spec.template.spec.domain.devices.disks\n      /objects[0].spec.template.spec.volumes\n      /objects[0].spec.template.spec.networks\n    name.os.template.kubevirt.io/fedora32: Fedora 32 or higher\n    name.os.template.kubevirt.io/fedora33: Fedora 32 or higher\n    name.os.template.kubevirt.io/silverblue32: Fedora 32 or higher\n    name.os.template.kubevirt.io/silverblue33: Fedora 32 or higher\n  labels:\n    os.template.kubevirt.io/fedora32: \"true\"\n    os.template.kubevirt.io/fedora33: \"true\"\n    os.template.kubevirt.io/silverblue32: \"true\"\n    os.template.kubevirt.io/silverblue33: \"true\"\n    workload.template.kubevirt.io/desktop: \"true\"\n    flavor.template.kubevirt.io/large: \"true\"\n    template.kubevirt.io/type: \"base\"\n    template.kubevirt.io/version: \"0.11.3\"\nobjects:\n- apiVersion: kubevirt.io/v1\n  kind: VirtualMachine\n  metadata:\n    name: ${NAME}\n    labels:\n      vm.kubevirt.io/template: fedora-desktop-large\n      vm.kubevirt.io/template.version: \"0.11.3\"\n      vm.kubevirt.io/template.revision: \"45\"\n      app: ${NAME}\n    annotations:\n      vm.kubevirt.io/os: \"fedora\"\n      vm.kubevirt.io/workload: \"desktop\"\n      vm.kubevirt.io/flavor: \"large\"\n      vm.kubevirt.io/validations: |\n        [\n          {\n            \"name\": \"minimal-required-memory\",\n            \"path\": \"jsonpath::.spec.domain.resources.requests.memory\",\n            \"rule\": \"integer\",\n            \"message\": \"This VM requires more memory.\",\n            \"min\": 1073741824\n          }\n        ]\n  spec:\n    dataVolumeTemplates:\n    - apiVersion: cdi.kubevirt.io/v1beta1\n      kind: DataVolume\n      metadata:\n        name: ${NAME}\n      spec:\n        storage:\n          accessModes:\n            - ReadWriteMany\n        source:\n          pvc:\n            name: ${SRC_PVC_NAME}\n            namespace: ${SRC_PVC_NAMESPACE}\n    running: false\n    template:\n      metadata:\n        labels:\n          kubevirt.io/domain: ${NAME}\n          kubevirt.io/size: large\n      spec:\n        domain:\n          cpu:\n            sockets: 2\n            cores: 1\n            threads: 1\n          resources:\n            requests:\n              memory: 8Gi\n          devices:\n            rng: {}\n            networkInterfaceMultiqueue: true\n            inputs:\n              - type: tablet\n                bus: virtio\n                name: tablet\n            disks:\n            - disk:\n                bus: virtio\n              name: ${NAME}\n            - disk:\n                bus: virtio\n              name: cloudinitdisk\n            interfaces:\n            - masquerade: {}\n              name: default\n        terminationGracePeriodSeconds: 180\n        networks:\n        - name: default\n          pod: {}\n        volumes:\n        - dataVolume:\n            name: ${NAME}\n          name: ${NAME}\n        - cloudInitNoCloud:\n            userData: |-\n              #cloud-config\n              user: fedora\n              password: ${CLOUD_USER_PASSWORD}\n              chpasswd: { expire: False }\n          name: cloudinitdisk\nparameters:\n- description: VM name\n  from: 'fedora-[a-z0-9]{16}'\n  generate: expression\n  name: NAME\n- name: SRC_PVC_NAME\n  description: Name of the PVC to clone\n  value: 'fedora'\n- name: SRC_PVC_NAMESPACE\n  description: Namespace of the source PVC\n  value: kubevirt-os-images\n- description: Randomized password for the cloud-init user fedora\n  from: '[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}'\n  generate: expression\n  name: CLOUD_USER_PASSWORD\n</code></pre> <p>Note that the template above defines free parameters (<code>NAME</code>, <code>SRC_PVC_NAME</code>, <code>SRC_PVC_NAMESPACE</code>, <code>CLOUD_USER_PASSWORD</code>) and the <code>NAME</code> parameter does not have specified default value.</p> <p>An OpenShift template has to be converted into the JSON file via <code>oc process</code> command, that also allows you to set the template parameters.</p> <p>A complete example can be found in the KubeVirt repository.</p> <p>!&gt; You need to be logged in by <code>oc login</code> command.</p> <pre><code>$ oc process -f cluster/vmi-template-fedora.yaml\\\n    -p NAME=testvmi \\\n    -p SRC_PVC_NAME=fedora \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n{\n    \"kind\": \"List\",\n    \"apiVersion\": \"v1\",\n    \"metadata\": {},\n    \"items\": [\n        {\n</code></pre> <p>The JSON file is usually applied directly by piping the processed output to <code>oc create</code> command.</p> <pre><code>$ oc process -f cluster/examples/vm-template-fedora.yaml \\\n    -p NAME=testvm \\\n    -p SRC_PVC_NAME=fedora \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n    | oc create -f -\nvirtualmachine.kubevirt.io/testvm created\n</code></pre> <p>The command above results in creating a Kubernetes object according to the specification given by the template \\(in this example it is an instance of the VirtualMachine object\\).</p> <p>It's possible to get list of available parameters using the following command:</p> <pre><code>$ oc process -f dist/templates/fedora-desktop-large.yaml --parameters\nNAME                  DESCRIPTION                                          GENERATOR           VALUE\nNAME                  VM name                                              expression          fedora-[a-z0-9]{16}\nSRC_PVC_NAME          Name of the PVC to clone                                                 fedora\nSRC_PVC_NAMESPACE     Namespace of the source PVC                                              kubevirt-os-images\nCLOUD_USER_PASSWORD   Randomized password for the cloud-init user fedora   expression          [a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}\n</code></pre>"},{"location":"user_workloads/templates/#starting-virtual-machine-from-the-created-object","title":"Starting virtual machine from the created object","text":"<p>The created object is now a regular VirtualMachine object and from now it can be controlled by accessing Kubernetes API resources. The preferred way how to do this from within the OpenShift environment is to use <code>oc patch</code> command.</p> <pre><code>$ oc patch virtualmachine testvm --type merge -p '{\"spec\":{\"running\":true}}'\nvirtualmachine.kubevirt.io/testvm patched\n</code></pre> <p>Do not forget about virtctl tool. Using it in the real cases instead of using kubernetes API can be more convenient. Example:</p> <pre><code>$ virtctl start testvm\nVM testvm was scheduled to start\n</code></pre> <p>As soon as VM starts, Kubernetes creates new type of object - VirtualMachineInstance. It has similar name to VirtualMachine. Example (not full output, it's too big):</p> <pre><code>$ kubectl describe vm testvm\nname:         testvm\nNamespace:    myproject\nLabels:       kubevirt-vm=vm-testvm\n              kubevirt.io/os=fedora33\nAnnotations:  &lt;none&gt;\nAPI Version:  kubevirt.io/v1\nKind:         VirtualMachine\n</code></pre>"},{"location":"user_workloads/templates/#cloud-init-script-and-parameters","title":"Cloud-init script and parameters","text":"<p>Kubevirt VM templates, just like kubevirt VM/VMI yaml configs, supports cloud-init scripts</p>"},{"location":"user_workloads/templates/#hack-use-pre-downloaded-image","title":"Hack - use pre-downloaded image","text":"<p>Kubevirt VM templates, just like kubevirt VM/VMI yaml configs, can use pre-downloaded VM image, which can be a useful feature especially in the debug/development/testing cases. No special parameters required in the VM template or VM/VMI yaml config. The main idea is to create Kubernetes PersistentVolume and PersistentVolumeClaim corresponding to existing image in the file system. Example:</p> <pre><code>---\nkind: PersistentVolume\napiVersion: v1\nmetadata:\n  name: mypv\n  labels:\n    type: local\nspec:\n  storageClassName: manual\n  capacity:\n    storage: 10G\n  accessModes:\n    - ReadWriteOnce\n  hostPath:\n    path: \"/mnt/sda1/images/testvm\"\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: mypvc\nspec:\n  storageClassName: manual\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 10G\n</code></pre>"},{"location":"user_workloads/templates/#using-datavolumes","title":"Using DataVolumes","text":"<p>Kubevirt VM templates are using dataVolumeTemplates. Before using dataVolumes, CDI has to be installed in cluster. After that, source Datavolume can be created.</p> <pre><code>---\napiVersion: cdi.kubevirt.io/v1beta1\nkind: DataVolume\nmetadata:\n  name: fedora-datavolume-original\n  namespace: kubevirt\nspec:\n  source:\n    registry:\n      url: \"image_url\"\n  storage:\n    resources:\n      requests:\n        storage: 30Gi\n</code></pre> <p>After import is completed, VM can be created: <pre><code>$ oc process -f cluster/examples/vm-template-fedora.yaml \\\n    -p NAME=testvmi \\\n    -p SRC_PVC_NAME=fedora-datavolume-original \\\n    -p SRC_PVC_NAMESPACE=kubevirt \\\n    | oc create -f -\nvirtualmachine.kubevirt.io/testvm created\n</code></pre></p>"},{"location":"user_workloads/templates/#additional-information","title":"Additional information","text":"<p>You can follow Virtual Machine Lifecycle Guide for further reference.</p>"},{"location":"user_workloads/virtctl_client_tool/","title":"Download and Install the virtctl Command Line Interface","text":""},{"location":"user_workloads/virtctl_client_tool/#download-the-virtctl-client-tool","title":"Download the <code>virtctl</code> client tool","text":"<p>Basic VirtualMachineInstance operations can be performed with the stock <code>kubectl</code> utility. However, the <code>virtctl</code> binary utility is required to use advanced features such as:</p> <ul> <li>Serial and graphical console access</li> </ul> <p>It also provides convenience commands for:</p> <ul> <li> <p>Starting and stopping VirtualMachineInstances</p> </li> <li> <p>Live migrating VirtualMachineInstances and canceling live migrations</p> </li> <li> <p>Uploading virtual machine disk images</p> </li> </ul> <p>There are two ways to get it:</p> <ul> <li> <p>the most recent version of the tool can be retrieved from the     official release     page</p> </li> <li> <p>it can be installed as a <code>kubectl</code> plugin using     krew</p> </li> </ul> <p>Example:</p> <pre><code>export VERSION=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)\nwget https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-linux-amd64\n</code></pre>"},{"location":"user_workloads/virtctl_client_tool/#install-virtctl-with-krew","title":"Install <code>virtctl</code> with <code>krew</code>","text":"<p>It is required to install <code>krew</code> plugin manager beforehand. If <code>krew</code> is installed, <code>virtctl</code> can be installed via <code>krew</code>:</p> <pre><code>$ kubectl krew install virt\n</code></pre> <p>Then <code>virtctl</code> can be used as a kubectl plugin. For a list of available commands run:</p> <pre><code>$ kubectl virt help\n</code></pre> <p>Every occurrence throughout this guide of</p> <pre><code>$ ./virtctl &lt;command&gt;...\n</code></pre> <p>should then be read as</p> <pre><code>$ kubectl virt &lt;command&gt;...\n</code></pre>"},{"location":"user_workloads/virtual_machine_instances/","title":"Virtual Machines Instances","text":"<p>The <code>VirtualMachineInstance</code> type conceptionally has two parts:</p> <ul> <li> <p>Information for making scheduling decisions</p> </li> <li> <p>Information about the virtual machine API</p> </li> </ul> <p>Every <code>VirtualMachineInstance</code> object represents a single running virtual machine instance.</p>"},{"location":"user_workloads/virtual_machine_instances/#api-overview","title":"API Overview","text":"<p>With the installation of KubeVirt, new types are added to the Kubernetes API to manage Virtual Machines.</p> <p>You can interact with the new resources (via <code>kubectl</code>) as you would with any other API resource.</p>"},{"location":"user_workloads/virtual_machine_instances/#virtualmachineinstance-api","title":"VirtualMachineInstance API","text":"<p>Note: A full API reference is available at https://kubevirt.io/api-reference/.</p> <p>Here is an example of a VirtualMachineInstance object:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: VirtualMachineInstance\nmetadata:\n  name: testvmi-nocloud\nspec:\n  terminationGracePeriodSeconds: 30\n  domain:\n    resources:\n      requests:\n        memory: 1024M\n    devices:\n      disks:\n      - name: containerdisk\n        disk:\n          bus: virtio\n      - name: emptydisk\n        disk:\n          bus: virtio\n      - disk:\n          bus: virtio\n        name: cloudinitdisk\n  volumes:\n  - name: containerdisk\n    containerDisk:\n      image: kubevirt/fedora-cloud-container-disk-demo:latest\n  - name: emptydisk\n    emptyDisk:\n      capacity: \"2Gi\"\n  - name: cloudinitdisk\n    cloudInitNoCloud:\n      userData: |-\n        #cloud-config\n        password: fedora\n        chpasswd: { expire: False }\n</code></pre> <p>This example uses a fedora cloud image in combination with cloud-init and an ephemeral empty disk with a capacity of <code>2Gi</code>. For the sake of simplicity, the volume sources in this example are ephemeral and don't require a provisioner in your cluster.</p>"},{"location":"user_workloads/virtual_machine_instances/#additional-information","title":"Additional Information","text":"<ul> <li> <p>Using instancetypes and preferences with a VirtualMachine:     Instancetypes and preferences</p> </li> <li> <p>More information about persistent and ephemeral volumes:     Disks and Volumes</p> </li> <li> <p>How to access a VirtualMachineInstance via <code>console</code> or <code>vnc</code>:     Console Access</p> </li> <li> <p>How to customize VirtualMachineInstances with <code>cloud-init</code>:     Cloud Init</p> </li> </ul>"},{"location":"user_workloads/vm_rollout_strategies/","title":"VM Rollout Strategies","text":"<p>In KubeVirt, the VM rollout strategy defines how changes to a VM object affect a running guest. In other words, it defines when and how changes to a VM object get propagated to its corresponding VMI object.</p> <p>There are currently 2 rollout strategies: <code>LiveUpdate</code> and <code>Stage</code>. Only 1 can be specified and the default is <code>Stage</code>.</p>"},{"location":"user_workloads/vm_rollout_strategies/#feature-gate","title":"Feature Gate","text":"<p>As long as the <code>VMLiveUpdateFeatures</code> is not enabled, the VM Rollout Strategy is ignored and defaults to \"Stage\". The feature gate is set in the KubeVirt custom resource (CR) like that:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    developerConfiguration:\n      featureGates:\n        - VMLiveUpdateFeatures\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#liveupdate","title":"LiveUpdate","text":"<p>The <code>LiveUpdate</code> VM rollout strategy tries to propagate VM object changes to running VMIs as soon as possible. For example, changing the number of CPU sockets will trigger a CPU hotplug.</p> <p>Enable the <code>LiveUpdate</code> VM rollout strategy in the KubeVirt CR:</p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"LiveUpdate\"\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#stage","title":"Stage","text":"<p>The <code>Stage</code> VM rollout strategy stages every change made to the VM object until its next reboot.  </p> <pre><code>apiVersion: kubevirt.io/v1\nkind: KubeVirt\nspec:\n  configuration:\n    vmRolloutStrategy: \"Stage\"\n</code></pre>"},{"location":"user_workloads/vm_rollout_strategies/#restartrequired-condition","title":"RestartRequired condition","text":"<p>Any change made to a VM object when the rollout strategy is <code>Stage</code> will trigger the <code>RestartRequired</code> VM condition. When the rollout strategy is <code>LiveUpdate</code>, only non-propagatable changes will trigger the condition.</p> <p>Once the <code>RestartRequired</code> condition is set on a VM object, no further changes can be propagated, even if the strategy is set to <code>LiveUpdate</code>. Changes will become effective on next reboot, and the condition will be removed.</p>"},{"location":"user_workloads/vm_rollout_strategies/#limitations","title":"Limitations","text":"<p>The current implementation has the following limitations:</p> <ul> <li>Once the <code>RestartRequired</code> condition is set, the only way to get rid of it is to restart the VM. In the future, we plan on implementing a way to get rid of it by reverting the VM template spec to its last non-RestartRequired state.</li> <li>Cluster defaults are excluded from this logic. It means that changing a cluster-wide setting that impacts VM specs will not be live-updated, regardless of the rollout strategy.</li> <li>The <code>RestartRequired</code> condition comes with a message stating what kind of change triggered the condition (CPU/memory/other). That message pertains only to the first change that triggered the condition. Additional changes that would usually trigger the condition will just get staged and no additional <code>RestartRequired</code> condition will be added.</li> </ul>"},{"location":"user_workloads/windows_virtio_drivers/","title":"Windows virtio drivers","text":"<p>Purpose of this document is to explain how to install virtio drivers for Microsoft Windows running in a fully virtualized guest.</p>"},{"location":"user_workloads/windows_virtio_drivers/#do-i-need-virtio-drivers","title":"Do I need virtio drivers?","text":"<p>Yes. Without the virtio drivers, you cannot use paravirtualized hardware properly. It would either not work, or will have a severe performance penalty.</p> <p>For more information about VirtIO and paravirtualization, see VirtIO and paravirtualization</p> <p>For more details on configuring your VirtIO driver please refer to Installing VirtIO driver on a new Windows virtual machine and Installing VirtIO driver on an existing Windows virtual machine.</p>"},{"location":"user_workloads/windows_virtio_drivers/#which-drivers-i-need-to-install","title":"Which drivers I need to install?","text":"<p>There are usually up to 8 possible devices that are required to run Windows smoothly in a virtualized environment. KubeVirt currently supports only:</p> <ul> <li> <p>viostor, the block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>viorng, the entropy source driver, applies to PCI Device in the     Other devices group.</p> </li> <li> <p>NetKVM, the network driver, applies to Ethernet Controller in     the Other devices group. Available only if a virtio NIC is     configured.</p> </li> </ul> <p>Other virtio drivers, that exists and might be supported in the future:</p> <ul> <li> <p>Balloon, the balloon driver, applies to PCI Device in the Other     devices group</p> </li> <li> <p>vioserial, the paravirtual serial driver, applies to PCI Simple     Communications Controller in the Other devices group.</p> </li> <li> <p>vioscsi, the SCSI block driver, applies to SCSI Controller in the     Other devices group.</p> </li> <li> <p>qemupciserial, the emulated PCI serial driver, applies to PCI Serial     Port in the Other devices group.</p> </li> <li> <p>qxl, the paravirtual video driver, applied to Microsoft Basic     Display Adapter in the Display adapters group.</p> </li> <li> <p>pvpanic, the paravirtual panic driver, applies to Unknown device in     the Other devices group.</p> </li> </ul> <p>Note</p> <p>Some drivers are required in the installation phase. When you are installing Windows onto the virtio block storage you have to provide an appropriate virtio driver. Namely, choose viostor driver for your version of Microsoft Windows, eg. does not install XP driver when you run Windows 10.</p> <p>Other drivers can be installed after the successful windows installation. Again, please install only drivers matching your Windows version.</p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-install-during-windows-install","title":"How to install during Windows install?","text":"<p>To install drivers before the Windows starts its install, make sure you have virtio-win package attached to your VirtualMachine as SATA CD-ROM. In the Windows installation, choose advanced install and load driver. Then please navigate to loaded Virtio CD-ROM and install one of viostor or vioscsi, depending on whichever you have set up.</p> <p>Step by step screenshots:</p> <p></p> <p></p> <p></p> <p></p> <p></p> <p></p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-install-after-windows-install","title":"How to install after Windows install?","text":"<p>After windows install, please go to Device Manager. There you should see undetected devices in \"available devices\" section. You can install virtio drivers one by one going through this list.</p> <p></p> <p></p> <p></p> <p></p> <p>For more details on how to choose a proper driver and how to install the driver, please refer to the Windows Guest Virtual Machines on Red Hat Enterprise Linux 7.</p>"},{"location":"user_workloads/windows_virtio_drivers/#how-to-obtain-virtio-drivers","title":"How to obtain virtio drivers?","text":"<p>The virtio Windows drivers are distributed in a form of containerDisk, which can be simply mounted to the VirtualMachine. The container image, containing the disk is located at: https://quay.io/repository/kubevirt/virtio-container-disk?tab=tags and the image be pulled as any other docker container:</p> <pre><code>docker pull quay.io/kubevirt/virtio-container-disk\n</code></pre> <p>However, pulling image manually is not required, it will be downloaded if not present by Kubernetes when deploying VirtualMachine.</p>"},{"location":"user_workloads/windows_virtio_drivers/#attaching-to-virtualmachine","title":"Attaching to VirtualMachine","text":"<p>KubeVirt distributes virtio drivers for Microsoft Windows in a form of container disk. The package contains the virtio drivers and QEMU guest agent. The disk was tested on Microsoft Windows Server 2012. Supported Windows version is XP and up.</p> <p>The package is intended to be used as CD-ROM attached to the virtual machine with Microsoft Windows. It can be used as SATA CDROM during install phase or to provide drivers in an existing Windows installation.</p> <p>Attaching the virtio-win package can be done simply by adding ContainerDisk to you VirtualMachine.</p> <pre><code>spec:\n  domain:\n    devices:\n      disks:\n        - name: virtiocontainerdisk\n          # Any other disk you want to use, must go before virtioContainerDisk.\n          # KubeVirt boots from disks in order ther are defined.\n          # Therefore virtioContainerDisk, must be after bootable disk.\n          # Other option is to choose boot order explicitly:\n          #  - https://kubevirt.io/api-reference/v0.13.2/definitions.html#_v1_disk\n          # NOTE: You either specify bootOrder explicitely or sort the items in\n          #       disks. You can not do both at the same time.\n          # bootOrder: 2\n          cdrom:\n            bus: sata\nvolumes:\n  - containerDisk:\n      image: quay.io/kubevirt/virtio-container-disk\n    name: virtiocontainerdisk\n</code></pre> <p>Once you are done installing virtio drivers, you can remove virtio container disk by simply removing the disk from yaml specification and restarting the VirtualMachine.</p>"}]}
\ No newline at end of file