Error installing on Azure Kubernetes Service

[karthikpawar]

What happened:
I have followed the given installation steps in the installation guide but the pods in the 'kubevirt' namespace are stuck in "CrashLoopBack".

The installation steps are:

export VERSION=$(curl -s https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
kubectl create -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml"
kubectl create -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml"

The output of the logs of the "virt-operator" pods is:

W1109 13:52:17.924128       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W1109 13:52:17.924453       1 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"component":"virt-operator","level":"info","msg":"we are on kubernetes","pos":"application.go:241","timestamp":"2024-11-09T13:52:17.952873Z"}
{"component":"virt-operator","level":"info","msg":"servicemonitor is not defined","pos":"application.go:259","timestamp":"2024-11-09T13:52:17.958575Z"}
{"component":"virt-operator","level":"info","msg":"prometheusrule is not defined","pos":"application.go:274","timestamp":"2024-11-09T13:52:17.965930Z"}
{"component":"virt-operator","level":"info","msg":"validatingAdmissionPolicyBindingEnabled is not defined","pos":"application.go:289","timestamp":"2024-11-09T13:52:17.973928Z"}
{"component":"virt-operator","level":"info","msg":"validatingAdmissionPolicyEnabled is not defined","pos":"application.go:304","timestamp":"2024-11-09T13:52:17.979419Z"}
{"component":"virt-operator","level":"info","msg":"Operator image: quay.io/kubevirt/virt-operator:v1.3.1","pos":"application.go:321","timestamp":"2024-11-09T13:52:17.980380Z"}
{"component":"virt-operator","level":"info","msg":"set log verbosity to 2","pos":"application.go:533","timestamp":"2024-11-09T13:52:17.980469Z"}
{"component":"virt-operator","level":"info","msg":"set log verbosity to 2","pos":"application.go:533","timestamp":"2024-11-09T13:52:17.980494Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorCRDInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.980704Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorServiceInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.980890Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorServiceMonitor","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.980917Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorClusterRoleInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981008Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorClusterRoleBindingInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981084Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorRoleBindingInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981109Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorMutatingWebhookInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981121Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer secretsInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981134Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorConfigMapInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981145Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer vmClusterInstancetypeInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981161Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981180Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer installStrategyJobsInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981190Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorAPIServiceInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981202Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer installStrategyConfigMapInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981213Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorPodDisruptionBudgetInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981230Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer namespaceInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981240Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorRoleInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981255Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorDeploymentInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981267Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorPodsInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981277Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981292Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorDaemonSetInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981304Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer vmClusterPreferenceInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981322Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorRoute","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981333Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorPrometheusRuleInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981345Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorValidatingAdmissionPolicyInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981362Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorServiceAccountInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981372Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorValidatingWebhookInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981385Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorSCC","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981396Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer FakeOperatorValidatingAdmissionPolicyBindingInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981410Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer extensionsConfigMapInformer","pos":"virtinformers.go:363","timestamp":"2024-11-09T13:52:17.981460Z"}
{"component":"virt-operator","level":"info","msg":"set log verbosity to 2","pos":"application.go:533","timestamp":"2024-11-09T13:52:17.990076Z"}
W1109 13:52:17.995330       1 reflector.go:547] pkg/controller/virtinformers.go:364: failed to list *v1beta1.VirtualMachineClusterInstancetype: the server could not find the requested resource (get virtualmachineclusterinstancetypes.instancetype.kubevirt.io)
E1109 13:52:17.995441       1 reflector.go:150] pkg/controller/virtinformers.go:364: Failed to watch *v1beta1.VirtualMachineClusterInstancetype: failed to list *v1beta1.VirtualMachineClusterInstancetype: the server could not find the requested resource (get virtualmachineclusterinstancetypes.instancetype.kubevirt.io)
W1109 13:52:17.995706       1 reflector.go:547] pkg/controller/virtinformers.go:364: failed to list *v1beta1.VirtualMachineClusterPreference: the server could not find the requested resource (get virtualmachineclusterpreferences.instancetype.kubevirt.io)
E1109 13:52:17.995824       1 reflector.go:150] pkg/controller/virtinformers.go:364: Failed to watch *v1beta1.VirtualMachineClusterPreference: failed to list *v1beta1.VirtualMachineClusterPreference: the server could not find the requested resource (get virtualmachineclusterpreferences.instancetype.kubevirt.io)
{"component":"virt-operator","level":"info","msg":"set log verbosity to 2","pos":"application.go:533","timestamp":"2024-11-09T13:52:18.164122Z"}
{"component":"virt-operator","level":"info","msg":"Attempting to acquire leader status","pos":"application.go:469","timestamp":"2024-11-09T13:52:18.181823Z"}
I1109 13:52:18.181885       1 leaderelection.go:250] attempting to acquire leader lease kubevirt/virt-operator...
{"component":"virt-operator","level":"info","msg":"set log verbosity to 2","pos":"application.go:533","timestamp":"2024-11-09T13:52:18.182042Z"}
{"component":"virt-operator","level":"info","msg":"Reinitialize virt-operator, ServiceMonitor has been introduced","pos":"application.go:484","timestamp":"2024-11-09T13:52:18.182175Z"}

The output of kubectl get crds | grep "kubevirt" is

kubevirts.kubevirt.io                                 2024-11-09T14:37:33Z

The output of kubectl get apiservices is :

NAME                                   SERVICE                      AVAILABLE   AGE
v1.                                    Local                        True        119m
v1.admissionregistration.k8s.io        Local                        True        119m
v1.apiextensions.k8s.io                Local                        True        119m
v1.apps                                Local                        True        119m
v1.authentication.k8s.io               Local                        True        119m
v1.authorization.k8s.io                Local                        True        119m
v1.autoscaling                         Local                        True        119m
v1.azmonitoring.coreos.com             Local                        True        108m
v1.batch                               Local                        True        119m
v1.certificates.k8s.io                 Local                        True        119m
v1.coordination.k8s.io                 Local                        True        119m
v1.crd.projectcalico.org               Local                        True        119m
v1.discovery.k8s.io                    Local                        True        119m
v1.eraser.sh                           Local                        True        117m
v1.events.k8s.io                       Local                        True        119m
v1.flowcontrol.apiserver.k8s.io        Local                        True        119m
v1.kubevirt.io                         Local                        True        101s
v1.networking.k8s.io                   Local                        True        119m
v1.node.k8s.io                         Local                        True        119m
v1.operator.tigera.io                  Local                        True        119m
v1.policy                              Local                        True        119m
v1.rbac.authorization.k8s.io           Local                        True        119m
v1.scheduling.k8s.io                   Local                        True        119m
v1.snapshot.storage.k8s.io             Local                        True        119m
v1.storage.k8s.io                      Local                        True        119m
v1alpha1.eraser.sh                     Local                        True        117m
v1alpha3.kubevirt.io                   Local                        True        101s
v1beta1.admissionregistration.k8s.io   Local                        True        119m
v1beta1.authentication.k8s.io          Local                        True        119m
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        119m
v1beta3.flowcontrol.apiserver.k8s.io   Local                        True        119m
v2.autoscaling                         Local                        True        119m

What you expected to happen:
The usual deployment of Kubernetes as mentioned in the docs.

How to reproduce it (as minimally and precisely as possible):

1. Create a AKS cluster with Kubernetes version v1.29
2. Follow Kubevirt installation steps

Additional context:
It works fine with self-managed Rancher k3s (Kubernetes v1.26)

Environment:

• KubeVirt version (use virtctl version): v1.3.1
• Kubernetes version (use kubectl version): v1.29.0
• VM or VMI specifications: N/A
• Cloud provider or hardware configuration: Azure (Azure Kubernetes Service)
• OS (e.g. from /etc/os-release): Linux
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: N/A