Kubevault CSI driver doesn't work on kubernetes v1.20+

[ghost]

I followed the documentation to set up kubevault with kubevault csi driver.

https://github.com/kubevault/docs/blob/master/docs/guides/secret-engines/kv/csi-driver.md

This works fine with Kubernetes v1.18 and v1.19 on minikube
When I try on kubernetes v1.20 the pvc seems to hanging the "Pending" state.

In the logs of csi-provisioner container in csi-vault-controller pod I see the following error:

E0309 14:48:19.255091 1 controller.go:1213] provision "trial/csi-pvc-kv" class "vault-kv-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
(see below) for the full output.

Is the CSI driver tested with kubernetes v1.20+?

If you need more information to debug this please don't hesitate me...

Kind regards,

--
staf

~$ kubectl get pvc -n trial
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-kv Pending vault-kv-storage 12m

$ kubectl logs csi-vault-controller-0 -n kube-system -c csi-attacher
I0309 14:46:21.750253 1 main.go:95] Version: v1.2.0-0-g834bae5
I0309 14:46:21.758848 1 connection.go:151] Connecting to unix:///csi/csi.sock
I0309 14:46:25.843549 1 common.go:111] Probing CSI driver for readiness
I0309 14:46:25.852523 1 main.go:145] CSI driver name: "secrets.csi.kubevault.com"
I0309 14:46:25.858138 1 main.go:172] CSI driver does not support ControllerPublishUnpublish, using trivial handler
I0309 14:46:25.859206 1 controller.go:113] Starting CSI attacher
I0309 14:46:25.867626 1 reflector.go:123] Starting reflector *v1beta1.VolumeAttachment (10m0s) from k8s.io/client-go/informers/factory.go:133
I0309 14:46:25.867760 1 reflector.go:161] Listing and watching *v1beta1.VolumeAttachment from k8s.io/client-go/informers/factory.go:133
I0309 14:46:25.868124 1 reflector.go:123] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:133
I0309 14:46:25.868188 1 reflector.go:161] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:133

~$ kubectl logs csi-vault-controller-0 -n kube-system -c csi-provisioner
I0309 14:46:02.540065 1 feature_gate.go:226] feature gates: &{map[]}
I0309 14:46:02.540173 1 csi-provisioner.go:98] Version: v1.3.0-0-g27750ab1
I0309 14:46:02.540201 1 csi-provisioner.go:112] Building kube configs for running in cluster...
I0309 14:46:02.551636 1 connection.go:151] Connecting to unix:///csi/csi.sock
W0309 14:46:12.551950 1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0309 14:46:22.551877 1 connection.go:170] Still connecting to unix:///csi/csi.sock
I0309 14:46:26.734729 1 connection.go:261] Probing CSI driver for readiness
I0309 14:46:26.740402 1 csi-provisioner.go:152] Detected CSI driver secrets.csi.kubevault.com
I0309 14:46:26.746470 1 controller.go:621] Using saving PVs to API server in background
I0309 14:46:26.747145 1 controller.go:769] Starting provisioner controller secrets.csi.kubevault.com_csi-vault-controller-0_39eacc6a-80e6-11eb-a6e1-0242ac110005!
I0309 14:46:26.747382 1 volume_store.go:90] Starting save volume queue
I0309 14:46:26.748205 1 reflector.go:123] Starting reflector *v1.PersistentVolumeClaim (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:800
I0309 14:46:26.748292 1 reflector.go:161] Listing and watching *v1.PersistentVolumeClaim from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:800
I0309 14:46:26.750226 1 reflector.go:123] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:803
I0309 14:46:26.750332 1 reflector.go:161] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:803
I0309 14:46:26.752297 1 reflector.go:123] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:806
I0309 14:46:26.752534 1 reflector.go:161] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:806
I0309 14:46:26.848196 1 controller.go:818] Started provisioner controller secrets.csi.kubevault.com_csi-vault-controller-0_39eacc6a-80e6-11eb-a6e1-0242ac110005!
I0309 14:48:19.247513 1 controller.go:979] Final error received, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress
I0309 14:48:19.247530 1 controller.go:902] Provisioning succeeded, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress
I0309 14:48:19.251624 1 controller.go:1196] provision "trial/csi-pvc-kv" class "vault-kv-storage": started
E0309 14:48:19.255091 1 controller.go:1213] provision "trial/csi-pvc-kv" class "vault-kv-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0309 14:48:19.255118 1 controller.go:979] Final error received, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress
I0309 14:48:19.255126 1 controller.go:902] Provisioning succeeded, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress
I0309 15:01:26.765440 1 controller.go:1196] provision "trial/csi-pvc-kv" class "vault-kv-storage": started
E0309 15:01:26.774916 1 controller.go:1213] provision "trial/csi-pvc-kv" class "vault-kv-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I0309 15:01:26.774956 1 controller.go:979] Final error received, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress
I0309 15:01:26.774969 1 controller.go:902] Provisioning succeeded, removing PVC d5d4f7e9-7350-413e-aec1-3ca616eb10cf from claims in progress