Disaster recovery of an on-prem cluster to GKE - failed to provision volume with StorageClass "pd-velero": claim Selector is not supported

[oboudry-mvp]

Hi all,

I have an on-prem K8S cluster with rook-ceph storage, and am successfully using velero as the backup solution. My backup target location is an on-prem Minio service. Snapshots are moved using Kopia to the same minio bucket.

I would like to use GKE as a DR solution. I already successfully did the following steps:

• Replicate the Minio bucket to Google Storage using rclone
• Deploy GKE using Terraform (following https://developer.hashicorp.com/terraform/tutorials/kubernetes/gke)
• Deploy velero in my GKE cloud K8S
• Restore all K8S objects from the backup

But the restore of snapshots fail with this error message: failed to provision volume with StorageClass "pd-velero": claim Selector is not supported

The On-Prem velero deployment was performed using the following instructions:

$ velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.8.0,velero/velero-plugin-for-csi:v0.6.1 \
    --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=true \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.77.201:9000 \
    --use-node-agent --snapshot-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.77.201:9000 \
    --features=EnableCSI

# VolumeSnapshotClass labelled for velero
apiVersion: v1
items:
- apiVersion: snapshot.storage.k8s.io/v1
  deletionPolicy: Retain
  driver: rook-ceph.rbd.csi.ceph.com
  kind: VolumeSnapshotClass
  metadata:
    annotations:
      velero.io/csi-volumesnapshot-class: "true"
    creationTimestamp: "2023-11-10T11:15:06Z"
    generation: 2
    labels:
      velero.io/csi-volumesnapshot-class: "true"
    name: csi-rbdplugin-snapclass
    resourceVersion: "3009450"
    uid: 9c4ca2ee-4387-4154-a92c-0411bd50f23f
  parameters:
    clusterID: rook-ceph
    csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph
kind: List
metadata:
  resourceVersion: ""

The GKE velero deployment was performed using the following instructions:

$ velero install \
    --provider gcp \
    --plugins velero/velero-plugin-for-csi:v0.6.1,velero/velero-plugin-for-gcp:v1.6.0 \
    --bucket marvinpac-velero-replica \
    --secret-file ~/credentials-velero-gke \
    --features=EnableCSI

# Apply of the follwoing:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-gce-vsc
  labels:
    "velero.io/csi-volumesnapshot-class": "true"
driver: pd.csi.storage.gke.io
deletionPolicy: Delete
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pd-velero
provisioner: pd.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: pd-standard
---
apiVersion: v1
kind: ConfigMap
metadata:
  # any name can be used; Velero uses the labels (below)
  # to identify it rather than the name
  name: change-storage-class-config
  # must be in the velero namespace
  namespace: velero
  # the below labels should be used verbatim in your
  # ConfigMap.
  labels:
    # this value-less label identifies the ConfigMap as
    # config for a plugin (i.e. the built-in restore item action plugin)
    velero.io/plugin-config: ""
    # this label identifies the name and kind of plugin
    # that this ConfigMap is for.
    velero.io/change-storage-class: RestoreItemAction
data:
  # add 1+ key-value pairs here, where the key is the old
  # storage class name and the value is the new storage
  # class name.
  rook-ceph-block: pd-velero
  rook-cephfs: pd-velero

On restore it stucks on "WaitingForPluginOperations"

$ velero restore describe corteza-from-daily-backup-all-but-system-20240321030049
Name:         corteza-from-daily-backup-all-but-system-20240321030049
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       WaitingForPluginOperations
Total items to be restored:  44
Items restored:              44

Started:    2024-03-22 08:20:36 +0000 UTC
Completed:  <n/a>

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    corteza:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version

Backup:  daily-backup-all-but-system-20240321030049

Namespaces:
  Included:  corteza
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

Restore Item Operations:  0 of 2 completed successfully, 0 failed (specify --details for more information)

HooksAttempted:   0
HooksFailed:      0

The PVCs are pending:

$ k describe pvc corteza-pv-claim
Name:          corteza-pv-claim
Namespace:     corteza
StorageClass:  pd-velero
Status:        Pending
Volume:
Labels:        app=corteza
               app.kubernetes.io/instance=corteza
               velero.io/backup-name=daily-backup-all-but-system-20240321030049
               velero.io/restore-name=corteza-from-daily-backup-all-but-system-20240321030049
               velero.io/volume-snapshot-name=velero-corteza-pv-claim-d4v64
Annotations:   backup.velero.io/must-include-additional-items: true
               volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
               volume.kubernetes.io/selected-node: gke-archives-storage-archives-storage-20cce987-49kb
               volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       corteza-c8dd499dd-nwlqs
Events:
  Type     Reason                Age                 From                                                                                              Message
  ----     ------                ----                ----                                                                                              -------
  Normal   WaitForFirstConsumer  40m (x2 over 40m)   persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Warning  ProvisioningFailed    11m (x15 over 40m)  pd.csi.storage.gke.io_gke-654746d25bc249b4b780-d847-04d1-vm_c88c7c7d-f9d5-4c0c-ad33-220de43d4a9c  failed to provision volume with StorageClass "pd-velero": claim Selector is not supported
  Normal   Provisioning          97s (x18 over 40m)  pd.csi.storage.gke.io_gke-654746d25bc249b4b780-d847-04d1-vm_c88c7c7d-f9d5-4c0c-ad33-220de43d4a9c  External provisioner is provisioning volume for claim "corteza/corteza-pv-claim"
  Normal   ExternalProvisioning  4s (x162 over 40m)  persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator

The PVs are not created, and in GKE I see the following log entry:

2024-03-22 09:24:55.000 HNEC
failed to provision volume with StorageClass "pd-velero": claim Selector is not supported
{
  insertId: "81po20fi7d1gl"
  jsonPayload: {
  apiVersion: "v1"
  eventTime: null
  involvedObject: {6}
  kind: "Event"
  lastTimestamp: "2024-03-22T08:24:55Z"
  message: "failed to provision volume with StorageClass "pd-velero": claim Selector is not supported"
  metadata: {6}
  reason: "ProvisioningFailed"
  reportingComponent: "pd.csi.storage.gke.io_gke-654746d25bc249b4b780-d847-04d1-vm_c88c7c7d-f9d5-4c0c-ad33-220de43d4a9c"
  reportingInstance: ""
  source: {1}
  type: "Warning"
}
logName: "projects/archives-storage/logs/events"
receiveTimestamp: "2024-03-22T08:25:00.825082410Z"
resource: {2}
severity: "WARNING"
timestamp: "2024-03-22T08:24:55Z"
}

Versions of K8S and Velero are as follows:

On GKE:

$ velero version
Client:
        Version: v1.13.1
        Git commit: ea5a89f83b89b2cb7a27f54148683c1ee8d57a37
Server:
        Version: v1.13.1
$ k version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.10
Kustomize Version: v5.0.1
Server Version: v1.27.8-gke.1067004

On-Site:

$ velero version                          Client:
        Version: v1.13.1
        Git commit: ea5a89f83b89b2cb7a27f54148683c1ee8d57a37
Server:
        Version: v1.13.1
$ k version --short                       Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.10
Kustomize Version: v5.0.1
Server Version: v1.27.10

I've not versed into the technicals of velero, I'm just a user. But I have the impression this is due to a limitation of GKE which does not accept the selector for linking PVC and PV. Am I right or wrong, and is there a workaround ?

Thanks for your help,

Olivier.

[draghuram]

I see that you are installing gcp plug-in1.6.0. What version of Velero are you installing there? My guess is that it is 1.12 because other wise, snapshot data mover backups are not supported. In that case, you need 1.8 version of GCP plug-in. More over, I don't see "--use-node-agent" option. In general, make sure that Velero version matches plug-inversions on both source and target.

Finally, you should use storage class mapping to map source PVC storage class to what is available in GCP (you may already be doing this).

• Raghu (CloudCasa)

[oboudry-mvp]

Hi Raghu,

I updated the gcp plugin version to v1.8.1, and added --use-node-agent and it now works like a charm.

Thank you very much for your help!

Olivier.

[draghuram]

Great. Glad to be of help.