Merge pull request rook#13622 from Madhu-1/csi-name-prefix

csi: option to customize csi driver name prefix

Documentation/Helm-Charts/ceph-cluster-chart.md

@@ -69,6 +69,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `cephObjectStores` | A list of CephObjectStore configurations to deploy | See [below](#ceph-object-stores) |
 | `clusterName` | The metadata.name of the CephCluster CR | The same as the namespace |
 | `configOverride` | Cluster ceph.conf override | `nil` |
+| `csiDriverNamePrefix` | CSI driver name prefix for cephfs, rbd and nfs. | `namespace name where rook-ceph operator is deployed` |
 | `ingress.dashboard` | Enable an ingress for the ceph-dashboard | `{}` |
 | `kubeVersion` | Optional override of the target kubernetes version | `nil` |
 | `monitoring.createPrometheusRules` | Whether to create the Prometheus rules for Ceph alerts | `false` |

Documentation/Helm-Charts/operator-chart.md

@@ -70,6 +70,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.csiCephFSPluginVolume` | The volume of the CephCSI CephFS plugin DaemonSet | `nil` |
 | `csi.csiCephFSPluginVolumeMount` | The volume mounts of the CephCSI CephFS plugin DaemonSet | `nil` |
 | `csi.csiCephFSProvisionerResource` | CEPH CSI CephFS provisioner resource requirement list | see values.yaml |
+| `csi.csiDriverNamePrefix` | CSI driver name prefix for cephfs, rbd and nfs. | `namespace name where rook-ceph operator is deployed` |
 | `csi.csiLeaderElectionLeaseDuration` | Duration in seconds that non-leader candidates will wait to force acquire leadership. | `137s` |
 | `csi.csiLeaderElectionRenewDeadline` | Deadline in seconds that the acting leader will retry refreshing leadership before giving up. | `107s` |
 | `csi.csiLeaderElectionRetryPeriod` | Retry period in seconds the LeaderElector clients should wait between tries of actions. | `26s` |

Documentation/Storage-Configuration/Advanced/ceph-configuration.md

@@ -45,14 +45,17 @@ sed -i.bak \
     -e "s/\(.*\):.*# namespace:cluster/\1: $ROOK_CLUSTER_NAMESPACE # namespace:cluster/g" \
     -e "s/\(.*serviceaccount\):.*:\(.*\) # serviceaccount:namespace:operator/\1:$ROOK_OPERATOR_NAMESPACE:\2 # serviceaccount:namespace:operator/g" \
     -e "s/\(.*serviceaccount\):.*:\(.*\) # serviceaccount:namespace:cluster/\1:$ROOK_CLUSTER_NAMESPACE:\2 # serviceaccount:namespace:cluster/g" \
-    -e "s/\(.*\): [-_A-Za-z0-9]*\.\(.*\) # driver:namespace:operator/\1: $ROOK_OPERATOR_NAMESPACE.\2 # driver:namespace:operator/g" \
     -e "s/\(.*\): [-_A-Za-z0-9]*\.\(.*\) # driver:namespace:cluster/\1: $ROOK_CLUSTER_NAMESPACE.\2 # driver:namespace:cluster/g" \
   common.yaml operator.yaml cluster.yaml # add other files or change these as desired for your config
 
 # You need to use `apply` for all Ceph clusters after the first if you have only one Operator
 kubectl apply -f common.yaml -f operator.yaml -f cluster.yaml # add other files as desired for yourconfig
 ```
 
+Also see the CSI driver
+[documentation](../Ceph-CSI/ceph-csi-drivers.md#Configure-CSI-Drivers-in-non-default-namespace)
+to update the csi provisioner names in the storageclass and volumesnapshotclass.
+
 ## Deploying a second cluster
 
 If you wish to create a new CephCluster in a separate namespace, you can easily do so

Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md

@@ -35,6 +35,59 @@ example, if the Rook operator is running in the namespace `my-namespace` the
 provisioner value should be `my-namespace.rbd.csi.ceph.com`. The same provisioner
 name must be set in both the storageclass and snapshotclass.
 
+To find the provisioner name in the example storageclasses and
+volumesnapshotclass, search for: `# csi-provisioner-name`
+
+### Configure custom Driver name prefix for CSI Drivers
+
+To use a custom prefix for the CSI drivers instead of the namespace prefix, set
+the `CSI_DRIVER_NAME_PREFIX` environment variable in the operator configmap.
+For instance, to use the prefix `my-prefix` for the CSI drivers, set
+the following in the operator configmap:
+
+```console
+kubectl patch cm rook-ceph-operator-config -n rook-ceph -p $'data:\n "CSI_DRIVER_NAME_PREFIX": "my-prefix"'
+```
+
+Once the configmap is updated, the CSI drivers will be deployed with the
+`my-prefix` prefix. The same prefix must be set in both the storageclass and
+snapshotclass. For example, to use the prefix `my-prefix` for the
+CSI drivers, update the provisioner in the storageclass:
+
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: rook-ceph-block-sc
+provisioner: my-prefix.rbd.csi.ceph.com
+...
+```
+
+The same prefix must be set in the volumesnapshotclass as well:
+
+```yaml
+apiVersion: snapshot.storage.k8s.io/v1
+kind: VolumeSnapshotClass
+metadata:
+  name: rook-ceph-block-vsc
+driver: my-prefix.rbd.csi.ceph.com
+...
+```
+
+When the prefix is set, the driver names will be:
+
+* RBD: `my-prefix.rbd.csi.ceph.com`
+* CephFS: `my-prefix.cephfs.csi.ceph.com`
+* NFS: `my-prefix.nfs.csi.ceph.com`
+
+!!! note
+    Please be careful when setting the `CSI_DRIVER_NAME_PREFIX`
+    environment variable. It should be done only in fresh deployments because
+    changing the prefix in an existing cluster will result in unexpected behavior.
+
+To find the provisioner name in the example storageclasses and
+volumesnapshotclass, search for: `# csi-provisioner-name`
+
 ## Liveness Sidecar
 
 All CSI pods are deployed with a sidecar container that provides a Prometheus
@@ -116,7 +169,6 @@ Refer to the [ephemeral-doc](https://kubernetes.io/docs/concepts/storage/ephemer
 See example manifests for an [RBD ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/pod-ephemeral.yaml)
 and a [CephFS ephemeral volume](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/pod-ephemeral.yaml).
 
-
 ## CSI-Addons Controller
 
 The CSI-Addons Controller handles requests from users. Users create a CR

deploy/charts/rook-ceph-cluster/templates/cephblockpool.yaml

@@ -16,7 +16,11 @@ metadata:
   name: {{ $blockpool.storageClass.name }}
   annotations:
     storageclass.kubernetes.io/is-default-class: "{{ if default false $blockpool.storageClass.isDefault }}true{{ else }}false{{ end }}"
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
 provisioner: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   pool: {{ $blockpool.name }}
   clusterID: {{ $root.Release.Namespace }}

deploy/charts/rook-ceph-cluster/templates/cephecblockpool.yaml

@@ -16,7 +16,11 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: {{ $cephEcStorage.name }}
-provisioner: {{ $cephEcStorage.provisioner }}
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
+provisioner: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ $cephEcStorage.parameters.clusterID }}
   dataPool: {{ $cephEcStorage.parameters.dataPool }}

deploy/charts/rook-ceph-cluster/templates/cephfilesystem.yaml

@@ -35,7 +35,11 @@ metadata:
   name: {{ $filesystem.storageClass.name }}
   annotations:
     storageclass.kubernetes.io/is-default-class: "{{ if default false $filesystem.storageClass.isDefault }}true{{ else }}false{{ end }}"
+{{- if $root.Values.csiDriverNamePrefix }}
+provisioner: {{ $root.Values.csiDriverNamePrefix }}.cephfs.csi.ceph.com
+{{- else }}
 provisioner: {{ $root.Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- end }}
 parameters:
   fsName: {{ $filesystem.name }}
   pool: {{ $filesystem.name }}-{{ default "data0" $filesystem.storageClass.pool }}

deploy/charts/rook-ceph-cluster/templates/volumesnapshotclass.yaml

@@ -1,3 +1,4 @@
+{{- $root := . -}}
 {{- $filesystemvsc := .Values.cephFileSystemVolumeSnapshotClass -}}
 {{- $blockpoolvsc := .Values.cephBlockPoolsVolumeSnapshotClass -}}
 
@@ -16,7 +17,11 @@ metadata:
 {{- if $filesystemvsc.annotations }}
 {{ toYaml $filesystemvsc.annotations | indent 4 }}
 {{- end }}
-driver: {{ .Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- if $root.Values.csiDriverNamePrefix }}
+driver: {{ $root.Values.csiDriverNamePrefix }}.cephfs.csi.ceph.com
+{{- else }}
+driver: {{ $root.Values.operatorNamespace }}.cephfs.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ .Release.Namespace }}
   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
@@ -42,7 +47,11 @@ metadata:
 {{- if $blockpoolvsc.annotations }}
 {{ toYaml $blockpoolvsc.annotations | indent 4 }}
 {{- end }}
-driver: {{ .Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- if $root.Values.csiDriverNamePrefix }}
+driver: {{ $root.Values.csiDriverNamePrefix }}.rbd.csi.ceph.com
+{{- else }}
+driver: {{ $root.Values.operatorNamespace }}.rbd.csi.ceph.com
+{{- end }}
 parameters:
   clusterID: {{ .Release.Namespace }}
   csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner

deploy/charts/rook-ceph-cluster/values.yaml

@@ -647,8 +647,6 @@ cephObjectStores:
 # if cephECBlockPools are uncommented you must remove the comments of cephEcStorageClass as well
 #cephECStorageClass:
 #  name: rook-ceph-block
-#  # Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-#  provisioner: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
 #  parameters:
 #    # clusterID is the namespace where the rook cluster is running
 #    # If you change this namespace, also change the namespace below where the secret namespaces are defined
@@ -687,3 +685,7 @@ cephObjectStores:
 #    imageFeatures: layering
 #  allowVolumeExpansion: true
 #  reclaimPolicy: Delete
+
+# -- CSI driver name prefix for cephfs, rbd and nfs.
+# @default -- `namespace name where rook-ceph operator is deployed`
+csiDriverNamePrefix:

deploy/charts/rook-ceph/templates/configmap.yaml

@@ -25,6 +25,9 @@ data:
   CSI_ENABLE_OMAP_GENERATOR: {{ .Values.csi.enableOMAPGenerator | quote }}
   CSI_ENABLE_HOST_NETWORK: {{ .Values.csi.enableCSIHostNetwork | quote }}
   CSI_ENABLE_METADATA: {{ .Values.csi.enableMetadata | quote }}
+{{- if .Values.csi.csiDriverNamePrefix }}
+  CSI_DRIVER_NAME_PREFIX: {{ .Values.csi.csiDriverNamePrefix | quote }}
+{{- end }}
 {{- if .Values.csi.pluginPriorityClassName }}
   CSI_PLUGIN_PRIORITY_CLASSNAME: {{ .Values.csi.pluginPriorityClassName | quote }}
 {{- end }}

deploy/charts/rook-ceph/values.yaml

@@ -147,6 +147,10 @@ csi:
   # @default -- `0`
   sidecarLogLevel:
 
+  # -- CSI driver name prefix for cephfs, rbd and nfs.
+  # @default -- `namespace name where rook-ceph operator is deployed`
+  csiDriverNamePrefix:
+
   # -- CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate
   # @default -- `RollingUpdate`
   rbdPluginUpdateStrategy:

deploy/examples/csi/cephfs/snapshotclass.yaml

@@ -3,7 +3,7 @@ apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshotClass
 metadata:
   name: csi-cephfsplugin-snapclass
-driver: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator
+driver: rook-ceph.cephfs.csi.ceph.com # csi-provisioner-name
 parameters:
   # Specify a string that identifies your cluster. Ceph CSI supports any
   # unique string. When Ceph CSI is deployed by Rook use the Rook namespace,

deploy/examples/csi/cephfs/storageclass-ec.yaml

@@ -2,8 +2,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-cephfs
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator
+provisioner: rook-ceph.cephfs.csi.ceph.com # csi-provisioner-name
 parameters:
   # clusterID is the namespace where the rook cluster is running
   # If you change this namespace, also change the namespace below where the secret namespaces are defined

deploy/examples/csi/cephfs/storageclass.yaml

@@ -2,8 +2,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-cephfs
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator
+provisioner: rook-ceph.cephfs.csi.ceph.com # csi-provisioner-name
 parameters:
   # clusterID is the namespace where the rook cluster is running
   # If you change this namespace, also change the namespace below where the secret namespaces are defined

deploy/examples/csi/nfs/snapshotclass.yaml

@@ -3,7 +3,7 @@ apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshotClass
 metadata:
   name: csi-nfsplugin-snapclass
-driver: rook-ceph.nfs.csi.ceph.com # driver:namespace:operator
+driver: rook-ceph.nfs.csi.ceph.com # csi-provisioner-name
 parameters:
   # Specify a string that identifies your cluster. Ceph CSI supports any
   # unique string. When Ceph CSI is deployed by Rook use the Rook namespace,

deploy/examples/csi/nfs/storageclass.yaml

@@ -2,8 +2,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-nfs
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.nfs.csi.ceph.com
+provisioner: rook-ceph.nfs.csi.ceph.com # csi-provisioner-name
 parameters:
   # nfsCluster is the name of the NFS cluster as managed by Ceph (sometimes called the NFS cluster ID).
   # With Rook, this should get the name of the CephNFS resource.

deploy/examples/csi/rbd/snapshotclass.yaml

@@ -3,7 +3,7 @@ apiVersion: snapshot.storage.k8s.io/v1
 kind: VolumeSnapshotClass
 metadata:
   name: csi-rbdplugin-snapclass
-driver: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
+driver: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
 parameters:
   # Specify a string that identifies your cluster. Ceph CSI supports any
   # unique string. When Ceph CSI is deployed by Rook use the Rook namespace,

deploy/examples/csi/rbd/storageclass-ec.yaml

@@ -30,8 +30,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-ceph-block
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
+provisioner: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
 parameters:
   # clusterID is the namespace where the rook cluster is running
   # If you change this namespace, also change the namespace below where the secret namespaces are defined

deploy/examples/csi/rbd/storageclass-test.yaml

@@ -18,8 +18,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-ceph-block
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.rbd.csi.ceph.com # driver:namespace:operator
+provisioner: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
 parameters:
   # clusterID is the namespace where the rook cluster is running
   # If you change this namespace, also change the namespace below where the secret namespaces are defined

deploy/examples/csi/rbd/storageclass.yaml

@@ -18,8 +18,7 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: rook-ceph-block
-# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
-provisioner: rook-ceph.rbd.csi.ceph.com
+provisioner: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
 parameters:
   # clusterID is the namespace where the rook cluster is running
   # If you change this namespace, also change the namespace below where the secret namespaces are defined
@@ -47,14 +46,14 @@ parameters:
   # https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
   # unmapOptions: force
 
-   # (optional) Set it to true to encrypt each volume with encryption keys
-   # from a key management system (KMS)
-   # encrypted: "true"
+  # (optional) Set it to true to encrypt each volume with encryption keys
+  # from a key management system (KMS)
+  # encrypted: "true"
 
-   # (optional) Use external key management system (KMS) for encryption key by
-   # specifying a unique ID matching a KMS ConfigMap. The ID is only used for
-   # correlation to configmap entry.
-   # encryptionKMSID: <kms-config-id>
+  # (optional) Use external key management system (KMS) for encryption key by
+  # specifying a unique ID matching a KMS ConfigMap. The ID is only used for
+  # correlation to configmap entry.
+  # encryptionKMSID: <kms-config-id>
 
   # RBD image format. Defaults to "2".
   imageFormat: "2"

deploy/examples/import-external-cluster.sh

@@ -22,8 +22,9 @@ RBD_STORAGE_CLASS_NAME=ceph-rbd
 CEPHFS_STORAGE_CLASS_NAME=cephfs
 ROOK_EXTERNAL_MONITOR_SECRET=mon-secret
 OPERATOR_NAMESPACE=rook-ceph                                 # default set to rook-ceph
-RBD_PROVISIONER=$OPERATOR_NAMESPACE".rbd.csi.ceph.com"       # driver:namespace:operator
-CEPHFS_PROVISIONER=$OPERATOR_NAMESPACE".cephfs.csi.ceph.com" # driver:namespace:operator
+CSI_DRIVER_NAME_PREFIX=${CSI_DRIVER_NAME_PREFIX:-$OPERATOR_NAMESPACE}
+RBD_PROVISIONER=$CSI_DRIVER_NAME_PREFIX".rbd.csi.ceph.com"       # csi-provisioner-name
+CEPHFS_PROVISIONER=$CSI_DRIVER_NAME_PREFIX=".cephfs.csi.ceph.com" # csi-provisioner-name
 CLUSTER_ID_RBD=$NAMESPACE
 CLUSTER_ID_CEPHFS=$NAMESPACE
 : "${ROOK_EXTERNAL_ADMIN_SECRET:=admin-secret}"

deploy/examples/operator-openshift.yaml

@@ -614,6 +614,12 @@ data:
   # (Optional) Retry period in seconds the LeaderElector clients should wait between tries of actions. Defaults to 26 seconds.
   # CSI_LEADER_ELECTION_RETRY_PERIOD: "26s"
 
+  # csi driver name prefix for cephfs, rbd and nfs. if not specified, default
+  # will be the namespace name where rook-ceph operator is deployed.
+  # search for `# csi-provisioner-name` in the storageclass and
+  # volumesnashotclass and update the name accordingly.
+  # CSI_DRIVER_NAME_PREFIX: "rook-ceph"
+
   # Rook Discover toleration. Will tolerate all taints with all keys.
   # (Optional) Rook Discover tolerations list. Put here list of taints you want to tolerate in YAML format.
   # DISCOVER_TOLERATIONS: |

deploy/examples/operator.yaml

@@ -61,6 +61,12 @@ data:
   # Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity.
   # CSI_SIDECAR_LOG_LEVEL: "0"
 
+  # csi driver name prefix for cephfs, rbd and nfs. if not specified, default
+  # will be the namespace name where rook-ceph operator is deployed.
+  # search for `# csi-provisioner-name` in the storageclass and
+  # volumesnashotclass and update the name accordingly.
+  # CSI_DRIVER_NAME_PREFIX: "rook-ceph"
+
   # Set replicas for csi provisioner deployment.
   CSI_PROVISIONER_REPLICAS: "2"
 

pkg/operator/ceph/csi/csi.go

@@ -320,5 +320,8 @@ func (r *ReconcileCSI) setParams(ver *version.Info) error {
 	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_ATTACH_REQUIRED", "true"), "false") {
 		CSIParam.NFSAttachRequired = false
 	}
+
+	CSIParam.DriverNamePrefix = k8sutil.GetValue(r.opConfig.Parameters, "CSI_DRIVER_NAME_PREFIX", r.opConfig.OperatorNamespace)
+
 	return nil
 }