Cannot create volumType: local volumes - Required value: Local volume requires node affinity

[travisgroth]

I'm trying to create local volumes instead of hostPath so that I can use Velero with my volumes but I'm running into an issue when the PVs are created.

Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    defaultVolumeType: local
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
    volume.kubernetes.io/selected-node: XXX.YYY.ZZZ
    volume.kubernetes.io/storage-provisioner: rancher.io/local-path
  finalizers:
  - kubernetes.io/pvc-protection
  name: config-tmp
  namespace: test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-path
  volumeMode: Filesystem
status:
  phase: Pending

Logs:

time="2024-11-19T12:22:57Z" level=info msg="create the helper pod helper-pod-delete-pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21 into local-path-storage"
I1119 12:22:57.863152       1 event.go:389] "Event occurred" object="test/config-tmp" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="Error creating provisioned PV object for claim test/config-tmp: PersistentVolume \"pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity. Deleting the volume."
time="2024-11-19T12:23:00Z" level=info msg="Volume pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21 has been deleted on /stor/k8s/ssd/local-path/pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21_test_config-tmp"
time="2024-11-19T12:23:00Z" level=info msg="Start of helper-pod-delete-pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21 logs"
time="2024-11-19T12:23:00Z" level=info msg="End of helper-pod-delete-pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21 logs"
I1119 12:23:00.961276       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21" failures=5 threshold=15
E1119 12:23:00.961352       1 controller.go:974] error syncing claim "e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21": PersistentVolume "pvc-e4f678d8-e5f2-4039-ac9b-2f6bfc7e3e21" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity
time="2024-11-19T12:23:50Z" level=info msg="Creating volume pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc at XXX.YYY.ZZZ:/stor/k8s/ssd/local-path/pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc_test_config-tmp"
time="2024-11-19T12:23:50Z" level=info msg="create the helper pod helper-pod-create-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc into local-path-storage"
I1119 12:23:50.466098       1 event.go:389] "Event occurred" object="test/config-tmp" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"test/config-tmp\""
time="2024-11-19T12:23:53Z" level=info msg="Volume pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc has been created on XXX.YYY.ZZZ:/stor/k8s/ssd/local-path/pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc_test_config-tmp"
time="2024-11-19T12:23:53Z" level=info msg="Start of helper-pod-create-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc logs"
time="2024-11-19T12:23:53Z" level=info msg="End of helper-pod-create-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc logs"
I1119 12:23:53.530549       1 volume_store.go:226] "Failed to save persistentvolume" PVC="test/config-tmp" StorageClass="local-path" persistentvolume="pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity"
I1119 12:24:03.536243       1 volume_store.go:226] "Failed to save persistentvolume" PVC="test/config-tmp" StorageClass="local-path" persistentvolume="pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity"
I1119 12:24:13.540896       1 volume_store.go:226] "Failed to save persistentvolume" PVC="test/config-tmp" StorageClass="local-path" persistentvolume="pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity"
I1119 12:24:23.545404       1 volume_store.go:226] "Failed to save persistentvolume" PVC="test/config-tmp" StorageClass="local-path" persistentvolume="pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity"
I1119 12:24:33.550386       1 volume_store.go:226] "Failed to save persistentvolume" PVC="test/config-tmp" StorageClass="local-path" persistentvolume="pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity"
E1119 12:24:33.550478       1 volume_store.go:242] "Error creating provisioned PV object for claim. Deleting the volume." err="PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity" PVC="test/config-tmp" StorageClass="local-path" claim="test/config-tmp"
time="2024-11-19T12:24:33Z" level=info msg="Deleting volume pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc at /stor/k8s/ssd/local-path/pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc_test_config-tmp"
time="2024-11-19T12:24:33Z" level=info msg="create the helper pod helper-pod-delete-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc into local-path-storage"
I1119 12:24:33.550971       1 event.go:389] "Event occurred" object="test/config-tmp" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="Error creating provisioned PV object for claim test/config-tmp: PersistentVolume \"pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc\" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity. Deleting the volume."
time="2024-11-19T12:24:36Z" level=info msg="Volume pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc has been deleted on /stor/k8s/ssd/local-path/pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc_test_config-tmp"
time="2024-11-19T12:24:36Z" level=info msg="Start of helper-pod-delete-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc logs"
time="2024-11-19T12:24:36Z" level=info msg="End of helper-pod-delete-pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc logs"
I1119 12:24:36.615172       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="32be5256-35b3-43a1-95f9-0276aec5a7dc" failures=0 threshold=15
E1119 12:24:36.615201       1 controller.go:974] error syncing claim "32be5256-35b3-43a1-95f9-0276aec5a7dc": PersistentVolume "pvc-32be5256-35b3-43a1-95f9-0276aec5a7dc" is invalid: spec.nodeAffinity: Required value: Local volume requires node affinity

This sounds like the PV isn't being created with the selected node set in affinity. Is there something else I'm missing or is this a bug?

I had this issue in 0.0.28 and it persists in 0.0.30. I tried upgrading when I noticed #454 but that doesn't seem to have fixed the issue.

Thanks!

[travisgroth]

I hadn't looked at my config for a while. I have this:

{
  "storageClassConfigs": {
    "local-path": {
      "sharedFileSystemPath": "/stor/k8s/ssd/local-path"
    },
    "local-path-disk": {
      "sharedFileSystemPath": "/stor/k8s/disk/local-path"
    }
  }
}

It seems that when a shared filesystem is being used, the affinity is not set in the volume spec, which leads to my error. I suppose implicitly this means only hostPath is supported for shared filesystem paths.

I don't actually use shared filesystems (and I need to correct my config) but I'm wondering if there's a reason shared local volumes aren't supported or if this is just leftover from hostPath assumptions, where no affinity was required in the spec.

[jcox10]

There was a bug in 0.29 with affinity, so you should try with 0.30 if you haven't already.