Pods using pvcs with mayastor intermittently fail to mount back their volumes

[nishant-dash]

Summary

Pod gets stuck initializing with unable to mount volume

Events:
  Type     Reason              Age    From                     Message
  ----     ------              ----   ----                     -------
  Normal   Scheduled           2m26s  default-scheduler        Successfully assigned cos/loki-0 to microk8s-2
  Warning  FailedAttachVolume  2m26s  attachdetach-controller  Multi-Attach error for volume "pvc-f17ed96a-a729-420d-8586-af31dd7eb212" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedMount         23s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[loki-loki-chunks-4bd159ef], unattached volumes=[loki-loki-chunks-4bd159ef], failed to process volumes=[]: timed out waiting for the condition

• Some deployment artifacts
  Loki pod
  PVC in question
  mk8s pvc layout
  storage classes

• Loki pod is scheduled to microk8s-2 (juju unit: microk8s/3), but the pvc is still on microk8s-3 (juju unit: microk8s/2) and mayastor is unable to move it to the new node.
  Relevant log snippet from mayastor csi controller pod

What Should Happen Instead?

Would expect that the pvc gets moved by mayastor sucessfully, other pods in this deployment and their pvcs are ok.

Reproduction Steps

This does not happen immediately after deploying the environment, but has happened a few times

1. Bootstrap juju 2.9.42 controller to microk8s 1.26/1.27 with mayastor addon
2. Deploy COS lite
3. After a while, some or all pods crash this way with pvcs stuck or unstable.

Introspection Report

inspection-report-20230809_103837-sanitized.tar.gz

Can you suggest a fix?

I believe we should try to further debug and check the mayastor addon since this behavior is present in both v1 and mayastor-aio-2.0.0-microk8s-1

Are you interested in contributing with a fix?

No