Add support for scaling kubernetes stateful sets

[kormide]

Allow the autoscaler to update stateful sets. This can be useful when workers need to be deployed as a stateful set with persistent volume claims for the FUSE mount's storage.

[EdSchouten]

Hey! Could you please elaborate a bit more on your use case? There's nothing inherently persistent to workers, so to me it's not clear why you'd want to use stateful sets there. The FUSE mount itself cannot be mounted through a PersistentVolumeClaim. Are you referring to the LocalBlobAccess instance that goes underneath? Why would you want that to be persistent?

[kormide]

Hey! Could you please elaborate a bit more on your use case? There's nothing inherently persistent to workers, so to me it's not clear why you'd want to use stateful sets there. The FUSE mount itself cannot be mounted through a PersistentVolumeClaim. Are you referring to the LocalBlobAccess instance that goes underneath? Why would you want that to be persistent?

To maximize performance, each worker is configured to use local node-attached SSDs for the FUSE mount, CAS, filepool, etc. That storage is served by several PVCs for block and filesystem devices which are satisfied from provisioned storage on the node. Each worker pod requires its own set of volumes, hence the need for a stateful set.

I suppose the use of PVCs here is unnecessary and I could mount and partition the SSDs using hostPath since, as you said, there's nothing stateful about a worker 🤔.

[kormide]

Yeah, I think I was using the wrong abstraction here. I'll close this for now and I'll reopen if I find a legitimate use case.