Skaha leaks kubernetes resources

[rptaylor]

Skaha does not appear to clean up the services and ingressroutes that it creates for workload pods so there is an unbounded growth of orphaned resources on clusters that it runs on. This contributes to load on the API and etcd services. Left long enough (especially running at large scale) there is a risk of consuming all etcd storage and bringing down the cluster. It would be difficult to recover from this condition (we already use the maximum recommended etcd size) so in practice it would likely require destroying and rebuilding the cluster.

Currently we have to occasionally remember to remind you :) to run a manual cleanup every few months to avoid this, which is a bit operationally fragile. Skaha should automatically clean up all resources it creates to avoid leaving orphaned resources indefinitely.