Prepare delayed downscale

[pstibrany]

This PR adds support for downscale delay when using reference resource as a source of number of replicas.

Two annotations are used for configuration:

• Delay duration: grafana.com/rollout-delayed-downscale
• URL: grafana.com/rollout-prepare-delayed-downscale-url.

How it works: When rollout-operator detects that statefulset configured with these annotations should be scaled down, it first calls POST <URL> on all pods that should be scaled down. Pods are expected to prepare for scaledown and return timestamp when such preparation happened (possibly in the past). Any errors from these calls (networking, non-200 status code, invalid JSON object) will cause that downscale is aborted (and retried later, if scaledown condition is still true).

Operator then computes Maximum of returned timestamps. If this maximum is in the past for more than "delay duration", scaledown proceeds. Otherwise, scaledown is not allowed at this point, and will be retried later.

All other pods that are NOT going to be scaled down also receive HTTP request to given URL, however with DELETE method. This tells pod to "cancel" any possible preparation for scaledown in the past.

For example, if statefulset has 100 pods, and 30 of them are going to be scaled down, then pods from 0 to 69 receive DELETE <url> call, while pods 70 .. 99 receive POST <url> call.

If rollout operator detects no change of replicas, or scale up, it will still perform DELETE <url> calls, to make sure that any possible previous preparations are cancelled. Since these DELETE calls are repeated all the time, errors from them are ignored.

JSON response expected by rollout-operator is simply {"timestamp": <unix-seconds as number>}.

Format of the URL annotation is full URL like http://pod/prepare-delayed-downscale, in which the "host" part of the URL will be replaced with pod's fully-qualified domain name, eg. ingester-zone-b-0.ingester-zone-b.namespace.svc.cluster.local..

How is this different from grafana.com/min-time-between-zones-downscale?

grafana.com/min-time-between-zones-downscale is applied to delay downscaling of zones based on downscale timestamp of another zone.

How is this different from grafana.com/prepare-downscale-http-path?

grafana.com/prepare-downscale-http-path together with grafana.com/prepare-downscale-http-port annotation are used to signal pod that is being scaled down. Ie. after call to grafana.com/prepare-downscale-http-path+port, pod will be downscaled right after.

grafana.com/rollout-prepare-delayed-downscale-url tells pod about downscale in the future, which can be minutes or hours later.

[pracucci]

Great work and very nice test! I left few minor comments only. Thanks!

[pstibrany]

Thanks for review, I've applied all your suggestions.