Setting ruler.evaluation-delay-duration to be deprecated.

[danielblando]

What this PR does:
Follow up from #6085

Deprecating -ruler.evaluation-delay-duration
Should use new flag ruler.query-offset

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Change looks great!
Integration test for ruler failed on TestRulerEvaluationDelay. Let's just remove that test.

[rapphil]

Was this flag experimental? If no, do we need to give a grace period before removing altogether and have just the warning message for now if this flag is being used (this is what I would understand as deprecating something).

[yeya24]

https://cortexmetrics.io/docs/configuration/v1guarantees/#flags-config-and-minor-version-upgrades

I am trying to understand this process better.

Upgrading cortex from one minor version to the next should “just work”; that being said, we don’t want to bump the major version every time we remove a flag, so we will will keep deprecated flags around for 2 minor release. There is a metric (deprecated_flags_inuse_total) you can alert on to find out if you’re using a deprecated flag.

just work means we need to keep the same functionality or there shouldn't be any issue to run Cortex with the existing config?

If the latter this PR should be ok.

[danielblando]

I think the biggest issue here is that now we have two flags doing the same thing. Looking at the code it looks like the delay would be the sum of two values. This seems more like a bug than a flag deprecate. We can go back to remove the new RuleQueryOffset flag and maintain RulerEvaluationDelay, but now as the query-offset time for Prometheus.

[yeya24]

Looks good. Just a small nit.