feat: TLS support for the Thanos web endpoint

[vyzigold]

Proposes #496 again after fixing an unreliable testcase.

This PR adds a new "WebTLSConfig" field to the ThanosQuerier Spec, which allows the user to specify secrets containing TLS certificates. The ThanosQuerier object is configured to use these secrets.

The secrets are then watched for changes and a reconciliation is triggered when one of them changes. In order to know if the contents of the TLS related secrets changed, a hash is computed from the contents of the secrets. The computed hash value is added to the querier deployment as an env variable. This means, that when that variable changes, a new querier pod is automatically created, which will use the new content of the secrets. (when the content of the secrets is the same, the old pod stays running).

The ServiceMonitor for monitoring the Querier is also modified to use TLS when TLS is enabled.

The first 2 commits of the PR (which add the WebTLSConfig struct and the TLS prometheus client for testing) are shared with #492

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vyzigold
Once this PR has been reviewed and has the lgtm label, please assign danielmellado for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Hi @vyzigold. Thanks for your PR.

I'm waiting for a rhobs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vyzigold]

This is essentially the same PR as the reverted #492 with 2 differences:

1. The last commit, which is the fix for the issue
2. The logging in: https://github.com/rhobs/observability-operator/pull/598/files#diff-62df723ef5f27edbbfaa7feeb8a0612efaa9ef8d91828004287a17f77f0c48aeR276 . I decided to make it a little quieter, there is no change to functionality of the function.

The issue:
I was able to replicate the issue locally. The issue would appear randomly - sometimes the tests would fail, sometimes they'd pass. I was able to determine that the issue is caused by the PR #492 and not by other TLS PRs. I was also able to determine that the new Thanos TLS test was the only one failing.

My theory about why the test was failing is a longer reconciliation after the querier is created. This could possibly be caused by the new "http-conf" CofigMap, which is being created and watched. I think it could trigger one more reconciliation loop right after startup. From the events during the tests we can see that the querier deployment is being created and then scaled up and down and up a few times. (same thing happens for the non TLS test, but my guess is it just settles faster and so it doesn't hit similar issue). From running the operator locally and watching the logs I can see that the reconciliation stops, so this isn't a case, where the operator would reconcile infinitely.

To fix the issue I introduced a new function "AssertDeploymentReadyAndStable". This checks the deployment is ready in 2 consecutive polls 5 seconds apart. This is to hopefully catch cases where the controller is still reconciling and the deployment is switching between ready and unready. I tested this implementation 5 times without any failure. I tested similar implementations of this like 10 more times and I also didn't get any failure. If you want we could perform a few rechecks on the PR to see it's stable even in the CI environment?