-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moving from CertificateRequests to Certificates: A Survey #49
Comments
We use it to allow certificate-manager to manage certificates for Routes. We do not use redhat's cert-utils-operator. We use an in-house ACME system to issue our certs. |
We use it as a way to request certs directly from the route resource, as using ingress adds a bit of unnecessary complexity (OCP users now need to create Ingress for the purpose of creating a Route, when many are only familiar with Routes). Personally I'd prefer if openshift-routes used Certificate resources, and allow cert-manager to handle using all the logic built into it already. IMO there is no difference in having the PK storage directly in the Route or Secret. Either way, it is being stored in etcd. By default cluster role definitions secrets are more secure than routes, as default cluster roles do not permit users with I suppose you could create custom roles to restrict access to the routes if you wanted to, while granting access to secrets, but it seems a bit backwards. But you could also create custom roles to restrict access to both secrets & routes if you truly wanted to restrict a user with access to a namespace from viewing PK's. I think the benefits of using a certificate resource outweigh any pros of embedding directly in routes to avoid storing PK in Routes. I think if cert-manager is okay with storing PK in secrets, then openshift-routes should too. |
Hello, we use openshift-routes as part of our internal PaaS together with Let's Encrypt (for a couple hundred websites). We were previously using the https://github.com/tnozicka/openshift-acme project (long deprecated) for the same purpose, but have switched to
I fully agree with this and am heavily in favor of switching to |
I'll put together a PR that provisions |
As discussed in cert-manager#49 , using `CertificateRequest` resources for interacting with cert-manager has several drawbacks: - this project (openshift-routes) has to duplicate logic for handling the creation of certificates that already exists in cert-manager. - observability is impaired because cert-manager does not offer metrics for `CertificateRequest` resources, only for `Certificates`. Therefore, this patch replaces the management of `CertificateRequests` with `Certificates` (and `Secrets`). In return, it requires slightly higher privileges because openshift-routes needs to be able to create and read `Certificates`. At the same time, the code complexity and LOC has been reduced. Signed-off-by: Jack Henschel <[email protected]>
I mentioned #55 during this morning's cert-manager open standup. I thought I understood the benefits of migrating to creating Certificates, but I struggled explaining the reasons. Would it be possible to meet at the next cert-manager community meeting (Thu 7 March 18:00 Paris time)? Before then, Ill try to gather the pros and cons of each approach. |
Some benefits of moving to Certificates
I understand initially there was one internal customer who didn't like the idea of having the private key in Personally I think there are some clear benefits of using I tried to summarize as best as I could, but please add to this if there are any other use cases I missed. |
Fair enough, that's a good point. But if this is the primary reason for creating Certificates, wouldn't it make more sense for openshift-routes to expose its own Routes metrics that mimic cert-manager's metrics to get the same result?
How big is this need? Is that something folks do often? I've read that it is something people do with Ingresses (cert-manager/cert-manager#841 (comment)) by pre-creating the Certificate resource instead of letting it be created by the ingress shim.
Ah, that's an excellent point. Ingresses and Gateways give you the opportunity to use a pre-existing Certificate resource, letting you fine-tune the Certificate, and among others letting you configure the encoding of the certificate. I also understand for need for the new annotation
Ingress and Gateway resources both let you re-encrypt traffic, so I do understand the need here. I can't see how people would be able to decrypt at the backend's site by reading the Route's
Just to understand things better, I compared the annotations supported by each:
In terms of annotations, there are only missing:
|
Routes are meant to behave like Ingress and Gateway resource, aren't they? Ingress and Gateway resources create Certificate resources, it would thus make sense that Routes create Certificates. As Craig pointed out, Ingress and Gateway resources support re-encryption and sharing certificates across resources. For the sake of consistency, I'd suggest we go with Certificates so that the experience between Ingresses, Gateways, and Routes are similar. |
On 7 March 2024, during the dev biweekly meetings (meeting notes), Trilok Geer – from Red hat – talked about OpenShift 4.16 (unreleased) and the new field externalCertificate. This ties well with the idea of creating Certificates instead of CertificateRequests in openshift-routes. Right now, On 16 May 2024, we talked a bit more on that topic (meeting notes):
|
@maelvls is the title the wrong way round? And should be:
|
@maelvls , thanks for sharing the collective date on annotations. This gives a better insight about compatibility with configurations supported by annotations.
Yes, the |
Just to chime in here: I fully agree with @jacksgt here, for exactly the same reasons as he points out. We've seen certificates that weren't able to be renewed at all unless manual action was taken, which kind of defeats the purpose of cert-manager altogether (removing Order and CertificateRequest was required in all cases we've seen). I'm hoping that using Certificates instead of CertificateRequests will improve this. We haven't been using it all that long yet, but the combination of cert-manager, openshift-routes addition, and Let's Encrypt has been problematic so far (curiously, no issues when Let's Encrypt is replaced with a local Step CA, with renewals every 30m). |
As discussed in cert-manager#49 , using `CertificateRequest` resources for interacting with cert-manager has several drawbacks: - this project (openshift-routes) has to duplicate logic for handling the creation of certificates that already exists in cert-manager. - observability is impaired because cert-manager does not offer metrics for `CertificateRequest` resources, only for `Certificates`. Therefore, this patch replaces the management of `CertificateRequests` with `Certificates` (and `Secrets`). In return, it requires slightly higher privileges because openshift-routes needs to be able to create and read `Certificates`. At the same time, the code complexity and LOC has been reduced. Signed-off-by: Jack Henschel <[email protected]>
As discussed in cert-manager#49 , using `CertificateRequest` resources for interacting with cert-manager has several drawbacks: - this project (openshift-routes) has to duplicate logic for handling the creation of certificates that already exists in cert-manager. - observability is impaired because cert-manager does not offer metrics for `CertificateRequest` resources, only for `Certificates`. Therefore, this patch replaces the management of `CertificateRequests` with `Certificates` (and `Secrets`). In return, it requires slightly higher privileges because openshift-routes needs to be able to create and read `Certificates`. At the same time, the code complexity and LOC has been reduced. Signed-off-by: Jack Henschel <[email protected]>
We've now merged #101 which makes the switch to using Certificates unilaterally. It'll be included in the next release which should be coming soon! |
Hello, thanks everyone for that piece of work! Any insight on how transition will occur from the CertificateRequests to the Certificates mode ? Will existing requests be picked up and certificates moved to new Certificates ? I guess I'll have to try :) Thanks ! |
First of all: please do try and report back! If there's anything we need to tweak now's the time. To answer your question: after upgrading, the same renewal logic as before will be used in that if a cert is found on the route and it's valid and doesn't need to be renewed, it'll be left in place. When the cert is renewed, it'll transparently be renewed just with a I triggered a renewal locally for testing by manually deleting the certificate from the Route, which immediately triggered a reissuance. |
Thanks for your input and considerations everyone, I'm really happy to see this has been released. :-) |
It looks good, certificates get created on (forced) renewal. I did notice that one of the annotations (cert-manager.io/private-key-rotation-policy: always) now requires a capital to work (so 'Always' instead of 'always'). It complained about that in the logging. Did I miss some notice about that, or am I doing something wrong? It would be awkward to have to replace that everywhere, but doable, I suppose. |
Well, I've got good and bad news. The bad is you probably have to update it everywhere, sorry. The good is that it never worked anyway - I don't think the old CertificateRequest approach supported that key at all, so it was being ignored entirely before! I added support for it while finishing up the Certificate work 😁 |
Well, then I hope everyone listens when I say they'll need to use a capital letter there, or drops the annotation entirely. I can't say I verified that it worked with CertificateRequests, so I'll take your word for it not working then. I'll definitely check now though 😄 |
Hello, I deployed the new release (v0.7.0) using Helm on our preproduction cluster (OKD 4.15) and we encountered an issue : there seem to be a new default On the openshift-routes controller logs we see:
I tried to set the Additionally all existing certificates were removed from the Routes data, hence all deployments using cert-manager are now running without certificate. This is not a smooth transition :D This behavior was clearly not expected, did I miss something ? Thanks ! EDIT: The ClusterIssuer used in this scenario is using letsencrypt to issue certificates. |
I noticed the same as you. Looks like a bug, the default value on Certificate resources should be empty if no annotation is provided which would then rely on cert-manager defaults of 2/3 duration. |
I gathered a little bit more intel on the side effect of this change, I believe the cert-manager admission webhook denies requests made by openshift-routes controller when the renewBefore parameter is unset. Should I create a new issue @maelvls ? |
Thanks for the reports! I can confirm that cert-manager will reject a cert with a renewBefore of zero, but I don't know why that value is getting set for you in your specific use case. I can't see anywhere where we'd set that value and you mentioned "there seem to be a new default renew-before setting with a value of 0s on every existing route that prevents cert-manager to provision the new certificate." I don't know where that value would be set. But I do think I can defend against it. I'll look into releasing an alpha release soon with a patch to omit the renewBefore if it's 0. That still doesn't explain where it comes from, but it might help with the problem at hand! |
Thanks for the work on this @SgtCoDFish ! Here are the relevant logs we get when this error occurs:
That's why I suspect an issue with either the value that is set by default for renewBefore when no annotation specifies it, or something not correctly handled by the cert-manager admission webhook. FYI, the cert-manager operator on the cluster in running v1.15.2 and is handled by OLM. Please tell me if I can help further! |
@Bengrunt & @ctml91 - I've just released openshift-routes v0.7.1-beta.0 which includes a potential fix for this. Are you able to test and confirm if this helps? |
@SgtCoDFish I've just deployed it on our preproduction cluster instead of release 0.7.0 and it now works. I'm just going to do some more extensive tests before rolling it out on the production cluster but everything seems fine so far. Thanks a lot ! |
We switched openshift-routes to Certificates. |
@Bengrunt thank you for testing! I'll release v0.7.1 proper later this week to give people a little more time to test 👌 |
Dear users of cert-manager/openshift-routes,
I'd like to hear your use-cases of cert-manager/openshift-routes. The initial reason openshift-routes was created was because an internal customer was using cert-utils-operator and didn't like the idea of having the private key in Secret resources. Also, cert-utils-operator doesn't create the certificate out of annotations like what's possible with the Ingress and Gateway resources. This is why cert-manager/openshift-routes doesn't create a Certificate resources.
In #34 and #42, I learned that this choice makes it hard for people to adopt this tool. For example, the project https://github.com/its4u/cert-manager-routes-controller/ does the same thing except it creates a Certificate resource rather than a CertificateRequest.
If you are using cert-manager/openshift-routes, why do you use it? Do you use it as a way to request certificates right from the Route resource?
The text was updated successfully, but these errors were encountered: