Proposal: make Let's Encrypt a first-class entity in Workflow

[Cryptophobia]

From @jmspring on January 19, 2017 20:31

At the Azure / Deis hackfest at in December, there was time to work on additional enhancements. One such was a PoC for adding support for Let's Encrypt certificates (to enable TLS) when new applications are deployed to Workflow.

The initial PoC takes on two parts:

1. An application that makes periodic checks for new applications: https://github.com/jmspring/deis_certificate_manager

2. A worker process that the (1) spawns to do the actual Let's Encrypt/ACME handshake:
   https://github.com/jmspring/deis_cert_letsencrypt_worker

This is just a PoC and real support should directly work with the ACME protocol and not necessarily shell out another process to do the handshake.

Further, some research into limitations in/around Let's Encrypt certificate issuance policies (limits per domain, etc) should be considered.

Also, the PoC only retrieves the initial certificate, Let's Encrypt certs have ~90 day life time so renewal as well as revocation as apps are removed should be considered as well.

@mboersma - Jason Hansen mentioned you were Workflow lead.

Copied from original issue: deis/workflow#708

[Cryptophobia]

From @bacongobbler on January 19, 2017 20:43

related: deis/router#134

[Cryptophobia]

From @jmspring on January 24, 2017 22:44

@bacongobbler, @mboersma - thoughts on moving forward on this? I have cycles to work on it.

[Cryptophobia]

From @bacongobbler on January 24, 2017 23:0

This sounds good to me!

Personally I'd like to see this fleshed out in the client as a new deis certs:generate command, which would essentially work with the ACME protocol to retrieve a certificate from Lets Encrypt and upload it to the server via deis certs:add. The POC could also look into re-running deis certs:generate to renew the certificate. That implementation may need to be performed by a new deis certs:update command, which currently does not exist. I know others have asked for that feature as well.

I think this is something that should be supported as a first-class citizen in the CLI rather than a sidekick container. Other than that, the idea of generating certs for domains using Let's Encrypt and using existing commands to upload/update/delete the cert would probably be the best approach.

What do you think?

[Cryptophobia]

From @jmspring on January 24, 2017 23:52

Part of the Let's Encrypt Handshake (for issuance) currently requires some form of validation of the domain. The PoC was one quick hack for such. I'll dive into the protocol a bit more this week and solidify my understanding both of issuance and renewal -- the certs are only 90 day certs, so renewal (as well as revocation will be important).

My concern if it is CLI only and requires said handshake, what the best approach is. As for updating nginx (per the thread you linked to -- the hash of a cert makes sense to trigger the update).

I'll get some more info by Thursday.

[Cryptophobia]

From @MartinodF on January 26, 2017 15:28

We're investigating deis and one of the main issues we found is the lack of Let's Encrypt integration. Our current solution on Kubernetes relies on kube-lego together with nginx-ingress.

kube-lego monitors the kubernetes cluster for ingresses with a specific annotation. It then reads the spec.tls.hosts field from those and compares the result with the available certificates. When some are missing or near the expiry date, it requests a new certificate, creating a new ingress rule which matches the acme challenge path and intercepting the Let's Encrypt validation request. It then saves the new certificates in a secret, ready to be used by the application.

I believe such an approach would work for deis too, wouldn't it?

[Cryptophobia]

From @bacongobbler on March 28, 2017 19:51

@MartinodF just following up on this comment, but with #732 in place, yes. You could just run ingress and kube-lego on the cluster as usual.

[Cryptophobia]

From @bacongobbler on April 5, 2017 22:53

Now with the experimental native ingress feature in v2.13.0, I've been able to get a Workflow cluster up and my apps being backed by letsencrypt running using a combination of kube-lego and the nginx ingress controller. I'll hopefully get to writing a blog post about the entire setup so others can learn about the internals and see where we can write a feature around it.

[Cryptophobia]

From @johnschultz on June 22, 2017 7:2

@bacongobbler really interested in how you got that to work. Is that blog post up somewhere or can you point me in the right direction? Thanks!

edit: apologies, didn't realize commenting would re-open the issue.

[Cryptophobia]

From @mariusmarais on June 22, 2017 8:1

@johnschultz Native ingress via Traefik with its built-in Let Encrypt support is working quite well for us too. Default app names work perfectly, but deis domain:xxx needs manual changes to the ingress for the app, which I'm assuming will be handled by deis/controller#1277. This is probably true for nginx too.

[Cryptophobia]

From @ccatlett2000 on July 5, 2017 0:38

Any progress on this? Yes, I can use native ingresses, but using Deis's router is preferable.

[Cryptophobia]

From @bacongobbler on July 5, 2017 4:56

No progress. The "hack" I did was merely just manually editing ingress routes and installing the nginx ingress controller, along with kube-lego. No code to contribute at this time.

[Cryptophobia]

From @ccatlett2000 on July 5, 2017 19:25

Does anyone who's setup native ingresses know if Deis adds the annotations for TLS (Or if there is a way to configure them)? Because the way that kube-lego works is that it looks for annotations to indicate which domain to pick.

[Cryptophobia]

From @bacongobbler on July 5, 2017 19:27

It does not; I had to manually edit the ingress resources to add those annotations in.