Deploy Kubernetes to DigitalOcean with Terraform

Provision a Kubernetes cluster in DigitalOcean via Terraform. By default, the k8s cluster deploys come with:

• Ingress Nginx
• Cert Manager (Automatically provision and manage TLS certificates)
• External DNS (automates the creation of CNAME records when new services are deployed)

The following optional services are deployed and should be removed from main.tf if they are not desired.

• Ntfy (Simple pub/sub based notification service)

The project is configured to be run using Terraform Cloud, and plans/applies are conducted remotely.

First time setup

This project can be cloned and used as-is, except that the value of terraform.cloud.organization in main.tf will need to be changed to reflect your own organization.

1. Create a Digital Ocean account

2. Create a Terraform Cloud account

3. Obtain a domain and create NS records pointing to DigitalOcean's name servers

4. Install CLI tools:

   • doctl
   • terraform
   • kubectl

5. Create a R/W DigitalOcean token

   For security, you can set the token to expire, but it will need to be manually rotated. This token will be used in the next step.

6. Connect to Terraform Cloud

   terraform login
   terraform init
   terraform plan

7. The first time you run terraform plan, you will be prompted to set values for certain variables from the CLI. At this point, a workspace will have been created in your Terraform organization for this project. To ensure future Terraform executions happen remotely, navigate to the Terraform Cloud workspace associated with this project and:

   1. On the Variables tab, add a "Workspace variable" for each of the required properties defined below.
   2. On the Settings -> General tab, ensure that:
      • "Execution Mode" is set to "Remote"
      • "Apply Method" is set to "Auto Apply"

8. Enable Continuous Deployment via Github

   • Create a Terraform Cloud token
   • In your Github repo, go to Settings -> Secrets -> Actions. Create a new repository secret called TF_API_TOKEN using the token value obtained in the previous step.

Creating the k8s cluster

To create the k8s cluster for the first time (or to apply changes), just run

terraform apply

Assuming none of the default values have been changed, this will provision:

• A 1-node Kubernetes cluster that can autoscale up to two nodes (Each node has 1 vCPU and 2GB RAM)
  • At time of writing, this would cost between $10-$20/month for one or two nodes, respectively
• A small load balancer that serves all traffic for the cluster
  • At time of writing, this costs $10/month

Once the terraform plan has been applied, you can update your kubectl config to connect to the new cluster by default by executing the following:

CLUSTER_NAME=$(terraform output --raw cluster_name)
doctl kubernetes cluster kubeconfig save "${CLUSTER_NAME}"

Parameters

See variables.tf

Using external-dns and cert-automation

This cluster comes with automated DNS record creation and TLS cert automation. Using this functionality requires that specific annotations be set on the deployment's ingress. The Helm chart contained within the example-deployment module provides an example of what these annotations look like.

Limitations / Caveats

• When a new service is deployed, there is a race condition between the creation of a DNS entry (external-dns) and the HTTP challenge required to create a TLS certificate (cert-manager). This results in cert-manager attempting to resolve the DNS before it propagates, receiving a NXDOMAIN response, then needing to wait for the record's TTL to expire before re-trying the challenge. You can read more about the issue here. To reliably avoid this issue, manually create the DNS entry in Digital Ocean long enough before deploying the corresponding service so that the record has time to propagate.

Development

Install dagger to run the local CI pipeline. Dagger depends on Docker engine.

To run the lint suite:

dagger do lint

If linting fails, many style guidelines can be automatically applied by:

dagger do fix

dagger may take a long time the first time it runs, as it must build docker images. You can increase the output verbosity to observe the build progress with the flag --log-format plain