Home K8s Gitops

Creating the k3d cluser

1. Modify cluster-setup/k3d/cluster-config.yaml to suit your needs. Especially the extraMounts.
2. Run the following to have the cluster up and running

k3d cluster create --config cluster-setup/k3d/cluster-config.yaml

Setting up for first time:

Setting up secrets

Create account on Doppler, and get a service token.

Store secrets the following secrets in the same env for which the service token was created:

• ARGO_GITHUB_USERNAME - GitHub username
• ARGO_GITHUB_PASSWORD - GitHub Token with Read access to the repo
• ARGO_UI_ADMIN_PASSWORD - Admin UI Password for Argo.
• ES_DOPPLER_SECRET_TOKEN - The service token just created. Should help with bootstrapping Doppler
• ARGO_DISCORD_SYNC_WEBHOOK - Discord Webhook URL where ArgoCD can push notifications in case of failures
• ARGO_GH_WEBHOOK_TOKEN - GitHub Token used for the GH Cli which forwards GitHub webhooks to ArgoCD
• GITHUB_ACTIONS_ARGO_WEBHOOK_SECRET - Any secret string to be used for configuring the GitHub webhook

To get all secrets that are to be configured run:

find apps/ -name config.json -exec $SHELL -c 'helm dependency update $(dirname {}) >/dev/null && helm template $(dirname {})' \; | yq -N 'select(.kind == "ExternalSecret") | .spec.data[].remoteRef.key' | sort | uniq

Create a copy of apps/external-secrets/templates/doppler-token-sample.yaml to apps/external-secrets/templates/doppler-token.yaml and add in the doppler service token for configuring external secrets.

Once everything is up and running and ArgoCD syncs, the doppler token should be refreshed from Doppler itself.

Starting up ArgoCD

Run the following to install crds first and then the remaining parts for basic setup:

helmfile template -f cluster-setup/first-run/ --environment with-crds --include-crds | yq '. | select (.kind == "CustomResourceDefinition" and .spec.group != "projectcontour.io")' | kubectl apply -f - && \
helmfile apply -f cluster-setup/first-run/

ArgoCD should then pick up all the apps from the repo automatically.

For any new changes, just make changes to repository and argo should pick them up.

Setting up LoadBalancer Forwarding for non-http services

Contour's HTTPProxy does not support TCP/UDP services (and for ports other than 80/443).

For that, MetalLb's LoadBalancer is utilized.

Required dependency is gomplate, can be installed via brew

To generate nginx.conf for TCP Forwarding via nginx, and start it as a TCP/UDP Proxy for LoadBalancers for the cluster, run the following at the root of the repo:

./startLbForwarding.sh