Complete refactor to add better Argo CD functionality (#79)

* add registry docker as an approved repo for bitnami mongo charts for infisical

* remove yaml dump on extra password for mongodb

* fix secret keys typo for infisical hostname key

* clean up defaults and update click in poetry lock

* solve when we delete stuff

* fix where we check if bitwarden is enabled

* catch other bitwarden error in dict

* change sync to happen on get item

* fix cert clusterissuer and clean up bitwarden debugs

* fix cluster issuers again

* only prompt for infisical smtp secret if it doesn't exist

* disable infisical by default again and enable eso. also alphabetize the env var dict for infisical in case it is ever production ready

* note again that infisical is not quite there yet and enable eso again

* fix return of config on delete

* fix custom fields for zitadel

* update fields dict

* add a wait for zitadel app to up before we configure it

* fix a bunch of bitwarden fields issues

* fix vouch bitwarden custom fields

* missed a vouch bitwarden item

* fix zitadel bitwarden db custom field for username

* switch back to cockroachdb as the default

* wait on zitadel first, followed by zitadel-web-app when bootstrapping zitadel

* attempt to fix zitadel get secret

* actually log the response from bw sync

* add getting data parameter the correct way for the zitadel secret

* add k8up as a default app, and add project source repo for bitarden eso provider

* fix string to byte encoding in zitadel

* adding zitadel api debug lines

* process api token more thoroughly

* actually grab the key we need

* adding some debug lines for the zitadel api calls

* add function to create token and download the zitadel-tools if needed

* add deptry to the dev dependencies

* clean up token for zitadel

* generate api url from hostname in zitadel class

* fix where we write the json file for zitadel, moves to cache

* fix return of zitadel token

* add more debug notes and don't verify zitadel ssl connection due to homelab setup

* strip trailing new line in the api key

* doing some linting and formatting of doc comments for configure_keycloak

* adding api token

* add more notes on jwt for later

* adding proper jwt creation for hitting zitadel's oauth api to get a token

* improve how we rename a context with k3s and how we search for one generally

* add a debug line for what clusters equals for cluster initialization

* fix jwt tokens, add crypto library, and clean up k3s default context name

* clean up api token request

* clean up creation of token for zitadel api

* remove update project setttings and get project id functions in favor of creating a project

* make verify optional

* fix call to self.project_id

* fix res.json() calls and pass in 10s instead of 10 for timeout of action

* clean up call to k8s secret

* add deleting argocd secret plugin pod after secret update and fix secret update decoding

* clean up k8s api stuff

* use loads for new secret get func

* fix dict processing for zitadel

* decoded data didn't need to be loaded as it's already loaded else where

* updating how we update secrets

* try to fix how we uninstall k3s entirely

* make sure we delete both the context and cluster for a k3s cluster when we run --delete

* fix -o json for deletion of secrets

* change when we configure logging

* clean up how we do deletion... again

* fix variables for init

* cleanup deletion of secret

* fix delete cluster function

* change when we create json token

* add a typeerror catch

* adding better logging

* fix users prompt for zitadel api

* fix green highlight for zitadel user prompts

* allow a special character in the create password function

* change available genders to match api request spec for zitadel

* randomly assign a special character to zitadel human user and place in bitwarden if bitwarden is being used

* fix bitwarden import typo for zitadel api

* more typos in bitwarden class via zitadel api

* fix envs passed into the bitwarden sync method

* don't return userId for user grants

* fix call to vouch

* fix secret keys dict for vouch

* clean up federated function definitions and secret variable creation

* add a struct for custom bitwarden field to cut down on duplicate code and make it easier to read

* fix query of secret_keys dict

* fix more vouch flow bugs

* seperate out configuration of vouch oidc app from keycloak and zitadel

* add new init sections of each major app, add keycloak class, allow automatic setting of zitadel and keycloak first user through config file

* update to new password manager dict

* clean up metallb creation

* fix cert manager init

* fix cert manager secret key grab

* still default zitadel to psotgres for now

* more pretty headings and logging

* default enable vouch

* fix vouch with zitadel and keycloak and change it so that argo apps are created with - instead of _

* fix k8up project scoped repos

* add prometheus as a default app

* fix zitadel vouch app creation

* fix vouch.. again

* make zitadel obj return for main install function

* add grant for zitadel admin

* fix name of ingress nginx controller from nginx ingress

* fix default prometheus diectory

* add abilityt to set vouch-proxy emails and domains on init

* make sure we only prompt for vouch emails and domains if they are not present in the config file

* fix project for prometheus

* fix how we calculate source repos for a project

* make args for argo proj creation mandatory

* add argocd source repos to attempt to fix argo managing itself in it's own project

* attempt to fix metallb projects

* add more project source repos to the default config for the generation of the project

* update k8s distro structure in default config.yml again

* make check_contexts not optional

* change name of secret and bitwarden item we create for argocd oidc provider

* attempt to fix argocd secret update for oidc param

* change admin role key to be super-user according to terraform docs

* fix yaml import

* fix typo for getting k8s distros

* try to fix update secret in-line yaml

* add back k9s functionality

* fix creation of argo cd oidc secret

* change cert-manager to cert_manager

* create initial secret token for appset communication

* update to newest version of argocd appset generator

* update to latest argocd appset generator version

* fix typo in argocd appset generator secret for helm value

* add latest argocd appset generator plugin

* update to latest version of argocd helm chart

* add more vouch logging

* make all argo apps apply outofsync only, add auto-generated jwt secret to vouch secrets

* clean up vouch a bit more by adding comments, fixing doc strings, and adjusting required parameters

* fix creating vouch jwt secret

* add some additional info at the end

* add preferred domain option

* make password creation tool allow for variable characters, make vouch jwt secret default to 44 characters

* fix final printed panel to be more spaced out

* update to latest argocd secrets generator helm chart

* set preferred domain to empty string

* switch monitoring to the prometheus namespace

* reorg of code again

* cleaning up more imports about reorg

* more import cleanup

* clean up more imports

* fix even more imports

* more fixing of imports

* fix incorrect imports and config app reference for bweso

* clean up secrets management printing

* fix arguments for secrets management

* fix bitwarden obj to eso

* clean up console logging functions and add emoji arg

* cleaning up some more headers

* more emojis

* more pretty printing

* clean up federated apps a bit

* fix up final part of install_with_argo call for nexdtcloud

* try to nest vouch preferred domain quotes harder

* try to fix vouch empty value

* fix nextcloud postgres credentials generation

* add github.io domain for prometheus repo

* update which namespaces are allowed for just prometheus which is special

* add new user grant to vouch user for zitadel

* clean up vouch user id issue

* add new create user org membership

* update zitadel admin role grant

* tidy up membership grant for new user

* we're zitadel users, scale it back a bit, we want to explore the entire project

* fix api call name

* finally fix zitadel admin role creation

* never check in .env files

* fixing emoji header to not have any clouds

* fix keycloak vouch client id

* add logout url to the secrets for argocd

* fix logout url param name

* update global cluster issuer to be a thing

* enable prometheus push gateway fdqn to be set

* clean up nextcloud deployment

* update more args to be typed and non-optional when needed

* finish fixing typing on all args

* consolidate base into __init__

* updating version of argocd helm chart

* update vouch urls

* add additional redirect URIs

* add end session url to vouch for zitadel

* clean up zitadel urls to be more readable

* clean up vouch some more, because we forgot the port for the callback url

* more edits to vouch in an attempt to make it work with zitadel

* callback url can be multiple so changing it back for vouch

* allow user to set max pods per node with k3s

* clean up default global cluster issuer

* use sudo to create the kubelet file

* nooooow we can set default pod limits

* fix call to username for nextcloud

* make sure to verify that a provider was also configured or we can't initialize vouch

* wait on ESO, just in case

* update how we process redirect uris, and when we check for existing provider if we're already in vouch init phase

* add flow type triggers to action creation

* fix argocd zitadel login

* add logging for when we do the zitadel vouch app user grant

* fixed spacing of zitadel app action

* update user grant for vouch, instead of creating a new one

* add even more logging of vouch oauth bitwarden item creation

* fix vouch user type iteration and fix verify to be false by default

* try to simplify all the custom fields for the vouch oauth bitwarden item and add preferredDomain

* updating it so we don't verify any certs if global cluster issuer is set to staging

* fix max_pods to actually be passed in so it is configurable

* fix missed tls verify param pass in

* add new zitadel link after user builds smol-k8s-lab

* fix bug where we never setup the cluster issuers

* fix vouch user grant and explicitly specify key word arguments for surrounding functions so I don't get confused

* update argocd fullnameOverride

* adding nextcloud backups secrets

* allow for an existing restic repo password

* change all federated apps to be app specific py files

* update serverInfoToken secret key for nextcloud

* update redis creds creation comments for nextcloud

* add smtp host to list of things we create keys in a secret for in nextcloud

* clean up nextcloud postgres credentials creation

* add timezone to secret keys we request for k8up when init is enabled

* add init enabled

* bump default metallb version

* adding cilium to play with and moving both that and metallb to networking module

* add cilium as an optional app and tidy up helm module/class

* adding cilium to default apps

* disable cilium by default

* install cillium BEFORE metallb

* install cilium to match argocd applicationset values

* fix cilium installation dict

* updating argocd apps to use correct Helm class name

* update vouch secret keys processing

* clean up argocd.py to have two distinct functions for argocd vs the plugin generator

* add argocd login {argo_cd_domain} --core to make argocd talk directly to k8s

* fix args in call to configure_secret_plugin_generator()

* update readme for all the new apps we support

* add minio icon

* add kepler, mastodon, and nextcloud logos

* adding cilium, k8tz, k8up, and keycloak logos

* Update README.md - add logo descriptions

* add matrix, vouch, and zitadel logos

* adding alpha to kepler and minio, and adding minio hostname

* move k8s_apps into it's own directory outside of notes

* add back k8s tools notes

* more updates to docs

* Update pyproject.toml - this really is a 1.0.0 and there's no avoiding it

* updating a quick example app

* add k8tz as an alpha app

* Update docs/index.md - fix grammar

Co-authored-by: Max! <[email protected]>

* update screenshot for brand new version :)

* update argocd version

* upgrade cert manager and ingress default versions

---------

Co-authored-by: Max! <[email protected]>

.gitignore

@@ -11,3 +11,6 @@
 smol_k8s_lab.egg-info/**
 dist/**
 build/**
+
+# ignore any local env testing
+.env

docs/index.md

@@ -21,7 +21,7 @@ We always install the latest version of kubernetes that is available from the di
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/k3s_icon.ico" width="26">][k3s] <br /> [k3s] | The certified Kubernetes distribution built for IoT & Edge computing |
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/kind_icon.png" width="32">][KinD] <br /> [KinD] | kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. |
 
-We tend to test first on kind and k3s first, then k0s.
+We tend to test on k3s first, then kind.
 
 
 ### Stack We Install on K8s
@@ -30,7 +30,6 @@ Version is the helm chart version, or manifest version.
 |           Application           |    Version    |                      Description                      |
 |:-------------------------------:|:-------------:|:------------------------------------------------------|
 | 🐄 [Local Path Provisioner] |   k3s latest  | Default simple local file storage for persistent data |
-| [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/metallb_icon.png" width="32px" alt="metallb logo, blue arrow pointing up, with small line on one leg of arrow to show balance">][metallb] <br /> [metallb] | 0.13.7 | loadbalancer for metal, since we're mostly selfhosting |
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/nginx.ico" width="32px" alt="nginx logo, white letter N with green background">][nginx-ingress] <br /> [nginx-ingress] | 4.4.0 | The ingress controller allows access to the cluster remotely, needed for web traffic |
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/cert-manager_icon.png" width="32px" alt="cert manager logo">][cert-manager] <br /> [cert-manager] | 1.10.1 | For SSL/TLS certificates |
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/k9s_icon.png" alt="k9s logo, outline of dog with ship wheels for eyes" width="32px">][k9s] <br /> [k9s] | latest | Terminal based dashboard for kubernetes |
@@ -40,8 +39,9 @@ Version is the helm chart version, or manifest version.
 
 | Application/Tool |    Version    | Description |
 |:----------------:|:-------------:|:------------|
-| [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/eso_icon.png" width="32" alt="ESO logo, outline of robot with astricks in a screen in it's belly">][ESO] <br /> [ESO] | 0.6.1 | external-secrets-operator integrates external secret management systems like GitLab|
+| [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/metallb_icon.png" width="32px" alt="metallb logo, blue arrow pointing up, with small line on one leg of arrow to show balance">][metallb] <br /> [metallb] | 0.13.7 | loadbalancer for metal, since we're mostly selfhosting |
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/argo_icon.png" width="32" alt="argo CD logo, an organer squid wearing a fishbowl helmet">][Argo CD] <br /> [Argo CD] | 5.16.2 | Gitops - Continuous Deployment |
+| [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/eso_icon.png" width="32" alt="ESO logo, outline of robot with astricks in a screen in it's belly">][ESO] <br /> [ESO] | 0.6.1 | external-secrets-operator integrates external secret management systems like GitLab|
 | [<img src="https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/docs/icons/kyverno_icon.png"  width="32" alt="kyvero logo">][Kyverno] <br /> [Kyverno] | latest | Kubernetes native policy management to enforce policies on k8s resources |
 
 
@@ -79,8 +79,9 @@ If you're stuck, checkout the [Notes section](https://jessebot.github.io/smol-k8
 [cert-manager]: https://cert-manager.io/docs/
 [k9s]: https://k9scli.io/topics/install/
 
-[ESO]: https://external-secrets.io/v0.5.9/
+<!-- k8s opitonal argocd apps link references -->
 [Argo CD]: https://github.io/argoproj/argo-helm
+[ESO]: https://external-secrets.io/v0.5.9/
 [Kyverno]: https://github.com/kyverno/kyverno/
 
 <!-- smol-k8s-lab dependency lib link references -->

docs/notes/apps_on_k8s/apps.md → docs/k8s_apps/apps.md

@@ -2,7 +2,6 @@
 layout: default
 title: Apps on K8s
 description: "Quick notes on various apps that run ON TOP OF K8s"
-parent: Notes
 has_children: true
-permalink: /notes/apps
+permalink: /apps
 ---

docs/notes/apps_on_k8s/argocd.md → docs/k8s_apps/argocd.md

@@ -2,9 +2,8 @@
 layout: default
 title: Argo CD
 description: "Quick notes on Argo CD"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/argocd
+permalink: /apps/argocd
 ---
 
 ## ArgoCD

docs/notes/apps_on_k8s/cert_manager.md → docs/k8s_apps/cert_manager.md

@@ -2,9 +2,8 @@
 layout: default
 title: cert-manager
 description: "cert-manager notes"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/cert-manager
+permalink: /apps/cert-manager
 ---
 
 ## Cert Manager

docs/notes/apps_on_k8s/core_dns.md → docs/k8s_apps/core_dns.md

@@ -2,9 +2,8 @@
 layout: default
 title: CoreDNS
 description: "Troubleshooting CoreDNS notes"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/coredns
+permalink: /apps/coredns
 ---
 
 ### Troubleshooting hellish networking issues with coredns

docs/notes/apps_on_k8s/k8up.md → docs/k8s_apps/k8up.md

@@ -2,9 +2,8 @@
 layout: default
 title: K8up
 description: "Quick notes on K8up"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/k8up
+permalink: /apps/k8up
 ---
 
 ## K8up

docs/notes/apps_on_k8s/metallb.md → docs/k8s_apps/metallb.md

@@ -2,12 +2,11 @@
 layout: default
 title: MetalLB
 description: "Troubleshooting MetalLB notes"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/metallb
+permalink: /apps/metallb
 ---
 
-## Assigning IPs 
+## Assigning IPs
 Running into issues with metallb assigning IPs, but them some of them not working with nginx-ingress controller? This person explained it really well, but it required hostnetwork to be set on the nginx-ingress chart values.yml. Check out thier guide [here](https://ericsmasal.com/2021/08/nginx-ingress-load-balancer-and-metallb/).
 
 ## Why am I getting deprecation notices on certain apps?

docs/notes/apps_on_k8s/nextcloud.md → docs/k8s_apps/nextcloud.md

@@ -2,9 +2,8 @@
 layout: default
 title: Nextcloud
 description: "Quick notes on Nextcloud"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/nextcloud
+permalink: /apps/nextcloud
 ---
 
 ## Nextcloud

docs/notes/apps_on_k8s/prometheus.md → docs/k8s_apps/prometheus.md

@@ -2,9 +2,8 @@
 layout: default
 title: Prometheus
 description: "Quick notes on hosting prometheus on Kubernetes"
-grand_parent: Notes
 parent: Apps on K8s
-permalink: /notes/apps/prometheus
+permalink: /apps/prometheus
 ---
 
 Here's a bare bones setup of prometheus that works via an argo app template which uses helm to install the community helm chart with grafana: [https://gitlab.com/vleermuis_tech/goobernetes/prometheus](https://gitlab.com/vleermuis_tech/goobernetes/prometheus)

docs/notes/k8s_toolbox/helm.md → docs/k8s_toolbox/helm.md

@@ -2,9 +2,8 @@
 layout: default
 title: Helm
 description: "Notes on helm, a package manager for k8s"
-grand_parent: Notes
 parent: Toolbox
-permalink: /notes/toolbox/helm
+permalink: /toolbox/helm
 ---
 
 # Helm

docs/notes/k8s_toolbox/k9s.md → docs/k8s_toolbox/k9s.md

@@ -2,9 +2,8 @@
 layout: default
 title: K9s
 description: "Notes on K9s, a terminal based dashboard for k8s"
-grand_parent: Notes
 parent: Toolbox
-permalink: /notes/toolbox/k9s
+permalink: /toolbox/k9s
 ---
 
 ## K9s

docs/notes/k8s_toolbox/kubectl.md → docs/k8s_toolbox/kubectl.md

@@ -2,9 +2,8 @@
 layout: default
 title: Kubectl
 description: "Notes on Kubectl, the default cli for k8s"
-grand_parent: Notes
 parent: Toolbox
-permalink: /notes/toolbox/kubectl
+permalink: /toolbox/kubectl
 ---
 
 # Kubectl

docs/k8s_toolbox/toolbox.md

@@ -0,0 +1,62 @@
+---
+layout: default
+title: Toolbox
+description: "Notes on tools for interacting with k8s"
+has_children: true
+permalink: /toolbox
+---
+
+## Toolbox
+
+Notes on tools for interacting with k8s.
+
+### Install some kubectl plugins with krew
+Krew is a plugin manager for `kubectl` plugins. You can install it with `brew install krew` and update plugins with `kubectl krew update`
+
+These together make namespace switching better. Learn more about kubectx + kubens [here](https://github.com/ahmetb/kubectx).
+
+```bash
+kubectl krew install ctx
+kubectl krew install ns
+```
+
+This will help with generating example k8s resources:
+
+```bash
+kubectl krew install example
+```
+
+This one helps find deprecated stuff in your cluster:
+
+```bash
+kubectl krew install deprecations
+```
+
+To install plugins from a krew file, you just need a file with one plugin per line. You can use [this one](https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/deps/kubectl_krew_plugins):
+
+```bash
+curl -O https://raw.githubusercontent.com/jessebot/smol-k8s-lab/main/deps/kubectl_krew_plugins
+
+kubectl krew install < kubectl_krew_plugins
+```
+
+### k8s shell aliases
+
+Add some [helpful k8s aliases](https://github.com/jessebot/dot_files/blob/main/.bashrc_k8s):
+
+```bash
+# copy the file
+curl -O https://raw.githubusercontent.com/jessebot/dot_files/main/.bashrc_k8s
+
+# load the file for your current shell
+source ~/.bashrc_k8s
+```
+
+To have the above file sourced every new shell, copy this into your `.bashrc` or `.bash_profile`:
+
+```bash
+# include external .bashrc_k8s if it exists
+if [ -f $HOME/.bashrc_k8s ]; then
+    . $HOME/.bashrc_k8s
+fi
+```