Jobbernetes

This README is a work in progress. Please refer to the thesis for more information.

Note: developed and tested on WSL2 running Ubuntu 20.04.2 LTS.

Prerequisites:

• Docker
• K3d
• Kubectl
• Kufecfg
• Jsonnet
• Python 3

Quick Start:

# Create cluster and deploy everything
./scripts/deploy/demo.py

# Start ingesting data into the work queue (using the InputService)
#   Create 2 threads
#   On every thread, send a new UUID every 0.06 seconds
./scripts/tools/send_text.py 0.06 -p 2

Navigate to http://grafana.localhost/ and open the Jobbernetes dashboard and http://dataviewer.jn.localhost/images to see the generated images.

Ingress

See also: ./kubernetes-specs/lib/components.libsonnet

• Grafana
• RabbitMQ
• Rancher
• DataViewer
• InputService
• ExternalDataService

Caveats:

Grafana cannot tail logs from Loki because it connects to a wss web socket. Unfortunately, the ingress is not configured to handle this. Firstly, it should not use encryption, and secondly, I suspect the ingress is not setting the correct headers.

We need to use port-forwarding to tail the logs:

./scripts/tools/port_forward.py

Access as follows: http://localhost:9092

Commands

# General
kubectl get po          # po  = pods
kubectl get svc         # svc = services
kubectl get ing         # ing = ingress

# Show more information
kubectl get <type> -o wide

# Show k8s object as yaml (spec + status)
kubectl get <type> <name> -o yaml

# Secrets
kubectl get secrets
kubectl get secret <name> -o jsonpath="{.data}"
kubectl get secret <name> -o jsonpath="{.data.password}" | base64 --decode

# Logs
kubectl logs -f <pod> [<container>]

# Port Forwarding
kubectl port-forward --namespace default svc/<name> <host>:<k8s>

# Storage
kubectl get sc      # sc = StorageClass
kubectl get pv      # pv = PersistentVolume
kubectl get pvc     # sc = PersistentVolumeClaim

Switch context

Also this: https://github.com/ahmetb/kubectx

# https://k3d.io/usage/kubeconfig/
set -gx KUBECONFIG (k3d kubeconfig write jobbernetes-cluster)

Manually trigger CronJobs:

https://stackoverflow.com/questions/40401795/how-can-i-trigger-a-kubernetes-scheduled-job-manually

kubectl create job --from=cronjob/<cronjob-name> <job-name>

sysctl

sudo sysctl -w sysctl vm.swappiness=2
sudo sysctl -w fs.inotify.max_user_instances=512

Show swap usage per process

for file in /proc/*/status ; awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; end | sort -k 2 -n -r

Dependencies

Python

pip install requests