GitHub Action
Kubernetes Chaos
This action provides a way to perform different chaos experiments on the Kubernetes environment. It contains Litmus Chaos experiments to run the chaos and find a weakness in the system. For more details about chaos engineering in Kubernetes using Litmus visit litmus-docs .
Kubernetes 1.11 or later.
There is a number of chaos experiments that can be performed using github-chaos-actions
, you can select the one which you want to perform, and for more details about the experiment please visit the experiment docs section.
We just need to follow these simple steps to run a chaos experiment using this action:
-
Install Litmus: Before running any experiment we need to setup litmus in the cluster. If litmus is not already installed then we can install it from
github-chaos-action
by just passing an ENVINSTALL-LITMUS
withtrue
value. This will bring up litmus with all infra components running in litmus namespace. -
Deploy Application: After having litmus installed in the cluster we need to have an application running on which the chaos will be performed. The user has to create an application and pass the application details through action's ENV. The details involved application kind (deployment,statefulset,daemonset), application label, and namespace.
-
Select experiment: Select an experiment from the list of experiments mentioned in the below section which you want to perform on an application. Get the details of the experiment and how to run the actions for a particular experiment.
The different experiments that can be performed using github-chaos-actions
are:
-
Pod Delete: This chaos action causes random (forced/graceful) pod delete of application deployment replicas. It tests deployment sanity (high availability & uninterrupted service) and recovery workflows of the application pod. Check a sample usage of pod delete chaos action and for more details about the experiment please visit pod delete docs.
-
Container Kill: This chaos action executes SIGKILL on the container of random replicas of application deployment. It tests the deployment sanity (high availability & uninterrupted service) and recovery workflows of an application. Check a sample usage of container kill chaos action and for more details about the experiment please visit container kill docs.
-
Node CPU Hog: This chaos action causes CPU resource exhaustion on the Kubernetes node. The experiment aims to verify the resiliency of applications that operate under resource constraints wherein replicas may sometimes be evicted on account on nodes turning unschedulable (Not Ready) due to lack of CPU resources. Check a sample usage of node cpu hog chaos action and for more details about the experiment please visit node cpu hog docs .
-
Node Memory Hog: This chaos action causes Memory exhaustion on the Kubernetes node. The experiment aims to verify the resiliency of applications that operate under resource constraints wherein replicas may sometimes be evicted on account on nodes turning unschedulable due to lack of Memory resources. Check a sample usage of node memory hog chaos action and for more details about the experiment please visit node memory hog docs.
-
Pod CPU Hog: This chaos action causes CPU resource consumption on specified application containers by starting one or more md5sum calculation process on the special file /dev/zero. It Can test the application's resilience to potential slowness/unavailability of some replicas due to high CPU load. Check a sample usage of pod cpu hog chaos action and for more details about the experiment please visit pod cpu hog docs.
-
Pod Memory Hog: This chaos action causes Memory resource consumption on specified application containers by using dd command which will be used to consume memory of the application container for a certain duration of time. It can test the application's resilience to potential slowness/unavailability of some replicas due to high Memory load. Check a sample usage of pod memory hog chaos action and for more details about the experiment please visit Pod Memory hog docs.
-
Disk Fill: This chaos action causes Disk Stress by filling up the Ephemeral Storage of the Pod using one of it containers. It forced the Pod to get Evicted if the Pod exceeds it Ephemeral Storage Limit.It tests the Ephemeral Storage Limits, to ensure those parameters are sufficient.Check a sample usage of disk fill chaos action and for more details about the experiment please visit Disk Fill hog docs.
-
Pod Network Corruption: This chaos action Injects packet corruption on the specified container by starting a traffic control (tc) process with netem rules to add egress packet corruption. Corruption is injected via pumba library with command Pumba netem corruption bypassing the relevant network interface, packet-corruption-percentage, chaos duration, and regex filter for the container name. Check a sample usage of pod network corruption chaos action and for more details about the experiment please visit pod network corruption docs.
-
Pod Network Latency: This chaos action causes flaky access to application replica by injecting network delay using Pumba. It injects latency on the specified container by starting a traffic control (tc) process with netem rules to add egress delays. It Can test the application's resilience to lossy/flaky network. Check a sample usage of pod network latency chaos action and for more details about the experiment please visit pod network latency docs.
-
Pod Network Loss: This chaos action injects chaos to disrupt network connectivity to Kubernetes pods. The application pod should be healthy once chaos is stopped. It causes loss of access to application replica by injecting packet loss using Pumba. Check a sample usage of pod network loss chaos action and for more details about the experiment please visit pod network loss docs
A sample pod delete experiment workflow:
.github/workflows/main.yml
name: CI
on:
push:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- name: Running Litmus pod delete chaos experiment
uses: mayadata-io/github-chaos-actions@master
env:
##Pass kubeconfig data from secret in base 64 encoded form
KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }}
##If litmus is not installed
INSTALL_LITMUS: true
##Give application info under chaos
APP_NS: default
APP_LABEL: run=nginx
APP_KIND: deployment
EXPERIMENT_NAME: pod-delete
##Custom image can also been used
EXPERIMENT_IMAGE: litmuschaos/ansible-runner:latest
TOTAL_CHAOS_DURATION: 30
CHAOS_INTERVAL: 10
FORCE: false
##Select true if you want to uninstall litmus after chaos
LITMUS_CLEANUP: true
Get the details of the chaos action tunables for pod delete (above example) here
KUBE_CONFIG_DATA
– required: A base64-encoded kubeconfig file with credentials for Kubernetes to access the cluster. You can get it by running the following command:
cat $HOME/.kube/config | base64
Some comman environment variables used for running the github-chaos-actions
are:
Variables | Description | Specify In Chaos Action | Default Value |
---|---|---|---|
EXPERIMENT_NAME | Give the experiment name you want to run(check the list of experiments available under experiment folder) | Mandatory | No default value |
APP_NS | Provide namespace of application under chaos | Optional | Default value is default |
APP_LABEL | Provide application label of application under chaos. | Optional | Default value is run=nginx |
APP_KIND | Provide the kind of application | Optional | Default value is deployment |
INSTALL_LITMUS | Keep it true to install litmus if litmus is not already installed. | Optional | Default value is not set to true |
LITMUS_CLEANUP | Keep it true to uninstall litmus after chaos | Optional | Default value is not set to true |
EXPERIMENT_IMAGE | We can provide cumstom image for running litmus chaos experiment | Optional | Default value is litmuschaos/ansible-runner:latest |