brew tap tchoupinax/brew
brew install tchoupinax/brew/kubernetes-labels-migrator
If you need help, the help flag can show you available flags (kubernetes-labels-migrator -help
).
kubernetes-labels-migrator \
-deployment="my-application" \
-namespace="default" \
-label="kubernetes.io/app" \
-value="my-application"
Let's say you are managing your application with the GitOps philosophy (ArgoCD is really great for this job). You want to add or edit a label. This change has already been made in your code, but ArgoCD refused to sync due to the "immutable field" error. Here's what you should do:
⚠️ First of all, ensure that nothing intervenes during the operation! kubernetes-labels-migrator should be free until the end. If you have tools like ArgoCD that synchronize your resources, please ensure this feature is disabled.- Commit the changes to the resource and observe the difference.
- Launch the script and watch the magic happen.
- According to your changes, the difference should have been resolved.
Deployment's labels are immutable. That means, if you have to edit, rename or add labels to a deployed application, you have to delete the deployment. On a production application, it is very annoying.
To be able to add labels without any downtime, we have to use a blue-green strategy. This projects aims to do this operation automatically and without human manual operation. The production must be safe!
Labels are immutable by design but moreover they can be used as selector for a service. That means if you break a label, you could cut the link between a Kubernetes service and a pod. That definitively not something you want to happen in your production. To prevent from meeting this issue, we make this operation.
We have a deployment called api
with labels app=api
and kubernetes.io/name=Api
. However, this label is bad typed and should be totally in lowercase. So we want to rename it. A service, called api
too, targets pods with labels app=api
kubernetes.io/name=Api
.
- Clone deployment: we will create the same deployment, called the same and suffixed by
changing-label-tmp
. This deployment won't have the label we want to edit or remove (here it'skubernetes.io/name
). - Edit the service: we will update the service to set selectors to match a subset of labels that matches both deployment. It's mandatory to ensure the service will loadbalance traffic between pods accross both deployment.
- Wait all pods are ready: you might know that Kubernetes waits until a pod is alive (liveness) and ready (readyness) to send traffic to it. That the goal here and we are querying Kubernetes's API to ensure both deployment have all of their pod ready.
- Delete the old deployment: as we want to rewrite label, we have to delete and create again the deployment. Labels are immutable and it's the only way to do it.
- Create the deployment with new label: this deployment has all final labels but continue to work with the current service. Remember this service targets a subset of selector.
- Wait all pods are ready: we repeat the operation about the new deployment.
- Remove temporary deployment: now that the deployment is ready, we can cut the temporary one and leaves in place the new deployment with modified label.
- [Optional] Add the label to service selector: some labels are recommenced by Kubernetes to be present as default label and thoses labels might be added to the service selector. That means we are making the matching set stronger.
- See documentation
- For the moment, only
kubernetes.io/name
is considered to be added to the service
Additional documentations:
Using Vegeta, we can overload the service to detect if one or more request is thrown during the process.
Ingress
or any others way to reach the service.
The following command requires two projects to work:
To install these project, you can enter these commands:
# MacOS
brew install rs/tap/jplot
brew install rs/tap/jaggr
Then, enter this command
echo 'GET YOUR_URL' | \
vegeta attack -rate 200 -duration 10m | vegeta encode | \
jaggr @count=rps \
hist\[100,200,300,400,500\]:code \
p25,p50,p95:latency \
sum:bytes_in \
sum:bytes_out | \
jplot rps+code.hist.100+code.hist.200+code.hist.300+code.hist.400+code.hist.500 \
latency.p95+latency.p50+latency.p25 \
bytes_in.sum+bytes_out.sum
- There is not diff in ArgoCD if the label does not impact matched labels
1.0.0
- Basic documentation
- Successful test with zero downtime
- Handle complexe resources (see Managed resources)
- Better handling of labels added to the service and well understand of K8S's good pratices
- Matching labels are
app.kubernetes.io/instance
andapp.kubernetes.io/name
, - Documentation about potentials issues from scale (DB)
- Handle the fact the edited label is the only one used for the service - A warning is displayed
- Successful migration in production with relatively high volume