kubernetes-operator

Kubernetes Operator Study Journey

1. Try using existing operators

1. prometheus-operator
2. postgres-operator
3. strimzi
4. argocd: appcontroller.go
5. grafana-operator
6. mysql-operator
7. terraform-k8s

And more

2. Understand what is Kubernetes operator.

1. Kubernetes Controller components.
2. How Kubernetes Controlloer works.
3. Custom Resource.

Kubernetes Operator

A Kubernetes operator is an application-specific controller that extends the functionality of the Kubernetes API to create, configure, and manage instances of complex applications on behalf of a Kubernetes user.

From What is a Kubernetes operator?

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

From https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

An Operator is like an automated Site Reliability Engineer for its application.

From Kubernetes Operators ~ Automating the Container Orchestration Platform ~

Operator vs. Controller

• Controller（Custom Controller）:Custom Resourceの管理を行うController。Control Loop（Reconciliation Loop）を実行するコンポーネント
• Operator: CRDとCustom Controllerのセット。etcd operatorやmysql operatorなどのように、特定のソフトウェアの管理を自動化するためのソフトウェア

From 実践入門Kubernetesカスタムコントローラーへの道

• Controllers can act on core resources such as deployments or services, which are typically part of the Kubernetes controller manager in the control plane, or can watch and manipulate user-defined custom resources.
• Operators are controllers that encode some operational knowledge, such as application lifecycle management, along with the custom resources defined in Chapter 4.

From Programming Kubernetes

• A controller is a loop that reads desired state ("spec), observed cluster state (others' "status"), and external state, and the reconciles cluster state and external state with the desired state, writing any observations down (to our own "status").
• All of Kubernetes functions on this model.
• An operator is a controller that encodes human operational knowledge: how do I run and manage a specific piece of complex software.
• All operators are controllers, but not all controllers are operators.

From Tutorial: Zero to Operator in 90 Minutes! - Solly Ross, Google (YouTube)

For more detail:

• CNCF Operator White Paper - Final Version
• CNCF White Paper

3. Create a sample operator following a tutorial

There are several ways to create an operator. You can try any of them:

1. operator-sdk
   1. go-based: https://github.com/nakamasato/memcached-operator
   2. helm-based: https://github.com/nakamasato/nginx-operator
   3. ansible-based: https://github.com/nakamasato/memcached-operator-with-ansible
2. kubebuilder
   1. Tutorial: Building CronJob
3. metacontroller
4. KUDO (Kubernetes Universal Declarative Operator)
5. つくって学ぶKubebuilder

You can also reference example controllers:

1. Sample Controller
2. Istio Example Controller
3. Foo Controller with Kubebuilder
4. Memcached Operator with Operator SDK

4. Understand more detail about each component

Simplified:

Detailed:

More Detailed:

from https://github.com/kubernetes/sample-controller/blob/master/docs/images/client-go-controller-interaction.jpeg

1. client-go:
   1. clientset is a client for the built-in API resources.
   2. informer: watch the changes of objects and reflect the changes to the in-memory-cache.
      1. factory: informers.NewSharedInformerFactory
      2. watcher
      3. lister
      4. indexer
      5. event handler
      6. reflector
   3. lister: Get data from in-memory cache.
   4. indexer: in-memory cache
   5. workqueue: A queue to store items that the controller will process.
2. code-generator:
   1. Generate codes for clientset for a custom resource.
3. apimachinery:
   1. Scheme: connects Kubernetes API and Go Types, API version conversion
4. controller-runtime
   1. builder
   2. cache
   3. client
   4. cluster
   5. controller
   6. eventhandler
   7. inject
   8. log
   9. manager
   10. reconciler
   11. source
   12. webhook

Reference:

• https://adevjoe.com/post/client-go-informer/
• https://www.huweihuang.com/kubernetes-notes/code-analysis/kube-controller-manager/sharedIndexInformer.html

5. Create your own operator

After creating a sample operator, you should have deeper understanding of Kubernetes operator. Now you can think about what kind of problem that you want to resolve by utilizing operator pattern.

To clarify a problem to resolve with a new operator, you can reference existing operators:

operator	role	language
prometheus-operator	Manage Prometheus, Alertmanager and their configuration	Golang
mysql-operator	Manage MySQL cluster	Python
postgres-operator	Manage PostgreSQL cluster (version upgrade, live volume resize, ...)	Golang
strimzi-kafka-operator	Manage Kafka cluster, user, and topic	Java
...	...	...

Considerations:

• Finalizer

• docs: Owners and Dependents: example

• Reconciliation Loop

  • [operator-sdk] Based on the return value of Reconcile() the reconcile Request may be requeued and the loop may be triggered again: (Building a Go-based Memcached Operator using the Operator SDK)

    // Reconcile successful - don't requeue
    return reconcile.Result{}, nil
    // Reconcile failed due to error - requeue
    return reconcile.Result{}, err
    // Requeue for any reason other than error
    return reconcile.Result{Requeue: true}, nil

  • operator-framework/operator-sdk#4209 (comment)
  • How can I have separate logic for Create, Update, and Delete events? When reconciling an object can I access its previous state? -> You should not have separate logic. Instead design your reconciler to be idempotent.
    • Q: How do I have different logic in my reconciler for different types of events (e.g. create, update, delete)? in controller-runtime

• Testing

  • KUbernetes Testing TooL (kuttl): https://kuttl.dev/ KUTTL is built to support some kubernetes integration test scenarios and is most valuable as an end-to-end (e2e) test harness.
  • Ginkgo (A Golang BDD Testing Framework): https://onsi.github.io/ginkgo/
  • Gomega (Ginkgo's preferred matcher library): https://onsi.github.io/gomega/
  • kubetest2: https://github.com/kubernetes-sigs/kubetest2: Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes. It is intended to be the next significant iteration of kubetest.

• Managing Errors

  • https://cloud.redhat.com/blog/kubernetes-operators-best-practices
  1. Return the error in the status of the object.
  2. Generate an event describing the error.

• Webhook

  • Admission Webhook
    • two types:
      • Mutating Webhook: Make some modifications for a request. e.g. set default value. (Defined with MutatingAdmissionConfiguration)
      • Validating Webhook: Validate a request. (Defined by ValidatingAdmissionConfiguration)
    • Request: AdmissionReview
    • Response: AdmissionReview with response.allowed boolean field.
  • Conversion Webhook
    • hub & spoke
    • isConvertible: need to have Hub & all non-Hub types must be able to convert to/from Hub
    • request & response: ConversionReview

• Indexing

6. Tools

• https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller/controllerutil
• https://github.com/spf13/cobra: a library for creating powerful modern CLI applications & a program to generate applications and command files.
  • Cobra is used in many Go projects such as Kubernetes, Hugo, and Github CLI to name a few. This list contains a more extensive list of projects using Cobra.

7. Study Golang for better code quality

1. golang-standanrds/project-layout
2. Learn Go with tests
3. GoとDependency Injectionの現在
4. Go Blog
5. Gopher Reading List
6. Type Embedding

8. Other Topics

1. Write Kubernetes Operator in other languages
   • kopf for Python
   • fabric8io/kubernetes-client for Java
   • java-operator-sdk/java-operator-sdk Build Kubernetes Operators in Java Without Hassle
2. Optimistic Concurrency Control

9. Keep learning

1. 47 Things To Become a Kubernetes Expert
2. Kubernetes API Basics - Resources, Kinds, and Objects
3. Kubernetes API Conventions
4. How To Call Kubernetes API using Simple HTTP Client
5. How To Call Kubernetes API using Go - Types and Common Machinery
6. How To Extend Kubernetes API - Kubernetes vs. Django
7. 在不生成 crd client 代码的情况下通过 client-go 增删改查 k8s crd 资源
8. kubebuilder vs operator-sdk (2019-04-10)
9. client-go 中的 informer 源码分析
10. Operator Best Practices