From 2a72e8f6b81fa43bd6e84cde0d6c45d308c883ac Mon Sep 17 00:00:00 2001 From: Peng Liu Date: Fri, 13 Jan 2023 04:08:20 -0500 Subject: [PATCH] Add MicroShift Service of Loadbalancer Type Support Signed-off-by: Peng Liu --- .../loadbalancer-service-support.md | 230 ++++++++++++++++++ 1 file changed, 230 insertions(+) create mode 100644 enhancements/microshift/loadbalancer-service-support.md diff --git a/enhancements/microshift/loadbalancer-service-support.md b/enhancements/microshift/loadbalancer-service-support.md new file mode 100644 index 0000000000..dbc4c1ea65 --- /dev/null +++ b/enhancements/microshift/loadbalancer-service-support.md @@ -0,0 +1,230 @@ +--- +title: loadbalancer-service-support +authors: + - "@pliurh" +reviewers: + - "@copejon, MicroShift contributor" + - "@fzdarsky, MicroShift architect" + - "@ggiguash, MicroShift contributor" + - "@dhellmann, MicroShift contributor" + - "@oglok, MicroShift contributor" + - "@zshi-redhat, MicroShift " + - "@pmtk, MicroShift" + - "@mangelajo, MicroShift" +approvers: + - "@dhellmann" +api-approvers: + - None +creation-date: 2023-01-13 +last-updated: 2023-01-13 +tracking-link: + - https://issues.redhat.com/browse/NP-604 +--- + +# MicroShift Service of Loadbalancer Type Support + +## Summary + +Customers using loadbalancer type Service don't want to make changes to their +workloads/deployment for MicroShift as they use other k8s flavors as well. It is +also promised by us that users can create workload manifests that can work on +both OpenShift and MicroShift (as long as they do not use APIs unsupported by +MicroShift). + +## Motivation + +Normally, the loadbalancer type Service is supported by a Kubernetes cluster +with external load balancers. In MicroShift, as there is only one node in the +cluster, we don't really need to have a load balancer. However, from user +experience perspective, it's important for customers to be able to deploy the +workload on both OpenShift MicroShift with the same manifests. And there are +workload types that require a load balancer service (for example, if they are +not using HTTP/S), so that other features like Ingress or node port service will +not work. Therefore we need to find a way to support the loadbalancer type +Service on MicroShift. + +### User Stories + +As a Microshift user, I want to be able to create a Service manifest with the +type of `LoadBalancer` and use it on both OCP and MicroShift. I don't want to +allocate a new IP address to the Node as the loadbalancer VIP. I want to use the +NodeIP as the ingress IP of the loadbalancer Service. + +### Goals + +* Allow users to deploy the load balancer type Service to MicroShift. +* The node IP shall be used as the ingress IP of the Service. +* As we will support pluggable CNI in MicroShift. The implementation shall be +also compatible with CNIs other than OVN-Kubernetes. + +### Non-Goals + +* + +## Proposal + +To support the loadbalancer type Service, we need to create a new controller +component that watch the Service objects. It shall be responsible for: + +1. work with CNIs to plumb the ingress traffic to the endpoint pods. +2. publish the ingress endpoint information in the Service's + `.status.loadBalancer` field. + +### Workflow Description + +**cluster user** is a human user responsible for provisioning Services to a +MicroShift cluster. + +1. The cluster user deploys a Service of type: LoadBalancer +2. The loadbalancer service controller watches the creation of the service. It +will update the Service's `.status.loadBalancer` field, and help the CNIs to +plumbing the ingress traffic to the endpoint pods. + +### API Extensions + +None + +### Implementation Details + +This new controller is part of the microshift binary. It shall be be limited to +the OVN-K CNI implementation, for now. Updating it to support other CNI +implementations will be left to a future enhancement. + +### Risks and Mitigations + +The pluggable CNI design of MicroShift is still under discussion, we need to +make sure that CNI type can be aware in MicroShift binary. + +### Drawbacks + +It is not a CNI agnostic design. The loadbalancer Service controller will behave +differently between OVN-Kubernetes and other CNIs. + +## Design Details + +The Service of loadblancer type is designed to exposes the Service externally +using a cloud provider's load balancer. Normally, there is a loadbalancer +Service controller which is responsible for: + +1. provision a loadbalancer for the Service. +2. publish the ingress endpoint information in the Service's + `.status.loadBalancer` field. + +However, in the context of MicroShift, as we only get one node in the cluster, +it doesn't make sense to use a real load balancer to forward the ingress +traffic. So instead of provisioning the loadbalancer instance, the controller +is only responsible for helping the CNIs to plumbing the ingress traffic. + +### Traffic Plumbing + +Since different CNIs get different implementation on how to support Kubernetes +Services. We need to do the traffic plumbing differently for different CNI. The +first version of the controller shall _not_ be aware of the CNI type from the +configuration. Enhancing it to support other CNI implementations is deferred to +a future release. + +#### OVN-Kubernetes + +OVN-Kubernetes has already implemented the [loadbalancer ingress +IP](https://github.com/openshift/ovn-kubernetes/blob/master/docs/external-ip-and-loadbalancer-ingress.md) +support. The ovnkube-node component which is running on each host watches the +Service's `.status.loadBalancer` field, and insert iptables rules that does DNAT +to the clusterIP of the Service. Therefore, from the loadbalancer Service +controller perspective, it will only be responsible for the updating the status +of the Service. The ovn-kubernetes will do the rest of the job. + +#### Notes for Future Work to Support Other CNIs + +For other CNIs, which don't have the similar function as OVN-Kubernetes, neither +do they bypass the kernel for the ingress traffic, we can take the similar +approach as K3S. The K3S's loadbalancer Service controller provisions a +[klipper-lb](https://github.com/k3s-io/klipper-lb) pod to each nodes for each +loadbalancer Service. The pod can insert iptables rules which DNAT ingress +traffic to the nodeIP of the Service. + +Another option is manipulating the iptables rule by this controller itself. +Since there is only one node, and the microshsift binary has the access to the +host network. + +For those CNIs that relays on kube-proxy to support the kubernetes Service. We +can also consider to use kube-proxy to do the job. + +### Update Status +As, we don't want to ask user to allocated additional VIP as the loadbalancer +IP. We can say that the ingress ip of the Service's `.status.loadBalancer` field +shall always be the existing IPs of the node. When there are multiple IPs, the +kubelet `NodeIP` shall be used by default. + +The controller shall also check if the Service port has already been allocated +to other loadbalancer Service before updating the Service status. To avoid the +risk of a race condition, we shall implement the controller with a work queue +and one worker. All the service update/creation requests will be handled in +sequence. + +### Open Questions [optional] + +* Do we want to allow users to choose the IP other than the NodeIP as the + Ingress IP of loadbalancer Service, when there are multiple IPs available? + + A: We will leave this as an open question for now and wait for an RFE to help + us shape requirements. + +* How do we make this implementation compatible with other loadbalancer + implementations such as metal-lb? + + A: To avoid the possible conflict between loadbalancer implementations, we + plan to expose a configuration flag that allows a user to turn the built-in + controller off. But it will not be implemented until we get a definite + requirement to support other loadbalancer implementations. + +### Test Plan + +N/A + +### Graduation Criteria + +N/A + +#### Dev Preview -> Tech Preview + +N/A + +#### Tech Preview -> GA + +N/A + +#### Removing a deprecated feature + +N/A + +### Upgrade / Downgrade Strategy + +N/A + +### Version Skew Strategy + +N/A + +### Operational Aspects of API Extensions + +N/A + +#### Failure Modes + +If the port of one Service has already been occupied by either a node process +or other loadbalancer Services, the controller shall leave the +`.status.loadBalancer` of the Service blank, thus the Service will be put in +pending state. + +#### Support Procedures + +N/A + +## Implementation History + +N/A + +## Alternatives + +Implement a k8s Cloud Controller Manager for MicroShift. The Cloud Controller +Manager will be running in the pod of the MicroShift Cluster.