diff --git a/enhancements/microshift/loadbalancer-service-support.md b/enhancements/microshift/loadbalancer-service-support.md new file mode 100644 index 00000000000..c06d3eb92f4 --- /dev/null +++ b/enhancements/microshift/loadbalancer-service-support.md @@ -0,0 +1,199 @@ +--- +title: loadbalancer-service-support.md +authors: + - "@pliurh" +reviewers: + - "@copejon, MicroShift contributor" + - "@fzdarsky, MicroShift architect" + - "@ggiguash, MicroShift contributor" + - "@dhellmann, MicroShift contributor" + - "@oglok, MicroShift contributor" + - "@zshi-redhat, MicroShift " +approvers: + - "@dhellmann" +api-approvers: + - None +creation-date: 2023-01-13 +last-updated: 2023-01-13 +tracking-link: + - https://issues.redhat.com/browse/NP-604 +--- + +# MicroShift Service of Loadbalancer Type Support + +## Summary + +Customers using loadbalancer type Service, doesn't want to make changes to their +workloads/deployment for MicroShift as they use other k8s flavors as well. It is +also promised by us that users do not have to modify workload manifests that +work on OpenShift to run on MicroShift (as long as they do not use APIs +unsupported by MicroShift). + +## Motivation + +Normally, the loadbalancer type Service is supported by a Kubernetes cluster +with external load balancers. In Microshift, as there is only one node in the +cluster, we don't really need to have a load balancer. However, from user +experience perspective, it's important for customers to be able to migrate +the workload to Microshift without modification. So we need to find a way to +support the loadbalancer type Service without a load balancer. + +### User Stories + +As a Microshift user, I want to reuse the Service manifest that with type of +`LoadBalancer` without modification. I don't want to allocate a new IP address +to the Node as the loadbalancer VIP. I want to use the NodeIP as the ingress IP +of the loadbalancer Service. + +### Goals + +* Allow users to deploy the load balancer type Service to Microshift. +* The node IP shall be used as the ingress IP of the Service. +* As we will support pluggable CNI in Microshift. The implementation shall be +also compatible with CNIs other than OVN-Kubernetes. + +### Non-Goals + +* + +## Proposal + +To support the loadbalancer type Service, we need to create a new controller +component that watch the Service objects. It shall be responsible for: + +1. work with CNIs to plumb the ingress traffic to the endpoint pods. +2. publish the ingress endpoint information in the Service's + `.status.loadBalancer` field. + +### Workflow Description + +**cluster user** is a human user responsible for provisioning Services to a +Microshift cluster. + +1. The cluster user deploys a Service of type: LoadBalancer +2. The loadbalancer service controller watches the creation of the service. It +will update the Service's `.status.loadBalancer` field, and help the CNIs to +plumbing the ingress traffic to the endpoint pods. + +### API Extensions + +None + +### Implementation Details + +This new controller is part of the microshift binary. It shall be able to know +what CNI is used by the cluster, and behave differently based on the CNI type. + +### Risks and Mitigations + +The pluggable CNI design of Microshift is still under discussion, we need to +make sure that CNI type can be aware in Microshift binary. + +### Drawbacks + +It is not a CNI agnostic design. The loadbalancer Service controller will behave +differently between OVN-Kubernetes and other CNIs. + +## Design Details + +The Service of loadblancer type is designed to exposes the Service externally +using a cloud provider's load balancer. Normally, there is a loadbalancer +Service controller which is responsible for: + +1. provision a loadbalancer for the Service. +2. publish the ingress endpoint information in the Service's + `.status.loadBalancer` field. + +However, in the context of Microshift, as we only get one node in the cluster, +it doesn't make sense to use a real load balancer to forward the ingress +traffic. So instead of provisioning the loadbalancer instance, the controller +is only responsible for helping the CNIs to plumbing the ingress traffic. + +### Traffic Plumbing + +#### OVN-Kubernetes + +OVN-Kubernetes has already implemented the [loadbalancer ingress +IP](https://github.com/openshift/ovn-kubernetes/blob/master/docs/external-ip-and-loadbalancer-ingress.md) +support. The ovnkube-node component which is running on each host watches the +Service's `.status.loadBalancer` field, and insert iptables rules that does DNAT +to the clusterIP of the Service. Therefore, from the loadbalancer Service +controller perspective, it will only be responsible for the updating the status +of the Service. The ovn-kubernetes will do the rest of the job. + +#### Other CNIs + +For other CNIs, which don't have the similar function as OVN-Kubernetes, neither +do they bypass the kernel for the ingress traffic, we can take the similar +approach as K3S. The K3S's loadbalancer Service controller provisions a +[klipper-lb](https://github.com/k3s-io/klipper-lb) pod to each nodes for each +loadbalancer Service. The pod can insert iptables rules which DNAT ingress +traffic to the nodeIP of the Service. + + +### Update Status +As, we don't want to ask user to allocated additional VIP as the loadbalancer +IP. We can say that the ingress ip of the Service's `.status.loadBalancer` field +shall always be the existing IPs of the node. When there are multiple IPs, the +kubelet `NodeIP` shall be used by default. + +The controller shall also check if the Service port has already been allocated +to other loadbalancer Service before updating the Service status. + +### Open Questions [optional] + +* Do we want to allow users to choose the IP other than the NodeIP as the +Ingress IP of loadbalancer Service, when there are multiple IPs available? + +### Test Plan + +N/A + +### Graduation Criteria + +N/A + +#### Dev Preview -> Tech Preview + +N/A + +#### Tech Preview -> GA + +N/A + +#### Removing a deprecated feature + +N/A + +### Upgrade / Downgrade Strategy + +N/A + +### Version Skew Strategy + +N/A + +### Operational Aspects of API Extensions + +N/A + +#### Failure Modes + +If the port of one Service has already been occupied by either a node process +or other loadbalancer Services, the controller shall leave the +`.status.loadBalancer` of the Service blank, thus the Service will be put in +pending state. + +#### Support Procedures + +N/A + +## Implementation History + +N/A + +## Alternatives + +Similar to the `Drawbacks` section the `Alternatives` section is used to +highlight and record other possible approaches to delivering the value proposed +by an enhancement.