Add MicroShift Service of Loadbalancer Type Support

Signed-off-by: Peng Liu <[email protected]>

enhancements/microshift/loadbalancer-service-support.md

@@ -0,0 +1,199 @@
+---
+title: loadbalancer-service-support.md
+authors:
+  - "@pliurh"
+reviewers:
+  - "@copejon, MicroShift contributor"
+  - "@fzdarsky, MicroShift architect"
+  - "@ggiguash, MicroShift contributor"
+  - "@dhellmann, MicroShift contributor"
+  - "@oglok, MicroShift contributor"
+  - "@zshi-redhat, MicroShift "
+approvers:
+  - "@dhellmann"
+api-approvers:
+  - None
+creation-date: 2023-01-13
+last-updated: 2023-01-13
+tracking-link:
+  - https://issues.redhat.com/browse/NP-604
+---
+
+# MicroShift Service of Loadbalancer Type Support
+
+## Summary
+
+Customers using loadbalancer type Service, doesn't want to make changes to their
+workloads/deployment for MicroShift as they use other k8s flavors as well. It is
+also promised by us that users do not have to modify workload manifests that
+work on OpenShift to run on MicroShift (as long as they do not use APIs
+unsupported by MicroShift).
+
+## Motivation
+
+Normally, the loadbalancer type Service is supported by a Kubernetes cluster
+with external load balancers. In Microshift, as there is only one node in the
+cluster, we don't really need to have a load balancer. However, from user
+experience perspective, it's important for customers to be able to migrate
+the workload to Microshift without modification. So we need to find a way to
+support the loadbalancer type Service without a load balancer.
+
+### User Stories
+
+As a Microshift user, I want to reuse the Service manifest that with type of
+`LoadBalancer` without modification. I don't want to allocate a new IP address
+to the Node as the loadbalancer VIP. I want to use the NodeIP as the ingress IP
+of the loadbalancer Service. 
+
+### Goals
+
+* Allow users to deploy the load balancer type Service to Microshift.
+* The node IP shall be used as the ingress IP of the Service.
+* As we will support pluggable CNI in Microshift. The implementation shall be
+also compatible with CNIs other than OVN-Kubernetes.
+
+### Non-Goals
+
+* 
+
+## Proposal
+
+To support the loadbalancer type Service, we need to create a new controller
+component that watch the Service objects. It shall be responsible for:
+
+1. work with CNIs to plumb the ingress traffic to the endpoint pods.
+2. publish the ingress endpoint information in the Service's
+   `.status.loadBalancer` field.
+
+### Workflow Description
+
+**cluster user** is a human user responsible for provisioning Services to a
+Microshift cluster.
+
+1. The cluster user deploys a Service of type: LoadBalancer 
+2. The loadbalancer service controller watches the creation of the service. It
+will update the Service's `.status.loadBalancer` field, and help the CNIs to 
+plumbing the ingress traffic to the endpoint pods.
+
+### API Extensions
+
+None
+
+### Implementation Details
+
+This new controller is part of the microshift binary. It shall be able to know
+what CNI is used by the cluster, and behave differently based on the CNI type.
+
+### Risks and Mitigations
+
+The pluggable CNI design of Microshift is still under discussion, we need to
+make sure that CNI type can be aware in Microshift binary.
+
+### Drawbacks
+
+It is not a CNI agnostic design. The loadbalancer Service controller will behave
+differently between OVN-Kubernetes and other CNIs.
+
+## Design Details
+
+The Service of loadblancer type is designed to exposes the Service externally
+using a cloud provider's load balancer. Normally, there is a loadbalancer
+Service controller which is responsible for:
+
+1. provision a loadbalancer for the Service.
+2. publish the ingress endpoint information in the Service's
+   `.status.loadBalancer` field.
+
+However, in the context of Microshift, as we only get one node in the cluster,
+it doesn't make sense to use a real load balancer to forward the ingress
+traffic. So instead of provisioning the loadbalancer instance, the controller
+is only responsible for helping the CNIs to plumbing the ingress traffic.
+
+### Traffic Plumbing
+
+#### OVN-Kubernetes
+
+OVN-Kubernetes has already implemented the [loadbalancer ingress
+IP](https://github.com/openshift/ovn-kubernetes/blob/master/docs/external-ip-and-loadbalancer-ingress.md)
+support. The ovnkube-node component which is running on each host watches the
+Service's `.status.loadBalancer` field, and insert iptables rules that does DNAT
+to the clusterIP of the Service. Therefore, from the loadbalancer Service
+controller perspective, it will only be responsible for the updating the status
+of the Service. The ovn-kubernetes will do the rest of the job.
+
+#### Other CNIs
+
+For other CNIs, which don't have the similar function as OVN-Kubernetes, neither
+do they bypass the kernel for the ingress traffic, we can take the similar
+approach as K3S. The K3S's loadbalancer Service controller provisions a
+[klipper-lb](https://github.com/k3s-io/klipper-lb) pod to each nodes for each
+loadbalancer Service. The pod can insert iptables rules which DNAT ingress
+traffic to the nodeIP of the Service.
+
+
+### Update Status
+As, we don't want to ask user to allocated additional VIP as the loadbalancer
+IP. We can say that the ingress ip of the Service's `.status.loadBalancer` field
+shall always be the existing IPs of the node. When there are multiple IPs, the
+kubelet `NodeIP` shall be used by default.
+
+The controller shall also check if the Service port has already been allocated
+to other loadbalancer Service before updating the Service status.
+
+### Open Questions [optional]
+
+* Do we want to allow users to choose the IP other than the NodeIP as the
+Ingress IP of loadbalancer Service, when there are multiple IPs available?
+
+### Test Plan
+
+N/A
+
+### Graduation Criteria
+
+N/A
+
+#### Dev Preview -> Tech Preview
+
+N/A
+
+#### Tech Preview -> GA
+
+N/A
+
+#### Removing a deprecated feature
+
+N/A
+
+### Upgrade / Downgrade Strategy
+
+N/A
+
+### Version Skew Strategy
+
+N/A
+
+### Operational Aspects of API Extensions
+
+N/A
+
+#### Failure Modes
+
+If the port of one Service  has already been occupied by either a node process
+or other loadbalancer Services, the controller shall leave the
+`.status.loadBalancer` of the Service blank, thus the Service will be put in
+pending state. 
+
+#### Support Procedures
+
+N/A
+
+## Implementation History
+
+N/A
+
+## Alternatives
+
+Similar to the `Drawbacks` section the `Alternatives` section is used to
+highlight and record other possible approaches to delivering the value proposed
+by an enhancement.