Introduce AEP with Provisioning Request CRD

cluster-autoscaler/proposals/provisioning-request.md

@@ -0,0 +1,326 @@
+# Provisioning Request CRD
+
+author: kisieland
+
+## Background
+
+Currently CA does not provide any way to express that a group of pods would like
+to have a capacity available.  
+This is caused by the fact that each CA loop picks a group of unschedulable pods
+and works on provisioning capacity for them, meaning that the grouping is random
+(as it depends on the kube-scheduler and CA loop interactions).  
+This is especially problematic in couple of cases:
+
+  - Users would like to have all-or-nothing semantics for their workloads.
+    Currently CA will try to provision this capacity and if it is partially
+    successful it will leave it in cluster until user removes the workload.
+  - Users would like to lower e2e scale-up latency for huge scale-ups (100
+    nodes+). Due to CA nature and kube-scheduler throughput, CA will create
+    partial scale-ups, e.g. `0->200->400->600` rather than one `0->600`. This
+    significantly increases the e2e latency as there is non-negligible time tax
+    on each scale-up operation.
+
+## Proposal
+
+### High level
+
+Provisioning Request (abbr. ProvReq) is a new namespaced Custom Resource that
+aims to allow users to ask CA for capacity for groups of pods.  
+It allows users to express the fact that group of pods is connected and should
+be threated as one entity.  
+This AEP proposes an API that can have multiple provisioning classes and can be
+extended by cloud provider specific ones.  
+This object is meant as one-shot request to CA, so that if CA fails to provision
+the capacity it is up to users to retry (such retry functionality can be added
+later on).
+
+### ProvisioningRequest CRD
+
+The following code snippets assume [kubebuilder](https://book.kubebuilder.io/)
+is used to generate the CRD:
+
+```go
+// ProvisioningRequest is a way to express additional capacity
+// that we would like to provision in the cluster. Cluster Autoscaler
+// can use this information in its calculations and signal if the capacity
+// is available in the cluster or actively add capacity if needed.
+type ProvisioningRequest struct {
+	metav1.TypeMeta `json:",inline"`
+	// Standard object metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
+	//
+	// +optional
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+	// Spec contains specification of the ProvisioningRequest object.
+	// More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status.
+	//
+	// +kubebuilder:validation:Required
+	Spec ProvisioningRequestSpec `json:"spec"`
+	// Status of the ProvisioningRequest. CA constantly reconciles this field.
+	//
+	// +optional
+	Status ProvisioningRequestStatus `json:"status,omitempty"`
+}
+
+// ProvisioningRequestList is a object for list of ProvisioningRequest.
+type ProvisioningRequestList struct {
+	metav1.TypeMeta `json:",inline"`
+	// Standard list metadata.
+	//
+	// +optional
+	metav1.ListMeta `json:"metadata"`
+	// Items, list of ProvisioningRequest returned from API.
+	//
+	// +optional
+	Items []ProvisioningRequest `json:"items"`
+}
+
+// ProvisioningRequestSpec is a specification of additional pods for which we
+// would like to provision additional resources in the cluster.
+type ProvisioningRequestSpec struct {
+	// PodSets lists groups of pods for which we would like to provision
+	// resources.
+	//
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:MinItems=1
+	// +kubebuilder:validation:MaxItems=32
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
+	PodSets []PodSet `json:"podSets"`
+
+	// ProvisioningClass describes the different modes of provisioning the resources.
+	// Supported values:
+	// * check-capacity.kubernetes.io - check if current cluster state can fullfil this request,
+	//   do not reserve the capacity.
+	// * atomic-scale-up.kubernetes.io - provision the resources in an atomic manner
+    // * ... - potential other classes that are specific to the cloud providers
+	//
+	// +kubebuilder:validation:Required
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
+	ProvisioningClass string `json:"provisioningClass"`
+
+	// AdditionalParameters contains all other parameters custom classes may require.
+	//
+	// +optional
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable"
+	AdditionalParameters map[string]string `json:"additionalParameters"`
+}
+
+type PodSet struct {
+	// PodTemplateRef is a reference to a PodTemplate object that is representing pods 
+	// that will consume this reservation (must be within the same namespace). 
+	// Users need to make sure that the  fields relevant to scheduler (e.g. node selector tolerations)
+	// are consistent between this template and actual pods consuming the Provisioning Request.
+	//
+	// +kubebuilder:validation:Required
+	PodTemplateRef Reference `json:"podTemplateRef"`
+	// Count contains the number of pods that will be created with a given
+	// template.
+	//
+	// +kubebuilder:validation:Minimum=1
+	// +kubebuilder:validation:Maximum=16384
+	Count int32 `json:"count"`
+}
+
+type Reference struct {
+	// Name of the referenced object.
+	// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names
+	//
+	// +kubebuilder:validation:Required
+	Name string `json:"name,omitempty"`
+}
+
+// ProvisioningRequestStatus represents the status of the resource reservation.
+type ProvisioningRequestStatus struct {
+	// Conditions represent the observations of a Provisioning Request's
+	// current state. Those will contain information whether the capacity
+    // was found/created or if there were any issues. The condition types
+    // may differ between different provisioning classes.
+	//
+	// +listType=map
+    // +listMapKey=type
+    // +patchStrategy=merge
+    // +patchMergeKey=type
+    // +optional
+	Conditions []metav1.Condition `json:"conditions"`
+
+	// AdditionalStatus contains all other status values custom provisioning classes may require.
+	//
+	// +optional
+	// +kubebuilder:validation:MaxItems=64
+	AdditionalStatus map[string]string `json:"additionalStatus"`
+}
+```
+
+### Provisioning Classes
+
+#### check-capacity.kubernetes.io class
+
+The `check-capacity.kubernetes.io` is one-off check to verify that the in the cluster
+there is enough capacity to provision given set of pods.
+
+Note: If two of such objects are created around the same time, CA will consider
+them independently and place no guards for the capacity.
+Also the capacity is not reserved in any manner so it may be scaled-down.
+
+#### atomic-scale-up.kubernetes.io class
+
+The `atomic-scale-up.kubernetes.io` aims to provision the resources required for the
+specified pods in an atomic way. The proposed logic is to:
+1. Try to provision required VMs in one loop.
+2. If it failed, remove the partially provisioned VMs and back-off.
+3. Stop the back-off after a given duration (optional), which would be passed
+   via `AdditionalParameters` field, using `ValidUntilSeconds` key and would contain string
+   denoting duration for which we should retry (measured since creation fo the CR).
+
+Note: that the VMs created in this mode are subject to the scale-down logic.
+So the duration during which users need to create the Pods is equal to the
+value of `--scale-down-unneeded-time` flag.
+
+### Adding pods that consume given ProvisioningRequest
+
+To avoid generating double scale-ups and exclude pods that are meant to consume
+given capacity CA should be able to differentiate those from all other pods.  
+To do so users need to specify the following pod annotation (it is not required
+in ProvReq’s template, though it can be specified):
+
+```yaml
+annotations:
+    "cluster-autoscaler.kubernetes.io/consume-provisioning-request": "provreq-name"
+```
+
+If it is provided for the pods that consume the ProvReq with `check-capacity.kubernetes.io` class,
+the CA will not provision the capacity, even if it was needed (as some other pods might have been 
+scheduled on it) and will result in visibility events passed to the ProvReq and pods.
+If it is not passed the CA will behave normally and just provision the capacity if it needed.
+
+Note: CA will match all pods with this annotation to a corresponding ProvReq and
+ignore them when executing a scale-up loop (so that is up to users to make sure
+that the ProvReq count is matching the number of created pods).
+If the ProvReq is missing, all of the pods that consume it will be unschedulable indefinitely.
+
+### CRD lifecycle
+
+1.  A ProvReq will be created either by the end user or by a framework.
+    At this point needed PodTemplate objects should be also created.
+2.  CA will pick it up, choose a nodepool (or create a new one if NAP is
+    enabled), and try to create nodes.
+3.  If CA successfully creates capacity, ProvReq will receive information about
+    this fact in `Conditions` field.
+4.  At this moment, users can create pods in that will consume the ProvReq (in
+    the same namespace), those will be scheduled on the capacity that was
+    created by the CA.
+5.  Once all of the pods are scheduled users can delete the ProvReq object,
+    otherwise it will be garbage collected after some time.
+6.  When pods finish the work and nodes become unused the CA will scale them
+    down.
+
+Note: Users can create a ProvReq and pods consuming them at the same time (in a
+"fire and forget" manner), but this may result in the pods being unschedulable
+and triggering user configured alerts.
+
+### Canceling the requests
+
+To cancel a pending Provisioning Request with atomic class, all that the users need to do is
+to delete the Provisioning Request object.
+After that the CA will no longer guard the nodes from deletion and proceed with standard scale-down logic.
+
+### Conditions
+
+The following Condition states should encode the states of the ProvReq:
+
+  - Provisioned - VMs were created successfully (Atomic class)
+  - CapacityAvailable - cluster contains enough capacity to schedule pods (Check
+    class)
+    * `CapacityAvailable=true` will denote that cluster contains enough capacity to schedule pods
+	* `CapacityAvailable=false` will denote that cluster does not contain enough capacity to schedule pods
+  - Failed - failed to create or check capacity (both classes)
+
+The Reasons and Messages will contain more details about why the specific
+condition was triggered.
+
+Providers of the custom classes should reuse the conditions where available or create their own ones
+if items from the above list cannot be used to denote a specific situation.
+
+### CA implementation details
+
+The proposed implementation is to handle each ProvReq in a separate scale-up
+loop. This will require changes in multiple parts of CA:
+
+1.  Listing unschedulable pods where:
+      - pods that consume ProvReq need to filtered-out
+      - pods that are represented by the ProvReq need to be injected (we need to
+        ensure those are threated as one group by the sharding logic)
+2.  Scale-up logic, which as of now has no notion atomicity and grouping of
+    pods. This is simplified as the ScaleUp logic was recently put [behind an
+    interface](https://github.com/kubernetes/autoscaler/pull/5597).
+      - This is a place where the biggest part of the change will be made. Here
+        many parts of the logic are assuming best-effort semantics and the scale
+        up size is lowered in many situations:
+          - Estimation logic, which stops after some time-out or number of
+            pods/nodes.
+          - Size limiting, which caps the scale-up to match the size
+            restrictions (on node group or cluster level).
+3.  Node creation, which needs to support atomic resize. Either via native cloud
+    provider APIs or best effort with node removal if CA is unable to fulfill
+    the scale-up.
+      - This is also quite substantial change, we can provide a generic
+        best-effort implementation that will try to scale up and clean-up nodes
+        if it is unsuccessful, but it is up to cloud providers to integrate with
+        provider specific APIs.
+4.  Scale down path is not expected to change much. But users should follow
+    [best
+    practices](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node)
+    to avoid CA disturbing their workloads.
+
+## Testing
+
+The following e2e test scenarios will be created to check whether ProvReq
+handling works as expected:
+
+1.  A new ProvReq with `check-capacity.kubernetes.io` provisioning class is created, CA
+    checks if there is enough capacity in cluster to provision specified pods.
+2.  A new ProvReq with `atomic-scale-up.kubernetes.io` provisioning class is created, CA
+    picks an appropriate node group scales it up atomically.
+3.  A new atomic ProvReq is created for which a NAP needs to provision a new
+    node group. NAP creates it CA scales it atomically.
+      - Here we should cover some of the different reasons why NAP may be
+        required.
+4.  An atomic ProvReq fails due to node group size limits and NAP CPU and/or RAM
+    limits.
+5.  Scalability tests.
+      - Scenario in which many small ProvReqs are created (strain on the number
+        of scale-up loops).
+      - Scenario in which big ProvReq is created (strain on a single scale-up
+        loop).
+
+## Limitations
+
+The current Cluster Autoscaler implementation is not taking into account [Resource Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/). \
+The current proposal is to not include handling of the Resource Quotas, but it could be added later on.
+
+## Future Expansions
+
+### ProvisioningClass CRD
+
+One of the expansion of this approach is to introduce the ProvisioningClass CRD,
+which follows the same approach as
+[StorageClass object](https://kubernetes.io/docs/concepts/storage/storage-classes/).
+Such approach would allow administrators of the cluster to introduce a list of allowed
+ProvisioningClasses. Such CRD can also contain a pre set configuration, i.e.
+administrators may set that `atomic-scale-up.kubernetes.io` would retry up to `2h`.
+
+Possible CRD definition:
+```go
+// ProvisioningClass is a way to express provisioning classes available in the cluster.
+type ProvisioningClass struct {
+	// Name denotes the name of the object, which is to be used in the ProvisioningClass 
+	// field in Provisioning Request CRD.
+	// 
+	// +kubebuilder:validation:Required
+	Name string `json:"name"`
+
+	// AdditionalParameters contains all other parameters custom classes may require.
+	//
+	// +optional
+	AdditionalParameters map[string]string `json:"additionalParameters"`
+}
+```