Introduce AEP with Provisioning Request CRD

[kisieland]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds a new AEP to support a new CRD in cluster autoscaler.
This novel API will allow users to specify a dependency between a group of pods.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

N/A

[qianlei90]

FYI：

• Cluster autoscaler improvements for AI workloads #5170 (comment)
• kueue want to use ResourceClaim API to interactive with CA：https://docs.google.com/document/d/1fYpdB_a3YYAOHfFkx6GnEBRuAZIVfN2TeILrwQWF8w4/edit#heading=h.33c0ab4ic73i

[mwielgus]

@qianlei90 Yes, it should help with the issue.

[qelo]

What are the rules for scale down when nodes are provisioned but pods didn't start (or didn't start yet)?

[mwielgus]

It would be great if this was somehow configurable in the longer run.

[kisieland]

The current proposal would be to use the same logic for scale-down so it would be configurable via --scale-down-unneeded-time flag.
For now there is no way for CA to know which VMs come from the atomic scale-up so we cannot differ.

[qelo]

Is there a strong need for GENERIC_CHECK_CAPACITY ? There is no scaling in that case , so it seems somewhat outside of "autoscaler" responsbilities. Could we maybe start with just GENERIC_ATOMIC?

Also, what happens if there are two CHECK_CAPACITY requests but there is capacity for one? Does one get capacity and the other one fails? When does the hold expire and is it consistent with how ProvisioningRequest scale downs for GENERIC_ATOMIC_SCALE_UP?

[mwielgus]

I think that there may be quite a few use cases. For example, trying to run low priority jobs only when there is capacity on the statically allocated nodes. Currently K8S doesn't have gang scheduling, and this may work as a best-effort replacement.
The problem with invalidation exist in all cases where shared/non-exclusive resources are provisioned. This should be addressed in the doc. Possible options are:

• Invalidate request that is still there.
• Treat PR as a point-in-time query and don't bother about invalidation.

Given that we already have 2 possible behaviors for a single capacity, the params/classes may be actually needed from day 1.

[severinson]

This capability is interesting for the Armada project, where we'd like to control resource provisioning separately from pod scheduling. Some context: with Armada we typically have long queues of pods that can't currently be scheduled, but we may not want to provision enough capacity to run those pods immediately.

[asm582]

Can other external controllers instead of CA use provisioning CRD to allocate resources?

[kisieland]

@asm582

Can other external controllers instead of CA use provisioning CRD to allocate resources?

CA is meant as the only component reacting to the CRD, that said if it is not installed there is nothing blocking another component stepping to fill the void (eg. the Karpenter component).
Note: One potential way we can have different components reacting to CRDs in the same cluster is to split them via ProvisioningClass.

As for interacting with the CRD any other component can create CRDs to ask CA to provision the VMs.

[asm582]

Note: One potential way we can have different components reacting to CRDs in the same cluster is to split them via ProvisioningClass.

Thanks @kisieland, can you please explain the below statement a bit more:

Note: One potential way we can have different components reacting to CRDs in the same cluster is to split them via ProvisioningClass.

[kisieland]

@asm582

We can have ProvisioningClasses GenericAtomicScaleUp and CloudProviderSomOtherProvisioningMode.
CRs with first of which would be handled by the CA and the CRs with the second one would be handled by some other component.

[kisieland]

/label api-review

[alculquicondor]

I forgot to send my review last week :(

[kisieland]

Thanks @alculquicondor addressed your comments

[jonathan-innis]

From reading through this, this strikes me as a static capacity request for "ghost pods". i.e. I want negative preemption pods to represent my capacity without those pods actually existing and being interacted with by the kube-scheduler. Is there a similar hierarchical relationship that we can build here similar to how we have varying levels of abstraction for Pod/ReplicaSet/Deployment. Is there a way to represent a capacity request as a single pod and then build off of it with a replica count? Then CAS recognizes either real pods or these static capacity ghost pods. There's also a question in my head about the difference in use-cases here. One use-case is I always want overhead capacity for a warm pool at all times; the other is I eventually want that static capacity to be filled and I don't want to further scale up my capacity when that pre-provisioned static capacity is utilized.

[kisieland]

@liggitt PTAL

[mwielgus]

@liggitt @jpbetz Do you have any more comments? If no (or there is no response) i will be merging the PR soon.

[kisieland]

Re @jpbetz:

... Is there a way to represent a capacity request as a single pod and then build off of it with a replica count?

Provisioning Requests can represent one pod. But building more of those objects and expecting CAS to group them beets the purpose of the ProvReq object in the first place, why use such ProvReqs, when one can just use pods?
The issue that ProvReq aims to represent groups of Pods is to avoid CA making decisions of partial state of the cluster.
If we used labels to group pods CAS will not know when all of the pods were created and might have started work that is not needed.

One use-case is I always want overhead capacity for a warm pool at all times; the other is I eventually want that static capacity to be filled and I don't want to further scale up my capacity when that pre-provisioned static capacity is utilized.

Here we propose to implement the second mode, but there is nothing prevent you or anybody else from adding a constan size buffer mode, all the required fields are here, only CAS logic would need to be slightly modified.

[mwielgus]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kisieland, mwielgus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/OWNERS [mwielgus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kerthcet]

Hi, I'm not quite familiar with autoscaler source code, but do care about this feature, so I left some comments, maybe not mature, forgive me. 🥲

[liggitt]

@liggitt @jpbetz Do you have any more comments? If no (or there is no response) i will be merging the PR soon.

Sorry for the delay, we have a bi-weekly API review that will be reviewing the updates today.

I see a CRD was merged already in #6104 linking to this PR as API approval.
Lazy consensus does not constitute API approval for k8s.io APIs. Please move the CRD to the x-k8s.io domain if you want to proceed without API approval, or revert the PR until approval is given.

[mwielgus]

The API was meant to go to x-k8s.io from the very beginning. Thanks for noticing it. It will be fixed soon.

[kisieland]

Thanks @liggitt for noticing created #6108 to address this!