Support atomic JobSets

[osigurdson]

What would you like to be added:

It would be great if the pods of a given job set were scheduled atomically.

Why is this needed:

When running multiple JobSets, the pods of individual job sets do not start atomically. Shown in the output below, jobset 1 and jobset 3 have individual parts running but not the entire jobset. This could lead to deadlock as JobSets can never get the necessary resources to complete.

$ k get pods
NAME READY STATUS RESTARTS AGE
js-1-part1-0-0-qz88c 0/1 Pending 0 4s
js-1-part2-0-0-dhgnv 1/1 Running 0 4s
js-2-part1-0-0-jqd9m 0/1 Pending 0 4s
js-2-part2-0-0-qxcsx 0/1 Pending 0 4s
js-3-part1-0-0-lqpv4 0/1 Pending 0 4s
js-3-part2-0-0-dvh5g 1/1 Running 0 4s
js-4-part1-0-0-472hl 1/1 Running 0 4s
js-4-part2-0-0-ds9js 1/1 Running 0 4s
js-5-part1-0-0-plpm5 1/1 Running 0 4s
js-5-part2-0-0-zprwq 1/1 Running 0 4s

[danielvegamyhre]

@osigurdson for atomic/group scheduling, we usually recommend users use Kueue. Here is an example from the docs. Let us know if this looks like it will address the issue/use case you've described here.

[osigurdson]

Thanks @danielvegamyhre. I will give it a try.

[osigurdson]

It seemed to lead to a lot of thrashing:

$ k get pods
NAME READY STATUS RESTARTS AGE
js-1-part1-0-0-56lx5 0/1 Terminating 0 1s
js-1-part1-0-0-5shsf 1/1 Terminating 0 11s
js-1-part1-0-0-66f7t 1/1 Terminating 0 5s
js-1-part1-0-0-6cql9 0/1 Terminating 0 0s
js-1-part1-0-0-7nbx6 1/1 Terminating 0 6s
js-1-part1-0-0-8vlx6 1/1 Terminating 0 8s
js-1-part1-0-0-bjfj4 1/1 Terminating 0 3s
js-1-part1-0-0-c2ckz 0/1 Terminating 0 0s
js-1-part1-0-0-fwq2q 1/1 Terminating 0 9s
js-1-part1-0-0-kgd4t 1/1 Terminating 0 1s
js-1-part1-0-0-kqbms 0/1 Terminating 0 12s
js-1-part1-0-0-qg4d2 1/1 Terminating 0 7s
js-1-part1-0-0-x8bcf 1/1 Terminating 0 6s
js-1-part2-0-0-2mknq 1/1 Terminating 0 4s

Perhaps misconfigured but a basic Job object ran ok.

kueue.yaml
1 apiVersion: kueue.x-k8s.io/v1beta1
2 kind: ResourceFlavor
3 metadata:
4 name: default-flavor
5 ---
6 apiVersion: kueue.x-k8s.io/v1beta1
7 kind: ClusterQueue
8 metadata:
9 name: clusterqueue
10 spec:
11 #queueingStrategy: StrictFIFO
12 namespaceSelector: {}
13 resourceGroups:
14 - coveredResources: ["cpu", "memory", "pods"]
15 flavors:
16 - name: "default-flavor"
17 resources:
18 - name: "cpu"
19 nominalQuota: 1000
20 - name: "memory"
21 nominalQuota: 2000Gi
22 - name: "pods"
23 nominalQuota: 1000
24 ---
25 apiVersion: kueue.x-k8s.io/v1beta1
26 kind: LocalQueue
27 metadata:
28 name: localqueue
29 spec:
30 clusterQueue: clusterqueue
31 ---

jobset.yaml
1 apiVersion: jobset.x-k8s.io/v1alpha2
2 kind: JobSet
3 metadata:
4 name: js-1
5 labels:
6 kueue.x-k8s.io/queue-name: localqueue
7 spec:
8 replicatedJobs:
9 - name: part1
10 template:
11 spec:
12 parallelism: 1
13 completions: 1
14 backoffLimit: 0
15 template:
16 spec:
17 containers:
18 - name: bb
19 image: busybox
20 args: ["sleep", "10"]
21 resources:
22 limits:
23 cpu: 2
24 - name: part2
25 template:
26 spec:
27 parallelism: 1
28 completions: 1
29 backoffLimit: 0
30 template:
31 spec:
32 containers:
33 - name: bb
34 image: busybox
35 args: ["sleep", "10"]

job.yaml
1 apiVersion: batch/v1
2 kind: Job
3 metadata:
4 name: testjob1
5 labels:
6 kueue.x-k8s.io/queue-name: localqueue
7 spec:
8 suspend: true
9 template:
10 spec:
11 containers:
12 - name: dummy
13 image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
14 args: ["60s"]
15 resources:
16 requests:
17 cpu: 4
18 memory: "200Mi"
19 restartPolicy: Never

This seemed to work ok and created an associated workload, etc.
$ k get jobs
NAME STATUS COMPLETIONS DURATION AGE
testjob1 Complete 1/1 67s 4m59s
--- ~/dev/cf/kueue/jobs
$ k get workloads
NAME QUEUE RESERVED IN ADMITTED FINISHED AGE
job-testjob1-8bfb7 localqueue clusterqueue True True 5m2s

Some logs from jobset controller manager

2024-10-25T03:29:01Z LEVEL(-2) Reconciling JobSet {"controller": "jobset", "controllerGroup": "jobset.x-k8s.io", "controllerKind": "JobSet", "JobSet": {"name":"js-1","namespace":"default"}, "namespace": "default", "name": "js-1", "reconcileID": "6843f06a-02c0-454d-a466-47c4d150c49e", "jobset": {"name":"js-1","namespace":"default"}}
2024-10-25T03:29:01Z ERROR resuming jobset {"controller": "jobset", "controllerGroup": "jobset.x-k8s.io", "controllerKind": "JobSet", "JobSet": {"name":"js-1","namespace":"default"}, "namespace": "default", "name": "js-1", "reconcileID": "6843f06a-02c0-454d-a466-47c4d150c49e", "jobset": {"name":"js-1","namespace":"default"}, "error": "Operation cannot be fulfilled on jobs.batch "js-1-part2-0": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/jobset/pkg/controllers.(*JobSetReconciler).reconcile
/workspace/pkg/controllers/jobset_controller.go:215
sigs.k8s.io/jobset/pkg/controllers.(*JobSetReconciler).Reconcile
/workspace/pkg/controllers/jobset_controller.go:120
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-25T03:29:01Z ERROR Reconciler error {"controller": "jobset", "controllerGroup": "jobset.x-k8s.io", "controllerKind": "JobSet", "JobSet": {"name":"js-1","namespace":"default"}, "namespace": "default", "name": "js-1", "reconcileID": "6843f06a-02c0-454d-a466-47c4d150c49e", "error": "Operation cannot be fulfilled on jobs.batch "js-1-part2-0": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222