Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) #625

mimowo · 2024-07-25T08:19:13Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #624
Fixes #535

Special notes for your reviewer:

This approach relies on updating the Jobs on resume, rather the deleting Jobs on suspend.
Alternative implementation was done in: #625

Does this PR introduce a user-facing change?

Allow restoring PodTemplate on suspend, and fix resuming JobSet after restoring 
the PodTemplate. This fixes the integration with Kueue to support eviction of workloads
and re-admitting in another ResourceFlavor (with other nodeSelectors, or without nodeSelectors).
It is achieved by deleting the Jobs while the JobSet is suspended.

netlify · 2024-07-25T08:19:35Z

✅ Deploy Preview for kubernetes-sigs-jobset canceled.

Name	Link
🔨 Latest commit	`8022696`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-jobset/deploys/66a8fb8490146c0008e03f32

mimowo · 2024-07-25T08:21:49Z

/cc @danielvegamyhre @tenzen-y @kannon92

tenzen-y · 2024-07-25T11:29:36Z

pkg/webhooks/jobset_webhook.go

 	// This is needed for integration with Kueue/DWS.
-	if ptr.Deref(oldJS.Spec.Suspend, false) {
+	if ptr.Deref(oldJS.Spec.Suspend, false) || ptr.Deref(js.Spec.Suspend, false) {


Suggested change

if ptr.Deref(oldJS.Spec.Suspend, false) || ptr.Deref(js.Spec.Suspend, false) {

if ptr.Deref(oldJS.Spec.Suspend, false) || (ptr.Deref(js.Spec.Suspend, false) && js.Spec.ReplicatedJobs[*].status.startTime == nil) {

Wouldn't it be helpful to validate if the jobs don't have startTime?
Because if any jobs have startTime, this operation should fail due to batch/job validation.
But, I'm not sure if we should add validations deeply depending on the batch/job specifications.

@danielvegamyhre @mimowo WDYT?

I think the flow is that we first suspend the parent JobSet, then it suspends the Jobs. So, if we added this condition, the startTime on running child jobs would prevent the update.

However, this is complex, and I think I should have added an e2e test in JobSet to verify the suspending works.

Yea I’m thinking integration and e2e test solidifying the requirements that JobSet needs for working with Kueue will be a great idea.

We obviously don’t want to bring in Kueue as a dependency so I think just verifying that the updates/patches work in integration/e2e will be welcome

I think the flow is that we first suspend the parent JobSet, then it suspends the Jobs. So, if we added this condition, the startTime on running child jobs would prevent the update.

Uhm, that makes sense.
But, I guess that users fall into the rabbit hole when they accidentally modify the scheduling directives against the ReplicatedJob with startTime because the jobset-controller just output errors in the controller logs, this is not directly user feedback.

Maybe we can improve these implied specifications for the JobSet users once we introduce kubernetes/kubernetes#113221.

So, I'm ok with the current implementation for now.

Actually, the e2e test would not work (see in the PR I already added it), because there is a deeper issue - JobSet does not delete the Jobs once suspended: #535. The fact that we keep the old jobs means we never create new jobs on resuming again if the PodTemplate is updated in JobSet.

I'm trying to implement it by deleting the Jobs on suspend, and it looks promising. There are some integration tests for startupPolicy to adjust I yet need to look into.

Let me know if you have some more context knowledge if traps I may encounter with this approach (I will probably continue next week), but essentially this is the only I see, and it would mimic the Job behavior, where suspended Jobs deletes the pods.

danielvegamyhre · 2024-07-25T15:32:30Z

Thanks for working on the fix for this @mimowo. Please take a look at my comment here #624 (comment) as well, I'd like to improve our coordination to prevent these kinds of issues from occurring in the future.

mimowo · 2024-07-25T17:12:00Z

/hold
for e2e or integration test inside the JobSet repo

danielvegamyhre · 2024-07-26T17:29:52Z

Let me know when this is ready for review and I'll take a look. Thanks!

mimowo · 2024-07-26T18:11:08Z

Let me know when this is ready for review and I'll take a look. Thanks!

Thank you @danielvegamyhre ! I think it is ready now, let me also know if some parts require additional explanation.

mimowo · 2024-07-26T18:14:26Z

/hold cancel
Two new e2e tests are added on suspending and resuming, one focuses on fast suspend -> resume -> suspend -> resume path to check for any race conditions. The other tracks deletion and creation of pods also.

mimowo · 2024-07-29T07:09:54Z

@danielvegamyhre I'm wondering about introducing DeleteJobsOnSuspend feature gate (true by default in Beta) that could be used in emergency (a bug found) to revert the behavior for a user without recompiling, but I didn't see any other feature gates in JobSet yet. OTOH, the amount of testing to cover both paths would be huge, plus it is rather a bug than a new feature, so I'm hesitant. WDYT?

danielvegamyhre · 2024-07-29T18:28:31Z

pkg/webhooks/jobset_webhook.go

+				rStatus := js.Status.ReplicatedJobsStatus
+				// Don't allow to mutate PodTemplate on unsuspending if there
+				// are still active or suspended jobs from the previous run
+				if len(rStatus) > index && (rStatus[index].Active > 0 || rStatus[index].Suspended > 0) {


Can we assign len(rStatus) > index to a well-named variable to make it more clear/readable why it matters if the number of replicatedJob statuses is greater than the current replicatedJob index?

Furthermore, could we abstract this block out into a helper function to make it clear at a glance what we are trying to check here, and make it more concise/readable?

Sure, instead of using len(rStatus) > index I basically want to check if the ReplicatedJobsStatus is initialized. If not, then I assume the JobSet didn't run yet, so there are no Jobs. I have also extracted the code to a function and documented. PTAL.

k8s-ci-robot · 2024-07-30T07:32:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mimowo
Once this PR has been reviewed and has the lgtm label, please ask for approval from danielvegamyhre. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/controllers/jobset_controller.go

pkg/webhooks/jobset_webhook.go

kannon92 · 2024-07-30T13:57:43Z

pkg/webhooks/jobset_webhook.go

+	// controller to delete the Jobs from the previous run if they could
+	// conflict with the Job creation on resume.
+	rStatus := js.Status.ReplicatedJobsStatus
+	if rStatus[index].Active > 0 || rStatus[index].Suspended > 0 {


Do you think we should include terminating?

Indeed, currently we only count as "active" Jobs which have at least one "active" pods. This means that it is possible that some Jobs which have only terminating pods may still exist.

One fixing idea would be to count Jobs as active if they have at least one active or terminating pod. Another is to introduce a new counter for "terminating" jobs - Jobs which have all pods terminating. WDYT?

I guess this is a corner case for now, so maybe could be done in a follow up?

I think it is not too bad in this PR, because a terminating Job (all pods are terminating) is going to be deleted eventually. Yes, if we resume the JobSet while it exists, then it can block creating replacement Job for a while, but IIUC the JobSet controller will create replacement for such a Job as soon as it is fully gone, and the replacement Job will have a good PodTemplate.

In order to prevent resuming a JobSet while there is a terminating Job we could:

include such a Job as active, if job.Status.Active > 0 || ptr.Deref(job.Status.Terminating, 0) > 0 here

introduce a new counter for Terminating Jobs. Terminating would be a fallback (so job.Status.Active == 0) counter if ptr.Deref(job.Status.Terminating, 0) > 0

I'm leaning towards option (2.). I could implement it either in this PR or a follow up.

I think a follow up is fine. Terminating on Jobs is currently a beta feature so logic may be somewhat complex.

Terminating on Jobs is currently a beta feature so logic may be somewhat complex.

I think we don't need any code in JobSet depending on the feature-gate in k8s. If the feature gate is disabled in k8s the field is not set. So, the code in JobSet would see nil, we could prevent panics with ptr.Deref(job.Status.Terminating, 0), but this should be enough, IIUC.

test/e2e/e2e_test.go

danielvegamyhre · 2024-08-01T02:01:44Z

@danielvegamyhre I'm wondering about introducing DeleteJobsOnSuspend feature gate (true by default in Beta) that could be used in emergency (a bug found) to revert the behavior for a user without recompiling, but I didn't see any other feature gates in JobSet yet. OTOH, the amount of testing to cover both paths would be huge, plus it is rather a bug than a new feature, so I'm hesitant. WDYT?

I don't think we need a feature gate for this.

mimowo · 2024-08-01T09:41:49Z

@danielvegamyhre @kannon92 thanks for reviewing. let me know what are the remaining things to address in this PR.
It seems this is the outstanding comment #625 (comment).

mimowo · 2024-08-05T14:00:52Z

/hold
I have synced with @danielvegamyhre and we discussed that it would be good to see if we can achieve the functionality needed for Kueue without deleting the Jobs on suspend, reusing bits introduced in https://github.com/kubernetes-sigs/jobset/pull/590/files#diff-68677f8530f00fbbb399ac36268f928ee584948dd09091644a8fad81a8658153R457-R478.

I agree this is worth exploring as this is closer to how JobSet operates currently, so the impact is more predictable. I will try this approach and update the PR or summarize if there are any blockers.

mimowo · 2024-08-06T11:04:53Z

@danielvegamyhre @kannon92
I have prepared another PR which fixes the issue by updating the Jobs on resume, rather than recreating them. I'm happy to move forward with that approach. PTAL: #640. I have also update the issue description to make it more clear.
Finally, there is a small PR with e2e test showing the issue: #635.

mimowo · 2024-08-06T17:22:29Z

/close
in favor of #640

k8s-ci-robot · 2024-08-06T17:22:34Z

@mimowo: Closed this PR.

In response to this:

/close
in favor of #640

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 25, 2024

k8s-ci-robot requested review from danielvegamyhre and kannon92 July 25, 2024 08:19

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 25, 2024

mimowo force-pushed the fix-suspending-jobset branch from c8d7013 to 24448c5 Compare July 25, 2024 08:21

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 25, 2024

k8s-ci-robot requested a review from tenzen-y July 25, 2024 08:21

mimowo mentioned this pull request Jul 25, 2024

Allow mutating schedulingGates when the Jobset is suspended #623

Merged

tenzen-y reviewed Jul 25, 2024

View reviewed changes

mimowo mentioned this pull request Jul 25, 2024

Add e2e test for JobSet fix on suspend kubernetes-sigs/kueue#2700

Merged

mimowo force-pushed the fix-suspending-jobset branch from 24c5d8b to 38e5520 Compare July 26, 2024 07:04

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 26, 2024

mimowo force-pushed the fix-suspending-jobset branch 6 times, most recently from 85077d4 to 0974200 Compare July 26, 2024 10:48

mimowo mentioned this pull request Jul 26, 2024

Do not create Jobs and the service when JobSet is suspended #535

Open

mimowo force-pushed the fix-suspending-jobset branch from 0974200 to f02353d Compare July 26, 2024 14:11

mimowo force-pushed the fix-suspending-jobset branch 2 times, most recently from b9678b7 to b40a6d2 Compare July 26, 2024 17:53

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 26, 2024

danielvegamyhre self-assigned this Jul 26, 2024

danielvegamyhre reviewed Jul 29, 2024

View reviewed changes

Fix restoring PodTemplate on JobSet suspend

b8d1c9b

mimowo force-pushed the fix-suspending-jobset branch from b40a6d2 to eb3f230 Compare July 30, 2024 07:32

Review remarks

495b8ff

mimowo force-pushed the fix-suspending-jobset branch from eb3f230 to 495b8ff Compare July 30, 2024 07:42

kannon92 reviewed Jul 30, 2024

View reviewed changes

Add a comment why Jobs are deleted.

8022696

mimowo mentioned this pull request Jul 30, 2024

support elastic jobset #622

Closed

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 5, 2024

mimowo changed the title ~~Fix restoring PodTemplate on JobSet suspend~~ Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) Aug 6, 2024

This was referenced Aug 6, 2024

Fix resuming JobSet after restoring PodTemplate (by Jobs update) #640

Closed

Allow to mutate PodTemplate when suspending a JobSet and support resuming such JobSet #624

Closed

mimowo mentioned this pull request Aug 6, 2024

E2e test to show that resuming JobSet after restoring PodTemplate does not work [EXPERIMENT] #635

Closed

k8s-ci-robot closed this Aug 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) #625

Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) #625

mimowo commented Jul 25, 2024 •

edited

Loading

netlify bot commented Jul 25, 2024 •

edited

Loading

mimowo commented Jul 25, 2024

tenzen-y Jul 25, 2024

mimowo Jul 25, 2024 •

edited

Loading

kannon92 Jul 25, 2024

tenzen-y Jul 26, 2024

mimowo Jul 26, 2024 •

edited

Loading

mimowo Jul 26, 2024 •

edited

Loading

danielvegamyhre commented Jul 25, 2024

mimowo commented Jul 25, 2024

danielvegamyhre commented Jul 26, 2024

mimowo commented Jul 26, 2024

mimowo commented Jul 26, 2024

mimowo commented Jul 29, 2024 •

edited

Loading

danielvegamyhre Jul 29, 2024

mimowo Jul 30, 2024

k8s-ci-robot commented Jul 30, 2024

kannon92 Jul 30, 2024

mimowo Jul 30, 2024

mimowo Aug 1, 2024

kannon92 Aug 1, 2024

mimowo Aug 1, 2024

danielvegamyhre commented Aug 1, 2024

mimowo commented Aug 1, 2024

mimowo commented Aug 5, 2024

mimowo commented Aug 6, 2024

mimowo commented Aug 6, 2024

k8s-ci-robot commented Aug 6, 2024

	if ptr.Deref(oldJS.Spec.Suspend, false) \|\| ptr.Deref(js.Spec.Suspend, false) {
	if ptr.Deref(oldJS.Spec.Suspend, false) \|\| (ptr.Deref(js.Spec.Suspend, false) && js.Spec.ReplicatedJobs[*].status.startTime == nil) {

Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) #625

Fix resuming Jobset after restoring PodTemplate (by deleting Jobs) #625

Conversation

mimowo commented Jul 25, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Jul 25, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-jobset canceled.

mimowo commented Jul 25, 2024

Choose a reason for hiding this comment

mimowo Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

mimowo Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

danielvegamyhre commented Jul 25, 2024

mimowo commented Jul 25, 2024

danielvegamyhre commented Jul 26, 2024

mimowo commented Jul 26, 2024

mimowo commented Jul 26, 2024

mimowo commented Jul 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielvegamyhre commented Aug 1, 2024

mimowo commented Aug 1, 2024

mimowo commented Aug 5, 2024

mimowo commented Aug 6, 2024

mimowo commented Aug 6, 2024

k8s-ci-robot commented Aug 6, 2024

mimowo commented Jul 25, 2024 •

edited

Loading

netlify bot commented Jul 25, 2024 •

edited

Loading

mimowo Jul 25, 2024 •

edited

Loading

mimowo Jul 26, 2024 •

edited

Loading

mimowo Jul 26, 2024 •

edited

Loading

mimowo commented Jul 29, 2024 •

edited

Loading