Always update TrainJob status on errors

[astefanutti]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #2351

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 12320339089

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 12320152919:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[tenzen-y]

Instead of this, I'm wondering if we should update the status regardless of errors there: https://github.com/kubeflow/training-operator/blob/3eb1a0c9e3616a7fed9864d8b1373d6d677a6c77/pkg/controller.v2/trainjob_controller.go#L99-

So, if we get any error in L100

training-operator/pkg/controller.v2/trainjob_controller.go

Lines 96 to 98 in 3eb1a0c

	if terminalCondErr := setTerminalCondition(ctx, runtime, &trainJob); terminalCondErr != nil {
	return ctrl.Result{}, errors.Join(err, terminalCondErr)
	}

. we just join the terminalCondErr with error, but we do not return error here.

In other words, we update the status in the case of status differences between old and new even if we get any errors from setCondition functions.

Because we do not restrict underlying Job to JobSet and we can not estimate that the NotFound error can be ignored.

[astefanutti]

You're right, that's much better!

I've just re-pushed. Let me know if that corresponds to what you had in mind.

[tenzen-y]

Great to see.

[tenzen-y]

Thank you!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment