Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct cluster drift using patches #822

Merged
merged 2 commits into from
Nov 30, 2023
Merged

Correct cluster drift using patches #822

merged 2 commits into from
Nov 30, 2023

Conversation

hiddeco
Copy link
Member

@hiddeco hiddeco commented Nov 28, 2023

This changes the cluster drift correction behavior from performing a
Helm upgrade to performing create and patch API requests based on the
JSON Patch data.

Doing this is much lighter than performing a full release cycle, and
deals with the issue of Helm being unable to restore state of Custom
Resources without the --force flag being set. Which has unwanted
side-effects like forcing objects through a deletion/creation cycle.

After a drift correction attempt a Kubernetes Event is emitted, which
contains a summary of the created and patched resources, and a
collection of any (potential) errors.

As the goal is to restore state as best as we can, the drift correction
will be re-attempted until all resources have been restored to the
desired state.

Fixes #805

@hiddeco hiddeco added enhancement New feature or request area/drift Drift detection/correction related issues and pull requests labels Nov 28, 2023
@hiddeco hiddeco force-pushed the correct-drift-apply branch from 8cf7783 to 84a08e2 Compare November 28, 2023 15:18
@hiddeco hiddeco force-pushed the correct-drift-apply branch from 84a08e2 to f6abde6 Compare November 28, 2023 21:13
@hiddeco hiddeco changed the title reconcile: correct cluster drift using patches Correct cluster drift using patches Nov 28, 2023
@hiddeco hiddeco force-pushed the correct-drift-apply branch 5 times, most recently from 270619b to 001b2d1 Compare November 28, 2023 22:43
@hiddeco hiddeco marked this pull request as ready for review November 28, 2023 22:44
@hiddeco hiddeco mentioned this pull request Nov 28, 2023
14 tasks
@hiddeco hiddeco force-pushed the correct-drift-apply branch 2 times, most recently from 39ae10d to 5055ebd Compare November 29, 2023 09:30
go.mod Outdated Show resolved Hide resolved
This changes the cluster drift correction behavior from performing a
Helm upgrade to performing create and patch API requests based on the
JSON Patch data.

Doing this is much lighter than performing a full release cycle, and
deals with the issue of Helm being unable to restore state of Custom
Resources without the `--force` flag being set. Which has unwanted
side-effects like forcing objects through a deletion/creation cycle.

After a drift correction attempt a Kubernetes Event is emitted, which
contains a summary of the created and patched resources, and a
collection of any (potential) errors.

As the goal is to restore state as best as we can, the drift correction
will be re-attempted until all resources have been restored to the
desired state.

Signed-off-by: Hidde Beydals <[email protected]>
@hiddeco hiddeco force-pushed the correct-drift-apply branch from 5055ebd to ccd8f88 Compare November 29, 2023 19:34
Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested this with various custom resources changes and found no issues except for the meta.helm.sh/release annotations missing from objects which have been deleted and restored.

Fixed in: 0131f22

@hiddeco hiddeco force-pushed the correct-drift-apply branch 4 times, most recently from d20e120 to 04cb4f8 Compare November 29, 2023 22:32
This ensures that the metadata labels and annotations Helm adds during
the creation of resources are included while diffing them.

As they are not part of the manifest but should be restored in case
they are e.g. removed or modified.

Signed-off-by: Hidde Beydals <[email protected]>
@hiddeco hiddeco force-pushed the correct-drift-apply branch from 04cb4f8 to 0131f22 Compare November 30, 2023 08:41
Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @hiddeco 🏅

PS. I have tested this with various custom resources edits/deletions.

@hiddeco hiddeco merged commit d310c8b into main Nov 30, 2023
6 checks passed
@hiddeco hiddeco deleted the correct-drift-apply branch November 30, 2023 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/drift Drift detection/correction related issues and pull requests enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Helm drift detection causes endless upgrade loop with CRDs
2 participants