-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework HelmRelease
reconciliation logic
#738
Conversation
b1ae813
to
7d044a7
Compare
d023277
to
42aac92
Compare
6854092
to
1307041
Compare
ae331af
to
9e467fb
Compare
9e467fb
to
596b071
Compare
6f9a0ed
to
015def3
Compare
I'm testing this PR here. Here are my findings so far:
|
7e7f7d0
to
5f82c20
Compare
This ensures that certain UIs can continue to display information to their users while they work on making better use of the new data available in `v2beta2`. Signed-off-by: Hidde Beydals <[email protected]>
3c86265
to
c9ddc0f
Compare
1b7c764
to
661c35b
Compare
This allows users to delay updating their `apiVersion` declarations, as the fields will be known in the previous version. If we would not do this, the fields would get wiped when the `v2beta1` resource is applied, potentially causing spurious upgrades. Signed-off-by: Hidde Beydals <[email protected]>
This allows the controller to be updated from `v2beta1` to `v2beta2` without triggering a release to settle state. It does this by looking at the previous successful release as recorded for the `v2beta1` object, and if found, recording a snapshot for it in the new `History` field of the status. This feature can be disabled by setting the `AdoptLegacyReleases` feature flag to `false`. Signed-off-by: Hidde Beydals <[email protected]>
661c35b
to
f03453d
Compare
This to allow better feedback to the user on why the controller decided to uninstall the release. Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
This improves continuity while the controller attempts to move the release forward. Signed-off-by: Hidde Beydals <[email protected]>
As it does not offer real value, and creates noise when the logs are included in an emitted event. Signed-off-by: Hidde Beydals <[email protected]>
This ensure that when a chart object has a temporary `Ready=False` state, the predicate will notice the change to `Ready=True` and cause an enqueue request. Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
Signed-off-by: Hidde Beydals <[email protected]>
f03453d
to
4a8d2ff
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Awesome work @hiddeco 🎖️ 🏅 🥇
I've tested the HelmRelease v2beta2 API and controller for two weeks using various configurations:
- install/upgrade/rollback/uninstall (with and without Helm tests)
- failure conditions (invalid charts, Helm test failures, controller OOM, Kubernetes API downtime)
- release auto-recovery when Helm storage has been locked
- multi-tenancy lockdown and account impersonation
- verify Kubernetes and Slack events for all actions
- install/upgrade of Flux monitoring stack (kube-prom-stack and other heavy charts)
- Helm release promotion with GitHub repository dispatch workflow
I've also run extensive benchmarks on Apple M1 Max and in GitHub hosted-runner, the results are impressive. Well done!
Objects | Type | Flux component | Duration Apple M1 | Duration GH Intel | Max Memory |
---|---|---|---|---|---|
100 | HelmChart | source-controller | 35s | 35s | 40Mi |
100 | HelmRelease | helm-controller | 42s | 42s | 140Mi |
500 | HelmChart | source-controller | 1m10s | 1m10s | 68Mi |
500 | HelmRelease | helm-controller | 1m58s | 4m40s | 350Mi |
1000 | HelmChart | source-controller | 1m45s | 1m45s | 110Mi |
1000 | HelmRelease | helm-controller | 5m10s | 14m10s | 620Mi |
For this benchmark 100, 500 and 1000 Helm releases are being installed/upgraded at the same time each with its own Helm chart.
This PR is practically a rewrite of the controller as a whole, it moves things into packages, introduces a
v2beta2
API with a changed Status structure, and handles the determination of the Helm action to take by observing writes to — and verifying the state of — the Helm storage.Changes
Snapshot
in the.status.history
list of theHelmRelease
.Snapshot
item for this release, to improve observability of the test results as run by the controller..spec.test.filters
, which works like--filter
of thehelm test
command.HelmRelease
is mutated in such a way that the release target would be different (change in release namespace, storage namespace and/or chart name), the previous release is first uninstalled..spec.maxHistory
has been changed to5
for performance improvements.Reconciling
andStalled
conditions are now being set to indicate when aHelmRelease
is undergoing reconciliation, or has reached a point where it is not expected to proceed without e.g. a change in values, a new chart version, etc.pending-install
,pending-upgrade
orpending-rollback
state will be unlocked by the controller to afailed
state, and retried with a Helm upgrade.Fixes #265
Fixes #270
Fixes #282
Fixes #312
Fixes #322
Fixes #323
Fixes #324
Fixes #367
Fixes #403
Fixes #439
Fixes #471
Fixes #487
Fixes #534
Fixes #536
Fixes #554
Fixes #555
Fixes #644
Fixes #739
Fixes #804