[RFC] Flux Bootstrap for OCI-compliant Container Registries

[stefanprodan]

Proposal for a Git-less bootstrap using OCI-compliant Container Registries as the desired state storage.

Spec preview: https://github.com/fluxcd/flux2/blob/rfc-flux-bootstrap-oci/rfcs/000X-flux-bootstrap-oci/README.md

Example:

flux bootstrap oci \
--url=oci://ghcr.io/stefanprodan/flux-manifests:production \
--username=stefanprodan \
--password=$GITHUB_TOKEN \
--kustomization=flux-manifests/kustomization.yaml \
--cluster-url=oci://ghcr.io/stefanprodan/fleet-manifests:production \
--cluster-path=clusters/production

[errordeveloper]

Thanks for drafting this, Stefan! Here are a few thoughts/suggestions from me :)

[errordeveloper]

Might be worse mentioning here regarding workload identity mods that may be needed, e.g. on EKS role ARN needs to be set as an annotation, I forgot if that was necessary in GKE also.

[stefanprodan]

We have all of those documented here https://fluxcd.io/flux/installation/configuration/workload-identity/, people will need to read the docs and adapt their kustomization.yaml.

[errordeveloper]

But perhaps there could be bootstrap argument to specify provider-specific attributes that would be handled accordingly based on provider flag?

[stefanprodan]

There is --provider=<aws|azure|gcp> but this is for the CLI to use the role of the machine where it's running. The IAM of the bastion host may be different from the one that you want to use for Flux source-controller. So we'll need yet another flag with the identity name. This is how Flux AIO works: https://timoni.sh/flux-aio/#__tabbed_1_3

[userbradley]

This is probably not the right place to bring this up but the workload identity for GKE is partially incorrect, you dont need to annotate service accounts any more with a GCP SA for GKE workload identity. You grant the Kubernetes service account access to what ever resources it needs via a member statement like below

principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/flux-system/sa/source-controller

I'm going to have a think on how I can update the docs on this one, but thought I'd raise it before I forget

[thejosephstevens]

We've recently adopted Flux for our multi-cloud architecture. In order to support that we're actually overriding env vars, volumes, and volume bindings directly in order to set up OIDC-based auth to each cloud on our workloads. If we can extend those parameters on the source controller, or if we can pass in a pod template that would allow us to easily inject OIDC auth for accessing the OCI backend (I suspect we could do this with the Helm install method, but support in the CLI would be nice, as the bootstrap command is great).

[errordeveloper]

Does this have to be strictly sequential and synchronous?

[stefanprodan]

Yes it does, CRDs and controllers must be up and running before the cluster sync is deployed, same procedure as for the Git bootstrap.

[errordeveloper]

I would imagine that users may wish to choose between pulling from upstream OCI artifact that is published as part of the Flux release or having a full copy of it. If they choose to use a copy, another command may be needed to keep their copy up to date. Does it make sense?

[stefanprodan]

You would always have a copy in your registry that includes customisations, same as with Git, bootstrap means vendoring the Flux manifests that in 99% of cases would need some fine tuning.

[errordeveloper]

Ah, that's true that in most cases right now people will end up with a copy. How do they bring the copy up to date, e.g. some component has new pod spec fields that have to be set or there is an RBAC change? Do they have to read changelog and implement such changes? If in OCI world this could be avoided by means of referencing an upstream artifact and local changes stored as a patch/kustonization, it might be very nice actually, I guess it wasn't feasible with Git as an upstream.

[stefanprodan]

For both Git and OCI bootstrap, the Flux update is fully automated in CI. See the Story 3 in this RFC. And also the docs here: https://fluxcd.io/flux/flux-gh-action/#automate-flux-updates

[errordeveloper]

... bootstrap means vendoring the Flux manifests that in 99% of cases would need some fine tuning.

Just want to clarify, can this fine tuning be done with an in-cluster Kustomization, or has that been proven somehow challenging?

[stefanprodan]

I've added the common flags to the RFC.

[errordeveloper]

Controller selection is already supported via flags common to all bootstrap sub commands.

Yes, I know, what I mean is that it would be kind of less natural specify controllers with kustomize, one would probably need to select bases as there is no meaningful parameters and if or switch statements (at least last time I checked), the CLI offers are more meaningful option.
So because you need to select controllers, you start with CLI that gives you a single file that you kustomize a little, but it's all much complicated then you might have wished and defeats the purpose of putting kustomize in at this stage.

[errordeveloper]

CUE on the other hand could do better in all of this and you could potentially remove the need for CLI and make custom configs easier to introduce. Does what I said earlier make more sense now?

[stefanprodan]

The way Flux can be customized at bootstrap is all via Kustomize patches and CLI options, this must be 100% compatible with the oci sub-command. I’m not considering using CUE or anything really that would diverge from the current Git bootstrap procedure. Users should be able to migrate from Git to OCI by simply reusing their current flux-system overlay including patches, image overrides, configGenerator, volumes, etc.

[errordeveloper]

That is for sure, I'm just thinking maybe self-managed flux in the future could use CUE for this, possibly even without exposing CUE to the user. CLI could still work the same way on the surface also. I do recall we once spoke of an installer operator too, you could use CUE there and in the CLI. Just an idea :)

[errordeveloper]

These are two alternative methods right? It's not very clear from the text at the moment.

[stefanprodan]

If you don't have access to the cluster (what the user story is about), this is the only way. If you have API access, then like with Git bootstrap, you can just rerun it to update. OCI bootstrap behaves the same as Git bootstrap.

[errordeveloper]

I just thought these two commands will write the same kind of output, except that install lets you select a subset of controllers... maybe I am missing something else.

[errordeveloper]

Is it the case right now that one has to rerun bootstrap on major/minor releases while patch releases are taken care of by in-cluster image version bumps?

[stefanprodan]

Every time we release Flux, users get a PR opened to update their manifests in Git. For OCI you would need some kind of semver range or some other manual gate e.g. a GitHub workflow dispatch to approve minor bumps and let only patch versions be automatically push to the registry.

[stefanprodan]

I just thought these two commands will write the same kind of output, except that install lets you select a subset of controllers... maybe I am missing something else.

All the flux bootstrap commands have the same args as flux install, so you can pick controllers, etc with bootstrap too. If you bootstrap with flux bootstrap oci --components=source-controller,kustomize-controller, to update your would run flux install --components=source-controller,kustomize-controller --export in CI.

[errordeveloper]

I am talking about'pull artifact' vs 'install --export' (per above)

[stefanprodan]

You need to pull to preserve the existing kustomization.yaml and any other extra resources you may have added at bootstrap. The install -export command only generates the components YAML.

[errordeveloper]

Ah, of course, this is a "rebase" ;)
Maybe the text needs to make that clearer.

[nagyv]

I wonder if it would make sense to have two registry auth. A read-only for the image pull secrets, and read-write for pushing the artifacts to the registry. Storing a read-write secret in the cluster for image pull secrets does not seem like a good idea.

[sestegra]

It is even possible to consider more pull secrets with specific permissions (least privileges principle):

• one with a read-only permission to be used by the Deployments of the Flux controllers to pull images
• one with a read-only permission (different namespace than images) to be used by the OCIRepository resource to pull OCI artifacts
• one with write permissions to be used by the bootstrap command line

Then, it is possible to consider different OCI registry: one for images and another for Flux artifacts, because the latter could contain sensible infrastructure information.

[stefanprodan]

I wonder if it would make sense to have two registry auth. A read-only for the image pull secrets, and read-write for pushing the artifacts to the registry.

Yes this is something the command could support. Currently our OCI implementation supports reading the Docker config file from the host OS, so we could use that for write operations and the flags for in-cluster secret.

@sestegra the pull secret for the container images is already supported, it's one of the common flags to all bootstrap commands.

[mo-saeed]

@stefanprodan is there any estimation when this can be released ?