Persistent customization strategy

[ecordell]

The operator creates kube resources on behalf of the users: deployments, services, serviceaccounts, rbac, etc, which may require some additional modification by the user:

• Adding extra labels or annotations to integrate with other tools (i.e. GKE workload identity)
• Directing workloads in specific ways (tolerations, nodeselectors, affinitty/anti-affinity, topologySpreadConstraints, etc)
• Capacity planning (resource requests / limits)
• Other unforseen future needs due to new SpiceDB features, tooling (HPA?), or the evolution of Kubernetes

All of these modifications are possible today by modifying operator-created resources after (or before!) they have been created. The operator uses Server Side Apply and will not touch fields it does not own. Users can query for which fields are owned by reading the fieldmanager metadata on a given resource.

But modifying the resources after creation makes git-ops workflows difficult, it would be nice if there was a way to persist such modifications in SpiceDBCluster or other native Kube resources.

There are some native methods for persisting this type of change, but only for specific fields of specific resources:

• resource requests can be added automatically via a limitrange on the namespace to set a default
• tolerations can be added with a default toleration setting for the namespace
• volumes and env vars could be injected with PodPreset (which are deprecated and no longer available)

With the background out of the way, this leaves some general approaches we could take:

1. Add new fields to SpiceDBCluster to cover any needs as they come up. This is the approach that most operators seem to take, but doing this for more than a couple of fields leads to huge schemas with dozens of options for customizing specific parts of downstream resources. I don't personally favor this approach - it seems at odds with the fieldmanager tracking that Kube introduced for SSA, and it brings things into the operator's scope that it doesn't actually have an opinion on (all such config is passed blindly to other resources).
2. Admission Controllers: this is the general form of the PodPreset solution, where external config can modify the resource before it is persisted. There are a couple of competing projects with no clear (to me) leader: Kyverno and Gatekeeper both support "mutation" policies that can inject arbitrary data into a resource on creation. This approach can be used with the operator today, but we have no example policies for users to lean on, and it requires installing and running one of these projects as well.
3. Embed generic customizations: instead of providing specific fields for specific customizations, we could provide a hook to allow users to provide arbitrary customizations. This could look like a single kustomization: <configMapName> field with Kustomize manifests (that the operator parses and applies, similar to kubebuilder-declarative-pattern), or it might look more like a Kyverno/Gatekeeper API but with a smaller, spicedb-operator focused scope.

[anthonyralston]

+1 for greater ability to customize the resources that are created by the SpiceDB Operator.

We recently decided against using the Operator and instead rolled our own manifests as there was no easy way to do the following:

• Fetch secrets from an external secret store (e.g. AWS Secrets Manager).
• Add annotations to pods.
• Customise readiness/liveness probes.
• Configure affinity/anti-affinity rules.

[ecordell]

Thanks @anthonyralston, I think we can start taking a closer look at addressing this soon.

Just for reference, did you try out at any of the options that do work today?

For example, you can add extra annotations to the pods by modifying the deployment after the operator creates it (they won't get overwritten!) or by using one of the webhook services mentioned above (kyverno / gatekeeper).

I can make some guesses why those options might not work for you, but I'd love to hear specifically why they don't if you can spare the time.

[ecordell]

v1.2.0 is released which now includes a patches feature to allow arbitrary customization of operator-managed resources.

Everything on this thread is addressed with the exception of:

• Fetch secrets from an external secret store (e.g. AWS Secrets Manager).

because you can't currently tell the operator to not mount a real kube secret. I made #155 to track the additional work needed to enable it.