Per-app infrastructure provisioning in app clusters using Kubernetes Operators

[martinothamar]

Status

Proposed

Context

We have been working on Maskinporten automation for apps, such that scopes that are required in apps
for integration with other APIs in a non-user context can be configured directly in Altinn Studio,
and abstractions for requesting tokens and authorizing HTTP clients are available in the Altinn.App.Core library.
In this effort we have been working to make sure the approach chosen for Maskinporten automation is generalizable to different types of infrastructure that we might want to offer service owners' apps.

At the moment, all infrastructure resources provisioned and configured on behalf of service owners are "global", so there is no support for provisioning infrastructure components per app. For Maskinporten, we need to provision 1 Maskinporten client per app.

We need a system and architectural pattern to provision infrastructure where

• Users
  • can configure features through Altinn Studio
  • can read/see current status of infrastructure in Altinn Studio
  • are sufficiently authorized, i.e. through ID-porten/Ansattporten
• Configuration is kept outside Git
  • So that Gitea access control doesn't become a loophole in terms of access control
• Appropriate transactional boundaries and semantics are in place to avoid 2-phase-commit-like issues
• "Infinite retries"/reconciliation/eventual consistency
  • Even if reconciliation fails 50 times in a row, we should eventually reach the desired state
• Good mechanism for providing infrastructure output as configuration input to apps
• Good tooling/API/SDK support for Kubernetes (k8s)
  • Since existing workloads and configuration exists in k8s

Decision

For use cases where we need to provision per-app infrastructure, we should use k8s Operators:

• Maskinporten automation/integration - build our own
• Azure KeyVault - Azure Service Operator, built by MS
• Azure PostgreSQL - Azure Service Operator, built by MS

When we need to build our own operator, we should use Kubebuilder.
The k8s operator pattern and controller-runtime libraries gives us APIs and a programming model
where the input is desired state/configuration, and we provide the logic (the reconciliation loop) that applies
the desired state to infrastructure, which in some cases is just additional k8s manifests,
and in others are external resources owned by external APIs. The controller runtime has

• Leader elections to ensure mutual exclusion/single writer over k8s and external resources (such as Maskinporten API)
• Flexible lifecycle management
  • Rescheduling on failure
  • Finalizer hooks
• Versioned model for developing resource definitions/configuration

The Kubebuilder framework brings in the controller-runtime libraries, testing setup,
codegen and CLI tools to make operator development simpler.

Consequences

• The Altinn platform can cover more ground, so that there is less of a reason for service owners to reach outside the platform (which is expensive)
• Get great libraries and infrastructure out of the box for providing idempotence, eventual consistency to operations
• Need to write Go, which is not common across teams

Diagram

The diagram below is what was agreed upon when designing the Maskinporten automation. The striped lines show how other infrastructure such as Azure Key vault and Azure PostgreSQL fit into the picture. Note that the DB in the top right corner is used as a config sync such that the infrastucture setup is not based on git.