Pull-based deploy to Kubernetes clusters

[martinothamar]

Status

Proposed

Context

While working on Maskinporten integration and automation in Altinn 3, we also discussed more general improvements around app clusters and integration with Altinn Studio. This is something that has been discussed before in the Platform team.

Azure DevOps (ADO) pipelines are currently used for deployment from Studio to service owner app clusters.
When users click the deploy button in Altinn Studio, a pipeline is run that is parameterized by Altinn Studio backend code to target the correct environment, git-repo, cluster and Azure subscription among other things.

The pipeline

• Pulls the app repo locally
• Initializes a helm values file for environment-specific configuration
  • Known based on input parameters from Studio and/or variables in ADO
• Runs helm upgrade

In the future we will be dynamically provisioning per-app infrastructure using Kubernetes operators (some of which won't live in the app Git repo), which will lead to more configuration that needs to work well at deployment-time.
For instance, the configuration for which Maskinporten scopes the app needs will be mutated at design-time,
but not stored in Gitea. So pulling the repo in the pipeline is not sufficient to build the whole artifact.

There are some drawbacks to the ADO approach

• Complexity of GUI-based release pipelines based on shell scripts
• Need PAT token / credentials to deploy from Studio
• Inflexible configuration options (ADO pipeline parameters, which are flat key-value pairs)
• Unclear history, since there is a single pipeline running deployments for all apps

Decision

Rather than extending ADO pipelines to accommodate our new configuration needs, we suggest that Altinn Studio

• Builds the complete artifact directly, containing
  • Code from the Gitea repo
  • Static configuration from the Gitea repo
  • Dynamic configuration stored in Studio PostgreSQL DB
• Uploads the artifact to a Azure Blob Storage container
  • Using Managed Identity (MI)/Workload Identity (WI) to avoid credentials
• TBD: when an app is created, the corresponding Flux config has to be created in k8s, which means 1 blob container per app probably

Then in the app clusters we can

• Configure Flux to pull from Azure Blob Storage
  • Using MI/WI

Building the artifact could consist of

• Create values.yaml files from dynamic configuration that is not in the Gitea repo
• Render the helm-chart (?), create a flux HelmRelease or similar
• Zip the output

For more concrete technical information, read the platform team RFC: https://github.com/Altinn/altinn-platform/blob/rfc/pull-based-cd/rfcs/0000-pull-based-cd.md

Consequences

• Less logic in ADO pipelines
• Less credentials (due to Azure MI/WI)
• More logic and complexity (including knowledge of environments) in Altinn Studio
• Altinn Studio still has to tell Flux where to pull from
• GitOps is a paradigm shift, there is no GUI pipeline anymore, and there are different tools to view the internal state of things (but Altinn Studio will still have the same list of release/deploy information in the UI)

[martinothamar]

cc @sduranc @bengtfredh - part 2 of k8s/platform related ADRs. Feel free to correct and rewrite as needed

[bengtfredh]

@martinothamar
Have a look at this rfc.
https://github.com/Altinn/altinn-platform/blob/rfc/pull-based-cd/rfcs/0000-pull-based-cd.md
It is focused around deploy platform components, but we want to extend it to cover service owner apps the same way.

[altinnadmin]

@martinothamar Nice! I think we should change the title of this decision to reflect the main architectural change: Pull-based deploy to Kubernetes app clusters.

[martinothamar]

Have a look at this rfc.

Cool! Added the "For more concrete technical information, read the platform team RFC..." paragraph with a link

Nice! I think we should change the title of this decision to reflect the main architectural change

Done

[sduranc]

LGTM. Although, it sounds more like it's specific for Altinn Studio.

Uploads the artifact to a Azure Blob Storage container

Currently we're pushing the OCI artifacts to ACR, using OCI repositories. But that's just implementation details, so it shouldn't matter, ie. we could also end up pushing the OCI artifact to buckets in some point, if that makes sense.

[altinnadmin]

LGTM. Although, it sounds more like it's specific for Altinn Studio.

@sduranc Yup, we agreed to make this more general, like #6, and review it next week.