Prefect Helm Charts

This repository contains the official Prefect Helm charts for installing and configuring Prefect on Kubernetes. These charts support multiple use cases of Prefect on Kubernetes depending on the values provided and chart selected including:

Prefect worker

Workers are lightweight polling services that retrieve scheduled runs from a work pool and execute them.

Workers each have a type corresponding to the execution environment to which they will submit flow runs. Workers are only able to join work pools that match their type. As a result, when deployments are assigned to a work pool, you know in which execution environment scheduled flow runs for that deployment will run.

Prefect server

Prefect server is a self-hosted open source backend that makes it easy to observe and orchestrate your Prefect flows. It is an alternative to using the hosted Prefect Cloud platform. Prefect Cloud provides additional features including automations and user management.

Prometheus Prefect Exporter

The Prometheus Prefect Exporter is a tool to pull relevant Prefect metrics from a hosted Prefect Server instance

Prefect agent

Prefect Agents have been deprecated. prefect-helm version 2024.8.30163822 and earlier will contain the prefect-agent chart.

Usage

Helm must be installed to use the charts. Please refer to Helm's documentation to get started.

TL;DR

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm search repo prefect
helm install my-release prefect/prefect-<chart>

Installing released versions

Charts are automatically versioned and released together. The appVersion and prefectTag version are pinned at package time to the current release of Prefect 2.

The charts are hosted in a Helm repository deployed to the web courtesy of GitHub Pages.

1. Add the Prefect Helm repository to Helm and list available charts and versions:

   helm repo add prefect https://prefecthq.github.io/prefect-helm
   helm repo update
   helm search repo prefect

2. Install the Helm chart

   Each chart will require some specific configuration values to be provided, see the individual chart README's for more information on configuring and installing the chart.

   If chart installation fails, run the same command with --debug to see additional diagnostic information.

   Refer to the Helm install documentation for all options.

The helm charts are tested against Kubernetes 1.26.0 and newer minor versions.

Installing development versions

Development versions of the Helm chart will always be available directly from this Github repository.

1. Clone repository

2. Change to this directory

3. Download the chart dependencies

   helm dependency update

4. Install the chart

   helm install . --generate-name

Upgrading

1. Look up the name of the last release

   helm list

2. Run the upgrade

   # Choose a version to upgrade to or omit the flag to use the latest version
   helm upgrade prefect-server prefect/prefect-server --version 2023.09.07

   For development versions, make sure your cloned repository is updated (git pull) and reference the local chart

   helm upgrade prefect-server .

3. Upgrades can also be used enable features or change options

   helm upgrade prefect-server prefect/prefect-server --set newValue=true

Important notes about upgrading

• Updates will only update infrastructure that is modified.
• You will need to continue to set any values that you set during the original install (e.g. --set namespaceOverride=prefect or --values path/to/values.yaml).
• If you are using the postgresql subchart with an autogenerated password, it will complain that you have not provided that password for the upgrade. Export the password as the error asks then set it within the subchart using:

  helm upgrade ... --set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD

Chart Provenance

You can verify the integrity and origin of Prefect Helm charts by using Helm provenance.

The official Prefect Helm Chart public signing key must be used to verify the provenance of the Prefect Helm charts. Prefect uses Keybase to host the public signing key. The key must first be downloaded and then imported into a local keyring.

Download the Public Signing Key

curl https://keybase.io/prefecthq/pgp_keys.asc | gpg --import

Verify a Chart

A Prefect Helm chart can be verified either by:

• Downloading the chart and running helm verify.
• Using the --verify option during chart installation.

Verify a Downloaded Chart

You can use the helm verify command to verify a downloaded chart. To download a verifiable chart, use the helm pull --prov command. For example:

helm pull --prov prefect/prefect-worker

You can then use the helm verify command to verify the downloaded chart.

helm verify prefect-worker-<version>.tgz
Signed by: Jamie Zieziula <[email protected]>
Signed by: Jamie Zieziula <[email protected]>
Using Key With Fingerprint: 062AA2CFEFB2A02D54975B528DBDDA7074C40C6A
Chart Hash Verified: sha256:90eecaf6650ec08045b82c1f3dbf75529105de1a7fb0e64927ab648752746cea

Verify a Chart During Installation

You can verify a chart during installation by using the --verify option to either the helm install or helm upgrade command.

# During installation
helm install prefect-worker --verify prefect/prefect-worker

# During upgrade
helm upgrade prefect-worker --verify prefect/prefect-worker

Options:

See comments in values.yaml.

Security context

By default, the worker, and server run as an unprivileged user with a read-only root filesystem. You can customize the security context settings for both the worker and server in the values.yaml file for your use case.

If you need to install system packages or configure other settings at runtime, you can configure a writable filesystem and run as root by configuring the pod and container security context accordingly:

podSecurityContext:
  runAsUser: 0
  runAsNonRoot: false
  fsGroup: 0

containerSecurityContext:
  runAsUser: 0
  # this must be false, since we are running as root
  runAsNonRoot: false
  # make the filesystem writable
  readOnlyRootFilesystem: false
  # this must be false, since we are running as root
  allowPrivilegeEscalation: false

If you are running in OpenShift, the default restricted security context constraint will prevent customization of the user. In this case, explicitly configure the runAsUser settings to null to use OpenShift-assigned settings:

podSecurityContext:
  runAsUser: null
  fsGroup: null

containerSecurityContext:
  runAsUser: null

The other default settings, such as a read-only root filesystem, are suitable for an OpenShift environment.

Version support policy

Prefect follows the upstream Kubernetes support policy, meaning that we test against the three most recent minor version releases of Kubernetes. The charts may be compatible with older releases of Kubernetes, however, we do not test against those versions and may choose to reject issues or patches to add support.

Troubleshooting

The database deploys correctly but other services fail with "bad password"

If you are using the subchart deployed database with persistence enabled, it is likely the password for the user has persisted between deployments in the PVC for the database but the secret has been regenerated by the Helm chart and does not match. Deploy and set the 'postgresql.auth.existingSecret' option or set a constant password at postgresql.auth.password.

Contributing

Contributions to the Prefect Helm Charts are always welcome! We welcome your help - whether it's adding new functionality, tweaking documentation, or anything in between. In order to successfully contribute, you'll need to fork this repository and commit changes to your local prefect-helm repo. You can then open a PR against this upstream repo that the team will review!

To get started, install the required dependencies by running:

make all

This will install tools like helm, pre-commit, and mise.

You'll also need to install dependencies for the charts you're working on. You can use the following commands to install dependencies for each chart:

# Server Chart
make buildserver
# Worker Chart
make buildworker
# Prometheus Prefect Exporter Chart
make buildprom
# All Charts
make buildall

Testing & validation

Make sure that any new functionality is well tested! You can do this by installing the chart locally, see above for how to do this.

You can also create and run test suites via helm-unittest. Related test files are stored under ./charts/<chart>/tests/*_test.yaml. Refer to the helm-unittest repository for more information.

The following helper script will run the tests via the helm-unittest Docker image in case you don't have the binary installed locally:

make helmtest

When helm-unittest is available via the mise registry, we'll add it to .mise.toml for easy local installation.

Opening a PR

A helpful PR explains WHAT changed and WHY the change is important. Please take time to make your PR descriptions as helpful as possible. If you are opening a PR from a forked repository - please follow these docs to allow prefect-helm maintainers to push commits to your local branch.