Skip to content

Latest commit

 

History

History
89 lines (60 loc) · 5.37 KB

token-rotation.md

File metadata and controls

89 lines (60 loc) · 5.37 KB
Raw

Rotate Service Account Tokens

The Kubernetes community considers bound service account tokens as best practice although non-expiring ones are still supported.

Prow Components and CI-Tools

Bound SA tokens are used for Prow components and ci-tools except the following cases in the next section. Each SA's token is created by prowjob/periodic-ci-secret-generator and bound to the same object secret/token-bound-object-{0|1} in the same namespace. To expire the tokens,

  • Bind all tokens to the other secret in the generator's configuration file, e.g., bind to secret/token-bound-object-1 if secret/token-bound-object-0 currently. Create a PR and Merge it.

  • Trigger prowjob/periodic-ci-secret-generator to generator the tokens bound to the new secret.

make job JOB=periodic-ci-secret-generator

  • Delete the secret that the old tokens were previously bound to. It will expire the old tokens. Since the old secret's manifests is still in the release repo, it will be created with a new uid and to be prepared the next rotation.
oc --context ${CLUSTER} delete secret -A -l ci.openshift.io/token-bound-object=$(TOKEN_BOUND_OBJECT_NAME_SUFFIX)  --dry-run=none --as system:admin

Non-Expiring Tokens

We use non-expiring tokens DPTP-3087 in the following cases:

  • config-updater: Their kubeconfigs are used to manage manifests on all clusters, including the secrets of kubeconfigs for other service accounts. We break the checken-and-egg issue this way.
  • The kubeconfigs that are used by the cronjobs on the psi clusters to communicate with the clusters in CI production. The secrets on psi are not managed by our tools because it is inside Red Hat Intranet.

The version of Non-Expiring Tokens is configured in _token.yaml. Before generating new tokens for those service accounts which use non-expiring tokens, we have to bump the token version there:

# need to install yq https://github.com/kislyuk/yq
$ make increase-token-version

Then, populate the new version in other places:

make refresh-token-version

Then create a pull request based on the changes and merge it.

To keep the maintenance simple, we should use non-expiring tokens as the last resort, i.e., only use it when bound tokens do not work for the case. If we have to use it for a new service account, an additional required step is to ignore the SAin the service account refresher controller of dptp-controller-manager to void deleting the secret holding its non-expiring token.

Generate New Tokens

Config-Updater

The tokens in dptp/config-updater have to be refreshed manually. If it is for a cluster managed by us, we can get the kubeconfig by

make CLUSTER=${CLUSTER} API_SERVER_URL=$API_SERVER_URL config-updater-kubeconfig

If the cluster is not managed by the test-platform team such as vsphere, we could use oc extract secret/config-updater -n ci --to=- --keys sa.config-updater.${CLUSTER}.config to get the new token or ask the owners of the cluster to provide the new token if secret/config-updater on app.ci is still valid. Then the secret will be refreshed with the new token after the next run of prowjob/periodic-ci-secret-bootstrap.

In case secret/config-updater does not work, we have to fix the secret manually because prowjob/periodic-ci-secret-bootstrap depends on it.

$ make secret-config-updater

Service Accounts Used On PSI

If the version is modified, the secrets on psi have to be refreshed manually with the new tokens with the steps below.

$ make -C ./clusters/psi apply_credentials

Expire a Previous token

After merging of the pull request, we expire the token with a previous version on every cluster:

$ make CLUSTER=${CLUSTER} EXPIRE_TOKEN_VERSION=1 DRY_RUN=none expire-token-version

Note that we have to always use a new name for the secrets (e.g, config-updater-token-version-n) that contain the non-expiring token because it would reactivate the expired token otherwise. That is the reason we cannot bounce between two secrets like we do for the bound SA's tokens. Instead, we increase the number in the secret's names each time.

On any cluster, we should keep only the latest version of those secrets.

$ make list-token-secrets 
oc --context app.ci -n ci get secret -l ci.openshift.io/token-version --show-labels
NAME                                        TYPE                                  DATA   AGE   LABELS
config-updater-token-version-1              kubernetes.io/service-account-token   4      13d   ci.openshift.io/non-expiring-token=true,ci.openshift.io/token-version=version-1
sync-rover-groups-updater-token-version-1   kubernetes.io/service-account-token   4      13d   ci.openshift.io/non-expiring-token=true,ci.openshift.io/token-version=version-1