OpenShift can be configured to use temporary credentials for different components with GCP Workload Identity. It enables an authentication flow allowing a component to assume an IAM Service Account resulting in short-lived credentials. It also automates requesting and refreshing of credentials using an OIDC identity provider. OpenShift can sign ServiceAccount tokens trusted by the provider which can be projected into a Pod and used for authentication. The following is a diagram showing how it works.
We start by creating an RSA key pair. The private key is provided to the OpenShift cluster via the installer. The kube-apiserver uses this key to sign service account tokens. These tokens are projected into pods and rotated by kubelet every 1 hour. The public key is stored along with openid-configuration in the google cloud storage bucket, which serves as the identity provider. We also create a workload identity pool along with a provider that has an issuer url (the public url of the storage bucket). Every in-cluster OpenShift component has the gcp credentials configuration file encoded in the kube secret. This configuration file has a path to an oidc token, details about the workload identity provider that can authenticate this token, and also the GCP service account that needs to be impersonated (a sample json of this config file is provided in the next section for reference). The google client shares the OIDC token with the GCP security token service (STS), which validates if the OIDC token is from a trusted provider before issuing an access token. We also set up an IAM policy that allows identities in the workload identity pool to impersonate the service account. With these policies in place, OpenShift components can use the signed access token to authenticate with the GCP service account.
If we check the credentials secret, we have the following base64 encoded content in the service_account.json key of the data field.
Without Workload Identity, the type of the credentials is service_account. These credentials include a private RSA key, in the private_key field, to be able to authenticate to gcp. This private key needs to be kept secure and is not rotated.
{
"type": "service_account",
"project_id": "test-project",
"private_key_id": "216fbde61a93dabd942ad738062f0ff0811b69f8",
"private_key": "<private_key>",
"client_email": "[email protected]",
"client_id": "100684158900346633229",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/[email protected]"
}With Workload Identity, the type of the credentials is external_account. audience is the target audience which is the workload identity provider. The service_account_impersonation_url key contains the resource url of the service account that can be impersonated with these credentials. credentials_source.file is the path to the oidc token, which is exchanged for a google access token. The oidc token is rotated every one hour and thus credentials are short lived.
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken",
"credential_source": {
"file": "/path/to/oidc/token",
"format": {
"type": "text"
}
}
}-
Set the variable
$RELEASE_IMAGE$RELEASE_IMAGEshould be a recent and supported OpenShift release image that you want to deploy in your cluster. Please refer to the support matrix for compatibilities.A sample release image would be
RELEASE_IMAGE=quay.io/openshift-release-dev/ocp-release:${RHOCP_version}-${Arch}Where
RHOCP_versionis the OpenShift version (e.g4.10.0-fc.4or4.9.3) and theArchis the architecture type (e.gx86_64) -
Extract the GCP Credentials Request objects from the above release image. You must use version 4.7 or newer of the
ocCLI.mkdir credreqs ; oc adm release extract --cloud=gcp --credentials-requests $RELEASE_IMAGE --to=./credreqs -
Extract the
openshift-installandccoctlbinaries from the release image.oc adm release extract --command=openshift-install $RELEASE_IMAGE CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' ${RELEASE_IMAGE}) && oc image extract ${CCO_IMAGE} --file='/usr/bin/ccoctl' --registry-config=${PULL_SECRET_PATH:-.}/pull-secret -
Create an install-config.yaml
./openshift-install create install-config -
Make sure that we install the cluster in Manual mode
echo "credentialsMode: Manual" >> install-config.yaml -
Create install manifests
./openshift-install create manifests -
Create GCP resources using the ccoctl tool (you will need GCP credentials with sufficient permissions). The following command will generate public/private ServiceAccount signing keys, create the cloud storage bucket, upload the OIDC config into the bucket, set up a workload identity pool/provider, and create an IAM service account for each GCP Credentials Request. It will also dump the files needed by the installer in the
output_dirdirectoryccoctl gcp create-all --name=<gcp_infra_name> --region=<gcp_region> --project=<gcp-project-id> --credentials-requests-dir=/path/to/credreqs/directory/created/in/step/2 --output-dir=<output_dir> -
Copy the manifests created in the step 7 and put them in the same location as install-config.yaml in the
manifestsdirectorycp _output/manifests/* /path/to/dir/with/install-config.yaml/manifests/ -
Copy the private key for the ServiceAccount signer and put it in the same location as install-config.yaml
cp -a _output/tls /path/to/dir/with/install-config.yaml -
Run the OpenShift installer
./openshift-install create cluster --log-level=debug
- Connect to the newly installed cluster and verify that the OpenShift cluster does not have
rootcredentials. Below command should throw secret not found erroroc get secrets -n kube-system gcp-credentials - Verify that components are assuming the IAM Service Account specified in the secret manifests, instead of creds minted by the cloud-credential-operator. The following command should show you the
typeasexternal_accountandservice_account_impersonation_urlshould contain the email of the service account used by the image registry operator.sample output of the above commandoc get secrets -n openshift-image-registry installer-cloud-credentials -o json | jq -r '.data."service_account.json"' | base64 -d{ "type": "external_account", "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/test-pool/providers/test-provider", "subject_token_type": "urn:ietf:params:oauth:token-type:jwt", "token_url": "https://sts.googleapis.com/v1/token", "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/test-openshift-image-registry-42ssv@test-project.iam.gserviceaccount.com:generateAccessToken", "credential_source": { "file": "/var/run/secrets/openshift/serviceaccount/token", "format": { "type": "text" } } }
Make sure you clean up the following resources after you uninstall your cluster. You can use the gcp_infra_name used in installation step 7 to identify these resources.
- Workload Identity Pool and Provider
- Cloud storage bucket used to store OpenID Connect configuration and the public key
- IAM service accounts created by ccoctl tool along with project policy bindings
You can use ccoctl gcp delete --name=<name> --project=<gcp-project-id> --credentials-requests-dir <path-to-directory-with-list-of-credentials-requests> to delete all the above resources.