Enable use of multiple deployment identities so can switch between GitHub Actions & local Dev Container builds

[jonnyry]

Summary

Deploying using GitHub Actions then switching and deploying (to the same TRE instance) using a DevContainer errors during the DevContainer deployment due to KeyVault permissions issue.

Problem occurs because the deployment methods use different user accounts (service principal vs user account) & Key Vault RBAC permissions are set up for one during the first deployment, and terraform errors when trying to read the Key Vault during the plan stage of the second deployment.

Steps to recreate

1. Deploy new TRE instance using GitHub Actions

Key Vault is set up with 'Key Vault Secrets Administrator' role applied to the CICD service principal account

2. Redeploy to same TRE instance using DevContainer, error occurs:

Terraform cannot read KV secrets during the plan stage as there is no 'Key Vault Secrets Administrator' RBAC role set up for the current user.

╷
│ Error: making Read request on Azure KeyVault Secret porter-db-connection-string: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=04b07795-8ddb-461a-bbee-xxxxxxxxx;oid=8c1b696d-3423-415e-b4ba-xxxxxxxxx;iss=https://sts.windows.net/bcc87841-98cf-40e6-a2a0-aaaaaaaaaaa/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'\r\nResource: '/subscriptions/10420f14-6516-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-xxxxxxxx/providers/microsoft.keyvault/vaults/kv-xxxxxxxx/secrets/porter-db-connection-string'\r\nAssignment: (not found)\r\nDenyAssignmentId: null\r\nDecisionReason: null \r\nVault: kv-xxxxxxxx;location=uksouth\r\n" InnerError={"code":"ForbiddenByRbac"}
│ 
│   with azurerm_key_vault_secret.cosmos_mongo_connstr,
│   on cosmos_mongo.tf line 94, in resource "azurerm_key_vault_secret" "cosmos_mongo_connstr":
│   94: resource "azurerm_key_vault_secret" "cosmos_mongo_connstr" {
│ 
╵
╷
│ Error: making Read request on Azure KeyVault Secret api-client-id: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=04b07795-8ddb-xxxx-xxxx-xxxxxxxxx;oid=8c1b696d-3423-xxxx-xxxx-xxxxxxxxxxx;iss=https://sts.windows.net/bcc87841-98cf-40e6-a2a0-xxxxxxxxx/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'\r\nResource: '/subscriptions/10420f14-xxxx-xxxx-xxxx-xxxxxxxxx/resourcegroups/rg-nwsdetest/providers/microsoft.keyvault/vaults/kv-nwsdetest/secrets/api-client-id'\r\nAssignment: (not found)\r\nDenyAssignmentId: null\r\nDecisionReason: null \r\nVault: kv-xxxxxxxxx;location=uksouth\r\n" InnerError={"code":"ForbiddenByRbac"}
│ 
│   with azurerm_key_vault_secret.api_client_id,
│   on keyvault.tf line 57, in resource "azurerm_key_vault_secret" "api_client_id":
│   57: resource "azurerm_key_vault_secret" "api_client_id" {
│ 
╵

(etc)

Workaround

Manually assign a Key Vault Administrator role to your user account at a scope higher than the Key Vault itself, such as the resource group or the subscription. If you assign to the Key Vault itself, terraform will error when trying to replace the role on the DevContainer deployment:

│ Error: authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="RoleAssignmentExists" Message="The role assignment already exists."
│ 
│   with azurerm_role_assignment.keyvault_deployer_role,
│   on keyvault.tf line 14, in resource "azurerm_role_assignment" "keyvault_deployer_role":
│   14: resource "azurerm_role_assignment" "keyvault_deployer_role" {
│ 

[marrobi]

Switching to a feature, as not sure its a change in behaviour. The ask is to enable multiple deployment identities to exist in parallel.

I think the best way to do this would be to have an array of additional_deployment_identities in config.yaml. the includes identities that you want to have the same roles assigned a the identity doing the current deployment.

For example when we create the keyvault administrator role:

resource "azurerm_role_assignment" "keyvault_deployer_role" {
  scope                = azurerm_key_vault.kv.id
  role_definition_name = "Key Vault Administrator"
  principal_id         = data.azurerm_client_config.current.object_id // deployer - either CICD service principal or local user
}

We do it for the current deployment ID and any also additional ones in config.yaml.

[jonnyry]

@marrobi just to say I'm relatively comfortable with the workaround for this one - so not a priority for me. Just wanted to get it documented in case other people experience it - I'd (wrongly) assumed terraform would be able to switch out the deployer identities on the KV.

[marrobi]

@jonnyry thanks, I thought it was going to be an easy fix, but our scripts that read the config.yaml don't support arrays. I have it working in python, but converting the bah scripts to Python is more than I want to embark on right now.