Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable use of multiple deployment identities so can switch between GitHub Actions & local Dev Container builds #4138

Open
jonnyry opened this issue Nov 14, 2024 · 3 comments
Labels
feature has workaround a workaround is available for this issue

Comments

@jonnyry
Copy link
Collaborator

jonnyry commented Nov 14, 2024

Summary

Deploying using GitHub Actions then switching and deploying (to the same TRE instance) using a DevContainer errors during the DevContainer deployment due to KeyVault permissions issue.

Problem occurs because the deployment methods use different user accounts (service principal vs user account) & Key Vault RBAC permissions are set up for one during the first deployment, and terraform errors when trying to read the Key Vault during the plan stage of the second deployment.

Steps to recreate

1. Deploy new TRE instance using GitHub Actions

Key Vault is set up with 'Key Vault Secrets Administrator' role applied to the CICD service principal account

2. Redeploy to same TRE instance using DevContainer, error occurs:

Terraform cannot read KV secrets during the plan stage as there is no 'Key Vault Secrets Administrator' RBAC role set up for the current user.

╷
│ Error: making Read request on Azure KeyVault Secret porter-db-connection-string: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=04b07795-8ddb-461a-bbee-xxxxxxxxx;oid=8c1b696d-3423-415e-b4ba-xxxxxxxxx;iss=https://sts.windows.net/bcc87841-98cf-40e6-a2a0-aaaaaaaaaaa/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'\r\nResource: '/subscriptions/10420f14-6516-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/rg-xxxxxxxx/providers/microsoft.keyvault/vaults/kv-xxxxxxxx/secrets/porter-db-connection-string'\r\nAssignment: (not found)\r\nDenyAssignmentId: null\r\nDecisionReason: null \r\nVault: kv-xxxxxxxx;location=uksouth\r\n" InnerError={"code":"ForbiddenByRbac"}
│ 
│   with azurerm_key_vault_secret.cosmos_mongo_connstr,
│   on cosmos_mongo.tf line 94, in resource "azurerm_key_vault_secret" "cosmos_mongo_connstr":
│   94: resource "azurerm_key_vault_secret" "cosmos_mongo_connstr" {
│ 
╵
╷
│ Error: making Read request on Azure KeyVault Secret api-client-id: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=04b07795-8ddb-xxxx-xxxx-xxxxxxxxx;oid=8c1b696d-3423-xxxx-xxxx-xxxxxxxxxxx;iss=https://sts.windows.net/bcc87841-98cf-40e6-a2a0-xxxxxxxxx/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'\r\nResource: '/subscriptions/10420f14-xxxx-xxxx-xxxx-xxxxxxxxx/resourcegroups/rg-nwsdetest/providers/microsoft.keyvault/vaults/kv-nwsdetest/secrets/api-client-id'\r\nAssignment: (not found)\r\nDenyAssignmentId: null\r\nDecisionReason: null \r\nVault: kv-xxxxxxxxx;location=uksouth\r\n" InnerError={"code":"ForbiddenByRbac"}
│ 
│   with azurerm_key_vault_secret.api_client_id,
│   on keyvault.tf line 57, in resource "azurerm_key_vault_secret" "api_client_id":
│   57: resource "azurerm_key_vault_secret" "api_client_id" {
│ 
╵

(etc)

Workaround

Manually assign a Key Vault Administrator role to your user account at a scope higher than the Key Vault itself, such as the resource group or the subscription. If you assign to the Key Vault itself, terraform will error when trying to replace the role on the DevContainer deployment:

│ Error: authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="RoleAssignmentExists" Message="The role assignment already exists."
│ 
│   with azurerm_role_assignment.keyvault_deployer_role,
│   on keyvault.tf line 14, in resource "azurerm_role_assignment" "keyvault_deployer_role":
│   14: resource "azurerm_role_assignment" "keyvault_deployer_role" {
│ 
@jonnyry jonnyry added the bug Something isn't working label Nov 14, 2024
@jonnyry jonnyry changed the title Key Vault permissions issue during deployment when switching between CICD and local DevContainer builds Key Vault permissions issue during deployment when switching between GitHub Actions local builds for same instance Nov 14, 2024
@jonnyry jonnyry changed the title Key Vault permissions issue during deployment when switching between GitHub Actions local builds for same instance Key Vault permissions issue during deployment when switching between GitHub Actions local builds Nov 14, 2024
@jonnyry jonnyry changed the title Key Vault permissions issue during deployment when switching between GitHub Actions local builds Key Vault permissions issue during deployment when switching between GitHub Actions & local Dev Container builds Nov 14, 2024
@marrobi marrobi added feature and removed bug Something isn't working labels Nov 15, 2024
@marrobi marrobi changed the title Key Vault permissions issue during deployment when switching between GitHub Actions & local Dev Container builds Enable use of multiple deployment identites so can switch between GitHub Actions & local Dev Container builds Nov 15, 2024
@marrobi marrobi changed the title Enable use of multiple deployment identites so can switch between GitHub Actions & local Dev Container builds Enable use of multiple deployment identities so can switch between GitHub Actions & local Dev Container builds Nov 15, 2024
@marrobi
Copy link
Member

marrobi commented Nov 15, 2024

Switching to a feature, as not sure its a change in behaviour. The ask is to enable multiple deployment identities to exist in parallel.

I think the best way to do this would be to have an array of additional_deployment_identities in config.yaml. the includes identities that you want to have the same roles assigned a the identity doing the current deployment.

For example when we create the keyvault administrator role:

resource "azurerm_role_assignment" "keyvault_deployer_role" {
  scope                = azurerm_key_vault.kv.id
  role_definition_name = "Key Vault Administrator"
  principal_id         = data.azurerm_client_config.current.object_id // deployer - either CICD service principal or local user
}

We do it for the current deployment ID and any also additional ones in config.yaml.

@marrobi marrobi self-assigned this Nov 15, 2024
@jonnyry
Copy link
Collaborator Author

jonnyry commented Nov 19, 2024

@marrobi just to say I'm relatively comfortable with the workaround for this one - so not a priority for me. Just wanted to get it documented in case other people experience it - I'd (wrongly) assumed terraform would be able to switch out the deployer identities on the KV.

@marrobi marrobi added the has workaround a workaround is available for this issue label Nov 28, 2024
@marrobi marrobi removed their assignment Nov 28, 2024
@marrobi
Copy link
Member

marrobi commented Nov 28, 2024

@jonnyry thanks, I thought it was going to be an easy fix, but our scripts that read the config.yaml don't support arrays. I have it working in python, but converting the bah scripts to Python is more than I want to embark on right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature has workaround a workaround is available for this issue
Projects
None yet
Development

No branches or pull requests

2 participants