Builds failing when MSI to authenticate with Azure Storage

[danielsollondon]

If you are using managed identity to access azure storage (similar to this Quickstart) we have an issue with the service that is causing builds to fail, this is agnostic of source OS.

I will test, and update the docs, please give me 24hrs.

Apologies for the inconvenience here.

Thanks,

[danielsollondon]

Hi, there is an incompatibility using the AIB Service Principal (SPN), in combination with a user identity, which you may use to authenticate with Azure Storage. However, we have made a simpler option available now, where you can grant the user identity rights to the resource group where you want to read or write images too. I have updated the example here, so you can see how to assign a user identity permissions.

https://github.com/danielsollondon/azvmimagebuilder/tree/master/quickquickstarts/7_Creating_Custom_Image_using_MSI_to_Access_Storage#create-a-custom-image-that-will-use-an-azure-user-assigned-managed-identity-to-seemlessly-access-files-azure-storage

We will be releasing more information on this very soon, again, apologies for the inconvenience here.

[drewkg]

Is this new way compatible with the 'File' customizer, allowing a file to be downloaded from an Azure Storage Account with access under the context of a User Assigned Identity?

[sajithvasu]

@danielsollondon : I'm still not able to make this work. Azure doesn't have the role "Azure Image Builder Service Image Creation Role" that was mentioned in the doc Also, I removed AIB role and added user identity as contributor at Resource Group. Now the 1st step of template building itself is failing after 4 mins with the default failure message shown below..I am using east us2 region for my build.

{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "InternalOperationError",
"message": "Internal error occurred."
}
]
}
}

[sajithvasu]

Ignore my above message.. I figured out the failure reason. Template fails if we enable firewall setting on storage account. Even if we enable "Allow trusted Microsoft services to access this storage account" still it fails. So for testing, i had to temporary disable firewall rules on storage and it worked for 2019 windows datacenter using file customizer.

FYI..
UserIdentity requires Storage account blob reader on storage and contributor on image resource group to be successful. It does not require "Azure Virtual Machine Image Builder" (app ID: cf32a0cc-373c-47c9-9156-0db11f6a6dfc) on the resource groups.

[danielsollondon]

@sajithvasu - thank you for letting me know, I am about to do a docs refresh soon, so I will incorporate this, I will leave the issue open until then.

[cbroglia]

Ciao, I confirm that I too have incurred in the same issue with firewall settings on the storage account. I had restricted the Storage Account using Service Endpoint, and enabling the VNET that Image Builder would use to build the VM.
But doing so, when I call New-AzResourceGroupDeployment, the deployment to the resource group fails.
Is there a plan to support restricting access to the Storage Account not only using a Managed Identity, but also with Service Endpoint? Various enterprise customers don't let Storage Account open to the Internet, according to security policies.

[christopherbantle]

I am also encountering an error when creating my template. In the template, I am accessing Azure Storage using a managed identity passed to Image Builder. I am quite sure that the error is related to Azure Storage access, as when I remove the steps that are accessing scripts in my Storage account, I am able to create the template without error. Also, when I go to inspect the failure in the activity log, I can see that the provisioning error code is NoCustomizerScript.

In my case, the managed identity being passed to Image Builder has been assigned the Contributor role for a resource group that houses both the images that I am building and the Storage account where I am hosting the scripts referenced in my template. I am just wondering if there is a validation check that is specifically looking for the Storage Blob Data Reader role, which hasn't been assigned in this case, because the permissions provided by this role are already being inherited from the Contributor role?

[cbroglia]

@christopherbantle you need to assign explicitly at least the Storage Blob Data Reader to have permissions to access the blobs, this is standard requisite for accessing blob storage, is not related to Azure Image Builder. You can test by yourself from the azure portal if you click "Switch to Azure AD permissions" in the explorer container window.

Instead the topic regarding firewall rules remain pending.

[shawntmeyer]

Daniel,

I am able to download any script directly with the file customizer or the powershell script uri using the User Assigned Identity which has been granted the Storage Blob Data Reader role on the resource group with the image assets storage account (contains blobs with zip files). However, when I run a master customization script that tries to download additional zip files and other sources to the image, it does not work. This script works great when I append a SAS Token to the uri for each blob but not without. I don't know if the issue is with the script running elevated and therefore possibly not with the User Assigned Identity.

[danielsollondon]

Hi Shawn - please can you raise a case for this: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/image-builder-troubleshoot#getting-support

[inaun]

Ignore my above message.. I figured out the failure reason. Template fails if we enable firewall setting on storage account. Even if we enable "Allow trusted Microsoft services to access this storage account" still it fails. So for testing, i had to temporary disable firewall rules on storage and it worked for 2019 windows datacenter using file customizer.

FYI.. UserIdentity requires Storage account blob reader on storage and contributor on image resource group to be successful. It does not require "Azure Virtual Machine Image Builder" (app ID: cf32a0cc-373c-47c9-9156-0db11f6a6dfc) on the resource groups.

@mc

Ciao, I confirm that I too have incurred in the same issue with firewall settings on the storage account. I had restricted the Storage Account using Service Endpoint, and enabling the VNET that Image Builder would use to build the VM. But doing so, when I call New-AzResourceGroupDeployment, the deployment to the resource group fails. Is there a plan to support restricting access to the Storage Account not only using a Managed Identity, but also with Service Endpoint? Various enterprise customers don't let Storage Account open to the Internet, according to security policies.

@microsoft -- any progress here? Not only do some corporate policies require using Private Link, Microsoft's own documentation calls this out as a best security practice. Are there any plans to update the Image Builder to allow customers to follow best practices for securing the Storage Account used for artifacts accessed by Image Builder file customizer?

[AlexanderSehr]

@danielsollondon is the service still under development? I saw that the corresponding repository has not been updated in a while. I build a solution for my customers based on the AIB some time ago and wanted to upgrade it to private networking since Deployment Scripts now support Private Networking too - and essentially aspire to have everything as private as possible. Yet the described network limitation seem to prevent me from doing so (Scenario: AIB in existing subnet with Storage Endpoint to Storage Account with Active Firewall that clearlists the subnet. Data Blob Reader Permissions set up. Yet I always get an unauthorized issue so I can only assume the ServiceEndpoint is not enough for the AIB)