Interruptible workloads on Azure Spot VM instances

This reference implementation provides you with a workload that is meant to guide you to explore the Azure Spot VM managed service from a development and architectural perspective to get the fundamentals, and most of the moving parts so you feel comfortable while architecting your own workload as a next step. As an application architect, you will know how to design a solution to support interruptions. As a developer, you want to use this example to reference when writing your workload code.

Azure Architecture Center guidance

This project has a companion article that describe challenges, design patterns, and best practices for Azure Spot VM as part of your architecture. You can find this article on the Azure Architecture Center at Interruptible workloads using Azure Spot VM. If you haven't reviewed it, we suggest you read it as it will give important context to the considerations applied in this implementation. Ultimately, this is the direct implementation of that specific architectural guidance.

Architecture

Azure must provision spare capacity along all of its regions so it can respond on demand when new resources requested to be created by customers. While that capacity remains idle, you are given with the chance to opportunistically deploy on top of that ephemeral compute in your subscription at discount prices and capped at Pay as you go prices using Azure Spot VM/VMSS.

While Azure Spot VM represents a great opportunity for having significant cost savings, this compute Infrastructure as a Service is provided without an SLA once created. In other words, Azure might evict Virtual Machines with Spot priority at any point in time even right after the machine has started. Therefore, designing workloads for being reliably interruptible is paramount for running on Azure VMs with spot pricing.

In this reference implementation, you are building a reliability interruptible workload, that will be deployed on a single Azure VM with spot pricing. This workload will be disrupted by simulating eviction events, and reliably responding to that event.

Installation

This reference implementation contains a simple and asynchronously queue-processing worker (C#, .NET 6) implemented in combination with Azure Queue Storage and demonstrates how to query the Azure Scheduled Events REST endpoint that allows the workload to be signaled prior to eviction so it can anticipate such disruption event and prepare for interruption limiting its impact.

This interruptible workload is installed on Azure Spot VM by using VM Applications.

Prerequisites

1. An Azure subscription. You can open an account for free.

2. Azure CLI installed or you can perform this from Azure Cloud Shell by clicking below.

   az login

3. Ensure you have latest version

   az upgrade

   

4. .NET 6.0 SDK

5. (Optional | Local Development) Docker

6. (Optional | Local Development) OpenSSL

7. (Optional) JQ

Note 💡 The steps shown here and elsewhere in the reference implementation use Bash shell commands. On Windows, you can install Windows Subsystem for Linux to run Bash by entering the following command in PowerShell or Windows Command Prompt and then restarting your machine: wsl --install

Planning

At this point, you are tasked with being flexile. You need to align architecture decisions with your organization business goals like budget. The design must meet the non-functional requirements at the capacity level for your workload.

1. Get acquainted with the VM sizes Azure can offer you, and try to pick out some of them. The following command list VM SKUs in US East 2 that has a number of cores not greater than 8 by excluding from the results not supported options when using Azure Spot VM/VMSS instances:

   Note 💡 In the future when creating your own interruptible workload ensure you right size your compute requirements, and include the filters in the following query or consider using the Virtual machine selector tool.

   az vm list-sizes -l eastus2 --query "sort_by([?numberOfCores <=\`8\` && contains(name,'Standard_B') == \`false\` && contains(name,'_Promo') == \`false\`].{Name:name, Cores:numberOfCores, RamMB:memoryInMb, DiskSizeMB:resourceDiskSizeInMb}, &Cores)" --output table

   The command above display an output similar to the following:

   Name                  Cores    RamMB    DiskSizeMB
   --------------------  -------  -------  ------------
   Standard_D1_v2        1        3584     51200
   Standard_F1           1        2048     16384
   ...
   Standard_D2_v2        2        7168     102400
   Standard_D11_v2       2        14336    102400
   ...
   Standard_D12_v2       4        28672    204800
   Standard_F4           4        8192     65536
   ...
   Standard_NC6s_v3      6        114688   344064
   Standard_NV6          6        57344    389120
   ...
   Standard_E8as_v4      8        65536    131072
   Standard_D4           8        28672    409600
   ...
   

2. Before laying out an infrastructure proposal, you have to be aware about pricing. You can navigate to the Azure Spot advisor to contrast alternatives you have found from the previous step to apply another filter more budget related for a final cherry-pick. Alternatively, if you had installed JQ you could execute the following command:

   curl -X GET 'https://prices.azure.com/api/retail/prices?api-version=2021-10-01-preview&$filter=serviceName%20eq%20%27Virtual%20Machines%27%20and%20priceType%20eq%20%27Consumption%27%20and%20armRegionName%20eq%20%27eastus2%27%20and%20contains(productName,%20%27Linux%27)%20and%20contains(skuName,%20%27Low%20Priority%27)%20eq%20false' --header 'Content-Type: application/json' --header 'Accept: application/json' | jq -r '.Items | sort_by(.skuName) | group_by(.armSkuName) | [["Sku Retail[$/Hour] Spot[$/Hour] Savings[%]"]] + [["-------------------- ------------ ------------ ------------"]] + map([.[0].armSkuName, .[0].retailPrice, .[1].retailPrice, (100-(100*(.[1].retailPrice / .[0].retailPrice)))]) | .[] | @tsv' | column -t

   Note 💡 You could modify this query by changing the filter for example to incorporate the VM sizes you are mostly interested in as well as specific regions.

   You should get an output similar as shown below:

   Sku                        Retail[$/Hour]  Spot[$/Hour]  Savings[%]
   --------------------       ------------    ------------  ------------
   Standard_DC16ds_v3         1.808           0.7232        60
   Standard_DC16s_v3          1.536           0.6144        60
   Standard_DC1ds_v3          0.113           0.0452        60
   ...
   Standard_NC48ads_A100_v4   7.346           2.9384        60
   Standard_NC96ads_A100_v4   14.692          5.8768        60
   Standard_ND96amsr_A100_v4  32.77           16.385        50
   

   Note 💡 Provided you have chosen a Max Price and Capacity eviction policy, it is a good practice to regularly use the Azure Retail Prices API to check whether the Max Price you set is doing well against Current Price. You might want to consider scheduling this query and respond with Max Price changes as well as gracefully deallocate the Virtual Machine accordingly.

Expected Results

Following the steps below will result in the creation of the following Azure resources that will be used throughout this Reference Implementation.

Object	Purpose
A Resource Group	Contains all of your organization's related networking, and compute resources.
A single Azure Spot VM instance	Based on how flexible you can be you selected an Azure VM size, and it gets deployed so your interruptible workloads can be installed and executed from there. In this Reference Implementation, the Standard_D2s_v3 size was chosen and the VM is assigned a System Managed Identity to give it Azure RBAC permissions as a Storage Queue Consumer.
A Storage Account (blob)	This Azure Storage Account is the home for blobs containing the interruptible workload. Therefore it can be later referenced by using SAS uris.
A VM Application version	A packaged interruptible workload is distributed using VM Applications as medium to make it available to the Spot VM. Specifically, it is created a version named 0.1.0 that is linked to the interruptible workload that is being uploaded to Azure Blob Storage.
A Virtual Network	The private Virtual Network that provides with connectivity over internet to the Azure VM so it can be accessed. For more information, please take a look at Virtual networks and virtual machines in Azure. For VNET enabled VMs like this, the Azure Scheduled Events Metadata Service is available from a static nonroutable IP.
A Network Card Interface	The must have NIC that will allow the interconnection between a virtual machine and a virtual network subnet.
A Spot VM Subnet	The subnet that the VM is assigned thought its NIC. The subnet allows the NIC to be assigned with a private IP address within the configured network address prefix.
A Bastion Subnet	The subnet that the Azure Bastion is assigned to. The subnet supports applying NSG rules to support expected traffic flows, like opening port 22 against the Spot VM private IP.
An Azure Bastion	The Azure Bastion that allows you to securely communicate with over Internet from your local computer to the Azure Spot VM.
A Public IP address	The public IP address of the Azure Bastion host.
A Storage Account (diagnostics)	The Azure Storage Account that stores the Azure Spot VM boot diagnostics telemetry.
A Storage Account (queue)	The Azure Storage Account that is a component of the interruptible workload, that represents work to be completed.
An Azure Monitor Application Insights	This is where all interruptible traced messages are sent. This will be helpful to observe whether the interruptible workload is processing and eventually gracefully shutdown.

Note 💡 Please note that the expected resources for the Spot instance you about to create are equal to what you would create for a regular Azure Virtual Machine. Nothing is changed but the selected Priority which is set to Spot in this case, while creating an on-demand it would have been set to Regular.

Clone the repository

1. Clone this repository

   git clone https://github.com/mspnp/interruptible-workload-on-spot.git

2. Navigate to the interruptible-workload-on-spot folder

   cd ./interruptible-workload-on-spot/

(Optional | Local Development) Execute the interruptible workload locally

You might want to get a first hand experience with the interruptible workload by running this locally. This will help you to get familiarized with the app, or you could skip this step and deploy this into Azure.

1. Generate a new self signed certificate to be able to listen over https when using Azurite emulator for local Azure Storage development:

   mkdir certs \
   && openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ../certs/127.0.0.1-azurite.key -out ../certs/127.0.0.1-azurite.crt -addext "subjectAltName=IP:127.0.0.1" -subj "/C=CO/ST=ST/L=LO/O=OR/OU=OU/CN=CN" --passout pass: \
   && openssl pkcs12 -export -out ../certs/127.0.0.1-azurite.pfx -inkey ../certs/127.0.0.1-azurite.key -in ../certs/127.0.0.1-azurite.crt --passout pass:  \
   && sudo cp ../certs/127.0.0.1-azurite.crt /usr/local/share/ca-certificates \
   && sudo update-ca-certificates \
   && openssl verify /usr/local/share/ca-certificates/127.0.0.1-azurite.crt \
   && cp /etc/ssl/certs/127.0.0.1-azurite.pem ../certs

   Note The instructions provided above must be used only for development purposes.

   Note Listening over https is required by Azurite emulator to enable OAuth support as well as trusting the self signed cert to be able to make secure calls using the SDK in development

   Warning The instructions provided above are valid for Ubuntu machines or WLS, while you could opt to use dotnet dev-certs if you are in Windows or MacOS. For more information, please let's take a look at https://github.com/Azure/Azurite#pfx

2. Run Azurite emulator for local Azure Storage Queue development

   docker run -d -v $(pwd)/certs:/workspace -p 10001:10001 --net="host" mcr.microsoft.com/azure-storage/azurite azurite-queue --queueHost 0.0.0.0 --oauth basic --cert /workspace/127.0.0.1-azurite.pem --key /workspace/127.0.0.1-azurite.key --debug /workspace/debug.log --loose --skipApiVersionCheck --disableProductStyleUrl

3. Setup the local Azure Storage Queue using the REST APIs

   Set the HTTP headers

   x_ms_date="x-ms-date:$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")"
   x_ms_version="x-ms-version:2021-08-06"

   Create a shared key signature for the create queue endpoint

   signature_create_queue=$(printf "PUT\n\n\n\n\n\n\n\n\n\n\n\n${x_ms_date}\n${x_ms_version}\n/devstoreaccount1/devstoreaccount1/messaging" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$(printf 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==' | base64 -d -w0 | xxd -p -c256)" -binary |  base64 -w0)

   Make an HTTP call to create the Azure Storage Queue named messaging

   curl -X PUT -k -v -H "${x_ms_date}" -H "${x_ms_version}" -H "Authorization: SharedKey devstoreaccount1:$signature_create_queue" https://127.0.0.1:10001/devstoreaccount1/messaging

   Create a shared key signature for the create queue message endpoint

   signature_create_queue_messages=$(printf "POST\n\n\n67\n\napplication/x-www-form-urlencoded\n\n\n\n\n\n\n${x_ms_date}\n${x_ms_version}\n/devstoreaccount1/devstoreaccount1/messaging/messages" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$(printf 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==' | base64 -d -w0 | xxd -p -c256)" -binary |  base64 -w0)

   Generate 100 messages

   for i in {1..100}; do curl -X POST -k -v -H "${x_ms_date}" -H "${x_ms_version}" -H "Authorization: SharedKey devstoreaccount1:$signature_create_queue_messages" https://127.0.0.1:10001/devstoreaccount1/messaging/messages -d '<QueueMessage><MessageText>Hello World</MessageText></QueueMessage>';done;

4. Run the worker application

   dotnet run --project src/

   Note When running in Development mode after querying 10 times, the Azure Event Schedule detects an eviction notice emulating an Azure infrastructure event claiming your Spot VM instance. The app proceed to shutdown the workload.

Deploy the Azure prerequisites

1. Create the Azure Spot VM resource group

   Adjust the location as necessary. The resource group and its resources will all reside in this region.

   az group create -n rg-vmspot -l eastus2

2. Create the prerequisite deployment

   az deployment group create -g rg-vmspot -f prereq.bicep

The bicep file deploys the following prerequisites:

• Application Insights instance
• Azure Bastion
• Log Analytics workspace
• User-assigned managed identity with the Storage Queue Data Message Processor role. The role gives read and processing access to the Queue service.
• Network security group assigned to the Bastion subnet (AzureBastionSubnet)
• Public IP address for Azure Bastion
• Storage account (Standard, general purpose v1, with locally-redundant storage) with an empty blob container named apps.
• Virtual network (10.200.0.0/16) with 2 subnets: snet-spot (10.200.0.64/27) and AzureBastionSubnet (10.200.0.0/26)

Package the workload

1. Build the sample worker

   dotnet build ./src -c Release --self-contained --os linux -o worker

2. Copy the systemd configuration file

   cp interruptible-workload.service worker/.

3. Copy the orchestration file

   cp orchestrate.sh worker/.

   Note Once the interruptible workload package gets downloaded into the Spot VM using VM Applications, this file will be executed to kick off the orchestration. The orchestration deploys a single interruptible workload instance. It installs this as a service on the VM and immediately starts the service for the first time.

4. Configure the Azure Application Insights Connection String

   AI_CONNSTRING=$(az deployment group show -g rg-vmspot -n prereq --query properties.outputs.aiConnectionString.value -o tsv)
   
   sed -i "s#\(ConnectionString\": \"\)#\1${AI_CONNSTRING//&/\\&}#g" ./worker/appsettings.json

   Note The general recommendation is not to embed secrets in your application or configuration files but to use a secret storage management solution such us Azure KeyVault. In this reference implementation, we embed this connection string for simplicity.

5. Configure the Azure Storage Account Queue name

   SA_NAME=$(az deployment group show -g rg-vmspot -n prereq --query properties.outputs.saName.value -o tsv)
   
   sed -i "s#\(QueueStorageAccountName\": \"\)#\1${SA_NAME//&/\\&}#g" ./worker/appsettings.json

6. Package the worker sample with the interruptible-workload.service and orchestrate.sh you copied earlier

   tar -czf worker-0.1.0.tar.gz worker/*
   rm -rf worker/

Upload the packaged workload and the orchestration script

1. Upload the package worker sample to the container apps

   az storage blob upload --account-name $SA_NAME --container-name apps --name worker-0.1.0.tar.gz --file worker-0.1.0.tar.gz

Deploy the Azure App Infrastructure

1. Generate a valid SAS uri expiring in seven days packaged workload

   SA_WORKER_URI=$(az storage blob generate-sas --full-uri --account-name $SA_NAME --container-name apps --name worker-0.1.0.tar.gz --account-key $(az storage account keys list -n $SA_NAME -g rg-vmspot --query [0].value) --expiry  $(date -u -d "7 days" '+%Y-%m-%dT%H:%MZ') --permissions r -o tsv)

2. Create the app deployment

   az deployment group create -g rg-vmspot -f app.bicep -p saWorkerUri=$SA_WORKER_URI

   Note This deployment will create the Azure resources that are required to install applications onto the spot VM. It creates the version 0.1.0 and references this version to Azure Storage Blob where you uploaded the packaged workload.

Populate the queue

1. Put 100 messages into the Azure Storage Queue

   for i in {1..100}; do az storage message put -q messaging --content $i  --account-name $SA_NAME;done;

   Note Later these messages are processed by the interruptible workload

Deploy the Azure Spot VM

1. Generate new Spot VM authentication ssh keys by following the instructions from Create and manage SSH keys for authentication to a Linux VM in Azure. Alternatively, quickly execute the following command:

   ssh-keygen -m PEM -t rsa -b 4096 -C "azureuser@vm-spot" -f ~/.ssh/opsvmspots.pem -q -N ""

2. Ensure you have read-only access to the private key.

   chmod 400 ~/.ssh/opsvmspots.pem

3. Get the public ssh cert

   SSH_PUBLIC=$(cat ~/.ssh/opsvmspots.pem.pub)

4. Get the Spot subnet Azure resource id

   SNET_SPOT_ID=$(az deployment group show -g rg-vmspot -n prereq --query properties.outputs.snetSpotId.value -o tsv)

5. Get the Spot subnet Azure resource id

   RA_NAME=$(az deployment group show -g rg-vmspot -n prereq --query properties.outputs.raName.value -o tsv)

6. Create the Azure Spot VM deployment

   az deployment group create -g rg-vmspot -f main.bicep -p snetId=$SNET_SPOT_ID raName=$RA_NAME sshPublicKey="${SSH_PUBLIC}"

   Note This template deploys the VM as a spot VM. It gives the VM permissions to access the Azure Storage Queue using Azure RBAC. Additionally, it will set the VM Application version named 0.1.0 onto the new spot VM. As a result, the interruptible workload starts when the spot VM starts since the interruptible workload is installed as a service.

   Warning Please note that your interruptible workload is set as a code as part of the VM creation. Therefore, if your application depends on managed identities like in this reference implementation, it is recommended to use User assigned identities. Otherwise, you might face race conditions. and as a result it may acquire a mismatched Azure identity token. If you need to make use of System assigned identities in your architecture, the recommendation is to make the workloads resilient to 403 responses, and ensure they implement the pattern to re-acquire tokens awaiting for the System identity to be assigned with the proper roles.

Simulate an eviction event

1. Test your Spot VM and see how the interruptible workload respond to disruption

   az vm simulate-eviction -g rg-vmspot -n vm-spot

   Below you can see an example of the response of the metadata endpoint (Azure Instance Metadata Service) when an eviction is scheduled for your Spot VM

   {
      "DocumentIncarnation": 1,
      "Events":[
         {
            "EventId": "C1BF8322-4CBF-4FC4-9C45-B40CF3106D0E",
            "EventStatus": "Scheduled",
            "EventType": "Preempt",
            "ResourceType": "VirtualMachine",
            "Resources": ["vm-spot"],
            "NotBefore": "Tue, 09 Aug 2022 17:07:32 GMT",
            "Description": "",
            "EventSource": "Platform",
            "DurationInSeconds": -1
         }
      ]
   }

   Note the NotBefore datetime field indicates that the machine is planed for eviction after that specified time and not before.

2. Validate the interruptible workload gracefully shutdown by looking at the tracing data in Azure Monitor

   az monitor app-insights query -g rg-vmspot --app aiworkload --analytics-query 'traces | project timestamp, message | order by timestamp' --offset 0h10m --query "tables[0].rows"

   Warning It takes few minutes to dump the traced messages into log analytics. You could choose to wait some time before executing the query or just go to Azure Portal and access the Application Insights Live Metrics instance.

3. Start the stopped Spot VM.

   az vm start --resource-group rg-vmspot --name vm-spot

   Note If you remote ssh the VM you could confirm the Interruptible Workload service is now started and running again.

Clean up

1. Delete the Azure Spot VM resource group

   az group delete -n rg-vmspot -y

Toublehshooting

While none of the following actions are required for you to follow as part of this reference implementation, you might find it useful to have them handy or just to experiment with the Azure Spot Virtual Machine and the interruptible Workload. Please take into account that the general recommendation is that orchestration should occur without human intervention.

Remote ssh using Bastion into the Spot VM

1. SSH into the new Spot VM. For detailed steps, take a look at connect to a Linux VM

   az network bastion ssh -n bh -g rg-vmspot --username azureuser --ssh-key ~/.ssh/opsvmspots.pem --auth-type ssh-key --target-resource-id $(az vm show -g rg-vmspot -n vm-spot --query id -o tsv)

Manually copy the worker-0.1.0.tar.gz file into the Spot VM

1. Open a tunnel using Bastion between your machine and the remote Spot VM

   az network bastion tunnel -n bh -g rg-vmspot --target-resource-id $(az vm show -g rg-vmspot -n vm-spot --query id -o tsv) --resource-port 22 --port 50022

2. Copy the file using ssh copy

   scp -i ~/.ssh/opsvmspots.pem -P 50022 src/bin/Release/net6.0/worker-0.1.0.tar.gz azureuser@localhost:~/.

Check the interruptible workload status and its logs within the Spot VM

1. You can remote ssh by using the section above and then execute the following command

   sudo systemctl status interruptible-workload

   After the new VM App version installation is complete if you ssh remote you could execute you could get a status outcome similar to one shown below

   

2. You may also need to take a look at all logs under some circumstance

   journalctl -u interruptible-workload.service

Contributions

Please see our contributor guide.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

With ❤️ from Microsoft Patterns & Practices, Azure Architecture Center.