Add arm remote builder with ubuntu image

[joinemm]

• aarch64 VM added, the module is modified version of the azurerm-linux-vm module.
• arm machines are added as remote builders on jenkins-controller.
• Currently has lot of duplicated code, modules could be refactored in the future.

[henrirosten]

I'm getting the following error on trying to build an aarch64 target on jenkins-controller after taking the changes from this PR:

cannot build on 'ssh://[email protected]': error: failed to start SSH connection to '[email protected]': Host key verification failed.

known_hosts on jenkins-controller is populated here:

ghaf-infra/hosts/azure/jenkins-controller/configuration.nix

Line 137 in bfb1c92

${pkgs.openssh}/bin/ssh-keyscan -f /var/lib/builder-keyscan/scanlist -v -t ed25519 > /root/.ssh/known_hosts

The builders get added to the scanlist here:

ghaf-infra/terraform/jenkins-controller.tf

Lines 63 to 64 in bfb1c92

	content = join("\n", toset(module.builder_vm[*].virtual_machine_private_ip_address))
	"path" = "/var/lib/builder-keyscan/scanlist"

I believe you should add the arm_builder_vms to that list too.

[henrirosten]

@joinemm: I know you did did pretty extensive trials with attempting to spin up NixOS VMs on Azure.

Please reference the results of your trials, and point out what were the main reasons you had to go with Ubuntu+Nix instead of NixOS on aarch64 builders.

I assume we'll later resume to this topic, and having the results from your analysis will then be valuable.

[joinemm]

@joinemm: I know you did did pretty extensive trials with attempting to spin up NixOS VMs on Azure.

Please reference the results of your trials, and point out what were the main reasons you had to go with Ubuntu+Nix instead of NixOS on aarch64 builders.

I assume we'll later resume to this topic, and having the results from your analysis will then be valuable.

Reasons explained now in arm-builder-vm readme.

I've also updated the ubuntu-builder.sh script to install the same rclone service as the nixos configuration does, so it uses the azure binary cache.

[henrirosten]

Generally, this change looks good to me. I tested that the deployment works, and the resulting infra can build aarch64 targets. Also, the build results end-up in the deployed binary cache signed with the correct key.

However, there is one problem I noticed in testing: after taking the changes from this PR, the deployment to UAE no longer works. The reason is: D2ps - D64ps v5 Azure image sizes used in aarch64 builders are not available in UAE, so the attempt to deploy to that location fails with:

Error: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The requested VM size Standard_D4ps_v5 is not available in the current region. The sizes available in the current region are: ...

Bigger problem is that currently, there are no Azure image sizes that support arm-based VMs on UAE.

My proposal is:

• We merge this change now despite the above issue
• We later follow-up with a work-around to address the above issue: perhaps use the aarch64 builder from another location on UAE deployments etc.