Error when running jobs with ubuntu image

[0xff8]

Hi guys,

When I run my job with any macOS image everything works as expected. But when I am trying to execute the same job using linux image - job fails. I assume the problem is in the config_exec step and particularly with gitlab-tart-executor config . Is there a way to set build_dir and cache_dir via environment variables ? Or set it in config.toml to some value that will work for both linux and macOS images ?

.gitlab-ci.yaml configuration:

virtual-machine-arm64:
  stage: test
  image: ghcr.io/cirruslabs/ubuntu-runner-arm64:22.04
  variables:
    TART_EXECUTOR_SHELL: "/usr/bin/bash"
  tags: [ "arm64" ]
  script:
    - lsb_release -a
    - whoami

Error:

....
Using Custom executor...
2024/11/30 19:03:15 Pulling the latest version of ghcr.io/cirruslabs/ubuntu-runner-arm64:22.04...
2024/11/30 19:25:02 Cloning and configuring a new VM...
2024/11/30 19:25:02 Waiting for the VM to boot and be SSH-able...
2024/11/30 19:25:09 Was able to SSH!
2024/11/30 19:25:09 VM is ready.

Preparing environment
00:00
Running on ubuntu...

Getting source from Git repository
00:00
mkdir: cannot create directory ‘/private’: Permission denied
2024/11/30 19:25:09 Process exited with status 1

Any suggestions are welcomed :)

[fkorotkov]

This is likely because mounting script is macOS specific:

gitlab-tart-executor/internal/commands/prepare/prepare.go

Lines 214 to 216 in c778283

	mountPoint := fmt.Sprintf("/Users/%s/%s", config.SSHUsername, dirToMount)
	mkdirScript := fmt.Sprintf("mkdir -p %s", mountPoint)
	mountScript := fmt.Sprintf("mount_virtiofs tart.virtiofs.%s.%s %s", dirToMount, gitLabEnv.JobID, mountPoint)

It should be along these lines:

sudo mkdir /mnt/shared
sudo mount -t virtiofs com.apple.virtio-fs.automount /mnt/shared

We'll look into it as some point. Can I ask you why specifically use Tart for Linux jobs instead of having some executor on a Linux box with some Docker executor. Just curious to learn about your use case to have more understanding of use cases of our users.

[0xff8]

hi @fkorotkov ,

Thanks for the answer!
Currently, we have a virtual machine runner for amd64 VMs (based on KVM) and it is very flexible we can run any OS (Linux, Windows, BSDs) just specifying image: win2019 field in the .gitlab-ci.yaml (except macOS by obvious reasons)
We would like to have something similar for ARM64 platform , where developers can run various VMs using different images.
The best solution for us would be either having an algorithm in config script that can detect what image is being run (MacOS / Linux / Windows ) and then adjust the paths automatically.
Also, less convenient way to us, but more error prone is to pass some env variables defined for a job / globally like TART_EXECUTOR_BUILD_DIR / TART_EXECUTOR_CACHE_DIR and define the correct values there.

Right now I can only define these values as parameters and they will only be read once runner is started:

	cmd.PersistentFlags().StringVar(&buildsDir, "builds-dir", "",
		"path to a directory on host to use for storing builds, automatically mounts that directory "+
			"to the guest VM, mutually exclusive with \"--guest-builds-dir\"")
	cmd.PersistentFlags().StringVar(&cacheDir, "cache-dir", "",
		"path to a directory on host to use for caching purposes, automatically mounts that directory "+
			"to the guest VM, mutually exclusive with \"--guest-cache-dir\"")
	cmd.PersistentFlags().StringVar(&guestBuildsDir, "guest-builds-dir", "",
		"path to a directory in guest to use for storing builds, useful when mounting a block device "+
			"via \"--disk\" command-line argument (mutually exclusive with \"--builds-dir\")")
	cmd.PersistentFlags().StringVar(&guestCacheDir, "guest-cache-dir", "",
		"path to a directory in guest to use for caching purposes, useful when mounting a block device "+
			"via \"--disk\" command-line argument (mutually exclusive with \"--cache-dir\")")

[fkorotkov]

Thank you for details!

@edigaryev should we use tart get --format=json to check for either OS is darwin or linux to determine scripts to use?

[0xff8]

so guys, small update. I was able to run the VMs using the following setting in the config.toml:
config_args = ["config", "--guest-builds-dir", "/tmp", "--guest-cache-dir", "/tmp"].
in that case both linux image and macOS do not trigger errors.

But I found another bug, if I use variable TART_EXECUTOR_INSTALL_GITLAB_RUNNER the executor always downloads darwin version of the runner , so it fails then to execute it later inside the VM. Shall a different issue be created?

here is a sample .gitlab-ci.yaml to reproduce the problem:

virtual-machine-arm64:
  stage: test
  image: ghcr.io/cirruslabs/ubuntu-runner-arm64:22.04
  variables:
    TART_EXECUTOR_SHELL: "/usr/bin/bash"
    #TART_EXECUTOR_HOST_DIR: true
    TART_EXECUTOR_INSTALL_GITLAB_RUNNER: "${CI_RUNNER_VERSION}"
  tags: [ "virtual-machine-arm64" ]
  script:
    - lsb_release -a
    - gitlab-runner --version
    - whoami

Here is the cut out output:

Installing GitLab Runner using cURL...
Downloading GitLab Runner from https://gitlab-runner-downloads.s3.amazonaws.com/v17.6.0/binaries/gitlab-runner-darwin-arm64 to /usr/local/bin/gitlab-runner...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 99.1M  100 99.1M    0     0  6164k      0  0:00:16  0:00:16 --:--:-- 6583k
Making /usr/local/bin/gitlab-runner executable...
GitLab Runner was successfully installed!

......

$ gitlab-runner --version
/usr/bin/bash: line 159: /usr/local/bin/gitlab-runner: cannot execute binary file: Exec format error
2024/12/11 20:48:44 Process exited with status 1
.....

[edigaryev]

Hi @0xff8,

Please check out the latest 1.21.0 release, which should work well with linux VMs 🐧

The only thing that you should probably keep in your CI configuration for Linux jobs is TART_EXECUTOR_SHELL: "/usr/bin/bash".