--dir with volume tags results in unusable mount points due to Sandbox violations

[protonjohn]

We recently upgraded our gitlab-tart-executor and tart installations to 1.19.0 and 2.18.1, respectively. It seems that this new version of gitlab-tart-executor now uses mount tags with the --dir argument to specify which directory serves which function, for example:

tart run --dir /Users/host/caches:tag=tart.virtiofs.cachedir.12345 --dir /Users/host/builds:tag=tart.virtiofs.buildsdir.12345 gitlab-12345

This is nice because it lets us choose a mount point in the host that doesn't have a space in the filename, which breaks make and several other tools. Once the VM is running, gitlab-tart-executor then executes the following in the VM after setting it up:

mount_virtiofs tart.virtiofs.cachedir.12345 /Users/admin/cachedir
mount_virtiofs tart.virtiofs.buildsdir.12345 /Users/admin/buildsdir

These directories can be written to, but not read:

$ echo "Testing 1234" > cachedir/test
$ cat cachedir/test
cat: cachedir/test: Operation not permitted
$ sudo ls cachedir
ls: cachedir: Operation not permitted

This error goes away if I just mount one directory with the --dir option, don't specify a tag, and access it normally under /Volumes/My Shared Files. I noticed when I try to access files mounted this way (i.e., with virtiofs tags), I get a sandbox violation on the guest, but not on the host:

kernel: (Sandbox) System Policy: ls(1078) deny(1) file-read-data /Users/admin/cachedir

I tried to get the full sandbox violation from sandboxd to gather more information but I was unsuccessful. This issue also persists if I try to mount the virtiofs from the /Volumes directory, even as root.

This issue is very strange and unexpected, and I'd greatly appreciate your prompt help :)

Edited for clarity. I also have given the Tart app full disk access through the Privacy & Security pane.

[edigaryev]

Which macOS versions are you using (1) on host and (2) in the guest?

[protonjohn]

Extra note: If we revert to [email protected], everything works as it did previously.

We were able to reproduce the above behavior with both host and guest each running Sonoma 14.4.1, and also 14.6.1.

[edigaryev]

Sorry for the late reply, but I've tried to reproduce your issue on macOS Sonoma 15.0 with the following job definition:

image: ghcr.io/cirruslabs/macos-sonoma-base:latest

job:
  script: echo "Hello, World!" > /Users/admin/cachedir/test.txt && cat /Users/admin/cachedir/test.txt

variables:
  TART_EXECUTOR_ALWAYS_PULL: "false"

...and the following ~/.gitlab-runner/config.toml:

[...]
  [runners.feature_flags]
    FF_RESOLVE_FULL_TLS_CHAIN = false
  [runners.custom]
    config_exec = "gitlab-tart-executor"
    config_args = ["config", "--builds-dir", "/Users/edi/gitlab-builds", "--cache-dir", "/Users/edi/gitlab-cache"]
    prepare_exec = "gitlab-tart-executor"
    prepare_args = ["prepare"]
    run_exec = "gitlab-tart-executor"
    run_args = ["run"]
    cleanup_exec = "gitlab-tart-executor"
    cleanup_args = ["cleanup"]

The following tart run process invocation was observed using ps:

edi              69694   0.0  0.1 411295456  25760   ??  Ss    5:39PM   0:00.08 /opt/homebrew/Cellar/tart/2.18.4/libexec/tart.app/Contents/MacOS/tart run --no-graphics --dir /Users/edi/gitlab-builds:tag=tart.virtiofs.buildsdir.7905985866 --dir /Users/edi/gitlab-cache:tag=tart.virtiofs.cachedir.7905985866 gitlab-7905985866

And the job was successful:

Could you try this again with the macOS Sonoma 15.0?

Be wary of #85 (comment), though.

[torarnv]

I've seen similar things locally. The key thing was that whatever was running tart --dir .. needed to have the rights to read that host dir. In the case of running Tart from Terminal.app, the responsible process is Terminal.app. If Tart is launched via gitlab-runner running as a launch agent, gitlab-runner is the responsible process. Whatever the responsible process is, it needs to have full disk access enabled, or similar permissions.

[protonjohn]

I tried adding both gitlab-runner and gitlab-tart-executor to the list of programs with Full Disk Access permissions, as well as recompiling gitlab-runner to contain an LC_UUID as described in by @edigaryev. Neither seem to work, and I'm able to reproduce this on two separate machines, an M1 and M2 mini.

@edigaryev, did you mean 15.0 Sequoia? We can't do that just yet as we're still working out issues with Xcode 16.

[edigaryev]

@edigaryev, did you mean 15.0 Sequoia? We can't do that just yet as we're still working out issues with Xcode 16.

Oh right, I was talking about the possibility of running macOS 15.0 as the host OS.

Should this be unrelated to the Xcode version if you're running Xcode inside of VMs? 🤔

[protonjohn]

In a spherical-cow world, sure, but due to the 2-vm-per-device restriction we run certain things like on-device and simulator tests outside of the virtual machine, on the physical host. It's not related to our discussion but suffice it to say that running 15.0 as the host OS isn't currently an option.