applehv: Rosetta support

[tnk4on]

This PR adds Rosetta support to the AppleHV Podman machine(only v5).
Rosetta is only available on macOS with Apple Silicon.
With Rosetta, the execution performance is several times better than QEMU emulation.
https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta

Todo

• GOARCH and Provider check
• CLI option docs
• Appropriate location in machine configuration(or containers common)
• rosetta.InstallRosetta vfkit option
• containers/common PR Add Rosetta support for applehv(arm64) common#1877
• Release notes
• Consideration of SELinux context
• ship the unit file/script in the Containerfile Add files for Rosetta podman-machine-os#7
• Checking Test Code Execution

Note

[SELinux context] > resolved
Perhaps because of the use of vfkit, the SELinux context needs to be with *exec_t when mounting the Rosetta directory.
On lima and UTM which support Rosetta, it can be executed with nfs_t without error.

https://github.com/lima-vm/lima/blob/88c89165273d87b33753a593af867b20c1d1c67d/pkg/cidata/cidata.TEMPLATE.d/boot/05-rosetta-volume.sh#L22

Does this PR introduce a user-facing change?

Podman machine can now use Rosetta 2 (a.k.a Rosetta) on macOS with Apple Silicon. This is enabled by default. If you wish to change this option, you can set via containers.conf.

[no new tests needed]

[rhatdan]

Why would a user ever disable this? Shouldn't this be on by default?

[tnk4on]

To activate Rosetta, you must mount the Rosetta directory on your Linux OS and add the magic code to binfmt_misc. Some users may want to remove this as an afterthought.
The default can be turned on (true).

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[rhatdan]

Man page needs to be updated as well.

@edsantiago are we verifying man pages for podman machine?

[edsantiago]

Yes, podman machine is checked. CI just didn't get to that check because of other earlier failures. If CI had made it past those, it would've failed with:

xref-helpmsgs-manpages: 'podman machine init --help' lists '--rosetta', which is not in docs/source/markdown/podman-machine-init.1.md
xref-helpmsgs-manpages: 'podman machine inspect --format <TAB>' lists '.Rosetta', which is not in docs/source/markdown/podman-machine-inspect.1.md
xref-helpmsgs-manpages: 'podman machine set --help' lists '--rosetta', which is not in docs/source/markdown/podman-machine-set.1.md

[edsantiago]

Hi, before you push again, please rebase on current main. This will be necessary to check man pages and to avoid breaking future main. Thank you!

[Luap99]

A couple of points, AFAIK rossetta only emulates x86_64 on arm macs:

applehv is also support on intel macs so we cannot do the mounts/setup there at all. This code has to handle it.
Also do we really need to set option? Just keep things simple who has to realistically switch this?

Also I don't like that the flag is added on non apple platforms. It would make much more sense to hide this flag when machine is not running on macos.

Also @baude PTAL

[baude]

@tnk4on i did an initial pass on your PR, but given this is a big change, i hit the big topics first ... thorough code reviews will need to be done once we decide how this fits in podman machine.

my biggest comment, which is a repeated here and there in the review, is that rosetta should be enabled via containers common. this keeps it out of podman docs proper, out of the cli, and structs that are supposed to be general in use.

However, i would like the rest of the maintainers of machine to comment: @Luap99 @mheon @ashley-cui @n1hility @rhatdan ptal

@cfergeau PTAL how are you thinking about enabling rosetta?

And finally, should this be held for after we branch?

[tnk4on]

@baude I appreciate your review and your comments !
The latest commit(0ca90ce) was written before I read your comment, but I pushed it forward to move this PR.
I will continually work on improving this PR.

[rhatdan]

I agree that containers.conf in the machine section is the proper place for this. I don't think the CLI option is necessary since I don't think people will be changing back and forth. I do believe most users will want Rosetta=true, since building x86 images on Mac Arm platforms is going to be very common.

[ashley-cui]

Thanks for the contribution! This looks super promising! One small formatting nit: all files need to end with a newline.

[tnk4on]

Thank you all for your comments and reviews.
The implementation of common (containers.conf) and the removal of the --rosetta option(init/set) is working well in my environment.

My understanding is that I need to send a PR to containers/common ,right?

github.com/containers/common/pkg/config/default.go

Rosetta: true,

github.com/containers/common/pkg/config/config.go

// Rosetta
Rosetta bool `toml: "rosetta,omitempty"`

[n1hility]

Sorry for the late comment:

@rhatdan:I do believe most users will want Rosetta=true, since building x86 images on Mac Arm platforms is going to be very common.

I agree this should just always be on for apple silicon. The only reason to change this is in the rare case you want qemu-user-static to take precedence, which would be a fairly specialized situation. Only case I think of that happening is if there is some bug in rosetta, which breaks a container you need to run, so you need to temporarily apply a workaround.

@rhatdan: I agree that containers.conf in the machine section is the proper place for this. I don't think the CLI option is necessary since I don't think people will be changing back and forth.

IMO there are two issues with it being only specifiable in containers.conf:

1. The machine section is a global, yet the setting scope is really per-machine. Adding it could be reasonable as a default, but if its something that is rare then it might not be necessary
2. Introduces UX complexity, since to set it you first need to write a containers.conf file, and then run the CLI. If you need to change it you then need to set it, delete the machine and then recreate it. It also introduces a similar UX contract API interaction complexity with Podman Desktop, since it needs to incorporate containers.conf editing when this particular change is made.

I think it's inevitable that we will need provider specific settings, but agree it can create a problem with MxN parameter clutter across providers. I can think of two ways we could handle that

1. We could use a generic option for provider settings using name/value arrays (e.g. --provider-setting-name="foo" --provider-setting-value="blah" --provider-setting-name="other" --provider-setting-value="thing"). These would be passed straight into the provider as a map
2. We could have a phase were each provider gets a hook to register extra flags under an opaque type as part of the cobra init, and then perhaps we can add a common prefix convention to make the docs cleaner. Some made up hypothetical examples --applehv-rosetta=false, --applehv-disk-cache=automatic --hyperv-dynamic-memory=true --wsl-wayland=true

Although not suggesting we do this now. With Rosetta could just wait until someone actually needs it to be turned off.

[Luap99]

Does this download anything?
I rather have this not be part of the actual test code and part of the CI setup somewhere.
@cevich How easy is this do add to the macos runners by default.

[tnk4on]

Yes, download a small file. This requires an Internet connection.
https://support.apple.com/en-us/102527

% du -sh /Library/Updates/Rosetta/SoftwareUpdate/RosettaUpdateAuto.pkg
380K	/Library/Updates/Rosetta/SoftwareUpdate/RosettaUpdateAuto.pkg

I agree with including it in the CI setup.

[cevich]

Assuming it requires privileges to install, then it must be installed during the setup where sudo is available. Note: After making the addition there, it will take a few days before all the Macs in the testing pool have the change.

Though I'm afraid the softwareupdate command might fail, doesn't that require the machine to be "attached" (don't know the term) to the Apple mothership via some login?

In either case, somebody (probably me) with privileges to our EC2 dedicated hosts will be needed to test the change (assuming it's possible). Testing setup.sh changes cannot be automated in any trivial way, since there's a 3-hour turnaround for a fresh machine.

[tnk4on]

As far as I have tried, install-rosetta does not require sudo.
I don't know how often Apple updates, but I am concerned about doing a CI setup for every update of the rosetta pkg.

[Luap99]

Well I guess the answer is simple, the CI currently passes because that command works just fine as user. I assume you only added this because CI failed without it @tnk4on?

So yes in this case I think we really should add this this to the runner setup, because I Really do not want this in the test suite. This stuff can be run locally by users so we should never install dependencies without their knowledge.
I think we we can add it to contrib/cirrus/mac_runner.sh temporarily until we have it in the proper runner seup linked by @cevich

[cevich]

I think we we can add it to contrib/cirrus/mac_runner.sh

Keep in mind these Mac's are shared by multiple tasks and multiple PRs (not at the same time). If you add it into that script, maybe wrap it in a condition that checks if it's already installed first?

doing a CI setup for every update of the rosetta pkg.

The Macs servicing CI are recreated roughly every 24 hours. If/when it's installed in the big setup.sh script, any updates would be picked up over time. This isn't ideal, since multiple runs of a PR could net different results depending on which Mac you get.

[tnk4on]

@Luap99 @cevich
The other code fixes for this PR are complete, waiting for mac CI setup changes.
What will happen regarding CI setup changes?

[Luap99]

I suggest you move this command into contrib/cirrus/mac_runner.sh for now. Possibly checking if it is already installed before calling this command.

I don't like this being in the test code.

[Luap99]

We can work on a proper integration in the automation repo later but for now we should get this merged

[cevich]

I opened a Jira card to track this.

[Luap99]

just so we do not forget, if we go a ahead with containers/podman-machine-os#8 this must be removed here

[tnk4on]

Fixed it, and latest machine-mac CI passed !

[packit-as-a-service]

Cockpit tests failed for commit 02be9b4. @martinpitt, @jelly, @mvollmer please check.

[rhatdan]

@Luap99 @baude @ashley-cui PTAL

[rhatdan]

@tnk4on Any reason this has the Hold Label?

[tnk4on]

@rhatdan
My fix work on the latest commit is already done and awaiting review.

I think we are still not done deciding whether to pre-install rosetta on the mac CI mothership in this discussion(#21670 (comment)).
However, we can pass this PR through.

[rhatdan]

Enough users hitting issues on qemu-user-static, so I would like to get this on by default and see if it works.

[Luap99]

@tnk4on Also if you don't mind I prefer the commits to be squashed because the later commits fixes code from the previous commit in this PR so it doesn't really make sense to have them separated.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[rhatdan]

/approve
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhatdan, tnk4on

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rhatdan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mheon]

Let's get this in the next release
/cherry-pick v5.1

[openshift-cherrypick-robot]

@mheon: new pull request created: #22757

In response to this:

Let's get this in the next release
/cherry-pick v5.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.