Add podman farm build command

[umohnani8]

Add podman farm build command that sends out builds to
nodes defined in the farm, builds the images on the farm
nodes, and pulls them back to the local machine to create
a manifest list.

Signed-off-by: Urvashi Mohnani [email protected]

Does this PR introduce a user-facing change?

Add podman farm build command

[umohnani8]

While I finalize the tests, opened the PR to get reviews started on it. Added no new tests needed so that the existing tests can run to ensure no regressions are happening.

Docs changes are quite a bit so opened another PR for it here: #20051

Demo of farm build: https://asciinema.org/a/V6WufoZX0Kglvlgq2YfmqZhH7

Adding hold label
/hold

[cgwalters]

Just happened to see this scroll by, but one thing I wanted to mention here is in the coreos pipeline we basically implement a version of this, see e.g. this code.

However, one recent pain point is that we actually want to orchestrate this process inside an OCP cluster, and the container-in-container flow needed STORAGE_DRIVER=vfs which is a well known thing. However, IMO there's no reason to actually use containers-storage for this, we should basically support using an oci directory and that has no special host requirements and should in general be a bit more efficient than deserializing the images only to reserialize them.

[cgwalters]

I'm slightly surprised to see podman ship this FWIW, I would have said that multi-system orchestration is something we should defer to a higher level tool.

[rhatdan]

The goal is multi-arch builds. Assembling multi-arch images is a pain in the butt, we are attempting to make that easier. Especially on a MAC, where you could have two podman machines running one ARM and one x86.

[vrothberg]

I'm slightly surprised to see podman ship this FWIW, I would have said that multi-system orchestration is something we should defer to a higher level tool.

Thanks for checking, @cgwalters !

As Dan mentioned, farm build is focusing on simplifying mutli-arch builds only. It's not meant to orchestrate beyond that. It boils down to maintaining a list of podman system connections along with the knowledge of the hardware platforms the machines run on.

[Luap99]

@umohnani8 I think the main problem is that you try to recreate a local engine for the local build instead of using the already existing engine interface in the client. I.e it should be possible to pass down registry.ImageEngine() to your farm package for the local build instead of trying to create your own interface instance.

[Luap99]

On the cli it should look something like this:

diff --git a/cmd/podman/farm/build.go b/cmd/podman/farm/build.go
index 6acf998fb..7ecb75baf 100644
--- a/cmd/podman/farm/build.go
+++ b/cmd/podman/farm/build.go
@@ -9,8 +9,8 @@ import (
        "github.com/containers/podman/v4/cmd/podman/common"
        "github.com/containers/podman/v4/cmd/podman/registry"
        "github.com/containers/podman/v4/cmd/podman/utils"
+       "github.com/containers/podman/v4/pkg/domain/entities"
        "github.com/containers/podman/v4/pkg/farm"
-       "github.com/containers/storage"
        "github.com/sirupsen/logrus"
        "github.com/spf13/cobra"
 )
@@ -106,17 +106,13 @@ func build(cmd *cobra.Command, args []string) error {
                defaultFarm = f
        }
 
-       var storeOptions *storage.StoreOptions
+       var localEngine entities.ImageEngine
        if buildOpts.local {
-               opts, err := storage.DefaultStoreOptionsAutoDetectUID()
-               if err != nil {
-                       return err
-               }
-               storeOptions = &opts
+               localEngine = registry.ImageEngine()
        }
 
        ctx := registry.Context()
-       farm, err := farm.NewFarm(ctx, defaultFarm, storeOptions, buildOpts.local)
+       farm, err := farm.NewFarm(ctx, defaultFarm, localEngine)
        if err != nil {
                return fmt.Errorf("initializing: %w", err)
        }

Then you need to make pkg/farm just pass around this interface instead of passing around c/storage options.
The main advantage of this is that it will actually respect the podman cli options such as --root/--runroot and not just work with the defaults.

[umohnani8]

This is ready to be merge, can we please merge @rhatdan @nalind @vrothberg

Added remote test and the bloat for both podman and podman-remote is significantly less now.

[edsantiago]

Test logs are disturbingly noisy. And it still bothers me that this can only be tested in CI, not by any developer in a local environment.

But enough already. It is impossible to keep reviewing this. I will take on the task of fixing the test bugs. I'm OK with merging this and then making smaller fix PRs in the future.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: edsantiago, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [edsantiago,umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[TomSweeneyRedHat]

LGTM
and I concur with @edsantiago . It's time to merge this and adjust as necessary afterwards. I'd say mark this as Tech Preview for Fedora v4.8, and then we'll do the same in RHEL 8.10/9.4 and Podman v4.9.
I'm going to let Paul, Valentin, Nalin, or Matt push the shiny red merge button.

[rhatdan]

/lgtm
/unhold

[vrothberg]

I'd say mark this as Tech Preview for Fedora v4.8, and then we'll do the same in RHEL 8.10/9.4 and Podman v4.9.

Please wait with advertising farm build. We had a meeting earlier this week and decided to merge the PR as is but there will be follow ups changing quite a bit how it's working. In other words, it's not yet ready for tech preview.

[afbjorklund]

There were some questions about the new podman farm already in v4.7, where all it did was edit config files...

Usage:
  podman-remote-static farm [command]

Available Commands:
  create      Create a new farm
  list        List all existing farms
  remove      Remove one or more farms
  update      Update an existing farm

[TomSweeneyRedHat]

@vrothberg I'm mostly concerned with putting the Tech Preview on Build Farm for the Podman 4.8/4.9 that lands in RHEL 8.10/9.4 in early February. Does that work for you? I'm fine with not marking it Tech Preview for Fedora at the moment while pieces/parts are worked on further.

[rhatdan]

It is not going to be ready so the commands should be hidden. We do not document it and will not ship this until podman 5.*

[TomSweeneyRedHat]

@rhatdan @baude @mheon we have created Features/Epics saying Farm will be in RHEL 8.10/9.4 as Tech Preview, which would be late Jan/early Feb with Podman v4.9. Do we need to backtrack there?

[rhatdan]

I think it would be better to wait until 5.0

[cevich]

Respectfully, I'm going to disagree. I think this is an important feature that distinguishes podman. I could understand delaying if it's incomplete/not working in some way (is it?). Otherwise, is there really some important reason to delay getting it into the hands of users? Could we do a v4.10 for this?

[afbjorklund]

Could we do a v4.10 for this?

It is now included in v4.9.0, but it is mostly undocumented...

"The podman farm suite of commands for multi-architecture builds is now fully enabled and documented."

https://docs.podman.io/en/v4.9.0/markdown/podman-farm.1.html

Especially the need to upgrade all the servers to 4.9 or above.

But also the need to deploy a registry, or configure an existing one.