OCI layout extensions #2633

vrothberg · 2024-11-14T12:07:37Z

Taken over from #2567 but:

Added tests for the three commits
Use reflinks when adding a file

vrothberg · 2024-11-14T12:07:46Z

In draft for now.

oci/layout/oci_dest.go

mtrmac

Just a quick skim; I’m afraid I didn’t read the added tests yet.

oci/layout/oci_dest.go

vrothberg · 2024-11-19T09:43:39Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works. We're using the function from c/storage which is e2e tested. The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

mtrmac

Actual detailed review still pending, I’m afraid.

mtrmac · 2024-12-02T22:59:51Z

oci/layout/oci_dest.go

@@ -18,10 +18,12 @@ import (
 	"github.com/containers/image/v5/internal/private"
 	"github.com/containers/image/v5/internal/putblobdigest"
 	"github.com/containers/image/v5/types"
+	reflinkCopy "github.com/containers/storage/drivers/copy"


This drags in quite a few dependencies:

+github.com/containers/storage/drivers/copy +github.com/containers/storage/pkg/archive +github.com/containers/storage/pkg/chrootarchive +github.com/containers/storage/pkg/idtools +github.com/containers/storage/pkg/ioutils +github.com/containers/storage/pkg/longpath +github.com/containers/storage/pkg/mount +github.com/containers/storage/pkg/pools +github.com/containers/storage/pkg/promise +github.com/containers/storage/pkg/reexec +github.com/containers/storage/pkg/system +github.com/containers/storage/pkg/unshare +# github.com/docker/go-units v0.5.0 +## explicit +github.com/docker/go-units +# github.com/json-iterator/go v1.1.12 +## explicit; go 1.12 +github.com/json-iterator/go +# github.com/klauspost/compress v1.17.11 +## explicit; go 1.21 +github.com/klauspost/compress +github.com/klauspost/compress/flate +github.com/klauspost/compress/fse +github.com/klauspost/compress/huff0 +github.com/klauspost/compress/internal/cpuinfo +github.com/klauspost/compress/internal/snapref +github.com/klauspost/compress/zstd +github.com/klauspost/compress/zstd/internal/xxhash +# github.com/klauspost/pgzip v1.2.6 +## explicit +github.com/klauspost/pgzip +# github.com/moby/sys/capability v0.3.0 +## explicit; go 1.21 +github.com/moby/sys/capability +# github.com/moby/sys/mountinfo v0.7.2 +## explicit; go 1.17 +github.com/moby/sys/mountinfo +# github.com/moby/sys/user v0.3.0 +## explicit; go 1.17 +github.com/moby/sys/user +# github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd +## explicit +github.com/modern-go/concurrent +# github.com/modern-go/reflect2 v1.0.2 +## explicit; go 1.12 +github.com/modern-go/reflect2 +# github.com/opencontainers/runtime-spec v1.2.0 +## explicit +github.com/opencontainers/runtime-spec/specs-go +# github.com/ulikunitz/xz v0.5.12 +## explicit; go 1.12 +github.com/ulikunitz/xz +github.com/ulikunitz/xz/internal/hash +github.com/ulikunitz/xz/internal/xlog +github.com/ulikunitz/xz/lzma

Podman itself does not care, but podman-remote might, and out-of-Podman OCI-specific utilities probably do.
Can only the relevant code be split into a subpackage in c/storage?

AFAIUI, the following lines are the ones we need:
https://github.com/containers/storage/blob/main/drivers/copy/copy_linux.go#L51-L54

Couldn't we move that into the code here?

Cc: @giuseppe

Will be better on the next push since I moved the few lines into c/image/internal/reflink.

oci/layout/oci_dest.go

mtrmac

A full review now.

docs/containers-transports.5.md

oci/layout/oci_dest_test.go

oci/layout/oci_transport.go

mtrmac · 2024-12-03T16:34:28Z

oci/layout/oci_transport_test.go

@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
 		"relativepath",


Holding the line at “all of *_transport.go should have test coverage”, please port the two test cases added to TestGetManifestDescriptor from #1677 (and then see if more need to be added).

Isn't that already being tested now with fixtures/two_images_manifest ?

oci/layout/oci_transport_test.go

oci/layout/reader.go

oci/layout/reader_test.go

mtrmac · 2024-12-03T17:12:48Z

oci/layout/reader_test.go

+			require.Equal(t, test.names[i], ociRef.image)
+			if test.names[i] != "" {
+				require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))
+			}


else test that res.Reference is a sourceIndex reference with the right index?

Or, alternatively, use the returned reference to read a manifest and verify that it matches res.ManifestDescriptor.Digest.

I think we are doing that above, aren't we?

mtrmac · 2024-12-03T17:14:41Z

oci/layout/oci_dest.go

+	}
+	blobName := blobFile.Name()
+
+	copyRange := false


Is there a reason we would not want to use this?

mtrmac · 2024-12-03T17:22:48Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works.

…

The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

Tests writing to non-test directories , and possibly leaving them around on aborts/failures, seems rather unexpected to me, some might consider it (or, at least, be worried that it is) hostile. So I’d prefer to leave this to e2e tests, maybe the future tests of the Podman artifact CLI.

(Just because I was looking it up, Google says reflinks don’t work on ext4. That’s not really a blocker, we could gate this behind a filesystem type check.)

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg mentioned this pull request Nov 14, 2024

WIP: oci/layout API extensions #2567

Closed

vrothberg commented Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

mtrmac reviewed Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

vrothberg force-pushed the layout-list branch 3 times, most recently from ae7457e to 234b45f Compare November 19, 2024 09:41

vrothberg marked this pull request as ready for review November 19, 2024 12:11

mtrmac added 3 commits November 19, 2024 13:11

Add @sourceIndex syntax to oci/layout

2d2d47b

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Add oci/layout.List

5944589

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Add oci/layout.PutBlobFromLocalFile

6481647

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg force-pushed the layout-list branch from 234b45f to 6481647 Compare November 19, 2024 12:12

mtrmac reviewed Dec 2, 2024

View reviewed changes

mtrmac reviewed Dec 3, 2024

View reviewed changes

vrothberg mentioned this pull request Dec 10, 2024

vendor c/image PR and update code baude/podman#40

Open

vrothberg added a commit to vrothberg/libpod that referenced this pull request Dec 10, 2024

vendor c/image PR and update code

ef6acd3

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCI layout extensions #2633

OCI layout extensions #2633

vrothberg commented Nov 14, 2024 •

edited

Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

mtrmac Dec 2, 2024

vrothberg Dec 17, 2024

vrothberg Dec 17, 2024

mtrmac left a comment

mtrmac Dec 3, 2024

vrothberg Dec 16, 2024

mtrmac Dec 3, 2024

vrothberg Dec 17, 2024

mtrmac Dec 3, 2024

mtrmac commented Dec 3, 2024

		@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
		"relativepath",

OCI layout extensions #2633

Are you sure you want to change the base?

OCI layout extensions #2633

Conversation

vrothberg commented Nov 14, 2024 • edited Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

Choose a reason for hiding this comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Dec 3, 2024

vrothberg commented Nov 14, 2024 •

edited

Loading