Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DO NOT MERGE: Smoke-test of a costly “always compute old-style IDs” approach #24419

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

DO NOT MERGE: Smoke-test of a costly “always compute old-style IDs” approach #24419

wants to merge 14 commits into from

Conversation

mtrmac
Copy link
Collaborator

@mtrmac mtrmac commented Oct 30, 2024

This is a variant of #24287 which reverts (some of) the system test changes to again expect old-style IDs, and includes containers/storage#2155 / containers/image#2613 .

⚠️ WARNING: all of this is a quick sketch, absolutely untested.

This exists purely to give us options. With this, zstd:chunked loses some of the performance wins, and it should revert to old-style IDs. We then have the option to reintroduce the ID change and performance improvement (perhaps even based on measurements) later.

And, separately, this enforces that layers should match against the config’s DiffIDs, resolving the signing ambiguity.

FYI @giuseppe @edsantiago @mheon @baude

Does this PR introduce a user-facing change?

None

@openshift-ci openshift-ci bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none labels Oct 30, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 30, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mtrmac

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2024
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 30, 2024
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from c179f32 to 5345c94 Compare October 30, 2024 15:19
@packit-as-a-service Packit-as-a-Service
Copy link

Ephemeral COPR build failed. @containers/packit-build please check.

1 similar comment
@packit-as-a-service Packit-as-a-Service
Copy link

Ephemeral COPR build failed. @containers/packit-build please check.

@mtrmac mtrmac changed the title DO NOT MERGE: Smoke-test of costly “always compute old-style IDs” approach DO NOT MERGE: Smoke-test of a costly “always compute old-style IDs” approach Oct 30, 2024
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch 6 times, most recently from 3d26edf to 85bf387 Compare October 31, 2024 18:29
@mtrmac
Copy link
Collaborator Author

mtrmac commented Nov 4, 2024 •
edited
Loading

Note to self: If/after containers/storage#2156 merges, this PR will need to re-enable enable_partial_images.

Edit: done.

@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from 85bf387 to 776b38d Compare November 18, 2024 16:48
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2024
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from 776b38d to 93405a1 Compare November 18, 2024 16:50
@mtrmac
Copy link
Collaborator Author

mtrmac commented Nov 18, 2024 •
edited
Loading

The test images with all images converted to zstd:chunked are too old. For now I’m leaving that variant in https://github.com/mtrmac/libpod/tree/mitr-zstd-costly-old-ids-integration-chunked-test-images , and reverting to test with the current images here.

⚠️ That significantly decreases the scope of testing; many of the unexpected behavior changes were happening only with most images converted to zstd:chunked. With this, we are mostly back to the test coverage of #21903 , which is known to not demonstrate many of the surprises.

@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from 7a0271e to b14da5e Compare November 18, 2024 17:00
@edsantiago
Copy link
Member

The test images with all images converted to zstd:chunked are too old

Would you care to follow (and test and review) my new simplified automation_images doc? You will need to import https://github.com/containers/automation_images/pull/388/files but apart from that it really should be fairly simple.

@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from b14da5e to 4148a66 Compare November 18, 2024 17:35
@mtrmac
Copy link
Collaborator Author

mtrmac commented Nov 18, 2024

Thanks, I’ll definitely try that.

@mtrmac
Copy link
Collaborator Author

mtrmac commented Nov 22, 2024

The test images with all images converted to zstd:chunked are too old.

Note to self: containers/automation_images#395 .

@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch 5 times, most recently from ac0bb3f to 9b2c391 Compare November 29, 2024 00:31
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch 2 times, most recently from 07052cc to fca988f Compare November 29, 2024 15:29
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from fca988f to e8de55a Compare December 12, 2024 23:57
mtrmac added 14 commits December 14, 2024 01:22
@mtrmac
DO NOT MERGE: Nerf make-and-check-size
1591cea 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
DO NOT MERGE: Test with a zstd:chunked testimage
edb6d39 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
DO NOT MERGE: Test with a zstd:chunked container
e649851 
... from containers/automation_images#395 .

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Test with current c/storage and c/image
352928b 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Revert "Use the config digest to compare images loaded/pulled using d…
7c036da 
…ifferent methods"

This reverts commit 1d7ec1e.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Fix image ID query
8b47f71 
Read the full one, not the truncated one

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
DO NOT MERGE: Testing containers/storage#2155
787bac9 
> go mod edit -replace github.com/containers/storage=github.com/mtrmac/storage@wip-authentic

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
DO NOT MERGE: Test containers/image#2613
b658171 
> go mod edit -replace github.com/containers/image/v5=github.com/mtrmac/image/v5@wip-authentic

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
??? Work around zstd:chunked images in health checks
86ed70c 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
WIP: Conditional digests (BUT STANDARD IDs)
8a246ea 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
WIP: Conditional image IDs
dcaad59 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
HACK: Re-enable partial pulls by default
9090231 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
HACK: nerf tree_status.sh
3c1bdac 
Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac
Update expected errors when pulling encrypted images
d8a0cea 
containers/image#2646 will track actually
returning a meaningful error instead of these internal details.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac mtrmac force-pushed the mitr-zstd-costly-old-ids-integration branch from e8de55a to d8a0cea Compare December 14, 2024 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mtrmac @edsantiago @openshift-merge-robot