From 6c56aa24356633773b000f72e9a7b074306eda32 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 11 Apr 2024 14:52:28 -0500 Subject: [PATCH 1/5] export run image metadata Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 121 +++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 text/0000-export-run-image-metadata.md diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md new file mode 100644 index 000000000..6ef3802fa --- /dev/null +++ b/text/0000-export-run-image-metadata.md @@ -0,0 +1,121 @@ +# Meta +[meta]: #meta +- Name: Export Run Image Metadata +- Start Date: 2024-04-11 +- Author(s): joeybrown-sf +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: N/A + +# Summary +[summary]: #summary + +The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values can be critical to a platform rebase process. + +# Motivation +[motivation]: #motivation + +When publishing an image directly to a registry, parsing the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) is inefficient--it requires a call to the registry after the image is published. Adding these values to `report.toml` lets platforms efficiently access this data after an export or rebase without a call to the registry. + +# What it is +[what-it-is]: #what-it-is + +These values will be included when the lifecycle exporter/creator/rebaser binary writes `report.toml`. + +Here are two examples of `report.toml` content. (Other values are omitted for readability.) + +#### Image published to a registry: +``` +[image] +tags = ... +digest = ... +image-id = ... +manifest-size = ... + +[run-image] +image = "run/name:foo" +reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" +top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +``` + +#### Image exported to the docker daemon: +``` +[image] +tags = ... +digest = ... +image-id = ... +manifest-size = ... + +[run-image] +image = "run/name:foo" +reference = "5b90f9c0e189" +top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +``` + +# How it Works +[how-it-works]: #how-it-works + +This metadata is readily available when `report.toml` is created, so it will be straight-forward to extend `report.toml`. + +# Migration +[migration]: #migration + +N/A + +This is an additive change to a metadata file and will be backwards compatible. + +# Drawbacks +[drawbacks]: #drawbacks + +This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). It can be accessed by querying the config from docker daemon or registry. So we will be writing this data to two outputs. + +# Alternatives +[alternatives]: #alternatives + +Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. + +Introduce a new output file (`lifecycle-metadata.toml?`) that contains everything in [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). + +# Prior Art +[prior-art]: #prior-art + +These values are among those that `pack inspect` returns. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +N/A + +# Spec. Changes +[spec-changes]: #spec-changes + +The following would be appended to the [`report.toml` spec](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) (this section would be materially identical to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json)): + +> run-image.top-layer MUST contain the uncompressed digest of the top layer of the run-image. +> +>run-image.reference MUST uniquely identify the run image. It MAY contain one of the following +> - An image ID (the digest of the uncompressed config blob) +> - A digest reference to a manifest stored in an OCI registry + +# History +[history]: #history + + From 0af349b834dc7396052fd97d1ed2d91894033b1f Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 09:01:29 -0500 Subject: [PATCH 2/5] provide more motivation content & alternatives Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 6ef3802fa..8d9922674 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -12,12 +12,16 @@ # Summary [summary]: #summary -The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values can be critical to a platform rebase process. +The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. # Motivation [motivation]: #motivation -When publishing an image directly to a registry, parsing the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) is inefficient--it requires a call to the registry after the image is published. Adding these values to `report.toml` lets platforms efficiently access this data after an export or rebase without a call to the registry. +Platform operators may need a comprehensive understanding of images on their platform in order to make decisions about rebase and image publishing. Run image metadata is likely part of this comprehensive understanding for rebase. It is likely that this data may only be known after an image is created or rebased, and today it is only accessible via reading the image. Therefore, in order to access this metadata, platform operators must query the image. + +Querying the docker daemon or querying an image registry is suboptimal and we should make this data more accessible. It is suboptimal because it requires the platform to run an additional service to query the data it just published. If we make this data more accessible, we could potentially reduce image queries (registry calls) calls by a significant factor. + +Putting this data into `report.toml` is advantageous over other methods, especially when considering the kubernetes `terminationMessagePath` message [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core). In this pattern, the content of `report.toml` can be used as a container's termination message, making this data easily accessible after an image is exported or rebased within a kubernetes container. # What it is [what-it-is]: #what-it-is @@ -69,19 +73,27 @@ This is an additive change to a metadata file and will be backwards compatible. # Drawbacks [drawbacks]: #drawbacks -This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). It can be accessed by querying the config from docker daemon or registry. So we will be writing this data to two outputs. +This metadata is written to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json) and it can be accessed by querying a docker daemon or registry. So we will be writing this data to two outputs. # Alternatives [alternatives]: #alternatives -Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. +1. Do nothing and continue to require platforms to retrieve this information via alternative means--either querying the docker daemon or registry. + - Rebase process may remain suboptimal for some platform providers. + +2. Write all the metadata labels to `report.toml`. + - This could break platform operators that are using the `terminationMessagePath` [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core). Because some of the metadata includes unbounded arrays, we could explode this report beyond the max size of 4096 bytes. -Introduce a new output file (`lifecycle-metadata.toml?`) that contains everything in [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json). +3. Write another file that contains this metadata (and potentially more metadata). + - If we consider this approach and take it to the logical conclusion, we should consider writing a sparse image as output. A sparse image would contain all the metadata available, and it would be in a well-known format that other image tools like `crane`, `skopeo`, or `pack` can read. + - Writing to another file is not as simple as writing to `report.toml`. + - It increases the complexity of lifecycle exporter & rebaser. + - Writing to another file would not give platform operators the advantage of the `terminationMessagePath` [pattern](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#pod-v1-core), because they are likely already reading `report.toml`. # Prior Art [prior-art]: #prior-art -These values are among those that `pack inspect` returns. +These values are written to image labels. And they are among values that `pack inspect` returns. # Unresolved Questions [unresolved-questions]: #unresolved-questions From 7ccbed177ec86c33b54c939fcbef543e9662d280 Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 11:27:06 -0500 Subject: [PATCH 3/5] update to add mirrors and change casing Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 32 +++++++++++++++----------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 8d9922674..394a0c956 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -12,7 +12,13 @@ # Summary [summary]: #summary -The `report.toml` file created by the lifecycle exporter and rebaser should include `run-image.image`, `run-image.reference`, and `run-image.top-layer`. These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. +The `report.toml` file created by the lifecycle exporter and rebaser should include the following properties: +- `run-image.image` +- `run-image.mirrors` +- `run-image.reference` +- `run-image.top-layer` + +These values are not necessarily known prior to export or rebase they can be critical to a platform rebase process. # Motivation [motivation]: #motivation @@ -37,11 +43,10 @@ tags = ... digest = ... image-id = ... manifest-size = ... - -[run-image] -image = "run/name:foo" -reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" -top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.image = "run/name:foo" +run-image.reference = "index.docker.io/run/name@sha256:94f85561b0976bf1e2bef6b14de92299ebcd4c8148802cf9b217654651e4f416" +run-image.top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.mirrors = ["", ""] ``` #### Image exported to the docker daemon: @@ -51,11 +56,10 @@ tags = ... digest = ... image-id = ... manifest-size = ... - -[run-image] -image = "run/name:foo" -reference = "5b90f9c0e189" -top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.image = "run/name:foo" +run-image.reference = "5b90f9c0e189" +run-image.top-layer = "sha256:83ad2f0b091621ce19357e19d853c8be1b8f4d60d99c281fc2db75e0f56df42a" +run-image.mirrors = ["", ""] ``` # How it Works @@ -105,11 +109,13 @@ N/A The following would be appended to the [`report.toml` spec](https://github.com/buildpacks/spec/blob/main/platform.md#reporttoml-toml) (this section would be materially identical to the [`lifecycle.metadata` label](https://github.com/buildpacks/spec/blob/main/platform.md#iobuildpackslifecyclemetadata-json)): -> run-image.top-layer MUST contain the uncompressed digest of the top layer of the run-image. +> runImage.topLayer MUST contain the uncompressed digest of the top layer of the run-image. > ->run-image.reference MUST uniquely identify the run image. It MAY contain one of the following +> runImage.reference MUST uniquely identify the run image. It MAY contain one of the following > - An image ID (the digest of the uncompressed config blob) > - A digest reference to a manifest stored in an OCI registry +> +> runImage.image and runImage.mirrors MUST be resolved from run.toml from the given # History [history]: #history From fcbe0f3deb4f0d62ddf9381a3043de8119e6975a Mon Sep 17 00:00:00 2001 From: Joey Brown Date: Thu, 18 Apr 2024 12:18:12 -0500 Subject: [PATCH 4/5] add a reference to the RFC that introduces report.toml Signed-off-by: Joey Brown --- text/0000-export-run-image-metadata.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-export-run-image-metadata.md b/text/0000-export-run-image-metadata.md index 394a0c956..6bd8013f7 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0000-export-run-image-metadata.md @@ -99,6 +99,8 @@ This metadata is written to the [`lifecycle.metadata` label](https://github.com/ These values are written to image labels. And they are among values that `pack inspect` returns. +`report.toml` was introduced in [RFC 0040](/text/0040-export-report.md). That RFC includes some context for why `report.toml` exists. + # Unresolved Questions [unresolved-questions]: #unresolved-questions From fb53ceb92f323010516d3e9e932b185326092522 Mon Sep 17 00:00:00 2001 From: Jesse Brown Date: Mon, 8 Jul 2024 14:55:23 -0500 Subject: [PATCH 5/5] Update and rename 0000-export-run-image-metadata.md to 0129-export-run-image-metadata.md Updating for merge Signed-off-by: Jesse Brown Signed-off-by: Jesse Brown --- ...-image-metadata.md => 0129-export-run-image-metadata.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-export-run-image-metadata.md => 0129-export-run-image-metadata.md} (96%) diff --git a/text/0000-export-run-image-metadata.md b/text/0129-export-run-image-metadata.md similarity index 96% rename from text/0000-export-run-image-metadata.md rename to text/0129-export-run-image-metadata.md index 6bd8013f7..124b24088 100644 --- a/text/0000-export-run-image-metadata.md +++ b/text/0129-export-run-image-metadata.md @@ -3,10 +3,10 @@ - Name: Export Run Image Metadata - Start Date: 2024-04-11 - Author(s): joeybrown-sf -- Status: Draft -- RFC Pull Request: (leave blank) +- Status: Approved +- RFC Pull Request: https://github.com/buildpacks/rfcs/pull/313 - CNB Pull Request: (leave blank) -- CNB Issue: (leave blank) +- CNB Issue: https://github.com/buildpacks/lifecycle/issues/1372 - Supersedes: N/A # Summary