diff --git a/docs/containers-storage-composefs.md b/docs/containers-storage-composefs.md index 55e297569f..fdfb8ed14c 100644 --- a/docs/containers-storage-composefs.md +++ b/docs/containers-storage-composefs.md @@ -30,6 +30,33 @@ convert_images = "true" This value must be a "string bool", it cannot be a native TOML boolean. +## IMPLEMENTATION + +As is implied by the setting `use_composefs = "true"`, currently composefs +is implemented as an "option" for the `overlay` driver. Some file formats +remain unchanged and are inherited from the overlay driver, even when +composefs is in use. The primary differences are enumerated below. + +The `diff/` directory for each layer is no longer a plain unpacking of the tarball, +but becomes an "object hash directory", where each filename is the sha256 of its contents. This `diff/` +directory is the backing store for a `composefs-data/composefs.blob` created for +each layer which is the composefs "superblock" containing all the non-regular-file content (i.e. metadata) from the tarball. + +As with `zstd:chunked`, existing layers are scanned for matching objects, and reused +(via hardlink or reflink as configured) if objects with a matching "full sha256" are +found. + +There is currently no support for enforced integrity with composefs; +an attempt is made to enable fsverity for the backing files and the composefs file, +but it is not an error if unsupported. There is as of yet no defined mechanism to +verify the fsverity digest of the composefs block before mounting; some work on that is +ongoing. + +In order to mount a layer (or a full image, with all of its dependencies), any +layer that has a composefs blob is mounted and included in the "final" overlayfs +stack. This is optional - any layers that are not in "composefs format" but +in the "default overlay" (unpacked) format will be reused as is. + ## BUGS - https://github.com/containers/storage/issues?q=is%3Aissue+is%3Aopen+label%3Aarea%2Fcomposefs diff --git a/docs/containers-storage-zstd-chunked.md b/docs/containers-storage-zstd-chunked.md index cbca7e2b9e..cb18863a37 100644 --- a/docs/containers-storage-zstd-chunked.md +++ b/docs/containers-storage-zstd-chunked.md @@ -27,13 +27,26 @@ enable_partial_images = "true" | "false" Note that the value of this field must be a "string bool", it cannot be a native TOML boolean. -## INTERNALS +## IMPLEMENTATION -At the current time the format is not officially standardized or documented beyond -the comments and code in the reference implementation. At the current time -the file with the most information is [pkg/chunked/internal/compression.go](https://github.com/containers/storage/blob/39d469c34c96db67062e25954bc9d18f2bf6dae3/pkg/chunked/internal/compression.go). +Each layer has an associated "big data" key called `chunked-manifest-cache` that +is a custom binary format suitable for mmap() that contains index metadata +for each layer with the full sha256 digest of each file plus its "chunks" (as +computed by `zstd:chunked`). + +When any image is pulled all existing other layers are scanned using `chunked-manifest-cache` to see if they contain a file with a matching digest. If one is found, the other file is hardlinked if `use_hardlinks = "true`", +otherwise it is reflinked (if supported by the filesystem, or a full physical copy +is made). There is a best-effort attempt to enable fsverity on the file if configured +(see ). + +For more information, at the current time the file with the most information is [pkg/chunked/internal/compression.go](https://github.com/containers/storage/blob/39d469c34c96db67062e25954bc9d18f2bf6dae3/pkg/chunked/internal/compression.go). The above is a permanent link for stability, but be sure to check to see if there are newer changes too. +## STANDARDIZATION + +At the current time the format is not officially standardized or documented beyond +the comments and code in the reference implementation. + ## BUGS - https://github.com/containers/storage/issues?q=is%3Aissue+label%3Aarea%2Fzstd%3Achunked+is%3Aopen