Add information about embedded registry

docs/install/airgap.md

@@ -2,7 +2,7 @@
 title: Air-Gap Install
 ---
 
-RKE2 can be installed in an air-gapped environment with two different methods. You can either deploy via the `rke2-airgap-images` tarball release artifact, or by using a private registry.
+RKE2 can be installed in an air-gapped environment with three different methods. You can either deploy via the [rke2-airgap-images tarball release artifact](#tarball-method), by using a [private registry](#private-registry-method) or by using the [embedded registry mirror](#embedded-registry-mirror).
 
 ## Prerequisites
 
@@ -49,10 +49,23 @@ If your nodes do not have an interface with a default route, a default route mus
 4. [Install RKE2](#install-rke2)
 
 ## Private Registry Method
-Private registry support honors all settings from the [containerd registry configuration](containerd_registry_configuration.md). This includes endpoint override and transport protocol (HTTP/HTTPS), authentication, certificate verification, etc.
+Private registry support honors all settings from the [containerd registry configuration](private_registry.md). This includes endpoint override and transport protocol (HTTP/HTTPS), authentication, certificate verification, etc.
 
 1. Add all the required system images to your private registry. A list of images can be obtained from the `.txt` file corresponding to each tarball referenced above, or you may `docker load` the airgap image tarballs, then tag and push the loaded images.
-2. [Install RKE2](#install-rke2) using the `system-default-registry` parameter, or use the [containerd registry configuration](containerd_registry_configuration.md) to use your registry as a mirror for docker.io.
+2. [Install RKE2](#install-rke2) using the `system-default-registry` parameter, or use the [containerd registry configuration](private_registry.md) to use your registry as a mirror for docker.io.
+
+## Embedded Registry Mirror
+
+:::info Version Gate
+The Embedded Registry Mirror is available as an experimental feature as of January 2024 releases: v1.26.13+rke2r1, v1.27.10+rke2r1, v1.28.6+rke2r1, v1.29.1+rke2r1
+:::
+
+K3s includes an embedded distributed OCI-compliant registry mirror.
+When enabled and properly configured, images available in the containerd image store on any node
+can be pulled by other cluster members without access to an external image registry.
+
+The mirrored images may be sourced from an upstream registry, registry mirror, or airgap image tarball.
+For more information on enabling the embedded distributed registry mirror, see the [Embedded Registry Mirror](./registry_mirror.md) documentation.
 
 ## Install RKE2
 The following options to install RKE2 should only be performed after completing one of either the [Tarball Method](#tarball-method) or [Private Registry Method](#private-registry-method).
@@ -87,4 +100,4 @@ curl -sfL https://get.rke2.io --output install.sh
 ```bash
 INSTALL_RKE2_ARTIFACT_PATH=/root/rke2-artifacts sh install.sh
 ```
-3. Enable and run the service as outlined [here.](quickstart.md#2-enable-the-rke2-server-service)
+3. Enable and run the service as outlined [here.](quickstart.md#2-enable-the-rke2-server-service)

docs/install/containerd_registry_configuration.md → docs/install/private_registry.md

@@ -1,5 +1,5 @@
 ---
-title: "Containerd Registry Configuration"
+title: "Private Registry Configuration"
 ---
 
 Containerd can be configured to connect to private registries and use them to pull private images on each node.
@@ -10,6 +10,33 @@ Note that server nodes are schedulable by default. If you have not tainted the s
 
 Configuration in containerd can be used to connect to a private registry with a TLS connection and with registries that enable authentication as well. The following section will explain the `registries.yaml` file and give different examples of using private registry configuration in RKE2.
 
+## Default Endpoint Fallback
+
+Containerd has an implicit "default endpoint" for all registries.
+The default endpoint is always tried as a last resort, even if there are other endpoints listed for that registry in `registries.yaml`.
+Rewrites are not applied to pulls against the default endpoint.
+For example, when pulling `registry.example.com:5000/rancher/mirrored-pause:3.6`, containerd will use a default endpoint of `https://registry.example.com:5000/v2`.
+* The default endpoint for `docker.io` is `https://index.docker.io/v2`.  
+* The default endpoint for all other registries is `https://<REGISTRY>/v2`, where `<REGISTRY>` is the registry hostname and optional port.  
+
+In order to be recognized as a registry, the first component of the image name must contain at least one period or colon.
+For historical reasons, images without a registry specified in their name are implicitly identified as being from `docker.io`.
+
+:::info Version Gate
+The `disable-default-registry-endpoint` option is available as an experimental feature as of February 2024 releases: v1.26.13+rke2r1, v1.27.10+rke2r1, v1.28.6+rke2r1, v1.29.1+rke2r1
+:::
+
+Nodes may be configured with the `disable-default-registry-endpoint: true` option.
+When this is set, containerd will not fall back to the default registry endpoint, and will only pull from configured mirror endpoints,
+along with the distributed registry if it is enabled.
+
+This may be desired if your cluster is in a true air-gapped environment where the upstream registry is not available,
+or if you wish to have only some nodes pull from the upstream registry.
+
+Disabling the default registry endpoint applies only to registries configured via `registries.yaml`.
+If the registry is not explicitly configured via mirror entry in `registries.yaml`, the default fallback behavior will still be used.
+
+
 ## Registries Configuration File
 
 The file consists of two main sections:
@@ -39,7 +66,7 @@ Each mirror must have a name and set of endpoints. When pulling an image from a
 
 #### Rewrites
 
-Each mirror can have a set of rewrites. Rewrites can change the tag of an image based on a regular expression. This is useful if the organization/project structure in the mirror registry is different to the upstream one.
+Each mirror can have a set of rewrites, which use regular expressions to match and transform the name of an image when it is pulled from that mirror. This is useful if the organization/project structure in the mirror registry is different to the upstream one.
 
 For example, the following configuration would transparently pull the image `rancher/rke2-runtime:v1.23.5-rke2r1` from `registry.example.com:5000/mirrorproject/rancher-images/rke2-runtime:v1.23.5-rke2r1`:
 
@@ -52,6 +79,15 @@ mirrors:
       "^rancher/(.*)": "mirrorproject/rancher-images/$1"
 ```
 
+
+Note that when using mirrors and rewrites, images will still be stored under the original name.
+For example, `crictl image ls` will show `docker.io/rancher/rke2-runtime:v1.23.5-rke2r1` as available on the node, even if the image was pulled from a mirror with a different name.
+
+:::info Version Gate
+Rewrites are no longer applied to the Default Endpoint as of the February 2024 releases: v1.26.13+rke2r1, v1.27.10+rke2r1, v1.28.6+rke2r1, v1.29.1+rke2r1
+Prior to these releases, rewrites were also applied to the default endpoint, which would prevent RKE2 from pulling from the upstream registry if the image could not be pulled from a mirror endpoint, and the image was not available under the modified name in the upstream.
+:::
+
 ### Configs
 
 The configs section defines the TLS and credential configuration for each mirror. For each mirror you can define `auth` and/or `tls`. The TLS part consists of:

docs/install/registry_mirror.md

@@ -0,0 +1,150 @@
+---
+title: "Embedded Registry Mirror"
+---
+
+:::info Version Gate
+The Embedded Registry Mirror is available as an experimental feature as of January 2024 releases: v1.26.13+rke2r1, v1.27.10+rke2r1, v1.28.6+rke2r1, v1.29.1+rke2r1
+:::
+
+RKE2 embeds [Spegel](https://github.com/spegel-org/spegel), a stateless distributed OCI registry mirror that allows peer-to-peer sharing of container images between nodes in a Kubernetes cluster. The distributed registry mirror is disabled by default. For RKE2 to leverage it, you must enable both the [Distributed OCI Registry Mirror](#enabling-the-distributed-oci-registry-mirror) and the [Registry mirroring](#enabling-registry-mirroring) as explained in the following subsections.
+
+## Enabling The Distributed OCI Registry Mirror
+
+In order to enable the embedded registry mirror, server nodes must be started with the `--embedded-registry` flag, or with `embedded-registry: true` in the configuration file.
+This option enables the embedded mirror for use on all nodes in the cluster.
+
+When enabled at a cluster level, all nodes will host a local OCI registry on port 6443,
+and publish a list of available images via a peer to peer network on port 5001.
+Any image available in the containerd image store on any node, can be pulled by other cluster members without access to an external registry.
+Images imported via [air-gap image tar files](./airgap.md#tarball-method) are pinned in containerd to
+ensure that they remain available and are not pruned by Kubelet garbage collection.
+
+### Requirements
+
+When the embedded registry mirror is enabled, all nodes must be able to reach each other via their internal IP addresses, on TCP ports 5001 and 6443.
+If nodes cannot reach each other, it may take longer for images to be pulled, as the distributed registry will be tried first by containerd, before it falls back to other endpoints.
+
+## Enabling Registry Mirroring
+
+Enabling mirroring for a registry allows a node to both pull images from that registry from other nodes, and share the registry's images with other nodes.
+If a registry is enabled for mirroring on some nodes, but not on others, only the nodes with the registry enabled will exchange images from that registry.
+
+In order to enable mirroring of images from an upstream container registry, nodes must have an entry in the `mirrors` section of `registries.yaml` for that registry.
+The registry does not need to have any endpoints listed, it just needs to be present.
+For example, to enable distributed mirroring of images from `docker.io` and `registry.k8s.io`, configure `registries.yaml` with the following content on all cluster nodes:
+
+```yaml
+mirrors:
+  docker.io:
+  registry.k8s.io:
+```
+
+Endpoints for registry mirrors may also be added as usual.
+In the following configuration, images pull attempts will first try the embedded mirror, then `mirror.example.com`, then finally `docker.io`:
+```yaml
+mirrors:
+  docker.io:
+    endpoint:
+      - https://mirror.example.com
+```
+
+If you are using a private registry directly, instead of as a mirror for an upstream registry, you may enable distributed mirroring in the same way public
+registries are enabled - by listing it in the mirrors section: 
+```yaml
+mirrors:
+  mirror.example.com:
+```
+
+:::info Version Gate
+Wildcard support is available as of the March 2024 releases: v1.26.15+rke2r1, v1.27.12+rke2r1, v1.28.8+rke2r1, v1.29.3+rke2r1
+:::
+
+The `"*"` wildcard mirror entry can be used to enable distributed mirroring of all registries. Note that the asterisk MUST be quoted:
+```yaml
+mirrors:
+  "*":
+```
+
+If no registries are enabled for mirroring on a node, that node does not participate in the distributed registry in any capacity.
+
+For more information on the structure of the `registries.yaml` file, see [Private Registry Configuration](./private_registry.md).
+
+### Default Endpoint Fallback
+
+By default, containerd will fall back to the default endpoint when pulling from registries with mirror endpoints configured. If you want to disable this,
+and only pull images from the configured mirrors and/or the embedded mirror, see the [Default Endpoint Fallback](./private_registry.md#default-endpoint-fallback)
+section of the Private Registry Configuration documentation.
+
+Note that if you are using the `--disable-default-endpoint` option and want to allow pulling directly from a particular registry, while disallowing the rest,
+you can explicitly provide an endpoint in order to allow the image pull to fall back to the registry itself:
+```yaml
+mirrors:
+  docker.io:           # no default endpoint, pulls will fail if not available on a node
+  registry.k8s.io:     # no default endpoint, pulls will fail if not available on a node
+  mirror.example.com:  # explicit default endpoint, can pull from upstream if not available on a node
+    endpoint:
+      - https://mirror.example.com
+```
+
+### Latest Tag
+
+When no tag is specified for a container image, the implicit default tag is `latest`. This tag is frequently
+updated to point at the most recent version of the image. Because this tag will point at a different revisions
+of an image depending on when it is pulled, the distributed registry **will not** pull the `latest` tag from
+other nodes. This forces containerd go out to an upstream registry or registry mirror to ensure a consistent
+view of what the `latest` tag refers to.
+
+This aligns with the [special `imagePullPolicy` defaulting](https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting)
+observed by Kubernetes when using the `latest` tag for a container image.
+
+## Security
+
+### Authentication
+
+Access to the embedded mirror's registry API requires a valid client certificate, signed by the cluster's client certificate authority.
+
+Access to the distributed hash table's peer-to-peer network requires a preshared key that is controlled by server nodes.
+Nodes authenticate each other using both the preshared key, and a certificate signed by the cluster certificate authority.
+
+### Potential Concerns
+
+:::warning
+The distributed registry is built on peer-to-peer principles, and assumes an equal level of privilege and trust between all cluster members.
+If this does not match your cluster's security posture, you should not enable the embedded distributed registry.
+:::
+
+The embedded registry may make available images that a node may not otherwise have access to.
+For example, if some of your images are pulled from a registry, project, or repository that requires authentication via Kubernetes Image Pull Secrets, or credentials in `registries.yaml`,
+the distributed registry will allow other nodes to share those images without providing any credentials to the upstream registry.
+
+Users with access to push images into the containerd image store on one node may be able to use this to 'poison' the image for other cluster nodes,
+as other nodes will trust the tag advertised by the node, and use it without checking with the upstream registry.
+If image integrity is important, you should use image digests instead of tags, as the digest cannot be poisoned in this manner.
+
+## Sharing Air-gap or Manually Loaded Images
+
+Images sharing is controlled based on the source registry.
+Images loaded directly into containerd via air-gap tarballs, or loaded directly into containerd's image store using the `ctr` command line tool,
+will be shared between nodes if they are tagged as being from a registry that is enabled for mirroring.
+
+Note that the upstream registry that the images appear to come from does not actually have to exist or be reachable.
+For example, you could tag images as being from a fictitious upstream registry, and import those images into containerd's image store.
+You would then be able to pull those images from all cluster members, as long as that registry is listed in `registries.yaml`
+
+## Pushing Images
+
+The embedded registry is read-only, and cannot be pushed to directly using `docker push` or other common tools that interact with OCI registries.
+
+Images can be manually made available via the embedded registry by running `ctr -n k8s.io image pull` to pull an image,
+or by loading image archives created by `docker save` via the `ctr -n k8s.io image import` command.
+Note that the `k8s.io` namespace must be specified when managing images via `ctr` in order for them to be visible to the kubelet.
+
+## Metrics
+
+Spegel provides a few metrics that can be helpful to track its activity and can be queried in the supervisor. However, the supervisor metrics are disabled by default. To enable them, server nodes must be started with the `--supervisor-metrics` flag, or with `supervisor-metrics: true` in the configuration file.
+
+Once enabled, query:
+
+```
+kubectl get --server https://$SERVER_NAME:9345 --raw /metrics | grep spegel
+```

docs/install/windows_airgap.md

@@ -94,10 +94,10 @@ This will require a reboot for the `Containers` feature to properly function.
 4. [Install RKE2](#install-windows-rke2)
 
 ## Private Registry Method
-Private registry support honors all settings from the [containerd registry configuration](./containerd_registry_configuration.md). This includes endpoint override and transport protocol (HTTP/HTTPS), authentication, certificate verification, etc.
+Private registry support honors all settings from the [containerd registry configuration](./private_registry.md). This includes endpoint override and transport protocol (HTTP/HTTPS), authentication, certificate verification, etc.
 
 1. Add all the required system images to your private registry. A list of images can be obtained from the `.txt` file corresponding to each tarball referenced above, or you may `docker load` the airgap image tarballs, then tag and push the loaded images.
-2. [Install RKE2](#install-windows-rke2) using the `system-default-registry` parameter, or use the [containerd registry configuration](./containerd_registry_configuration.md) to use your registry as a mirror for docker.io.
+2. [Install RKE2](#install-windows-rke2) using the `system-default-registry` parameter, or use the [containerd registry configuration](./private_registry.md) to use your registry as a mirror for docker.io.
 
 ## Install Windows RKE2
 

sidebars.js

@@ -9,7 +9,8 @@ module.exports = {
         'install/requirements',
         'install/configuration',
         'install/methods',
-        'install/containerd_registry_configuration',
+        'install/private_registry',
+	'install/registry_mirror',
         'install/airgap',
         'install/ha',
         'install/windows_airgap',