[feat] pull-through caching/proxying for images

[DerekTBrown]

Background

• Motivation: [feature request] Pulling images from remote registries #355

• In my "real" Kubernetes clusters, I use ECR as an image registry. Nodes are automatically authorized/configured to pull from ECR through a combination of IAM and containerd settings.

• I want my Tilt Kubernetes clusters to mirror "real" clusters as much as possible. I want to avoid making Tilt-specific modifications to Kubernetes manifests to make them work (see discussion in [feature request] Pulling images from remote registries #355).

• There currently isn't a way to achieve both of these goals simulaneously:

  • I could use some combination of docker pull && docker push or kind load, though this makes the Tilt configuration clunky.
  • I could provide imagePullSecrets, but this causes the Tilt configuration to deviate from the "real" configuration (and would require modifying hundreds of helm charts).

• Note: I think there are other, similar usecases for providing cluster authentication credentials. For example, a user might want to avoid Docker registry ratelimits by providing a user token.

What does this PR do?

• This PR adds the ability for a user to configure pull-through proxy/cache registries, which are attached to the kind cluster (possible in other cluster provisioners, but I have chosen this as a starting point).
• Authentication can be passed to the proxy via the username and password properties, or via a templated env var to password.

Test Plan

• I created a cluster using a Docker Hub endpoint, passing in my username and password (token).
• I was successfully able to start docker containers with an image contained in a private repo.

[nicks]

hey @DerekTBrown ! thanks for diving into the tangled mess that is containerd configs.

so...this is complicated. my understanding is that containerd-team is in the middle of removing this functionality. see the "mirror config deprecation PSA" here:

https://github.com/kubernetes-sigs/kind/releases/tag/v0.20.0

and the big DEPRECATED node at the top of this doc --

https://github.com/containerd/containerd/blob/main/docs/cri/registry.md#configure-image-registry

and that even today, the code you have here will break in a bunch of cases because ctlptl sets config_path. but i haven't chatted with the containerd folks for a while, lemme try to sus out what the right way to do this is.

[nicks]

lol, there's also a big note on why they're removing it 🙈

NOTE: registry.configs.*.auth is DEPRECATED and will NOT have an equivalent way to store unencrypted secrets in the host configuration files.

[nicks]

one crazy idea -- Kind does let you pass extra args to the kubelet

https://kind.sigs.k8s.io/docs/user/configuration/#kubeadm-config-patches

so you might be able to mount a custom Credential provider that does what you want

https://kubernetes.io/docs/tasks/administer-cluster/kubelet-credential-provider/#configure-a-kubelet-credential-provider

i think this might be closer to what the "real" EKS does to authenticate against ECR these days?

[DerekTBrown]

NOTE: registry.configs.*.auth is DEPRECATED and will NOT have an equivalent way to store unencrypted secrets in the host configuration files.
lemme try to sus out what the right way to do this is.

I am not sure how I missed these deprecation warnings- I have seen this in practice a bunch.

I interpret the deprecation notice to mean that there will be some future interface, but that interface will require storing credentials in an encrypted format.

Assuming that is the case, I think this PR still makes sense: we would just need to migrate to that interface when it becomes available.

[DerekTBrown]

one crazy idea -- Kind does let you pass extra args to the kubelet

Assuming we do need to implement an alternative authentication strategy, I have a slightly different approach in mind. We could extend the Registry API spec slightly to allow configuring the Registry as a pull-through cache with authentication:

https://distribution.github.io/distribution/recipes/mirror/#run-a-registry-as-a-pull-through-cache

We would then follow the exact same pattern currently used to redirect localhost:* images to the Cluster registry, but instead to redirect to the Pull-through registry.

I think this has a few advantages:

• Because the authentication implementation lives in the registry layer, it is shared across all of the cluster providers; no need to implement authentication for each provider type (though some work would be required of course to do redirects).
• I think this could be extra useful when coupled with Persistent Storage for Registry #346, since users could easily cache upstream images on the machine between tilt ci runs (for example).

[nicks]

ok, i guess i'm fine with trying this approach while we wait to see what happens with containerd

[nicks]

(though i'm also fine with trying the proxy approach you linked to)

[DerekTBrown]

@nicks should be good to go!