Cache Starlark packages in the cluster, and clean old ones automatically, so Kurtosis is runnable offline

[mieubrisse]

Background & motivation

If I kurtosis run github.com/kurtosis-tech/eth2-package while I'm offline, the APIC will try to clone github.com/kurtosis-tech/eth2-package, and all its Starlark dependencies (this is the only behaviour we support right now). This will fail because the user is offline and the APIC can't reach Github.

Desired behaviour

Whenever Kurtosis clones a package once, it caches that package and its dependencies locally inside the Kurtosis cluster so that if the user goes offline in the future, they can still potentially run the Starlark package.

1. The Kurtosis cluster has a cross-enclave Starlark package LRU cache, that is added to whenever the APIC wants to pull a SL package. This is likely blocked by proper dependencies in Starlark packages, as well as hashes, so we have a unique key to cache off of.
2. The cache is automatically emptied of old entries so it doesn't infinitely fill the user's disk
3. The APIC resolves imports against the cache first

Thought: thinking about the logistics of sharing a volume across all the APICs, and having multiple APICs potentially modifying the thing at the same time, etc etc sounds like a gigantic nightmare. It really makes me think, yet again.... "Should we just merge the APICs and the Engine?"

How important is this to you?

Painful; the lack of this feature makes using Kurtosis frictionful.

[mieubrisse]

This came from a side-by-side debugging session with Barnabas, who was trying to run the eth2-package offline

[leeederek]

Hey @mieubrisse, is this painful because you currently cannot do local package imports? Would love to hear about the workflow a bit more. Thank you!

[mieubrisse]

@leeederek the problem is, if I kurtosis run github.com/kurtosis-tech/eth2-package while I'm offline, the APIC will try to clone github.com/kurtosis-tech/eth2-package, and all its Starlark dependencies (this is the only beahviour we support right now). This will fail because the user is offline and the APIC can't reach Github.

This ticket proposes that whenever Kurtosis clones a package once, it caches that package and its dependencies locally inside the Kurtosis cluster so that if the user goes offline in the future, they can still potentially run the Starlark package.

This is a ubiquitous pattern with languages - Go caches modules in your ~/go directory, NPM does this in the node_modules directory, Rust does it somewhere, etc.

[leeederek]

Maybe Q4 2023 / Q1 2024