CondaPkg environments are re-resolved each time a Pluto notebook is opened

[stillyslalom]

1. Video:
2. Gist: https://gist.github.com/stillyslalom/fcc3e5342244f097d42fa81f72f0b817
3. Because Pluto creates a new temporary environment each time a notebook is executed, any Conda environment required by the notebook has to be re-resolved each time. Even with all Conda packages cached, this is a fairly slow process - it takes about a minute for my package PyThermo.jl to re-load in an existing Pluto notebook, versus about 10 seconds in the REPL. Other CondaPkg users have encountered the same issue (using CondaPkg with Pluto leads to unnecesarry resolves JuliaPy/CondaPkg.jl#120), but it appears the problem needs to be resolved from the Pluto side.

[fonsp]

Hey! Thanks for bringing this up, I would like this to work better!

Can you take a look at previous discussion in JuliaPy/PythonCall.jl#102 ?

In particular, I wonder if this would be fixed by storing the contents of CondaPkg.toml in the notebook, just like Project.toml and Manifest.toml, like I suggested in my issue.

Can you check this? To reproduce: make an empty folder called myenv and activate it and set up PyThermo.

1. Copy myenv and call it myenv2, and remove the CondaPkg.toml file. Measure how long this env takes to start. This should match what Pluto does.
2. Copy myenv and call it myenv3, and keep the CondaPkg.toml file. Measure how long this env takes to start.

[fonsp]

@schlichtanders what do you think about this one?

[disberd]

@fonsp I believe that CondaPkg will also need the actual downloaded python dependencies for this to work without re-instantiating

From my understsanding by default CondaPkg stores the conda deps in a subfolder of the active project, just storing the CondaPkg.toml will not help with avoiding re-downloading the various dependencies which is why I believe @cjdoris suggested to manage a pool of projects directly from within Puto to make the process more seamless for the user

[cjdoris]

Saving CondaPkg.toml in the notebook is definitely a good idea to make Pluto notebooks automatically have their Conda dependencies available. This is a straight win.

But as observed above, there will still be a significant lag every time you start the notebook because a new Conda environment needs to be installed every time - instantiating a Conda environment is a lot slower than instantiating a Julia one. Counter to @disberd's comment, Conda has a central cache of packages so it won't usually need to re-download them, but still, copying all the packages into the actual env takes time.

There are a couple of ways to alleviate this:

• Change CondaPkg to treat Conda environments as immutable, and store them at a location which is some hash of the dependencies. This way, if two environment have the same dependencies, we can just re-use the same environment. I've not thought about this much but it would be a significant undertaking and would most likely be a breaking change.
• Change Pluto to re-use the same Julia environment for the same notebook. Specifically, store the Julia environment at a location which is determined only from the path of the notebook, e.g. you could put it alongside the notebook, or you could put it at ~/.julia/environments/pluto/<hash_of_the_path_of_the_notebook>. This seems like an easier (and non-breaking) change.

[fonsp]

Thanks for the advice @cjdoris ! With your second point, do you mean that if the CondaPkg.toml is in the same location for two Julia runs, then the conda environment is reused?

I believe that Stephan has experience with storing the CondaPkg.toml in the Pluto notebook! He is currently on holiday, so let's pick this up when he gets back.