Make it easier for users to track the environment build being used

[kcpevey]

Context

I have a desire as an end user, to specify the build rather than the environment that a notebook or script uses.

One possible workflow for this (though there are others):

1. I have a notebook that uses a conda-store environment. As an end user, I have no visibility into which build is loaded in my notebook.
2. Over time, I rebuild the environment several times, the latest build becomes the "active" build.
3. Notebooks still points to previous build from step 1, but I want to use the updated env.
4. Load a different env in the notebook
5. Load the original environment (effectively switching to the "active" build)
6. Realize that my notebook no longer works and decide to rollback.
7. Attempt to recall the date of the build that worked for the notebook and make it the "active" build.
8. Repeat steps 4-7 until I figure out the build that worked.

Rolling back to different builds of environments is "easy" in conda-store but the practical usage of it in Jupyter Notebooks (or python scripts) is less clear. It is handy to just say "this runs in this environment" and let conda-store handle which environment is active. But in reality, its often really useful to say "run this in this environment build" which is much more explicit. And if not that, then just make it easier for me to know which build I'm currently using.

I believe there are symlinks which are obfuscating the environment name from the build hash. I'm wondering if its possible for me as an end user to point to the build hash itself?

Or maybe we could add a symlink that adds part of the build hash to the env name? The same hash could be shown in the conda-store UI for easy visibility and cross reference. That would cause a LOT of noise so I'm sure that would take some effort to control.

Value and/or benefit

Improved practical usage of conda-store

Anything else?

No response

[trallard]

I am unsure if I follow the core problem described here since the environment and the build/version should always be traceable. That is the foundation of the active build mechanism.

@peytondmurray am I missing something here?

[kcpevey]

This is more from the JupyterLab side of things. If I have a conda-store environment loaded in a jupyter notebook, I can't tell which version of the environment it is. It may or may not be the current "active" version.

I was thinking that if the environment "display name" had some additional build info, it could help.

I'm not sure if this is something that can resolved on the conda-store side or something that would require an extension, but this seemed the most appropriate place to open the issue. I can move it as needed.

[trallard]

This seems outside of the scope of conda-store core/server and more of a problem for either the JLab extension or JLab launchpad

[kcpevey]

@krassowski do you have any thoughts on how/if this could be implemented (and where)?

[krassowski]

Hits close to home. I opened other issues about enhancing the interactive experience focusing on some other aspects (#49, conda-incubator/conda-store-ui#407).

Thoughts on UX

We could show the build name in the toolbar next to the kernel name, or at least on hover. Users should be able to open an older notebook from git and it should get the right build (but also a signal that a newer build exists).

In the future, using history slider could change environment versions, see the video of the slider here: jupyterlab/jupyter-collaboration#338

Implementation

For the JupyterLab UI to display the build information we would need to have this information available.

https://github.com/anaconda/nb_conda_kernels exposes a lot of information that it can infer from the filesystem about the installed kernels based on conda environments it sees.

The question is how conda-store can pass the information about the build version to it.

In environment name (file system path)

I do not have a good understanding of conda-store, but I think currently the only way of passing information about namespace and environment is by having them concatenated in the name of the environment. Adding a version slug/time/ID sounds reasonable but would force everyone to adopt the new format (e.g. if users had explicit references to old environment conda store paths).

By modifying the kernelspec directly

conda-store already special-cases ipykernel package which adds a kernel.json. Maybe it is not an unreasonable ask to add a metadata stanza with conda environment build version in these kernel.json files.

Using environment variables

It is hard for the frontend to extract information from env variables, but we could try to do this. Would it be hard for conda-store to add an environment variable population to activation scripts?

Using an entry point for parametrized kernels?

Maybe there could be something useful in jupyter/enhancement-proposals#87 approach.
It feels that if we wanted to have multiple versions of an environment exposed, it should be a parametrized kernel. Can conda-store swap the environment path in-flight based on environment variable?

[peytondmurray]

We could show the build name in the toolbar next to the kernel name, or at least on hover. Users should be able to open an older notebook from git and it should get the right build (but also a signal that a newer build exists).

I can also see how if you execute cells in a notebook with a certain build, they may run. If you then change the environment, they may not run, but other cells might. So in some sense it may make sense to think about cell-level information rather than notebook-level information about the current namespace, environment, and build.

Maybe it makes sense to store some metadata about which namespace, environment, and build was used to execute a cell, and have a visual indicator of the last execution in the cell output?

Then again, this is something that JupyterLab currently considers the user's responsibility. If you execute a notebook in a certain environment, then modify that environment, there's no guarantee about still having a functioning notebook. The same is true here, but maybe one of the options discussed above could alleviate this pain.

I do not have a good understanding of conda-store, but I think currently the only way to passing information about namespace and environment is by having them concatenated in the name of the environment. Adding a version slug/time/ID sounds reasonable but would force everyone to adopt the new format (e.g. if users had explicit references to old environment conda store paths).

This is something we should consider as we move forward with bringing parity to the python API, the CLI, and the REST API (currently you can query this for information about namespaces and environments). Continuing to add more to the current namespace/environment scheme to include version/time/ID etc might be problematic, especially when you consider the impact of potential regex/wildcards, etc.

[krassowski]

Maybe it makes sense to store some metadata about which namespace, environment, and build was used to execute a cell, and have a visual indicator of the last execution in the cell output

I like this as a future enhancement idea on jupyterlab-conda-store side. There is a metadata field for cells which could be used to store this info.

Then again, this is something that JupyterLab currently considers the user's responsibility. If you execute a notebook in a certain environment, then modify that environment, there's no guarantee about still having a functioning notebook.

True, there are no guarantees in plain Jupyter, but JupyterLab with conda nb kernels solves this problem by remembering the environment (technically kernelspec but in that case these are synonymous) in notebook metadata. Since an access to old builds are one of the "selling points" of conda-store, it would seem reasonable to consider this a natural feature which belongs here

[krassowski]

I also do not understand why after building an environment an older version remains "Active"

As a user I feel blocked with doing work on an instance where conda-store is deployed because of it. I do not know if I should come back after coffee or next day to see it updated. I know that I am using an older version of conda-store but I understood that the newer one could not be deployed due to a release error.

[kcpevey]

I also do not understand why after building an environment an older version remains "Active"

Immediately after a build is complete, it becomes the "active" build. However, the UI doesn't update to reflect this, it requires a screen refresh. I was hoping removing tabs would improve this behavior, but I haven't tested it yet. Did you see this on a nebari deployment (which doesn't have tabs removed yet) or were you running locally?

[krassowski]

This is from a nebari deployment.

[kcpevey]

This is from a nebari deployment.

So it still has tabs. The behavior in conda-store main may have fixed this.

[trallard]

So this is, as I originally thought, something that would be best suited in the JLab extension, so I will move the issue there.

In environment name (file system path)
I do not have a good understanding of conda-store, but I think currently the only way of passing information about namespace and environment is by having them concatenated in the name of the environment. Adding a version slug/time/ID sounds reasonable but would force everyone to adopt the new format (e.g. if users had explicit references to old environment conda store paths).

There are hard requirements based on the OS for path/prefix length. We recently did a good amount of work to align with OS-specific path requirements and include meaningful hashes in the env name/prefix, so this should already be available. The issue seems to be a representation one through the JLab extension.

[krassowski]

This is what jupyterlab can currently work with:

{
    "default": "python3",
    "kernelspecs": {
        "conda-env-nebari-git-nebari-git-dashboard-py": {
            "name": "conda-env-nebari-git-nebari-git-dashboard-py",
            "spec": {
                "argv": [
                    "python",
                    "-m",
                    "nb_conda_kernels.runner",
                    "/opt/conda",
                    "/home/conda/nebari-git/envs/nebari-git-dashboard",
                    "/home/conda/nebari-git/b41bf223-1727131653-101-dashboard/bin/python",
                    "-m",
                    "ipykernel_launcher",
                    "-f",
                    "{connection_file}"
                ],
                "env": {},
                "display_name": "nebari-git-nebari-git-dashboard",
                "language": "python",
                "interrupt_mode": "signal",
                "metadata": {
                    "debugger": true,
                    "conda_env_name": "nebari-git-nebari-git-dashboard",
                    "conda_env_path": "/home/conda/nebari-git/envs/nebari-git-dashboard",
                    "conda_language": "Python",
                    "conda_raw_kernel_name": "python3",
                    "conda_is_base_environment": false,
                    "conda_is_currently_running": false
                }
            },
            "resources": {
                "logo-64x64": "/user/[email protected]/kernelspecs/conda-env-nebari-git-nebari-git-dashboard-py/logo-64x64.png",
                "logo-32x32": "/user/[email protected]/kernelspecs/conda-env-nebari-git-nebari-git-dashboard-py/logo-32x32.png",
                "logo-svg": "/user/[email protected]/kernelspecs/conda-env-nebari-git-nebari-git-dashboard-py/logo-svg.svg"
            }
        }
    }
}

And this is what user sees in the conda-store-ui:

To me it looks like this issue cannot be resolved on jupyterlab-conda-store side just yet, because the kernelspecs do not include the required information.

I see two potential ways forward:

• a) reviving https://github.com/Quansight/nb_conda_store_kernels and providing more conda-store specific metadata in kernelspecs, as discussed in [BUG] - Newly built environments don't immediately show up as available to JupyterLab conda-store-ui#428 (comment)
• b) making this jupyterlab extension into a full-blown client of the conda-store API with bespoke logic for synchronizing environment state and metadata