conda with libmamba solver does not work in offline mode

[c4hm]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

We have an environment without connection to the internet with a shared Conda Python installation. Regular Python users do not have write access to the installation. They use conda create ... to create their Conda environments. With the classic solver it works fine. But with the libmamba solver it does not work.
For the example below I created a small environment to reproduce the bug.
It's easy to reproduce. You only need a read only Conda installation.

Conda Info

active environment : base
active env location : /tmp/software/python/3.11.4
shell level : 1
user config file : /tmp/c4hm387/.condarc
populated config files : /tmp/software/python/3.11.4/.condarc
/tmp/c4hm387/.condarc
conda version : 24.5.0
conda-build version : not installed
python version : 3.11.4.final.0
solver : libmamba (default)
virtual packages : __archspec=1=skylake_avx512
__conda=24.5.0=0
__cuda=11.4=0
__glibc=2.31=0
__linux=5.14.21=0
__unix=0=0
base environment : /tmp/software/python/3.11.4  (read only)
conda av data dir : /tmp/software/python/3.11.4/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64  (offline)
https://repo.anaconda.com/pkgs/main/noarch  (offline)
https://repo.anaconda.com/pkgs/r/linux-64  (offline)
https://repo.anaconda.com/pkgs/r/noarch  (offline)
package cache : /tmp/software/python/3.11.4/pkgs
/tmp/c4hm387/.conda/pkgs
envs directories : /tmp/c4hm387/.conda/envs
/tmp/software/python/3.11.4/envs
platform : linux-64
user-agent : conda/24.5.0 requests/2.31.0 CPython/3.11.4 Linux/5.14.21-150500.55.39-default sles/15.5 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
UID:GID : 34387:1490
netrc file : /tmp/c4hm387/.netrc
offline mode : True

Conda Config

==> /tmp/software/python/3.11.4/.condarc <==
ssl_verify: /var/lib/PKI/PKI-bundle.txt
channels: []
always_softlink: True
offline: True
solver: classic
 
==> /tmp/c4hm387/.condarc <==
ssl_verify: /var/lib/PKI/PKI-bundle.txt
channels: []
always_softlink: True
offline: True
solver: libmamba

Conda list

# packages in environment at /tmp/software/python/3.11.4:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    http://reposrv/localrepos/conda/conda-forge
_openmp_mutex             4.5                       2_gnu    http://reposrv/localrepos/conda/conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
brotli-python             1.1.0           py311hb755f60_1    http://reposrv/localrepos/conda/conda-forge
bzip2                     1.0.8                h7f98852_4    http://reposrv/localrepos/conda/conda-forge
c-ares                    1.28.1               hd590300_0    http://reposrv/localrepos/conda/conda-forge
ca-certificates           2023.05.30           h06a4308_0    defaults
certifi                   2023.5.7        py311h06a4308_0    defaults
cffi                      1.16.0          py311hb3a22ac_0    http://reposrv/localrepos/conda/conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
conda                     24.5.0          py311h38be061_0    http://reposrv/localrepos/conda/conda-forge
conda-content-trust       0.2.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
conda-forge-repodata-patches 20240528.05.30.30      hd8ed1ab_1    http://reposrv/localrepos/conda/conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
conda-package-handling    2.2.0              pyh38be061_0    http://reposrv/localrepos/conda/conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
cryptography              42.0.5          py311h63ff55d_0    http://reposrv/localrepos/conda/conda-forge
distro                    1.8.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
fmt                       10.2.1               h00ab1b0_0    http://reposrv/localrepos/conda/conda-forge
frozendict                2.4.4           py311h331c9d8_0    http://reposrv/localrepos/conda/conda-forge
icu                       73.2                 h59595ed_0    http://reposrv/localrepos/conda/conda-forge
idna                      3.6                pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
jsonpointer               2.0                        py_0    http://reposrv/localrepos/conda/conda-forge
keyutils                  1.6.1                h166bdaf_0    http://reposrv/localrepos/conda/conda-forge
krb5                      1.21.2               h659d440_0    http://reposrv/localrepos/conda/conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    http://reposrv/localrepos/conda/conda-forge
libarchive                3.7.2                h039dbb9_0    http://reposrv/localrepos/conda/conda-forge
libcurl                   8.7.1                hca28451_0    http://reposrv/localrepos/conda/conda-forge
libedit                   3.1.20191231         he28a2e2_2    http://reposrv/localrepos/conda/conda-forge
libev                     4.33                 h516909a_1    http://reposrv/localrepos/conda/conda-forge
libffi                    3.4.2                h7f98852_5    http://reposrv/localrepos/conda/conda-forge
libgcc-ng                 13.2.0               h807b86a_2    http://reposrv/localrepos/conda/conda-forge
libgomp                   13.2.0               h807b86a_2    http://reposrv/localrepos/conda/conda-forge
libiconv                  1.17                 h166bdaf_0    http://reposrv/localrepos/conda/conda-forge
libmamba                  1.5.8                had39da4_0    http://reposrv/localrepos/conda/conda-forge
libmambapy                1.5.8           py311hf2555c7_0    http://reposrv/localrepos/conda/conda-forge
libnghttp2                1.58.0               h47da74e_1    http://reposrv/localrepos/conda/conda-forge
libsolv                   0.7.25               hfc55251_0    http://reposrv/localrepos/conda/conda-forge
libssh2                   1.11.0               h0841786_0    http://reposrv/localrepos/conda/conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    http://reposrv/localrepos/conda/conda-forge
libuuid                   1.41.5               h5eee18b_0    defaults
libxml2                   2.12.6               h232c23b_0    http://reposrv/localrepos/conda/conda-forge
libzlib                   1.2.13               hd590300_5    http://reposrv/localrepos/conda/conda-forge
lz4-c                     1.9.4                hcb278e6_0    http://reposrv/localrepos/conda/conda-forge
lzo                       2.10              h516909a_1000    http://reposrv/localrepos/conda/conda-forge
menuinst                  2.0.2           py311h38be061_0    http://reposrv/localrepos/conda/conda-forge
ncurses                   6.4.20240210         h59595ed_0    http://reposrv/localrepos/conda/conda-forge
openssl                   3.2.1                hd590300_1    http://reposrv/localrepos/conda/conda-forge
packaging                 23.2               pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
pcre2                     10.37                he7ceb23_1    defaults
pip                       24.0               pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
pybind11-abi              4                    hd8ed1ab_3    http://reposrv/localrepos/conda/conda-forge
pycosat                   0.6.6           py311h459d7ec_0    http://reposrv/localrepos/conda/conda-forge
pycparser                 2.21               pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
pyopenssl                 24.0.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
pysocks                   1.7.1              pyha2e5f31_6    http://reposrv/localrepos/conda/conda-forge
python                    3.11.4               h955ad1f_0    defaults
python_abi                3.11                    2_cp311    http://reposrv/localrepos/conda/conda-forge
readline                  8.2                  h8228510_1    http://reposrv/localrepos/conda/conda-forge
reproc                    14.2.4.post0         hd590300_0    http://reposrv/localrepos/conda/conda-forge
reproc-cpp                14.2.4.post0         h59595ed_0    http://reposrv/localrepos/conda/conda-forge
requests                  2.31.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
ruamel.yaml               0.17.35         py311h459d7ec_0    http://reposrv/localrepos/conda/conda-forge
ruamel.yaml.clib          0.2.7           py311h2582759_1    http://reposrv/localrepos/conda/conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
sqlite                    3.41.2               h5eee18b_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
toolz                     0.12.0             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
tqdm                      4.66.2             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
truststore                0.8.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
typing_extensions         4.9.0              pyha770c72_0    http://reposrv/localrepos/conda/conda-forge
tzdata                    2023c                h71feb2d_0    http://reposrv/localrepos/conda/conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    http://reposrv/localrepos/conda/conda-forge
xz                        5.4.2                h5eee18b_0    defaults
yaml-cpp                  0.8.0                h59595ed_0    http://reposrv/localrepos/conda/conda-forge
zlib                      1.2.13               hd590300_5    http://reposrv/localrepos/conda/conda-forge
zstandard                 0.22.0          py311haa97af0_0    http://reposrv/localrepos/conda/conda-forge
zstd                      1.5.5                hc292b87_0    defaults

Additional Context

here comes the what I've done in my shell:

(base) c4hm387@conda:/tmp/c4hm387 > conda create -p /tmp/offlinetest python=3.11.4 pip mlflow
Channels:
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): failed
 
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
 
Traceback (most recent call last):
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/exception_handler.py", line 18, in __call__
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/cli/main.py", line 84, in main_subshell
exit_code = do_call(args, parser)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 200, in do_call
result = getattr(module, func_name)(args, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/notices/core.py", line 132, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/cli/main_create.py", line 137, in execute
return install(args, parser, "create")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/cli/install.py", line 377, in install
unlink_link_transaction = solver.solve_for_transaction(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
unlink_precs, link_precs = self.solve_for_diff(
^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda/core/solve.py", line 222, in solve_for_diff
final_precs = self.solve_final_state(
^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda_libmamba_solver/solver.py", line 207, in solve_final_state
index = IndexHelper(
^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda_libmamba_solver/index.py", line 136, in __init__
self._index = self._load_channels()
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda_libmamba_solver/index.py", line 337, in _load_channels
info = self._json_path_to_repo_info(url, jsons[url])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda_libmamba_solver/index.py", line 293, in _json_path_to_repo_info
repo = api.Repo(self._pool, noauth_url, str(path_to_use), escape_channel_url(noauth_url))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Permission denied
 
`$ /tmp/software/python/3.11.4/bin/conda create -p /tmp/offlinetest python=3.11.4 pip mlflow`
 
environment variables:
CIO_TEST=<not set>
CONDA_DEFAULT_ENV=base
CONDA_EXE=/tmp/software/python/3.11.4/bin/conda
CONDA_PREFIX=/tmp/software/python/3.11.4
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/tmp/software/python/3.11.4/bin/python
CONDA_ROOT=/tmp/software/python/3.11.4
CONDA_SHLVL=1
CURL_CA_BUNDLE=<not set>
LD_PRELOAD=<not set>
PATH=/tmp/software/python/3.11.4/bin:/tmp/software/python/3.11.4/condabin:/
usr/local/bin:/bin:/usr/bin:/usr/sbin:/usr/lib/mit/bin:/sbin
PYTHONSTARTUP=/etc/pythonstart
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
XFILESEARCHPATH=/usr/share/X11/%T/%N%S:/software/fs1/X11/%T/%N%S:/software/opensource/shared/X11/%T/%N%S
XNLSPATH=/usr/share/X11/nls
__GL_SHADER_DISK_CACHE_PATH=/tmp/scr/c4hm387/.nv
 
active environment : base
active env location : /tmp/software/python/3.11.4
shell level : 1
user config file : /tmp/c4hm387/.condarc
populated config files : /tmp/software/python/3.11.4/.condarc
/tmp/c4hm387/.condarc
conda version : 24.5.0
conda-build version : not installed
python version : 3.11.4.final.0
solver : libmamba (default)
virtual packages : __archspec=1=skylake_avx512
__conda=24.5.0=0
__cuda=11.4=0
__glibc=2.31=0
__linux=5.14.21=0
__unix=0=0
base environment : /tmp/software/python/3.11.4  (read only)
conda av data dir : /tmp/software/python/3.11.4/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64  (offline)
https://repo.anaconda.com/pkgs/main/noarch  (offline)
https://repo.anaconda.com/pkgs/r/linux-64  (offline)
https://repo.anaconda.com/pkgs/r/noarch  (offline)
package cache : /tmp/software/python/3.11.4/pkgs
/tmp/c4hm387/.conda/pkgs
envs directories : /tmp/c4hm387/.conda/envs
/tmp/software/python/3.11.4/envs
platform : linux-64
user-agent : conda/24.5.0 requests/2.31.0 CPython/3.11.4 Linux/5.14.21-150500.55.39-default sles/15.5 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
UID:GID : 34387:1490
netrc file : /tmp/c4hm387/.netrc
offline mode : True

An unexpected error has occurred. Conda has prepared the above report.
If you suspect this error is being caused by a malfunctioning plugin,
consider using the --no-plugins option to turn off plugins.
 
Example: conda --no-plugins install <package>
 
Alternatively, you can set the CONDA_NO_PLUGINS environment variable on
the command line to run the command without plugins enabled.
 
Example: CONDA_NO_PLUGINS=true conda install <package>
 
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers? [y/N]: N
 
No report sent. To permanently opt-out, use

$ conda config --set report_errors false

[travishathaway]

Thanks for perhaps identifying a potential bug with the conda-libmamba-solver.

Before we transfer this issue over to the applicable repository (https://github.com/conda/conda-libmamba-solver), I was wondering whether you have tried explicitly setting the envs_dirs setting to a user-writable location? (e.g. /home/user/.conda/envs).

For most Linux installations, this should be done automatically for you. Here's an example from an Ubuntu Server VM where conda is installed to the system location /opt/conda. You'll notice that both "package cache" and "envs directories" both contain entries where the user is able to write to:

     active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/thath/.condarc
 populated config files :
          conda version : 24.4.0
    conda-build version : not installed
         python version : 3.12.3.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=neoverse_n1
                          __conda=24.4.0=0
                          __glibc=2.39=0
                          __linux=6.8.0=0
                          __unix=0=0
       base environment : /opt/conda  (read only)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-aarch64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-aarch64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/thath/.conda/pkgs
       envs directories : /home/thath/.conda/envs
                          /opt/conda/envs
               platform : linux-aarch64
             user-agent : conda/24.4.0 requests/2.31.0 CPython/3.12.3 Linux/6.8.0-31-generic ubuntu/24.04 glibc/2.39 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[c4hm]

Sorry, cut & paste has destroyed the formatting. I have the same envs dirs as you.

      package cache : /tmp/software/python/3.11.4/pkgs
                      /tmp/c4hm387/.conda/pkgs
   envs directories : /tmp/c4hm387/.conda/envs
                      /tmp/software/python/3.11.4/envs

/tmp/c4hm387/ ist my (temporary) HOME and writable for me, /tmp/software not.

[travishathaway]

Thanks for the clarification.

I was unfortunately unable to recreate this myself.

@jaimergp, do you believe this belongs in the conda-libmamba-solver project as a bug?

[c4hm]

At least in your conda info output you are not in offline mode. I can try to reproduce it in a different environment.
Thanks for the quick response!

[c4hm]

I have to add one important point: Reproducing the problem is not as easy as I've described. It depends on one more condition: the user running conda has to be a member of the group owning the conda installation (base environment) (but none of the directories are group writable).

[c4hm]

I've done some futher debugging. The behavior only depends on whether the file urls.txt is writable. This is called PACKAGE_CACHE_MAGIC_FILE in libmamba/include/mamba/core/package_cache.hpp
I've changed the mode of this 'magic' file and the error message is gone. I think this is a bug in two ways:

1. conda set a 664 for that file (and some other file) and ignores my umask (but it does not ignore my umask for the pkgs/cache dir)
2. The assumption that a writable magic file means you can write other files or dirs might not always be correkt, as we can see.

[jaimergp]

I think (1) is related to conda-forge/miniforge#506. I'll need to check what we do about it in (2). Seeing defaults in your config makes me think you installed conda via Miniconda, right? We might need to check what we are doing in constructor about urls.txt

It's added to the preconda tarball in this block, which I guess then gets extracted with the same permissions obtained during creation in this series of calls.

Those --extract-tarball calls are implemented in both conda-standalone and micromamba. So we might need to check both sides.

[jaimergp]

Ok, a little summary of the things we'll need:

• conda-standalone needs this patch and a release. micromamba possibly too, so I submitted an issue in their repo. Then we can rebuild installers without the urls.txt problem.
• The assumption of "I can write to this file then the directory is writable" is encoded here. We could patch that function with a try/except block to see if we can write a dummy file, I guess.

It's not clear to me why libmamba needs to write anything while offline reading the cache.

File "/tmp/software/python/3.11.4/lib/python3.11/site-packages/conda_libmamba_solver/index.py", line 293, in _json_path_to_repo_info

    repo = api.Repo(self._pool, noauth_url, str(path_to_use), escape_channel_url(noauth_url))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Permission denied

I think it might be this step, where it always tries to write the SOLV cache. I created an issue and proposed a PR, but my C++ is limited.

[jaimergp]

Just learnt that this is fixed in libmamba v2, so when a new CLS is released with this PR merged, we can close this issue.

[jaimergp]

I created an issue and mamba-org/mamba#3313, but my C++ is limited.

libmamba 1.5.11 includes this fix too now.