Mixed conda / private pip lock file does not install inside docker container #623

metgrahamr · 2024-04-17T06:08:47Z

Checklist

I added a descriptive title
I searched open reports and couldn't find a duplicate

What happened?

I created a lock file using a mix of conda-forge packages and private pip packages in an AWS CodeArtifact repo in a similar way to here but as I am using conda-lock version 2.5.6 I don't have the issue with the value of the CodeArtifact password being passed through.
With the CodeArtifact password set as an environment variable, I can successfully use this lock file to create a new conda environment. However, if I try to do this from within a docker container then I get the following error

conda-lock install -p /opt/gauge /locks/conda-lock.yml
INFO:root:Downloading and Extracting Packages: ...working... done
INFO:root:
INFO:root:Downloading and Extracting Packages: ...working... done
INFO:root:Preparing transaction: ...working... done
INFO:root:Verifying transaction: ...working... done
INFO:root:Executing transaction: ...working... done
ERROR:root:ERROR: Exception:
ERROR:root:Traceback (most recent call last):
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
ERROR:root:    status = run_func(*args)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
ERROR:root:    return func(self, options, args)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run
ERROR:root:    requirement_set = resolver.resolve(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
ERROR:root:    collected = self.factory.collect_root_requirements(root_reqs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
ERROR:root:    reqs = list(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
ERROR:root:    cand = self._make_base_candidate_from_link(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
ERROR:root:    self._link_candidate_cache[link] = LinkCandidate(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
ERROR:root:    super().__init__(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
ERROR:root:    self.dist = self._prepare()
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
ERROR:root:    dist = self._prepare_distribution()
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
ERROR:root:    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
ERROR:root:    return self._prepare_linked_requirement(req, parallel_builds)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 596, in _prepare_linked_requirement
ERROR:root:    local_file = unpack_url(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 168, in unpack_url
ERROR:root:    file = get_http_url(
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 109, in get_http_url
ERROR:root:    from_path, content_type = download(link, temp_dir.path)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/download.py", line 134, in __call__
ERROR:root:    resp = _http_get_download(self._session, link)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/download.py", line 117, in _http_get_download
ERROR:root:    resp = session.get(target_url, headers=HEADERS, stream=True)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 602, in get
ERROR:root:    return self.request("GET", url, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/session.py", line 520, in request
ERROR:root:    return super().request(method, url, *args, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 589, in request
ERROR:root:    resp = self.send(prep, **send_kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 710, in send
ERROR:root:    r = dispatch_hook("response", hooks, r, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_vendor/requests/hooks.py", line 30, in dispatch_hook
ERROR:root:    _hook_data = hook(hook_data, **kwargs)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 500, in handle_401
ERROR:root:    username, password, save = self._prompt_for_password(parsed.netloc)
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/network/auth.py", line 455, in _prompt_for_password
ERROR:root:    username = ask_input(f"User for {netloc}: ") if self.prompting else None
ERROR:root:  File "/opt/gauge/lib/python3.9/site-packages/pip/_internal/utils/misc.py", line 251, in ask_input
ERROR:root:    return input(message)
ERROR:root:EOFError: EOF when reading a line
ERROR:root:
ERROR:root:ERROR conda.cli.main_run:execute(124): `conda run pip install --no-deps -r /tmp/tmp7ssjgkkc` failed. (See above for error)
INFO:root:Collecting arnparse@ https://aws:****@sample-111222333444.d.codeartifact.us-west-2.amazonaws.com/pypi/repo-name/simple/arnparse/0.0.2/arnparse-0.0.2-py2.py3-none-any.whl#sha256=b0906734e4b8f19e39b1e32944c6cd6274b6da90c066a83882ac7a11d27553e0 (from -r /tmp/tmp7ssjgkkc (line 1))
INFO:root:User for sample-111222333444.d.codeartifact.us-west-2.amazonaws.com:
Traceback (most recent call last):
  File "/opt/conda/bin/conda-lock", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1497, in click_install
    install(
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 1551, in install
    install_func(file=lockfile)
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/conda_lock.py", line 240, in do_conda_install
    _conda(["run"], ["pip", "install", "--no-deps", "-r", str(requirements_path)])
  File "/opt/conda/lib/python3.11/site-packages/conda_lock/invoke_conda.py", line 143, in _invoke_conda
    raise subprocess.CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/opt/conda/bin/conda', 'run', '--prefix', '/opt/gauge', 'pip', 'install', '--no-deps', '-r', '/tmp/tmp7ssjgkkc']' returned non-zero exit status 2.

I wondered whether the error was anything to do with this and tried adding --no-capture-output to the conda run command that gets called. This just caused the install to hang at the first package install. Running the pip install command manually with the requirements file in the /tmp directory prompted for a user and then password. If I entered aws and the CodeArtifact password at each prompt then pip was able to install each package.

It looks like when running inside a docker container the install command is not getting the password from the environment and is trying to prompt for the information. How do I get this lock file to work inside a docker container in the same way that it works outside?

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /root/.condarc
 populated config files : 
          conda version : 24.3.0
    conda-build version : not installed
         python version : 3.11.7.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.3.0=0
                          __glibc=2.31=0
                          __linux=5.15.146.1=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
       envs directories : /opt/conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.11.7 Linux/5.15.146.1-microsoft-standard-WSL2 debian/11.9 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.6
                UID:GID : 0:0
             netrc file : None
           offline mode : False

Conda Config

No response

Conda list

No response

Additional Context

I am using the continuumio/miniconda3:latest docker image and install conda-lock into the base environment using pip.

The text was updated successfully, but these errors were encountered:

maresb · 2024-04-17T11:25:27Z

Hi @metgrahamr! This seems like a tricky one. Thanks a lot for experimenting with --no-capture-output, that's pretty interesting.

I think we'll need to drill down quite a bit more before we can isolate the cause of this issue.

Could you clarify exactly what you mean by "within a docker container"? Is this from a bash prompt following a docker run command, or is this occurring in a Dockerfile via docker build?
How are you creating the environment variables with the credentials? Are they coming from the -e argument to docker run?
How exactly are you referencing the environment variables within your original environment definition? And how are those environment variables being translated within your lockfile? There is some quirkiness where sometimes you must use ${ENV_VAR} instead of $ENV_VAR when referencing the environment variables.

It would help a lot with the diagnosis if you could share relevant snippets (or better the whole thing minus personally identifiable info if not confidential) from your original environment definition, the generated lockfile, and the particular docker commands you're executing. Thanks!

metgrahamr · 2024-04-17T21:29:33Z

1. Could you clarify exactly what you mean by "within a docker container"? Is this from a `bash` prompt following a `docker run` command, or is this occurring in a `Dockerfile` via `docker build`?

I initially tried this in a Dockerfile as I want to use conda-lock to install all the dependencies for an AWS Lambda function. However, when it didn't work I also fired up a container with docker run and tried replicating what I was doing in the Dockerfile but got the same results.

2. How are you creating the environment variables with the credentials? Are they coming from the `-e` argument to `docker run`?

For the Dockerfile I used ARG and supplied the CodeArtifact token as a build argument. When using docker run I created an environment variable once inside the container. I checked that it was available in the environment, e.g. echo $CODEARTIFACT_AUTH_TOKEN and even tried CODEARTIFACT_AUTH_TOKEN=$(echo $CODEARTIFACT_AUTH_TOKEN) conda-lock install .... When I tried running the requirements file directly I was prompted for the user and password and this worked (although obviously not via the environment variable).

3. How exactly are you referencing the environment variables within your original environment definition? And how are those environment variables being translated within your lockfile? There is some quirkiness where sometimes you must use `${ENV_VAR}` instead of `$ENV_VAR` when referencing the environment variables.

ARG FUNCTION_DIR="/opt/gauge/"
ARG GENERATOR="gauge"

FROM continuumio/miniconda3:latest as BUILDER

ARG FUNCTION_DIR
ARG GENERATOR
ARG CODEARTIFACT_AUTH_TOKEN

RUN pip install conda-lock
COPY conda-lock.yml /locks/conda-lock.yml
RUN CODEARTIFACT_AUTH_TOKEN=${CODEARTIFACT_AUTH_TOKEN} conda-lock install -p ${FUNCTION_DIR} /locks/conda-lock.yml

I then pass the credentials in like this

docker build . -t gauge_lock --build-arg CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain sample --domain-owner 111222333444 --region us-west-2 --query authorizationToken --output text`

The env file looks like this

name: gauge
channels:
  - conda-forge
pip-repositories:
  - https://aws:$CODEARTIFACT_AUTH_TOKEN@sample-111222333444.d.codeartifact.us-west-2.amazonaws.com/pypi/repo/simple/
dependencies:
  - python=3.9
  - pyyaml
  - pip
  - pip:
    - aws-lambda-powertools
    - awslambdaric
    - fr-helpers  # a private package in the CodeArtifact repo

platforms:
  - linux-64

conda-lock.yml

Thanks for looking into this (and the project as a whole) it's much appreciated.

maresb · 2024-04-23T10:18:41Z

Hey @metgrahamr, thanks a lot for bearing with my questions. This seems pretty challenging to debug, and unfortunately I wasn't able to get to it over the weekend. Just wanted you to know that I haven't forgotten. Please feel free to ping me again later or continue debugging yourself.

metgrahamr · 2024-04-23T21:33:30Z

@maresb Thanks for the update and for keeping this under consideration.

maresb · 2024-07-25T09:18:39Z

Hey @metgrahamr, sorry I wasn't able to address this in a timely manner.

While trying to get the tests to pass for #637 I realized that the cache directory was causing problems. (~/.cache/pypoetry-conda-lock on Linux)

If you delete this directory, I suspect you'll be able to reproduce the issue outside of Docker.

In my recent commits on that PR I change the way we pass the authentication information to Poetry, so I hope we'll be able to merge/release soon, and that this will fix the issue for you.

metgrahamr · 2024-07-25T21:01:51Z

@maresb Thanks, exciting that you have likely found the issue. The project that I was considering this for has had to be put aside for higher priorities but it is due to bubble back up the list soon (and I would like to use conda-lock in other projects in a similar way) so hopefully I will get a chance to test in the near future.

Thanks again and I really appreciate the update.

maresb · 2024-07-26T07:09:47Z

Cool! By the way, have you considered pixi? It's not a drop-in replacement for conda-lock because it's project-centric, but their support for mixed conda/pip dependencies seems quite a bit more robust than ours. See #615 for more discussion. I'd be curious about your perspective.

metgrahamr · 2024-07-28T22:24:55Z

Thanks, pixi is on the list of things to look at but hadn't realised that dependency handling was more robust. Might bump up the priority!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mixed conda / private pip lock file does not install inside docker container #623

Mixed conda / private pip lock file does not install inside docker container #623

metgrahamr commented Apr 17, 2024

maresb commented Apr 17, 2024

metgrahamr commented Apr 17, 2024

maresb commented Apr 23, 2024

metgrahamr commented Apr 23, 2024

maresb commented Jul 25, 2024 •

edited

Loading

metgrahamr commented Jul 25, 2024

maresb commented Jul 26, 2024

metgrahamr commented Jul 28, 2024

Mixed conda / private pip lock file does not install inside docker container #623

Mixed conda / private pip lock file does not install inside docker container #623

Comments

metgrahamr commented Apr 17, 2024

Checklist

What happened?

Conda Info

Conda Config

Conda list

Additional Context

maresb commented Apr 17, 2024

metgrahamr commented Apr 17, 2024

maresb commented Apr 23, 2024

metgrahamr commented Apr 23, 2024

maresb commented Jul 25, 2024 • edited Loading

metgrahamr commented Jul 25, 2024

maresb commented Jul 26, 2024

metgrahamr commented Jul 28, 2024

maresb commented Jul 25, 2024 •

edited

Loading