Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

py3-parallel-ssh cannot be used - stuck on python3.11 #23756

Closed
smoser opened this issue Jul 10, 2024 · 3 comments
Closed

py3-parallel-ssh cannot be used - stuck on python3.11 #23756

smoser opened this issue Jul 10, 2024 · 3 comments
Assignees

Comments

@smoser
Copy link
Member

smoser commented Jul 10, 2024

py3-parallel-ssh
is currently unable to be used even though it can be built.

$ docker run --rm -it cgr.dev/chainguard/wolfi-base:latest
% apk add py3-parallel-ssh
fetch https://packages.wolfi.dev/os/x86_64/APKINDEX.tar.gz
(1/25) Installing libbz2-1 (1.0.8-r8)
(2/25) Installing libexpat1 (2.6.2-r1)
(3/25) Installing libffi (3.4.6-r3)
(4/25) Installing gdbm (1.24-r0)
(5/25) Installing xz (5.6.2-r0)
(6/25) Installing libgcc (13.3.0-r0)
(7/25) Installing libstdc++ (13.3.0-r0)
(8/25) Installing mpdecimal (4.0.0-r2)
(9/25) Installing ncurses-terminfo-base (6.5_p20240629-r0)
(10/25) Installing ncurses (6.5_p20240629-r0)
(11/25) Installing readline (8.2-r7)
(12/25) Installing sqlite-libs (3.45.1-r1)
(13/25) Installing python-3.12-base (3.12.4-r1)
(14/25) Installing python-3.12 (3.12.4-r1)
(15/25) Installing py3-pycparser (2.22-r1)
(16/25) Installing py3-cffi (1.16.0-r3)
(17/25) Installing py3-greenlet (3.0.3-r1)
(18/25) Installing py3.12-setuptools (70.3.0-r0)
(19/25) Installing py3-zope.event (5.0-r0)
(20/25) Installing py3-zope.interface (6.3-r0)
(21/25) Installing py3-gevent (24.2.1-r0)
(22/25) Installing python-3.11-base (3.11.9-r5)
(23/25) Installing py3-ssh-python (1.0.0-r1)
(24/25) Installing py3-ssh2-python (1.0.0-r1)
(25/25) Installing py3-parallel-ssh (2.12.0-r1)
OK: 170 MiB in 39 packages

$ python3.12 -c 'from pssh.clients import ParallelSSHClient'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.12/site-packages/pssh/clients/__init__.py", line 19, in <module>
    from .native.parallel import ParallelSSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/__init__.py", line 19, in <module>
    from .parallel import ParallelSSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/parallel.py", line 20, in <module>
    from .single import SSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/single.py", line 24, in <module>
    from ssh2.error_codes import LIBSSH2_ERROR_EAGAIN
ModuleNotFoundError: No module named 'ssh2'

The above demonstrates the failure.
Here is the "why". The package knows it depends directly on

    runtime:
      - py3-gevent
      - py3-ssh2-python
      - py3-ssh-python

But those dependencies are spread where some are installed in python3.11 path and others in python3.12

% pkgs="py3-pycparser py3-cffi py3-greenlet py3.12-setuptools 
  py3-zope.event py3-zope.interface py3-gevent py3-ssh-python 
  py3-ssh2-python py3-parallel-ssh"
% for p in $pkgs do
   echo == $p == ; apk info -L $p | grep .py$ | head -n 1; done
== py3-pycparser ==
usr/lib/python3.12/site-packages/pycparser/__init__.py
== py3-cffi ==
usr/lib/python3.12/site-packages/cffi/__init__.py
== py3-greenlet ==
usr/lib/python3.12/site-packages/greenlet/__init__.py
== py3.12-setuptools ==
usr/lib/python3.12/site-packages/_distutils_hack/__init__.py
== py3-zope.event ==
usr/lib/python3.12/site-packages/zope/event/__init__.py
== py3-zope.interface ==
usr/lib/python3.12/site-packages/zope/interface/__init__.py
== py3-gevent ==
usr/lib/python3.12/site-packages/gevent/__init__.py
== py3-ssh-python ==
usr/share/py3-ssh/.venv/lib/python3.11/site-packages/_distutils_hack/__init__.py
== py3-ssh2-python ==
usr/share/py3-ssh2/.venv/lib/python3.11/site-packages/_distutils_hack/__init__.py
== py3-parallel-ssh ==
usr/lib/python3.12/site-packages/pssh/__init__.p

The problem is:

  • py3-ssh2-python is stuck on python 3.11 (issue)
  • py3-ssh-python is stuck on python 3.11 (issue)
  • py3-gevent (and all it's deps have moved to 3.12).
  • py3-ssh2-python and py3-ssh-python are built in a venv against py3.11, so they couldn't be used as a apk dep anyway.
@smoser
Copy link
Member Author

smoser commented Jul 11, 2024

The whole 'python3' migration thing (python3.11 -> python3.12) is weird. I had some thoughts on it that I wanted to write down.

  1. We recently changed the SCA in melange to automatically add a specific python-3.N dependency when building a package. This is correct. A python package installs .py/.pyc files into a specific python-3.X site-packages, you can't just use python3.11 and import requests if py3-requests installed its files into /usr/lib/python3.12/site-packages/requests.
  2. I've removed several of the 'python-3' manual dependencies (package.dependencies.runtime). because a.) they're not strictly necessary and b.) at a future point after 'python3.13' is available, they'll actually be wrong and would drag in python3.13 on install.
  3. The python3 symlink is useful during package build, but problematic. When you install a package it generally writes a shebang referencing whatever
    executable ran the installer. Using python3.12 -m installer pkg.whl will write #!/usr/bin/python3.12 as shebang while python3 -m installer.pkg will write #!/usr/bin/python3. But during package build, you don't want to have to call python3.12 explicitly as that will become wrong at some point in the future.
  4. similarly to 3, images should ship python3, as no end user should be burdened with having to know what python3.X version they're using.

justinvreeland pushed a commit to justinvreeland/wolfi that referenced this issue Aug 13, 2024
In b7de8cb py3-ssh-python
py3-ssh2-python and py3-parallel-ssh were imported for use in
py3-cassandra-medusa.

Sice then py3-cassandra-medusa switched to a venv
49c9723. These packages are not
used or useable in their current state and upstream is not in a
position to be releasing python3.12 compatible versions right now.
@smoser
Copy link
Member Author

smoser commented Aug 14, 2024

@justinvreeland, did you want to do a pull request for your commit ?

@justinvreeland
Copy link
Member

@smoser Looks like I might've messed up some metadata but it should be here: #26270

@smoser smoser closed this as completed in a5e4968 Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants