Python3-keystone (OpenStack identity service) conflicts with keystone-engine when installed.

[neoh4x0r]

When the package Python3-keystone is installed on Debian (see https://packages.debian.org/sid/python3-keystone) pince will fail to import the keystone-engine related functions because the two packages have the same name and python seems to favor loading from /usr/lib/python3/dist-package instead of from the virtual environment. Without the Python3-keystone package installed, pince operates as expected.

I don't know if there is a way to force python to never load packages that exist outside of the virtual environment (or to prevent certain packages from being loaded from the system path) -- I mean other than explicitly importing the module from the VIRTUAL_ENV path.

Here's the error that occurs right after the program starts -- additionally, trying to attach to a process will cause pince to hang.

Last command: source gdb_python_scripts/gdbextensions.py
"Traceback (most recent call last):\n"
"  File \"gdb_python_scripts/gdbextensions.py\", line 28, in <module>\n"
"    from libpince import utils, typedefs, regexes\n"
"  File \"/home/william/Apps/Emulators/pince/libpince/utils.py\", line 21, in <module>\n"
"    from keystone import Ks, KsError, KS_ARCH_X86, KS_MODE_32, KS_MODE_64\n"
"ImportError: cannot import name 'Ks' from 'keystone' (/usr/lib/python3/dist-packages/keystone/__init__.py)\n"
done

[brkzlr]

I don't think there's much we can do if 2 different libraries use the same name, it's pretty much out of our control.

I did look into this though and it seems that it might be a Debian issue? I'm saying this because on my distro Arch, python-keystone points to the keystone assembler and python-keystoneauth1 is the OpenStack one. After installing the OpenStack one, it still works for me so either the packages don't conflict here or venv is working correctly.

You might have to look into why it loads the system ones instead of venv as it should be the other way around to my knowledge.

[brkzlr]

I've also double checked and, on Arch, they use the same name as the packages for the directories (assembler uses /usr/lib/python3.1x/site-packages/keystone and OpenStack uses keystoneauth1).

Checking the .deb file it seems that they place it in the keystone folder instead of keystoneauth1.
OpenStack has a GitHub page for the Python binding and it uses keystoneclient name and inside they're using both import keystoneauth1 and import keystoneclient so I think the fault lies with Debian not using standard names.

You should flag a bug with the package maintainer or anybody else that handles the package naming in Debian. Though your best bet is to find out why venv is being ignored as I doubt they would change anything about this.

[neoh4x0r]

You should flag a bug with the package maintainer or anybody else that handles the package naming in Debian. Though your best bet is to find out why venv is being ignored as I doubt they would change anything about this.

I sent a bug-report to debian about the python3-virtualenv module rather than about the python3-keystone package -- packages should always be loaded from the virtual environment and never from the system unless a named-package does not exist. I'm currently waiting for the bug-report number and confirmation of its receipt.

[brkzlr]

We're using python3-venv on Debian in the installer file, not virtualenv.

[neoh4x0r]

We're using python3-venv on Debian in the installer file, not virtualenv.

I'll have to wait for the bug confirmation against python3-virtualenv -- at which point I can request it to be removed and open a new one against python3-venv.

[brkzlr]

Make sure to double check that venv was used instead of virtualenv, just so we're on the same page. You did use our installer for PINCE instead of manually installing stuff right?

Also as a side note, while you wait for the bug to be fixed, I highly recommend using our AppImage instead of dealing with the dev install if you simply want to just use PINCE.

The installer is reserved for developers and code contributors, not for general use.

[neoh4x0r]

Make sure to double check that venv was used instead of virtualenv, just so we're on the same page. You did use our installer for PINCE instead of manually installing stuff right?

I used the installer script and I had both virtualenv and venv installed, but removing virtualenv did not make a difference as far as keystone being loaded from the system instead of the virtual path.

[brkzlr]

Try the AppImage then to see if the PYTHONHOME trick we do for a fake venv will work in your favour.

If that also does not work, then you're at the mercy of Debian maintainers, so good luck.

Let us know when that bug gets closed/resolved so we can also close this one.

[neoh4x0r]

Using the appimage, while python3-keystone (openstack) is installed, did not report errors about not being able to import the namespace Ks .

The bug opened against python3-venv (python3-defaults/3.11.2-1) can be found here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1091740

PS: Where do I find the version number associated with a pince installation? The latest appimage is from the v0.4.2 tag, but is this version stored somewhere in the installation?
Better yet, how do I build an appimage from the development source?

[neoh4x0r]

Here's an update from the Debian bug Report #1091740

Editing and running PINCE.sh so it only runs the python interpeter and does not execute PINCE.py I get the following:

Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keystone
>>> keystone.__file__
'.venv/PINCE/lib/python3.11/site-packages/keystone/__init__.py'
>>> import sys
>>> sys.path
['', '/usr/lib/python311.zip', '/usr/lib/python3.11', 
'/usr/lib/python3.11/lib-dynload', 
'.venv/PINCE/lib/python3.11/site-packages']

This means that original error message (about noting being able to import name 'Ks' from 'keystone' ) is strictly an issue with PINCE -- the package is trying to be loaded from /usr/lib/python3/dist-packages/keystone:

Last command: source gdb_python_scripts/gdbextensions.py
"Traceback (most recent call last):\n"
"  File \"gdb_python_scripts/gdbextensions.py\", line 28, in <module>\n"
"    from libpince import utils, typedefs, regexes\n"
"  File \"/home/william/Apps/Emulators/pince/libpince/utils.py\", line 21, in <module>\n"
"    from keystone import Ks, KsError, KS_ARCH_X86, KS_MODE_32, KS_MODE_64\n"
"ImportError: cannot import name 'Ks' from 'keystone' (/usr/lib/python3/dist-packages/keystone/__init__.py)\n"
done

PS: This failing behavior could happen if the system python is executed by some process instead of the python binary from the virtual environment.

[brkzlr]

Try manually running PINCE.py using the steps highlighted in the sh installer and see what does not work. Either the venv activation script is not working in Debian or something else is going on.

[neoh4x0r]

EDIT: My suspicions were correct -- the system path list, in the context of sourcing gdbextensions, will not load packages from the virtual environment because it is looking in the system path first.

Last command: source gdb_python_scripts/gdbextensions.py

keystone.__file__=/usr/lib/python3/dist-packages/keystone/__init__.py

sys.path=[
    /usr/share/gdb/python,
    /usr/lib/python311.zip,
    /usr/lib/python3.11, 
    /usr/lib/python3.11/lib-dynload,
    /home/william/.local/lib/python3.11/site-packages, 
    /usr/local/lib/python3.11/dist-packages,
    /usr/lib/python3/dist-packages, 
    /usr/lib/python3.11/dist-packages,
    /usr/lib/python311.zip,
    /usr/lib/python3.11, 
    /usr/lib/python3.11/lib-dynload, 
    /home/william/Apps/Emulators/pince/.venv/PINCE/lib/python3.11/site-packages,
    /home/william/Apps/Emulators/pince
]

---

I think this issue is the result of the gdbextensions.py being sent as a source command to gdb.

https://github.com/korcankaraokcu/PINCE/blob/master/libpince/debugcore.py#L521

def set_pince_paths():
    """Initializes $PINCE_PATH and $GDBINIT_AA_PATH convenience variables to make commands in gdbextensions.py
    and gdbutils.py work. GDB scripts need to know libpince and .config directories, unfortunately they don't start
    from the place where script exists
    """
    libpince_dir = utils.get_libpince_directory()
    pince_dir = os.path.dirname(libpince_dir)
    gdbinit_aa_dir = utils.get_user_path(typedefs.USER_PATHS.GDBINIT_AA)
    send_command("set $GDBINIT_AA_PATH=" + '"' + gdbinit_aa_dir + '"')
    send_command("set $PINCE_PATH=" + '"' + pince_dir + '"')
    send_command("source gdb_python_scripts/gdbextensions.py")

I think when gdb sources this file it running in a context outside of the virtual environment and that's why it attempts to load keystone from /usr/lib/python3/dist-packages.

It doesn't look like the gdbextensions even uses the keystone functionality, it only seems to call some other utility functions, ie that file may not need to import the keystone package.

I'm still investigation this issue...I will see if I can get the gdbextensions.py to report the path for the keystone library and the system path, to try to confirm my suspicions stated above.

[brkzlr]

That has nothing to do with your problem. We already make the gdb spawned inside PINCE respect the venv environment by importing a specific gdbinit_venv file inside libpince/debugcore.py at line 502.

Your problem is that the venv environment itself is not getting activated. This creates a snowball effect where you run into the issues you listed but they're not the main cause.

You modified PINCE.sh to spawn a Python interpreter instead of running PINCE.py and the venv directories were not included in sys path. That's where you should check first and foremost.

Manually run the commands inside PINCE.sh in your terminal and see if venv activates correctly. Basically:

• Activate venv environment by running . .venv/PINCE/bin/activate.
• Spawn a Python 3 interpreter and check sys path that way.
• If sys path correctly includes venv, try again using sudo -E --preserve-env=PATH and check if venv is still included.

Something broke in Debian's behaviour of including the venv and I have a hunch it's the preserve-env (which was added as a hack for Debian/Ubuntu only anyway) that broke.

[neoh4x0r]

Manually run the commands inside PINCE.sh in your terminal and see if venv activates correctly. Basically:

• Activate venv environment by running . .venv/PINCE/bin/activate.
• Spawn a Python 3 interpreter and check sys path that way.
• If sys path correctly includes venv, try again using sudo -E --preserve-env=PATH and check if venv is still included.

Something broke in Debian's behaviour of including the venv and I have a hunch it's the preserve-env (which was added as a hack for Debian/Ubuntu only anyway) that broke.

I see the virtual env path listed in the output and both have the same path list.

$ . .venv/PINCE/bin/activate
$ .venv/PINCE/bin/python3
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[
'', 
'/usr/lib/python311.zip', 
'/usr/lib/python3.11', 
'/usr/lib/python3.11/lib-dynload', 
'/home/william/Apps/Emulators/pince/.venv/PINCE/lib/python3.11/site-packages'
]
>>>quit()

$  . .venv/PINCE/bin/activate
$ sudo -E --preserve-env=PATH .venv/PINCE/bin/python3
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
[
'', 
'/usr/lib/python311.zip', 
'/usr/lib/python3.11', 
'/usr/lib/python3.11/lib-dynload', 
'/home/william/Apps/Emulators/pince/.venv/PINCE/lib/python3.11/site-packages'
]
>>>quit()

PS: I can create a minimal venv the keystone package isloaded correctly (as per the Debian Bug I opened):

$ python3 -m venv testvenv
$ . ./testvenv/bin/activate
$ ./testvenv/bin/python -m pip install -U keystone-engine
$ sudo -E --preserve-env=PATH ./testvenv/bin/python
Python 3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import keystone
>>> keystone.__file__
'/media/PNY64GB/debian-bug-1091740/testvenv/lib/python3.11/site-packages/keystone/__init__.py'
>>> import sys
>>> sys.path
[
'',
'/usr/lib/python311.zip',
'/usr/lib/python3.11', 
'/usr/lib/python3.11/lib-dynload', 
'/media/PNY64GB/debian-bug-1091740/testvenv/lib/python3.11/site-packages'
]

[brkzlr]

Hmm, do me a favour and modify libpince/gdbinit_venv.

Inside it we extend the sys path of gdb's Python with the one we have from the venv environment. Instead of extending it, just replace sys.path with the paths variable.

[neoh4x0r]

Hmm, do me a favour and modify libpince/gdbinit_venv.

Inside it we extend the sys path of gdb's Python with the one we have from the venv environment. Instead of extending it, just replace sys.path with the paths variable.

I think that was the problem.

After I changing it to sys.path = paths I got the following output:

Last command: source gdb_python_scripts/gdbextensions.py
keystone.__file__=
    /home/william/Apps/Emulators/pince/.venv/PINCE/lib/python3.11/site-packages/keystone/__init__.py
sys.path=[
'/usr/lib/python311.zip',
'/usr/lib/python3.11', 
'/usr/lib/python3.11/lib-dynload', 
 '/home/william/Apps/Emulators/pince/.venv/PINCE/lib/python3.11/site-packages', 
'/home/william/Apps/Emulators/pince'
]

It is no longer complaining about importing the namespaces from the keystone package.

[brkzlr]

That's also why it only happens on Debian as Debian is pretty much the only one from our supported list that names the OpenStack one keystone, so it will conflict with the system one.

I'll look into it tomorrow or just after new years to make sure changing that doesn't break our other distros or AppImage.

Feel free to use the AppImage meanwhile until we commit the fix. Thanks for testing.

[brkzlr]

Fixed with 9327a7a

Thanks for the report!