Subproject with static libraries fails the build in 0.13.0 onward (but not in 0.12.0)

[blochl]

When building with a subproject which provides both static and dynamic libraries, I get the following error with 0.13.0 onward:

meson-python: error: Could not map installation path to an equivalent wheel directory: '{libdir_static}/mylibrary.a'

With meson-python 0.12.0 everything works fine.

Looking at #35, I understand that there is no location in wheels for static libraries, and they aren't needed at runtime, but for me --skip-subprojects install option is not a solution, as I do need the dynamic libraries which this subproject installs. What can I do (besides just using version 0.12.0, where it works 😄 )?

[rgommers]

Shared libraries also have nowhere to go in a wheel, so you'll run into that same problem. I think what you want is only use the dependency from the subproject, e.g. libsimple_dep from the examples in https://mesonbuild.com/Subprojects.html.

I'll note that a static library is a lot easier to deal with in a wheel. If you want to use a shared library, there may be issues (I actually don't know how to do it, it kinda requires running auditwheel & co before returning control to the build frontend).

[blochl]

@rgommers , thanks for your reply!

1. But in the subproject the libraries' dependency is specified with install : true, and I can't change that (not my code)...
2. The shared libraries have to go in a wheel - it's a project which uses Python C extensions. And it works great with 0.12.0 - the wheel contains the shared libs, and installs them just fine. How do you suggest to overcome this?

[blochl]

@rgommers ,
Just checked: the shared libs go into the wheel and are installed just fine in all versions, including the latest (0.16.0).

[rgommers]

Just checked: the shared libs go into the wheel and are installed just fine in all versions,

I think they go under the data directory right, and then get installed to <prefix>/lib/? That's not a good default for redistributable packages (since it risks clobbering things in other virtualenv or in /usr/lib/), but I do agree that shared and static libraries should be treated the same way here.

Is your package public perhaps? We're lacking a test package for this config right now unfortunately - we should add that, with more docs, since this isn't the first question about subprojects with shared/static libraries.

1. But in the subproject the libraries' dependency is specified with install : true, and I can't change that (not my code)...

I think --skip-subprojects to avoid installing it should be taking care of that. With static libraries this works, since you don't want to install those separately but link them into the Python extension module you're building. With shared libraries, there's an issue - this kinda will only work if you'd run auditwheel over them, but for from-source installs this would actually need doing before returning the wheel to the build frontend. I think this issue is unsolved right now.

[blochl]

I think they go under the data directory right, and then get installed to <prefix>/lib/?

Yeah, to <prefix>/usr/local/lib/x86_64-linux-gnu/.

That's not a good default for redistributable packages (since it risks clobbering things in other virtualenv or in /usr/lib/)

I agree. But what else can be done if I want (or have to) use dynamic libraries with Python C extension?
I wasn't aware of auditwheel. Thanks for pointing me there! Looks like it can graft an .so into the wheel? Looks promising!
But where will it install it later on the system? And if it's doable, maybe meson-python can do it? Because it's a very real scenario for projects with Python C extensions.

Is your package public perhaps?

Unfortunately (and I mean it sincerely) it is not.
I use the PyFAI project for references to meson-python. It's an awesome project, which is both complex, has C in it, and uses meson-python. My project has a similar structure to it.

[rgommers]

I agree. But what else can be done if I want (or have to) use dynamic libraries with Python C extension?
I wasn't aware of auditwheel. Thanks for pointing me there! Looks like it can graft an .so into the wheel? Looks promising!
But where will it install it later on the system? And if it's doable, maybe meson-python can do it? Because it's a very real scenario for projects with Python C extensions.

So the default way of dealing with a shared library if it's not in a Meson subproject is:

1. In your meson.build, use dependency('foo') or some equivalent to discover and build against the shared library when producing a wheel.
2. Run auditwheel foo-xxx.whl to obtain a redistributable wheel (auditwheel takes care of stashing the shared library within the wheel itself and updating symbol names, rpath's, etc. as needed).

If the shared library is in a Meson subproject, you can do essentially the same thing now. Skip install of the subproject in your pyproject.toml, so that you're only linking against the shared library in the build directory. Then:

1. python -m build --no-isolation -Cbuild-dir=builddir (builds a wheel, say foo-xxx.whl, with meson-python)
2. auditwheel foo-xxx.whl

That should work. What does not work is using a temporary directory, e.g. when doing pip install foo --no-binary, because there's no way to invoke auditwheel that way, and the shared library disappears when the tmpdir is cleaned up.

Note: auditwheel is Linux-specific, there are similar tools on other platforms (delocate, delvewheel, repair-wheel). Using a shared library on Windows requires more effort, because of the lack of RPATH support on Windows. The delvewheel docs should explain (you need os.add_dll_directory or something like that).

---

I suspect that the bit we should add at some point in meson-python is an option to invoke auditwheel & co before handling the wheel over to pip/build, so that building in a tmpdir works as well.