Best practice for optionally bundling a library dependency

[minrk]

Split off from #410

Summary

In general, I'd like to get to a documented best practice for optionally bundling a library dependency. At a high level:

if library.found():
    link_it()
else:
    build_and_bundle_it()

In #410, @rgommers mentioned static linking for the bundle branch, which simplifies the bundling issue, since meson doesn't need to duplicate the delvewheel/delocate step. I believe this should work for me, but figuring out the Windows version needs some work

Specific case

I'm exploring updating the build system for pyzmq, which bundles libzmq if not found. This is slightly more complex than #410 because libzmq itself has a dependency on libsodium and both should be bundled. It is further complicated by the need to support Windows, and the fact that libsodium has visual studio solutions on Windows while ZeroMQ uses CMake. So far, it seems like it should be roughly:

if libzmq_found:
    locate libs, includes with pkg-config/etc. (standard, simple)
else:
    fetch libsodium, libzmq
    if Windows:
        build static libsodium with msbuild
        build static libzmq against static libsodium with cmake
    else:
        build static libsodium with autotools
        build static libzmq against static libsodium with autotools

if PyPy:
    _zmq = cffi extension
else:
    _zmq = cython extension

if libzmq_found:
    dynamically_link(_zmq, libzmq)
else:
    statically_link(_zmq, local libzmq)

more info

https://github.com/oscarbenjamin/mesontest provides a useful example for using wrapfiles to set up the fallback builds. It's gotten me pretty far for the simple posix case, at least.

[rgommers]

I believe this should work for me, but figuring out the Windows version needs some work

One of the thoughts I had, also because your build dependencies have different build system usage on Windows, is whether you actually want/need to support this on Windows? In general you need it because of folks building from source when running pip install pyzmq (e.g., because there are no wheels on PyPI). However, if you do have wheels on PyPI, it may be less work to simply document that Windows users who want to build from source need to prepare libzmq/libsodium in the same way as in your wheel build env.

There is anyway a second Windows issue: if you do pip install . locally it is entirely possible for libzmq to be found, linked against, and then go missing at runtime (this is not specific to Meson). The reason is that Windows doesn't have RPATH support, so pip is basically entirely broken for this use case - it will not work until you add manual handling of the external dependency or run delvewheel over the produced wheel. We have this exact problem with OpenBLAS in numpy/scipy, and always have some weird workarounds which involve copying the shared library around manually and adding the directory in which we put it to the DLL search path.

and the fact that libsodium has visual studio solutions on Windows while ZeroMQ uses CMake.

Other build systems: the most robust option is having upstream support for building with Meson, or a WrapDB file to do so. However, Meson also understands CMake - unless the CMake files from the upstream repo do really exotic things, Meson can parse those upstream CMakelists.txt files and auto-translate them to Meson on the fly when building.

Other build systems are harder, I believe that cannot be done as part of a subproject build. So libzmq with autotools may be an issue. However, there is a WrapDB file for cppzmq already, which may be of use.

if PyPy:

This is the second time in two days I've come across the need for determining whether the interpreter is PyPy. The one-liner from numpy/numpy#25531 to implement this is:

if not py.get_variable('SOABI').startswith('pypy')

but it looks like it would be nice to add a py.interpreter_name() method to Meson (@eli-schwartz WDYT?).

if dynamically_link else statically_link

There is no need for the explicit if-else at this point. You'll simply use a libzmq_dep dependency object, and Meson already knows whether that is static or dynamic and will link it correctly.

Instead, I think you do it in the discovery phase, something like:

libzmq_dep = dependency('zmq',
    fallback : ['zmq', 'libzmq_dep'],
    default_options: ['default_library=static']
)

as documented at https://mesonbuild.com/Subprojects.html#toggling-between-system-libraries-and-embedded-sources.

https://github.com/oscarbenjamin/mesontest provides a useful example for using wrapfiles to set up the fallback builds. It's gotten me pretty far for the simple posix case, at least.

We should indeed add something like this to the docs, going into a bit more detail about what is specific to distributing wheels on PyPI (since that's the hard part and not covered by the Meson docs).

[eli-schwartz]

but it looks like it would be nice to add a py.interpreter_name() method to Meson (@eli-schwartz WDYT?).

I have no real objections, but "interpreter" name is kind of vague and it should probably be "implementation" instead.

[minrk]

Lacking other reasons, it would probably be nice to match Python's own naming and values for this, which would suggest implementation and use the value from platform.python_implementation().

[rgommers]

Sounds good. Moving that to mesonbuild/meson#12727.

[minrk]

Thanks for the tips. Unfortunately, I do indeed need to support this on Windows. That is in fact the most important case.

I've gotten pretty far. Unfortunately, somethings that's causing me what seems like an unreasonable amount of trouble is getting one ExternalProject to depend on another one. ExternalProjects can depend on Targets, but can't depend on a Dependency, and can only get a Dependency from the ExternalProject object (the .target public attribute is not exposed), and I don't know enough meson to turn a Dependency back into a Target so it can be depended upon. All I want to do is make one subproject build before the other, but I can't seem to figure it out.

[eli-schwartz]

This is the downside of Meson's "external projects" support, because meson doesn't natively understand the external project's configure stage. It is (currently?) unable to inject one dependency into another. It's not a simple matter of building one before the other, either. Without a way to coordinate knowledge about targets that will be built in the future, there's no good way for the external project to look up a dependency.

I'd be interested to see the issue solved but I have no idea how one would attempt to model the problem.

[minrk]

This is probably not a good idea, but I went with building an empty static library that depends on the actual library, and then depending on that. This gave me a target that depends on the dependency. Fortunately, I didn't need to actually extract cflags, libs, etc. from the dependency as inputs to the second configure, since I also can't seem to figure out how to do that, which seems like it should be a simple attribute or method on the dependency object.

At least for now, this seems to be good enough. It would have been much simpler if the ExternalProject.target public attribute were accessible in meson, but it doesn't seem to be. That's all I really needed.

Since I need to have custom targets for the msbuild and (maybe not?) cmake builds on Windows, perhaps custom targets for ./configure; make; make install will work better than ExternalProject. I'm not sure.

[minrk]

zeromq/pyzmq#1922 has working builds, at least for the simple cases of Cython on linux/mac

[eli-schwartz]

https://pypi.org/project/pypkgconf/

[eli-schwartz]

Why is GitHub offering to translate that link from Portuguese to English.

[minrk]

Perfect, thanks! Looks like trading the compile in my existing script for emit_c_code is ~all I need to do.

[mattip]

A bit off-topic, but maybe adding a sample meson.build to the cffi documentation would be nice?

[kivkiv12345]

Using the linked meson.build as inspiration, I manged to bundle a library object with custom_target():

custom_target('<bundled_target_name>',
		output: '<bundled_target_file>.so',
		install: true,
		install_dir: py.get_install_dir() / '<bundled_target__init__py_subdir>',
		input: <bundled_lib>,
		command: ['cp', '@INPUT@', '@OUTPUT@'],
		build_by_default: true
	)

py.get_install_dir() fixes the meson-python: error: Could not map installation path to an equivalent wheel directory: '{prefix}' error. Funnily enough it points to /usr/local/lib/python3/dist-packages/, but the .whl is installed to ~/.local/lib/python3.12/site-packages/ which is quite neat.

[minrk]

Any idea how to prevent things from running on sdists? Meson is running various library detection, etc. steps during sdists, when of course it shouldn't be looking for any dependencies just to build a tarball. But I can't find any documentation about how to specify things to be run or not run during different stages, or even how to detect what stage is being run.

[dnicolodi]

AFAIK, this is the default behavior of Meson. If you want to include the subprojects source code, you need to specify the --include-subprojects option to meson dist.

[minrk]

Thanks for the help! I specifically don't want to include subprojects in the dist, so that's not what I'm after. I can see how dependency detection should run before dist if that were the case.

It seems like meson maybe isn't the tool for me, since its abstractions and strong opinions are fighting me at every turn.

[dnicolodi]

@minrk Yuo are not doing a very good job in explaining what you are trying to do, how you are trying to do it, and what does not work as expected. Therefore it is extremely difficult to help you. Simply stating that the solution you have tried does not work for solving your problem does not help without understanding what problem you are trying to solve.: you may be approaching the problem in the wrong way.

In the specific case of building an sdist, I don't understand where your problem is coming from. Apparently you are trying to build an sdist on a system where meson setup for your project fails. The first question is: why are you trying to build sdists on such a system? I don't see any reason to do so. The second question is: why is meson setup failing?

[eli-schwartz]

but if I can tell meson to skip looking for dependencies when building an sdist, that would probably be enough. But I can't seem to figure out what I can tell meson and/or what meson can tell me to distinguish the two runs.

The simple and straightforward way to do that would be to add a new meson option -Dsdist-stub=true that includes whatever meson.build content you need to make a sdist correctly (this might just be "project()" if you don't have any dist scripts) and then uses meson's subdir_done() function inside a conditional -- and pass that argument to -Csetup-args=.... when building an sdist only.

This still invites the question of what system are you trying to build an sdist on, if you specifically don't want to build a wheel on that system, and "maybe you should just not do that?" -- I do understand that that question is a personal opinion.

[eli-schwartz]

Also, to be clear -- meson is not strongly opinionated on how you should distribute your source tarballs, but includes a meson equivalent to the traditional autotools ./configure && make distcheck and this includes doing a full test run of the created tarball by default.

There are people who use meson and don't use meson dist :) but since python requires a dist tarball for PyPI and also requires the build backend information that isn't easily available otherwise be baked into the tarball, this is more complex than it has to be.

In fact, I'd argue that python is the opinionated thing here. Python is the thing saying "there is only one way to release sdists and that is to go through the full power of a build backend including the calculation of dynamic metadata, and expose that to all users as the first step of compiling the code".

[minrk]

There are people who use meson and don't use meson dist

Indeed, looks like I'd like to be one of those people! Maybe that's a feature request for meson-python - to do a more Python-friendly sdist instead of meson dist. I'm guessing I'm in the very small minority where it makes a difference, though, and a clear enough workaround is fine for me.

pass that argument to -Csetup-args=.... when building an sdist only.

I think that's what I was looking for, thank you, as long as I can put this in pyproject.toml so meson-python always passes it during sdists. That's firmly on meson-python's side of things, rather than meson itself. The important thing is that it isn't passed on the command-line by me, it's specified in pyproject.toml, so it happens consistently.

scikit-build-core has SKBUILD_STATE, to tell the build files a bit about what's going on, if they need to act on it. A similar signal from meson-python seems quite sensible and practical to me.

Yuo are not doing a very good job

Nice. I did my best to describe the use case and linked to the code itself and the builds that are failing, but to reiterate, I am trying to:

1. build two dependencies with autotools (optionally, as fallback)
2. one of those must not run configure before build completes for the other because it is a dependency (pyzmq -> libzmq -> libsodium)
3. neither should run configure during sdist (both because it's irrelevant, and because one has to be built before the other can run configure)

Right now, sdist is failing because ExternalProject is running configure for the second autotools project as part of setup without having built the first. Neither autotools project will be in the dist, so all of this is irrelevant anyway, which is why I want to skip it.

I think perhaps a purely custom target could do the job where ExternalProject doesn't seem to have the necessary controls.

[rgommers]

Indeed, looks like I'd like to be one of those people! Maybe that's a feature request for meson-python - to do a more Python-friendly sdist instead of meson dist.

I've heard this request a couple of times now, so I'll see if I can come up with a good problem summary in a new issue with an enhancement request. There are certainly trade-offs here that are being made. We get functionality for free from Meson, and dist scripts in particular can solve things that required hacky build_ext subclasses with distutils/setuptools. But on the other hand, the way things work now can be slower and at least surprising to folks coming from other Python build backends.

one of those must not run configure before build completes for the other because it is a dependency (pyzmq -> libzmq -> libsodium)

Just checking: this does work fine when you build a wheel? So it is possible to do configure + build for subproject 1 before configuring subproject 2? I imagine so because it's documented at https://mesonbuild.com/Subprojects.html#subprojects-depending-on-other-subprojects, but if that case prevents meson dist from working as intended, maybe that should be documented.

[minrk]

Just checking: this does work fine when you build a wheel?

It had, but possibly not in the failing environment, because when it succeeded, it is possible the libzmq build may have found already-installed libsodium elsewhere on my system, in which case it wouldn't fail, but would produce an incorrect result.

Ultimately, I've gone with cmake via scikit-build-core, which has most of the same challenges, but it happened to be easier for me to muddle my way toward what i needed, which turned out to be:

1. actually build static libsodium during the 'setup' step. This violates the intended separation of setup and build, but works out in practice because the 'setup' phase is exclusively run immediately prior to actual wheel builds and has all the same info available to do it, and is required because the 'setup' of libzmq requires libsodium to be built already
2. import libzmq-static cmake targets using FetchContent after libsodium is installed

Now that I've done it with cmake, I imagine the same approach ought to work in meson. I'd just need to specify the stub setup for sdist from pyproject.toml, and to understand how to import cmake into meson, which seems straightforward.

[eli-schwartz]

How does the cmake setup handle using a system libzmq (mandatory for redistributors that ship their own dependency management instead of using PyPI wheels)?

actually build static libsodium during the 'setup' step. This violates the intended separation of setup and build, but works out in practice because the 'setup' phase is exclusively run immediately prior to actual wheel builds and has all the same info available to do it, and is required because the 'setup' of libzmq requires libsodium to be built already

This is also a bit awkward as it doesn't parallelize well -- and parallelism is one of the biggest advantages of moving to a build system that groks C/C++ at a more sophisticated level than the very basics as handled by setuptools.

[minrk]

How does the cmake setup handle using a system libzmq (mandatory for redistributors that ship their own dependency management instead of using PyPI wheels)?

Basically the same as dependency('libzmq') and cc.find_library('libzmq') in the meson build. It looks for cmake config files, pkg-config, or finally a shared library on a path, and links with what it finds. This is the most common case.

The static build is only the 'fallback', used when no suitable system libzmq is found.