Tracking issue for manylinux_2_34 image

[h-vetinari]

As time goes by, projects start using new glibc features not present in old versions, hence requiring a newer manylinux baseline version to be defined.

For example:

• Nightly wheels broken triton-lang/triton#2616
• Error Repairing Wheel to manylinux2014_x86_64 ABI Due to Too-Recent Versioned Symbols #1574

Also, according to the manylinux-timeline, around ~40% of users on non-EOL Python versions have better than glibc 2.34 already.

IMO the discussion in #1282 (and the failure of Debian-based 2_24, c.f. #1012) showed comprehensively that we should stick with a RHEL-derivative also for the next manylinux standard - the devtoolset backports make all the difference w.r.t. to longevity, and all those flavours are ABI compatible.

This would also continue the pattern for all major manylinux standards so far (again excepting 2_24):

manylinux	glibc	RHEL
manylinux1	2.5	5
manylinux2010	2.12	6
manylinux2014	2.17	7
manylinux_2_28	2.28	8
manylinux_2_34	2.34	9

Updating the flavour table from #1282:

The big question is what base image to use. We can choose between C9S (centos:stream9), AlmaLinux, RockyLinux, or UBI. Personally I would prefer an image based entirely on open source software. [...]

Obviously Alma (as the base for 2_28) is a strong candidate, though since then, RHEL has announced it will not provide its sources anymore, creating some doubts initially how Alma will continue, though in the end, they decided they have everything they need to keep going. They'll keep ABI-compatibility (which is by far the most important), but will drop bug-for-bug compatibility (which is IMO not a big deal for us). Still, this might influence the decision; at the time of #1282, the main argument against using UBI was the lack of a gcc-toolset, but this has since been added (both to UBI 8 & 9).

base image	EOL¹	gcc-toolset-13²	aarch64	ppc64le	s390x	bug fix availability
CentOS Stream 9	2027-05-31³	yes	yes³	yes³	yes³	1st
UBI 9	2032-05-31	yes⁴	yes⁵	yes⁵	yes⁵	2nd
AlmaLinux 9	2032-05-31	yes	yes⁶	yes⁶	yes⁶	3rd
RockyLinux 9	2032-05-31	yes	yes⁷	yes⁷	yes⁷	3rd

^{1: including Maintenance Support, https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates
2: https://pkgs.org/download/gcc-toolset-13
3: https://www.centos.org/centos-stream/
4: at least gcc 12, probably on gcc 13 already
5: https://hub.docker.com/r/redhat/ubi9-minimal/tags
6: https://github.com/AlmaLinux/cloud-images / https://hub.docker.com/r/almalinux/9-minimal
7: https://hub.docker.com/r/rockylinux/rockylinux/tags}

---

The checklist is short, since wheel and pip, which both use packaging, and warehouse do not require additional PRs.

Publisher-side Support:

• feat: Add manylinux_2_34 image #1698
• Find BETA testers to check wether changes are required immediately, especially C++ projects have a chance to get a failure on repair, knowing how much they're affected will help to prioritize work on auditwheel.
• Upload the new images to quay.io

Documentation:

• ???

Additional projects to double-check for support, perhaps these are already done?

• Add an option to target manylinux_2_34 in cibuildwheel: feat: add manylinux_2_34 cibuildwheel#2098
• Add an option to target manylinux_2_34 in multibuild
• Add an option to target manylinux_2_34 in maturin -- likely completed by Add manylinux_2_34 and manylinux_2_35 policies PyO3/maturin#1273 already
• Distinguish between images in dockcross

[mayeut]

Thanks for the tracking issue.

manylinux_2_34 is already supported by auditwheel so there's no need for a new version: pypa/auditwheel#405

While UBI added support for gcc-toolset, it might still not support some -devel packages so, IMHO, it's fine to use AlmaLinux for now as is done for manylinux_2_28.

They'll keep ABI-compatibility (which is by far the most important), but will drop bug-for-bug compatibility (which is IMO not a big deal for us).

Indeed, we only aim for ABI-compatibility.

[snnn]

showed comprehensively that we should stick with a RHEL-derivative
I hold a different opinion on this and just left a few comments in the original issues to explain why. Mainly,

1. RHEL is not free. Neither the derivations can provide enough security updates for free.
2. From supply chain point of view you would not want to bind to a single supplier

I would suggest not going further on this road. Instead, give the existing users enough time to transit to an alternative solution. For example, as you said half of the pypi users already have glibc>=2.34, which means they also have a decent GCC compiler and very new libstdc++, which dims the need of using devtoolset when time goes by.

[h-vetinari]

From supply chain point of view you would not want to bind to a single supplier

So far all relevant¹ manylinux images have been based on RHEL-derivatives, and this system has worked very well. Red Hat itself - though a single vendor - is about as far as possible from a supply chain risk as any single company can be.

Your proposal in #1012 is not feasible (in my opinion). Your argument against UBI in #1282 may be correct, but then we can still stay with alma (which will fix security issues also in the rpms).

For example, as you said half of the pypi users already have glibc>=2.34, which means they also have a decent GCC compiler and very new libstdc++, which dims the need of using devtoolset when time goes by.

Try telling maintainers that they should halve their user base. Said differently, a lot of projects care about providing support that is a broad as possible across many different uses (many of which get stuck on old distros for whatever reason), and raising their glibc baseline is done extremely conservatively. This creates an obvious tension between needing to support old systems while wanting to use +/- contemporary compilers. The only sustainable solution to this over the relevant lifecycles has been the devtoolset.

Footnotes

1. I'm deliberately not counting 2_24. ↩

[snnn]

Functionally the existing Alma based images work very well, but if you run a security scanner to see if any of the components in the images needs be patched, it will be a different story. So my concern is more from supply chain point of view. If the toolchains you use has a known vulnerability, it could be get used to inject a backdoor in the package you are building. We do not see a reliable way to get security patches for the RHEL-derivatives, mainly because Red Hat doesn't want to provide them for free.

[dralley]

RHEL is not free. Neither the derivations can provide enough security updates for free.

RHEL Universal Base Image is free (and redistributable), fwiw.

https://catalog.redhat.com/software/base-images

I believe many of the downsides of CentOS Stream were addressed since the previous discussion also, although the 5 year lifecycle remains. For that reason either UBI or Alma images would (I assume) probably be preferred.

[snnn]

Yes, the image is free. But anything users installed to the image are not covered. manylinux_2_34 needs to install GCC compiler and a lot of other software from dnf repos. These software may or may not be able to get security updates, that's my biggest concern. For example, GCC depends on GMP. The gmp package in UBI's repo has a known security vulnerability. The UBI8 base image is well patched, but the packages hosted in UBI8's official repos are not. If manylinux doesn't need to manually install any RPM package(i.e. an official UBI image can provide everything we need), I would be less concerned.

[dralley]

@carlwgeorge ^ can you speak to those concerns

[carlwgeorge]

Your argument against UBI in #1282 may be correct,

They're not, please see my other reply for the full breakdown.

We do not see a reliable way to get security patches for the RHEL-derivatives, mainly because Red Hat doesn't want to provide them for free.

For packages in the UBI content set, Red Hat does provide them, for free, under freely redistributable terms. If something needs to be added to that content set for UBI to be a viable solution, I'm happy to advocate for that.

Yes, the image is free. But anything users installed to the image are not covered. manylinux_2_34 needs to install GCC compiler and a lot of other software from dnf repos. These software may or may not be able to get security updates, that's my biggest concern. For example, GCC depends on GMP. The gmp package in UBI's repo has a known security vulnerability. The UBI8 base image is well patched, but the packages hosted in UBI8's official repos are not. If manylinux doesn't need to manually install any RPM package(i.e. an official UBI image can provide everything we need), I would be less concerned.

This is all bogus, for the reasons I described in my other reply. For that GMP security vulnerability, do you know who has that patched? CentOS Stream 8, RHEL 8.8 EUS, and RHEL 8.6 EUS. That's it. RHEL 8.9, Alma 8, and Rocky 8 don't have it. Somehow that fix snuck out in the UBI 8 image (not the repos) ahead of the RHEL 8.10 release, which is likely a mistake but not evidence of UBI repo package not being maintained.

[snnn]

Great to know that. Thanks!

[ncoghlan]

From a docs PoV, it would be helpful to have a concise list of the key distro versions shipping glibc 2.34 or later (similar to the list I am suggesting to add for manylinux_2_28 and manylinux2014 in #1635 )

For glibc 2.34, that would be:

• Debian 12+
• Ubuntu 22.04+
• Fedora 35+
• CentOS/RHEL 9+

[mayeut]

I'd like to wait for gcc-toolset-14 for this one.
In any case, given the CI state of things, we need to wait for either #1619 or #1629.

[h-vetinari]

If musl 1.1 was already at 2.5% (of musllinux!) downloads ~2 months ago, is it really necessary/beneficial to wait until November to remove it?

Independently of that, I don't see why the release of other 2_34 images would necessarily need to wait for musllinux; those could come later as constraints get resolved.

[EwoutH]

Nice milestone: glibc 2.34 is now supported over 50% of consumer systems using Python.

glibc 2.35 is also supported on more than half of the systems, probably since Ubuntu 22.04 has glibc 2.35 bundled by default (and has a huge marketshare).

[EwoutH]

I'd like to wait for gcc-toolset-14 for this one.

Seems like something is in the pipeline: https://gitlab.com/redhat/centos-stream/rpms/gcc-toolset-14-gcc

Where would a published version appear?

[carlwgeorge]

CentOS Stream 9 has gcc-toolset-14 available for installation right now.

root@c9-container:~# dnf info gcc-toolset-14-gcc
Last metadata expiration check: 0:03:03 ago on Mon Aug 26 16:35:18 2024.
Available Packages
Name         : gcc-toolset-14-gcc
Version      : 14.2.1
Release      : 1.1.el9
Architecture : x86_64
Size         : 47 M
Source       : gcc-toolset-14-gcc-14.2.1-1.1.el9.src.rpm
Repository   : appstream
Summary      : GCC version 14
URL          : http://gcc.gnu.org
License      : GPL-3.0-or-later AND LGPL-3.0-or-later AND (GPL-3.0-or-later WITH GCC-exception-3.1) AND (GPL-3.0-or-later WITH Texinfo-exception) AND
             : (LGPL-2.1-or-later WITH GCC-exception-2.0) AND (GPL-2.0-or-later WITH GCC-exception-2.0) AND (GPL-2.0-or-later WITH GNU-compiler-exception) AND
             : BSL-1.0 AND GFDL-1.3-or-later AND Linux-man-pages-copyleft-2-para AND SunPro AND BSD-1-Clause AND BSD-2-Clause AND BSD-2-Clause-Views AND
             : BSD-3-Clause AND BSD-4-Clause AND BSD-Source-Code AND Zlib AND MIT AND Apache-2.0 AND (Apache-2.0 WITH LLVM-Exception) AND ZPL-2.1 AND ISC AND
             : LicenseRef-Fedora-Public-Domain AND HP-1986 AND curl AND Martin-Birgmeier AND HPND-Markus-Kuhn AND dtoa AND SMLNJ AND AMD-newlib AND OAR AND
             : HPND-merchantability-variant AND HPND-Intel
Description  : The gcc-toolset-14-gcc package contains the GNU Compiler Collection
             : version 14.

[mayeut]

I'll most likely merge the new image next week unless there are reviews delaying that.
It will use gcc-toolset-13 for now as there's no clear planning for gcc-toolset-14 (as a side note, since the comment in #1698, gcc-toolset-14 is available in AlmaLinux 8 but not in AlmaLinux 9).

[carlwgeorge]

RHEL 9.5 was released last week, so UBI 9 has gcc-toolset-14 now. This confirms the availability ordering described in the chart in the first comment.

• CentOS Stream 9, GCC 14 available since August
• UBI 9, GCC 14 available since last week
• AlmaLinux 9, GCC 14 not available and no ETA
• Rocky Linux 9, GCC 14 not available and no ETA

Is there anything missing from UBI that prevents it from being used?

[richli]

FYI, AlmaLinux released 9.5 today and it includes gcc-toolset-14