SIGILL in gmp on Bulldozer CPU

[Jannik2099]

I'm getting a SIGILL in libgmp.so.10.4.1 on a Bulldozer CPU. FreeCAD 0.20.1.29410
This happens with everything that uses gnutls, which includes git, so installing addons becomes a bit tedious.

Since there's no debug info available I have no idea how to get the invalid instruction.
I also don't remember gmp having any x86 asm beyond the baseline (i.e. no AVX2 which this cpu wouldn't support), so I have no idea how this'd happen.

Here's the output from coredumpctl info

           PID: 98423 (wget)
           UID: 1000 (jannik)
           GID: 1000 (jannik)
        Signal: 4 (ILL)
     Timestamp: Thu 2022-09-15 00:16:20 CEST (2s ago)
  Command Line: wget https://google.com
    Executable: /usr/bin/wget
 Control Group: /user.slice/user-1000.slice/[email protected]/app.slice/app-flatpak-org.freecadweb.FreeCAD-97506.scope
          Unit: [email protected]
     User Unit: app-flatpak-org.freecadweb.FreeCAD-97506.scope
         Slice: user-1000.slice
     Owner UID: 1000 (jannik)
       Boot ID: 69d8057f8a654395b299e0a6e12b1b00
    Machine ID: dc142e85a0e3df2dde020300623eeb1f
      Hostname: j-galaxy
       Storage: /var/lib/systemd/coredump/core.wget.1000.69d8057f8a654395b299e0a6e12b1b00.98423.1663193780000000.zst (present)
     Disk Size: 324.4K
       Message: Process 98423 (wget) of user 1000 dumped core.
                
                Module /usr/bin/wget with build-id dd1bdb70c26a1c7afe96ebd83ce747e95fb0c7cd
                Module /usr/lib/x86_64-linux-gnu/libresolv-2.33.so with build-id 37fbb9160d1a1f7d122eba2548b81e405dc3575e
                Module /usr/lib/x86_64-linux-gnu/libnss_dns-2.33.so with build-id 01822e282b2ebbf5373d20b99266bf7fb2daf6c2
                Module /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 with build-id 6dca6d6f8e253228cab3d4469a04548560f18ce4
                Module /usr/lib/x86_64-linux-gnu/libselinux.so.1 with build-id 7fab7ef7348913e9d1e36ece2a659e6df1d5bf2a
                Module /usr/lib/x86_64-linux-gnu/librt-2.33.so with build-id b6408705e6ec4347ab6399c880847a98af3b1c0e
                Module /usr/lib/x86_64-linux-gnu/libnss_resolve.so.2 with build-id a22a218f16745447153e70306c08f24735cab9db
                Module /usr/lib/x86_64-linux-gnu/libnss_files-2.33.so with build-id e527a5806160253a6313f0e47fe5a110275185be
                Module /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-client.so with build-id 2e0a902fbe3ddd2bcb887c07ccb305c9f39d0162
                Module /usr/lib/x86_64-linux-gnu/libdl-2.33.so with build-id 20be4ecfd5346af55ccafadf4211c2c47c496b2b
                Module /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0 with build-id 9409f4a8a0ef373035bf02ad335a73a0970e16b7
                Module /app/lib/libgmp.so.10.4.1 with build-id ec3673c11622aa40a08cb2a65a594b7fb1717e35
                Module /usr/lib/x86_64-linux-gnu/libhogweed.so.6.4 with build-id da2e1a3c32a622eaadc09325ae6ddeddbdab16da
                Module /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.2 with build-id 2cd130a9b1864418413e41daaf17745b858cabc9
                Module /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0 with build-id b081092317b45d39c72a209bac13f85375b19da3
                Module /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 with build-id f2fd7d4d26aedc34cb5063ba89349ec45ec6e137
                Module /usr/lib/x86_64-linux-gnu/libpthread-2.33.so with build-id 97172665d1042add3adc9b120b0760258ba0aba5
                Module /usr/lib/x86_64-linux-gnu/libc-2.33.so with build-id 833fec3acaa8ba63dfea79852744153ba716f8cb
                Module /usr/lib/x86_64-linux-gnu/libpsl.so.5.3.3 with build-id 887ee05d92fbac1ff52dc4ddeea64c414b0b618b
                Module /usr/lib/x86_64-linux-gnu/libz.so.1.2.11 with build-id 5885119a2a8097b900bdb4ca40560e892ae7ac8c
                Module /usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 with build-id e46dfac77f54d1d788dded3a3aadc2a130e2e771
                Module /usr/lib/x86_64-linux-gnu/libnettle.so.8.4 with build-id c5330a4ecc25167802fa655726f88a5ca5bb2619
                Module /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.7 with build-id a65a0e54327f3f66c006d950adb03596844c1724
                Module /usr/lib/x86_64-linux-gnu/libuuid.so.1.3.0 with build-id 6b708151eb4355799495e777d3bd851ca5ffc868
                Module /usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.10.0 with build-id 32d1f4cf529f3d41fe6f6adf7bca7f0cccd6238b
                Module /usr/lib/x86_64-linux-gnu/ld-2.33.so with build-id 856401969318527d829336ec391155c2ebc785c5
                Module linux-vdso.so.1 with build-id fffeaf9d978a45b3de7673a54130ff9ce31ad7d1
                Stack trace of thread 69:
                #0  0x00007f69a65c5952 n/a (/app/lib/libgmp.so.10.4.1 + 0x5f952)
                flathub/org.freecadweb.FreeCAD#1  0x00007f69a65e6b6b n/a (/app/lib/libgmp.so.10.4.1 + 0x80b6b)
                flathub/org.freecadweb.FreeCAD#2  0x00007f69a65e6d89 n/a (/app/lib/libgmp.so.10.4.1 + 0x80d89)
                flathub/org.freecadweb.FreeCAD#3  0x00007f69a658e0f4 n/a (/app/lib/libgmp.so.10.4.1 + 0x280f4)
                flathub/org.freecadweb.FreeCAD#4  0x00007f69a661e071 n/a (/usr/lib/x86_64-linux-gnu/libhogweed.so.6.4 + 0x10071)
                flathub/org.freecadweb.FreeCAD#5  0x00007f69a661e921 n/a (/usr/lib/x86_64-linux-gnu/libhogweed.so.6.4 + 0x10921)
                flathub/org.freecadweb.FreeCAD#6  0x00007f69a6cc9003 n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0x153003)
                flathub/org.freecadweb.FreeCAD#7  0x00007f69a6bf4a56 n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0x7ea56)
                flathub/org.freecadweb.FreeCAD#8  0x00007f69a6bf8647 n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0x82647)
                flathub/org.freecadweb.FreeCAD#9  0x00007f69a6c62ee7 n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0xecee7)
                flathub/org.freecadweb.FreeCAD#10 0x00007f69a6c636dd n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0xed6dd)
                flathub/org.freecadweb.FreeCAD#11 0x00007f69a6c64a9a n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0xeea9a)
                flathub/org.freecadweb.FreeCAD#12 0x00007f69a6c6515e n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0xef15e)
                flathub/org.freecadweb.FreeCAD#13 0x00007f69a6c75ba7 n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0xffba7)
                flathub/org.freecadweb.FreeCAD#14 0x00007f69a6bfe17c n/a (/usr/lib/x86_64-linux-gnu/libgnutls.so.30.30.0 + 0x8817c)
                flathub/org.freecadweb.FreeCAD#15 0x000055add331c7c0 n/a (/usr/bin/wget + 0x457c0)
                flathub/org.freecadweb.FreeCAD#16 0x000055add32ffc7c n/a (/usr/bin/wget + 0x28c7c)
                flathub/org.freecadweb.FreeCAD#17 0x0000003000000010 n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64

Here's a lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               AuthenticAMD
  Model name:            AMD FX(tm)-8350 Eight-Core Processor
    CPU family:          21
    Model:               2
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            0
    Frequency boost:     enabled
    CPU max MHz:         4000.0000
    CPU min MHz:         1400.0000
    BogoMIPS:            8029.20
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 ss
                         e4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd vmmcall bmi1 arat npt lbrv svm_lock nrip_save ts
                         c_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold

The gmp configuration from the manifest.json

    {
      "name" : "gmp",
      "config-opts" : [
        "--enable-fat"
      ],
      "sources" : [
        {
          "url" : "https://gmplib.org/download/gmp/gmp-6.2.1.tar.bz2",
          "sha256" : "eae9326beb4158c386e39a356818031bd28f3124cf915f8c5b1dc4c7a36b4d7c",
          "type" : "archive"
        }
      ]
    }

[Jannik2099]

After some messing around in gdb:

The memory at $pc is 0x00000000, which of course would translate to an invalid instruction.
So it's not actually due to an illegal instruction, but some memory bug earlier. I'll do more debugging on another day

[hfiguiere]

Maybe GMP doesn't detect the CPU properly and returns a NULL for the SIMD optimized code....

--enable-fat is supposed to enable them all and pick at runtime.

I don't have that hardware to debug it. (only a 7th Gen Core i7)

[hfiguiere]

the coredump here is about a wget....

[hfiguiere]

since it's in /usr/bin, it shouldn't know about gmp in /app/lib

[Jannik2099]

the coredump here is about a wget....

Yes, the git backtrace was needlessly long so I took the next best thing that was using gnutls.

since it's in /usr/bin, it shouldn't know about gmp in /app/lib

None of the libraries in the backtrace are part of my system, so they must all be from the runtime? I'm not too familiar with runtimes, so no idea why they are in /usr/lib, but my distro (gentoo) places libraries in /usr/lib64, so it's definitely not system libraries.

I'll attach gdb and step through later, perhaps that will give me symbol names.

[Jannik2099]

Ah, seems the coredump was pretty useless. It is a real SIGILL after all!

flatpak run --command=bash --devel org.freecadweb.FreeCAD , and then:

gdb wget https://google.com
run
0x00007ffff76d3952 in __gmpn_tdiv_qr () from /app/lib/libgmp.so.10
x/i $pc
=> 0x7ffff76d3952 <__gmpn_tdiv_qr+3090>:        mulx   %rcx,%rdi,%rsi

mulx is an instruction from BMI2, but my CPU only does BMI1.
So either the detection in gmp is borked, or some of the seccomp filters is changing cpuid - have there been previous occurences of that in flatpak?

[davidschillerJKU]

I think this can be fixed by removing GMP from the build:
flathub/org.freecadweb.FreeCAD#97 (comment)

[adrianinsaval]

it seems that for the time being we need to keep building gmp due to flathub/org.freecadweb.FreeCAD#87 but it looks like this was recently fixed in freedesktop sdk: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/merge_requests/22298 however I don't know when that will land on a release or how to know if we have it.