undefined symbol: EVP_idea_cbc, version OPENSSL_1_1_0

[lazzarello]

This has the same title as #9 mentioned by @dehann because I have the same problem. I found a work around and I can probably submit a patch...when we can figure out how and where it's actually happening. Read through the linked issue for background, which I won't repeat here.

The players:

• OpenSSL
• OpenSSL_jll
• PyCall
• py-julia

Base OS is Ubuntu 20.04 but I'm also testing it on Alpine Linux to build inside a docker container via a CI system. Let's stick with Ubuntu for this example. On my computer, here's all the things with the name libssl.so in them.

/home/lee/.julia/artifacts/3a83c270789c2c05ff731923907ab1ccac10dfeb/lib/libssl.so
/home/lee/.julia/artifacts/3a83c270789c2c05ff731923907ab1ccac10dfeb/lib/libssl.so.1.1
/home/lee/.julia/artifacts/3a83c270789c2c05ff731923907ab1ccac10dfeb/logs/OpenSSL/update_linkage_libssl.so.1.1_libcrypto.so.1.1.log.gz
/home/lee/.julia/artifacts/3a83c270789c2c05ff731923907ab1ccac10dfeb/logs/OpenSSL/update_linkage_openssl_libssl.so.1.1.log.gz
/home/lee/.julia/artifacts/3a83c270789c2c05ff731923907ab1ccac10dfeb/logs/OpenSSL/update_rpath_libssl.so.1.1_libcrypto.so.1.1.log.gz
/home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so
/home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so.1.1
# ... some stuff in snapcraft we can ignore
/usr/lib/x86_64-linux-gnu/libssl.so
/usr/lib/x86_64-linux-gnu/libssl.so.1.1

The last two on the list seem important, as those are what the OS distributes but it looks like the Julia binary is pulled from this package on install time and it is distributing its own C shared lib. That's the one that throws the EVP_idea_cbc symbol error. I suspect because it was compiled differently. What if...I just replace the Julia one? Let's try!

lee@MS3ZRMZ-LT:~$ ls -l /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so
lrwxrwxrwx 1 lee lee 13 Dec 18 21:33 /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so -> libssl.so.1.1
lee@MS3ZRMZ-LT:~$ rm /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so
lee@MS3ZRMZ-LT:~$ ln -s /usr/lib/x86_64-linux-gnu/libssl.so.1.1 /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so
lee@MS3ZRMZ-LT:~$ ls -l /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so
lrwxrwxrwx 1 lee lee 39 Jan 14 01:21 /home/lee/.julia/artifacts/53037ac9d528ee46c3526799b407ee52b7c224f3/lib/libssl.so -> /usr/lib/x86_64-linux-gnu/libssl.so.1.1

That removes the symbol error!

This obviously sucks and I spent about 6 hours debugging this across a number of projects in my organization. We're doing earth orbital simulation through custom Julia packages (some planned to be open source!) which are then called from Python data APIs using py-julia, which uses PyCall behind the scenes.

All three bug reports I've found in these projects mention either PyCall or py-julia. In my case, I suspect libssl is being called because I have some initialization function that downloads planetary body data models. That's pure speculation and really difficult to debug due to this data passing back and forth through eval().

PyCall has quite an essay under the installing section devoted to how a python interpreter is picked from Julia. I suspect this might be my problem and/or close to the other people's problems.

In my case I'm using pyenv and I have many python interpreters. I'm also using the poetry dependency manager, which does some python path tricks that I don't fully understand.

but I digress

It seems like it could be helpful for this library to search through the base OS for libssl.so before using the binary from the JLL package. That's the best I can do for now. I expect to reference this issue across the other projects, get thrown around from one project to the next and perhaps post on the Julia discourse. But I'm starting here.

[quinnj]

cc: @giordano in case he has any insights here

[giordano]

There isn't much to do when you mix up libraries from different sources: if they're incompatible you're out of luck.

[lazzarello]

I don't think this has anything to do with luck. It looks like all the players are in this room and given some time and attention we can fix this. I'll look into removing the custom compiled libssl.so in the OpenSSL_jll package later this week. @giordano are you the maintainer of this package?

[lazzarello]

UPDATE! with good news

Downgrading to a version of OpenSSL_jll from around the same time as the system library fixed the problem. The source of this problem is that the Julia binary wrapper for the C openssl code is too new to work with many versions of python, which needs to be compiled as a shared library to maintain compatibility with py-julia.

After installing all my dependencies I just ran julia -e 'using Pkg; Pkg.add(name="OpenSSL_jll", rev="OpenSSL-v1.1.1+2")' and the undefined symbol: EVP_idea_cbc error stopped. My tests pass once again! It seems the latest release from Nov 2022 removed this symbol, thus the scarcity of bug reports.

I don't know what this symbol means for the library other than that I know what a CBC algorithm is. That might be interesting to understand better.