Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any roadmap plans to make tick compatible with Apple silicon chips ? #520

Open
mkaanerkoc opened this issue Feb 15, 2024 · 19 comments
Open

Comments

@mkaanerkoc
Copy link

Hey everyone,

I wanted to install "tick" on my MacBook Air with an M1 processor, but I found out that it relies on the MKL library, which isn't compatible with my laptop.

I'm wondering if there's a way to still run tick on my computer, even if it's not super fast—I don't mind sacrificing some performance for it to work.

Can anyone help me figure this out?

@PhilipDeegan
Copy link
Member

PhilipDeegan commented Feb 15, 2024

I'm fairly positive that MKL is an optional dependency.
We don't as of yet have a Apple ARM distribution, mainly as github does not yet have free M1 github action runners

if you would like to try build on ARM I can help you try that.

@mkaanerkoc
Copy link
Author

Hey Philip!
Thanks for your response. Indeed quickly looking into build files I see that MKL is can be disabled via flags. I will look into by myself and gonna poke you if I need any help out there !

@fpavone
Copy link

fpavone commented Feb 22, 2024

Hi, I have the same problem.
@mkaanerkoc, what is the procedure to install it disabling MKL?

Thank you

@Roh-codeur
Copy link

hi, did anyone find a solution to this please? ta!

@PhilipDeegan
Copy link
Member

@Roh-codeur

If you see the pull request here https://github.com/X-DataInitiative/tick/pull/511/files#diff-f619751a4b718590788fed0b6056ac9750f93c013539291fbef6abdc3bcfa72bR55-R57

You should be able to build it yourself on ARM/macos

@Roh-codeur
Copy link

@PhilipDeegan : thanks a lot mate, I will give it a shot

@Roh-codeur
Copy link

I am afraid it didnt work

clang++ -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -Iinclude -Ithird_party/cereal/include -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/eb58e3dd0a08835e-normal_distribution.cpp.o -c cpp/base/math/normal_distribution.cpp
clang++ -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -Iinclude -Ithird_party/cereal/include -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/1154878851c2ac1e-t2exp.cpp.o -c cpp/base/math/t2exp.cpp
clang++ -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -Iinclude -Ithird_party/cereal/include -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/fc5415f1802bbce8-interruption.cpp.o -c cpp/base/interruption.cpp
clang++ -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -Iinclude -Ithird_party/cereal/include -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/8a24a07454c4fca2-exceptions_test.cpp.o -c cpp/base/exceptions_test.cpp
clang++ -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -Iinclude -Ithird_party/cereal/include -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/a1c15a0b62e666b4-time_func.cpp.o -c cpp/base/time_func.cpp
LINKER ARGUMENTS
	 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
Undefined symbols for architecture arm64:
  "_PyMem_RawFree", referenced from:
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::clear() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      ...
  "_PyMem_RawMalloc", referenced from:
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
  "__Py_Dealloc", referenced from:
      SArray<double>::_clear() in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::~SArray() in a1c15a0b62e666b4-time_func.cpp.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/Users/runner/.maiken/repo/mkn/kul/master/inc/mkn/kul/except.hpp 122 Process exit code: 1
Process exit code: 1
clang++ -shared -o /Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build/lib_base.cpython-36m-x86_64-linux-gnu.so bin/base/obj/*.o -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build -l_array.cpython-36m-x86_64-linux-gnu -Wl,-rpath,/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -Wl,-rpath,@loader_path/../../array/build -Wl,-install_name,@rpath/lib_base.cpython-36m-x86_64-linux-gnu.so -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
clang++ -shared -o /Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build/lib_base.cpython-36m-x86_64-linux-gnu.so bin/base/obj/*.o -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build -l_array.cpython-36m-x86_64-linux-gnu -Wl,-rpath,/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -Wl,-rpath,@loader_path/../../array/build -Wl,-install_name,@rpath/lib_base.cpython-36m-x86_64-linux-gnu.so -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2%                                                                                          

@PhilipDeegan
Copy link
Member

I forgot to do something for general use.

You should try exporting these variables first

export MKN_PYTHON_LIB_EMBED=1
export MKN_LIB_LINK_LIB=1

As seen here https://github.com/X-DataInitiative/tick/pull/511/files#diff-f619751a4b718590788fed0b6056ac9750f93c013539291fbef6abdc3bcfa72bR16

@Roh-codeur
Copy link

thanks for the point. I did try with that mate, didnt work. if I run the command as is:

./mkn -C lib build -g 0 -dtOa "-fPIC -std=c++17"

or
./mkn -C lib build -g 0 -dtOa "-fPIC -std=c++17" -F /Users/s-pass/miniconda/miniconda3/include/python3.12/ I get:

In file included from cpp/array/alloc.cpp:3:
In file included from include/tick/array/alloc.h:8:
In file included from include/tick/base/debug.h:23:
include/tick/base/defs.h:14:10: fatal error: 'Python.h' file not found

@PhilipDeegan
Copy link
Member

@Roh-codeur sorry I haven't tested any conda config

But you're on the right track trying to add the header directory

You can get more logging to see the compile commands with

export KLOG=3

To confirm your including the correct header directory. Normally it uses python3-config to find the header directories. If you don't have it you might have issues

@Roh-codeur
Copy link

Roh-codeur commented Aug 16, 2024

sorry to bother again and thanks for your patience. it appears that I have something in the lib which is x86 and not ARM compatible - do you think so?

Updated command:

./mkn -C lib build -g 0 -dtOa "-fPIC -std=c++17" -b /Users/s-pass/miniconda/miniconda3/include/python3.12


clang++ -Iinclude -Ithird_party/cereal/include -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -DNDEBUG -fwrapv -O2 -Wall -fstack-protector-strong -Wformat -Werror -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -ffast-math -Wno-uninitialized -fPIC -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/base/obj/a1c15a0b62e666b4-time_func.cpp.o -c cpp/base/time_func.cpp
LINKER ARGUMENTS
	 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
Undefined symbols for architecture arm64:
  "_PyMem_RawFree", referenced from:
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::clear() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      Array<double>::~Array() in a1c15a0b62e666b4-time_func.cpp.o
      ...
  "_PyMem_RawMalloc", referenced from:
      SArray<double>::SArray(unsigned long long) in a1c15a0b62e666b4-time_func.cpp.o
  "__Py_Dealloc", referenced from:
      SArray<double>::_clear() in a1c15a0b62e666b4-time_func.cpp.o
      SArray<double>::~SArray() in a1c15a0b62e666b4-time_func.cpp.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/Users/runner/.maiken/repo/mkn/kul/master/inc/mkn/kul/except.hpp 122 Process exit code: 1
Process exit code: 1
clang++ -shared -o /Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build/lib_base.cpython-36m-x86_64-linux-gnu.so bin/base/obj/*.o -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -L/Users/s-pass/dev/projects/third-party/third-part/tick/tick/base/build -l_array.cpython-36m-x86_64-linux-gnu -Wl,-rpath,/Users/s-pass/dev/projects/third-party/third-part/tick/tick/array/build -Wl,-rpath,@loader_path/../../array/build -Wl,-install_name,@rpath/lib_base.cpython-36m-x86_64-linux-gnu.so -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2


@PhilipDeegan
Copy link
Member

@Roh-codeur I think you're on the wrong branch.

You should be using this branch

https://github.com/PhilipDeegan/tick/tree/releases

And python3.6 is too old, you should use maybe 3.9

@Roh-codeur
Copy link

Roh-codeur commented Aug 17, 2024

  1. So, moved to the repo and branch
  2. Ran the below command:
    ./mkn -C lib clean build -g 0 -dtOa "-fPIC -std=c++17" -b /Users/s-pass/miniconda/miniconda3/include/python3.12 -L /Users/s-pass/miniconda/miniconda3/Lib
    Looks like the issue could be with the python version? which version should I use?
#include "..." search starts here:
#include <...> search starts here:
 include
 third_party/cereal/include
 /Users/s-pass/miniconda/miniconda3/include/python3.12
 /Users/s-pass/miniconda/miniconda3/lib/python3.12/site-packages/numpy/core/include
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
clang++ -Iinclude -Ithird_party/cereal/include -I/Users/s-pass/miniconda/miniconda3/include/python3.12 -I/Users/s-pass/miniconda/miniconda3/lib/python3.12/site-packages/numpy/core/include -DBUILDING_DLL -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_FILE_OFFSET_BITS=64 -DPYTHON_LINK -fwrapv -fstack-protector-strong -Wformat -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++17 -Wno-uninitialized -fPIC -v -std=c++17 -DNDEBUG -O3 -funroll-loops -march=native -mtune=native -o bin/tick.py/obj/206456e765e1dbe1-prox_l2sq.cpp.o -c cpp/prox/prox_l2sq.cpp
LINKER ARGUMENTS
	 /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -ldl  -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
ld: file cannot be mmap()ed, errno=22 path=/Users/s-pass/miniconda/miniconda3/Lib in '/Users/s-pass/miniconda/miniconda3/Lib'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/Users/runner/.maiken/repo/mkn/kul/master/inc/mkn/kul/except.hpp 122 Process exit code: 1
Process exit code: 1
clang++ -shared -o /Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
clang++ -shared -o /Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2%                                                    

@PhilipDeegan
Copy link
Member

PhilipDeegan commented Aug 18, 2024

@Roh-codeur

We can see the macos ARM python3.12 build is linking on my PR

clang++ -shared -o /Users/runner/work/tick/tick/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/runner/work/tick/tick/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so -L/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/config-3.12-darwin -lpython3.12 -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2

where as yours is

clang++ -shared -o /Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2

which doesn't have the -lpython3.12

I've updated my branch so you don't need to export those MKN_* variables, as I guess you're missing that

@Roh-codeur
Copy link

thanks a lot for your help with this.

so, I do export the MKN_* variables. I suspect the issue is with the -lpython3.12. I am not sure what this should point to. - any idea pls.

/Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -lpython3.12 -ldl  -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
ld: library 'python3.12' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)
/Users/runner/.maiken/repo/mkn/kul/master/inc/mkn/kul/except.hpp 122 Process exit code: 1
Process exit code: 1
clang++ -shared -o /Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -lpython3.12 -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
clang++ -shared -o /Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp/_tick_cpp.cpython-312-darwin.so bin/tick.py/obj/*.o -L/Users/s-pass/dev/projects/third-party/tick-fork/tick/tick_cpp -Wl,-install_name,@rpath/_tick_cpp.cpython-312-darwin.so /Users/s-pass/miniconda/miniconda3/Lib -L/Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin -lpython3.12 -ldl -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2%                                       

@PhilipDeegan
Copy link
Member

PhilipDeegan commented Aug 18, 2024

so, I do export the MKN_* variables.

You shouldn't need to anymore. Assuming you took the latest commit on my branch.

Is there a file like libpython3.12.so or libpython3.12.dynlib in the directory /Users/s-pass/miniconda/miniconda3/lib/python3.12/config-3.12-darwin ?

You might not have the development package which provides the shared library. I'm not a mac (or conda) person tho

you could try something like find /Users/s-pass/miniconda -name "libpython*"

@Roh-codeur

@Roh-codeur
Copy link

I took your latest changes mate.

libpython3.12.dynlib exists in /Users/s-pass/miniconda/miniconda3/lib

Do you think I am running the right command?

./mkn -C lib clean build -g 0 -dtOa "-fPIC -std=c++17" -b /Users/s-pass/miniconda/miniconda3/include/python3.12 -L /Users/s-pass/miniconda/miniconda3/lib -l /Users/s-pass/miniconda/miniconda3/lib/

@PhilipDeegan
Copy link
Member

maybe try

./mkn -C lib clean build -g 0 -dtOa "-fPIC -std=c++17" -b /Users/s-pass/miniconda/miniconda3/include/python3.12 -B /Users/s-pass/miniconda/miniconda3/lib

@Roh-codeur
Copy link

Roh-codeur commented Aug 18, 2024

maybe try

./mkn -C lib clean build -g 0 -dtOa "-fPIC -std=c++17" -b /Users/s-pass/miniconda/miniconda3/include/python3.12 -B /Users/s-pass/miniconda/miniconda3/lib

this worked!! thanks a lot mate! what do I do next to install the package itself please? thanks!

Update:
I tried the below from the home directory of the project, seems to work. thanks again for your help in setting this up! ta!

python setup.py install

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants