OpenCL, Fedora 40 Server, RTX 4090 GPU and i9-13900K CPU

[Karm]

Hello,

I got lucky to have a GPU desktop at my desk and naturally I'd like to experiment with

./bin/tornadovm-installer --jdk mandrel-jdk-21 --backend opencl,ptx,spirv 

It's RTX 4090 GPU and i9-13900K CPU, Fedora 40.

I know my CUDA devel workflow is fine from other projects. And even TornadoVM seems happy about it.
It seems to have a problem with OpenCL though:

[INFO] tornado ............................................ SUCCESS [  0.615 s]
[INFO] tornado-api ........................................ SUCCESS [  1.982 s]
[INFO] tornado-runtime .................................... SUCCESS [  1.744 s]
[INFO] tornado-matrices ................................... SUCCESS [  0.581 s]
[INFO] tornado-drivers .................................... SUCCESS [  0.294 s]
[INFO] tornado-drivers-common ............................. SUCCESS [  0.552 s]
[INFO] tornado-drivers-opencl ............................. SUCCESS [  2.608 s]
[INFO] tornado-drivers-opencl-jni ......................... FAILURE [  2.584 s]
[INFO] tornado-drivers-spirv .............................. SKIPPED
[INFO] tornado-drivers-ptx ................................ SUCCESS [  1.384 s]
[INFO] tornado-drivers-ptx-jni ............................ SUCCESS [  3.377 s]
[INFO] tornado-examples ................................... SUCCESS [  1.062 s]
[INFO] tornado-benchmarks ................................. SUCCESS [  4.966 s]
[INFO] tornado-unittests .................................. SUCCESS [  9.489 s]
[INFO] tornado-annotation ................................. SUCCESS [  0.412 s]
[INFO] tornado-assembly ................................... SKIPPED

like...it's trying to find 3.0 while the OpenCL that came with my Nvidia Linux drivers is 1.0?

/home/karm/workspace/TornadoVM/tornado-drivers/opencl-jni/target/linux-amd64-release/sources/headers/include/CL/cl_version.h:22:104: note: '#pragma message: cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0)'
   22 | #pragma message("cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0)")
      |                                                                                                        ^
[100%] Linking CXX shared library libtornado-opencl.so
/home/karm/workspace/TornadoVM/etc/dependencies/cmake-3.25.2-linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/tornado-opencl.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC  -O2 -fpermissive -fPIC -export-dynamic -O3 -DNDEBUG -shared -Wl,-soname,libtornado-opencl.so -o libtornado-opencl.so "CMakeFiles/tornado-opencl.dir/source/OCLCommandQueue.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLContext.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLDevice.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLEvent.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLKernel.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLPlatform.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLProgram.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OCLNvidiaPowerMetricHandler.cpp.o" "CMakeFiles/tornado-opencl.dir/source/OpenCL.cpp.o" "CMakeFiles/tornado-opencl.dir/source/utils.cpp.o" "CMakeFiles/tornado-opencl.dir/source/opencl_time_utils.cpp.o" "CMakeFiles/tornado-opencl.dir/source/SPIRVOCLDispatcher.cpp.o"  -lOpenCL -L/usr/local/cuda/lib64/stubs -lnvidia-ml 
/usr/bin/ld: cannot find -lOpenCL: No such file or directory
collect2: error: ld returned 1 exit status

I do have 1.0:

$  locate libOpenCL.so
/usr/lib64/libOpenCL.so.1
/usr/lib64/libOpenCL.so.1.0.0
/usr/local/cuda-12.6/targets/x86_64-linux/lib/libOpenCL.so
/usr/local/cuda-12.6/targets/x86_64-linux/lib/libOpenCL.so.1
/usr/local/cuda-12.6/targets/x86_64-linux/lib/libOpenCL.so.1.0
/usr/local/cuda-12.6/targets/x86_64-linux/lib/libOpenCL.so.1.0.0
/usr/share/man/man7/libOpenCL.so.7.gz

...and how about my CPU that allegedly supports 3.0?
Would I be able to craft a heterogeneous run utilizing both the Intel embedded GPU and the discrete Nvidia one?

I think I have the Intel OpenCL installed too:

$ rpm -qa | grep opencl
 opencl-filesystem-1.0-20.fc40.noarch
 intel-opencl-clang-15.0.0-11.fc40.x86_64
 intel-opencl-24.26.30049.6-1.fc40.x86_64
 intel-opencl-clang-devel-15.0.0-11.fc40.x86_64

$ rpm -qa | grep  spirv
 spirv-tools-libs-2024.3-3.fc40.x86_64
 spirv-llvm15.0-translator-15.0.0-9.fc40.x86_64
 spirv-tools-devel-2024.3-3.fc40.x86_64
 spirv-tools-2024.3-3.fc40.x86_64
 spirv-headers-devel-1.5.5-25.fc40.noarch

Do I hardcode CL_TARGET_OPENCL_VERSION in the CMakeLists to 1?

Thx for hints

[Karm]

🥳 Resolved

As usual, it helped to write the problem down, it occurred to me that the project might expect Intel OneAPI libs.

[INFO] tornado ............................................ SUCCESS [  0.623 s]
[INFO] tornado-api ........................................ SUCCESS [  1.955 s]
[INFO] tornado-runtime .................................... SUCCESS [  1.774 s]
[INFO] tornado-matrices ................................... SUCCESS [  0.554 s]
[INFO] tornado-drivers .................................... SUCCESS [  0.272 s]
[INFO] tornado-drivers-common ............................. SUCCESS [  0.610 s]
[INFO] tornado-drivers-opencl ............................. SUCCESS [  2.559 s]
[INFO] tornado-drivers-opencl-jni ......................... SUCCESS [  3.428 s]
[INFO] tornado-drivers-spirv .............................. SUCCESS [  1.177 s]
[INFO] tornado-drivers-ptx ................................ SUCCESS [  1.340 s]
[INFO] tornado-drivers-ptx-jni ............................ SUCCESS [  3.403 s]
[INFO] tornado-examples ................................... SUCCESS [  1.110 s]
[INFO] tornado-benchmarks ................................. SUCCESS [  3.924 s]
[INFO] tornado-unittests .................................. SUCCESS [  9.227 s]
[INFO] tornado-annotation ................................. SUCCESS [  0.487 s]
[INFO] tornado-assembly ................................... SUCCESS [02:13 min]
[INFO] ------------------------------------------------------------------------

So the missing bit was:

cat  /etc/yum.repos.d/oneAPI.repo
[oneAPI]
name=Intel® oneAPI repository
baseurl=https://yum.repos.intel.com/oneapi
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB

and

dnf install intel-igc.x86_64 intel-opencl-clang-devel.x86_64 intel-opencl-clang.x86_64 intel-oneapi-runtime-openmp-opencl-shared.x86_64  intel-oneapi-runtime-opencl.x86_64 intel-oneapi-runtime-dpcpp-sycl-opencl-cpu.x86_64

[stratika]

Thank you @Karm for the update.

I am wondering if it would work, in case you removed the spirv option. If CUDA is installed, most likely OpenCL driver is installed and should be detected.

Additionally, would clinfo report the GPU device as an OpenCL device?

[jjfumero]

Hi @Karm , thanks for the info.

Each backend requires different tooling:

If you have an NVIDIA GPU, you can use OpenCL and PTX:

./bin/tornadovm-installer --jdk mandrel-jdk-21 --backend opencl,ptx 

If you also want to access the SPIR-V, you need the intel-compute-runtime, and run it through Intel architectures (CPUs via Codeplay OCK, or GPUs via Intel oneAPI).

sudo dnf install intel-compute-runtime

./bin/tornadovm-installer --jdk mandrel-jdk-21 --backend opencl,ptx 

You can also select individual backends. Once you build TornadoVM, you can re-build as follows:

source setvars.sh
make BACKEND=opencl
make BACKEND=ptx
make BACKEND=spirv

# or even combinations., e.g., OpenCL and PTX
make BACKEND=opencl,ptx

Hope this clarifies the installation. Let us know otherwise.

In case you need to install the latest NVIDIA Driver, I have a blog explaining the process for Fedora 39. For Fedora 41 should be almost identical.
https://jjfumero.github.io/posts/2023/12/nvidia-cuda-drivers-fedora39