Instrumentation function count_instrs not found in binary

[ShihengCao]

I run the instr_count in test-apps and get this

 LD_PRELOAD=./tools/instr_count_bb/instr_count_bb.so ./test-apps/vectoradd/vectoradd
------------- NVBit (NVidia Binary Instrumentation Tool v1.5.5) Loaded --------------
NVBit core environment variables (mostly for nvbit-devs):
            NVDISASM = nvdisasm - override default nvdisasm found in PATH
            NOBANNER = 0 - if set, does not print this banner
---------------------------------------------------------------------------------
      START_GRID_NUM = 0 - Beginning of the kernel gird launch interval where to apply instrumentation
        END_GRID_NUM = 4294967295 - End of the kernel grid launch interval where to apply instrumentation
    COUNT_WARP_LEVEL = 1 - Count warp level or thread level instructions
    EXCLUDE_PRED_OFF = 0 - Exclude predicated off instruction from count
   ACTIVE_FROM_START = 1 - Start instruction counting from start or wait for cuProfilerStart and cuProfilerStop
       MANGLED_NAMES = 1 - Print kernel names mangled or not
        TOOL_VERBOSE = 0 - Enable verbosity inside the tool
----------------------------------------------------------------------------------------------------
ASSERT FAIL: function.cpp:805:void Function::gen_new_code(std::unordered_map<std::__cxx11::basic_string<char>, Function*>&): FAIL !(instr_func_map.find(c.instr_func_name) != instr_func_map.end()) MSG: instrumentation function count_instrs not found in binary!

when i run make, there are some warnings. i am not sure is it relevant with the problem.

make[1]: Entering directory '/home/caosh/nvbit_release/tools/instr_count'
nvcc -ccbin=g++ -D_FORCE_INLINES -dc -c -std=c++11 -I../../core -Xptxas -cloning=no -Xcompiler -Wall -arch=sm_70 -O3 -Xcompiler -fPIC instr_count.cu -o instr_count.o
nvcc -ccbin=g++ -D_FORCE_INLINES -I../../core -maxrregcount=24 -Xptxas -astoolspatch --keep-device-functions -arch=sm_70 -Xcompiler -Wall -Xcompiler -fPIC -c inject_funcs.cu -o inject_funcs.o
nvcc -ccbin=g++ -D_FORCE_INLINES -arch=sm_70 -O3 instr_count.o inject_funcs.o -L../../core -lnvbit -L /usr/local/cuda-12.3/lib64 -lcuda -lcudart_static -shared -o instr_count.so
/usr/bin/ld: warning: ../../core/libnvbit.a(instr.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(instr.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(nvbit.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(nvbit.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(nvbit_imp.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(nvbit_imp.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(gv10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(gv10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(ga10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(ga10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(gk11x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(gk11x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tu10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tu10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(gv11x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(gv11x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(gm10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(gm10x_hal.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf64.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf64.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf_init.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf_init.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_hashmap.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_hashmap.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf32.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf_caches.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_readelf_caches.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_rangemap.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_rangemap.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_list.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_list.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(function.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(function.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_hashset.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_hashset.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_rbtr.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: ../../core/libnvbit.a(tools_shared_rbtr.o): unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
make[1]: Leaving directory '/home/caosh/nvbit_release/tools/instr_count'

my env:

1. GPU: RTX 2060
2. Host CPU: i5-8500
3. OS: WSL2 Ubuntu18.04
4. GCC/G++ version: 7.5
5. CUDA version: 12.3
6. CUDA driver version: 546.33
7. nvcc version: 11.8

there are some tests i did, i compiled and ran the example in another machine and success, but when i copy the so file and the exexcute file to WSL, it faces the same problem ,which is instrumentation function count_instrs not found in binary.

[xudoong]

The error may by revelant to the NVCC and CUDA version. My machine is an A100 GPU with CUDA Driver version 545.23.08 and CUDA version 12.3. I encounted the same problem when building the tool with NVCC 12.1.1 and 11.8. However, when I changed to NVCC 11.4.0, this error just vanished and the tool worked well.

LD_PRELOAD=../../tools/instr_count_bb/instr_count_bb.so ./vectoradd 
------------- NVBit (NVidia Binary Instrumentation Tool v1.5.5) Loaded --------------
NVBit core environment variables (mostly for nvbit-devs):
            NVDISASM = nvdisasm - override default nvdisasm found in PATH
            NOBANNER = 0 - if set, does not print this banner
---------------------------------------------------------------------------------
      START_GRID_NUM = 0 - Beginning of the kernel gird launch interval where to apply instrumentation
        END_GRID_NUM = 4294967295 - End of the kernel grid launch interval where to apply instrumentation
    COUNT_WARP_LEVEL = 1 - Count warp level or thread level instructions
    EXCLUDE_PRED_OFF = 0 - Exclude predicated off instruction from count
   ACTIVE_FROM_START = 1 - Start instruction counting from start or wait for cuProfilerStart and cuProfilerStop
       MANGLED_NAMES = 1 - Print kernel names mangled or not
        TOOL_VERBOSE = 0 - Enable verbosity inside the tool
----------------------------------------------------------------------------------------------------
kernel 0 - _Z6vecAddPdS_S_i - #thread-blocks 98,  kernel instructions 50066, total instructions 50066
Final sum = 100000.000000; sum/n = 1.000000 (should be ~1)
Total app instructions: 50066

[coppock]

I got it to work on CUDA 11.7 with NVIDIA driver 545.23.08.

[ShihengCao]

Thanks for reply， I made it work with CUDA 10.2