possible overcommit

[EricDeveaud]

Hello,

number of processors to use is either hardcoded (4, 8) either set using multiprocessing.cpu_count()

problem is that multiprocessing.cpu_count() returns the number of available cpu, but this is not the same as the number of cpu available to the process. For example, you can run in a taskset context or a batch scheduler like slurm.

see:

$ nproc
96
$ taskset -c 1 nproc
1
$ taskset -c 1 python3 -c "import multiprocessing; print(multiprocessing.cpu_count())"
96

I would suggest to use len(os.sched_getaffinity(0)) instead of multiprocessing.cpu_count()

$ python3 -c "import os; print(len(os.sched_getaffinity(0)))"
96
$ taskset -c 1 python3 -c "import os; print(len(os.sched_getaffinity(0)))"
1

regards

Eric

[zhangrengang]

Hi Eric, thanks for your review and suggestion. I have changed it to len(os.sched_getaffinity(0)).