Can't get Deepspeed to work on Ubuntu 22.04

[AlpsAficionado]

Hello esteemed Deepspeed community.

I've spent several hours bashing my head against getting deepspeed to function properly on my system. I run oobabooga/text-generation-webui inside an Ubuntu 22.04 VM on my server (Nvidia Quadro RTX 8000 with 48 VRAM; 128GB system RAM; the VM is given total control of the 8000 via PCI passthrough/iommu, and has 96GB of system RAM allocated to it).

After hours of frustration, I was unable to get deepspeed to function with oobabooga on Ubuntu 22.04.

I was able to get it to run on Ubuntu 23.04; however, 22.04 is the current long-term supported version (and it still took me significant manual intervention to get it going in 23.04).

A rundown of my issues with 22.04:

1. Error: ModuleNotFoundError: No module named 'deepspeed'
   Solution: pip install deepspeed

2. Error: ModuleNotFoundError: No module named 'mpi4py'
   Solution: sudo apt install libopenmpi-dev ; pip install mpi4py

3. Error: 'pip install mpi4py' won't work; it crashes like so:

/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_class_finalize'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \mca_base_var_group_get_count'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_datatype_resize'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_hash_table_set_value_uint64'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \MPIR_being_debugged'
/home/user/oobabooga_linux/installer_files/env/compiler_compat/ld: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so: undefined reference to \opal_list_t_class'
collect2: error: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
error: Cannot link MPI programs. Check your configuration!!!

Solution: env LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ pip install --no-cache-dir mpi4py

4. Error:

module = importlib.util.module_from_spec(spec)
File "<frozen importlib._bootstrap>", line 571, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1176, in create_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /home/user/oobabooga_linux/installer_files/env/bin/../lib/libstdc++.so.6: version \GLIBCXX_3.4.30' not found (required by /home/user/.cache/torch_extensions/py310_cu117/async_io/async_io.so)

Solution: conda install -c conda-forge gcc=12.1.0 (rebuilds/reinstalls a whole bunch of crap, see below:

Downloading and Extracting Packages
_libgcc_mutex-0.1 | 3 KB | ###################################################################################### | 100%
ca-certificates-2023 | 145 KB | ###################################################################################### | 100%
gcc_impl_linux-64-12 | 56.2 MB | ####################################################################################2 | 98%
libgomp-12.2.0 | 455 KB | ###################################################################################### | 100%
kernel-headers_linux | 707 KB | ###################################################################################### | 100%
libgcc-ng-12.2.0 | 931 KB | ###################################################################################### | 100%
libgcc-devel_linux-6 | 3.3 MB | ###################################################################################### | 100%
openssl-1.1.1t | 1.9 MB | ###################################################################################### | 100%
requests-2.29.0 | 55 KB | ###################################################################################### | 100%
libsanitizer-12.1.0 | 6.2 MB | ###################################################################################### | 100%
libstdcxx-ng-12.2.0 | 4.3 MB | ###################################################################################### | 100%
sysroot_linux-64-2.1 | 31.4 MB | ####################################################################################6 | 98%
gcc-12.1.0 | 24 KB | ###################################################################################### | 100%
certifi-2023.5.7 | 149 KB | ###################################################################################### | 100%
_openmp_mutex-4.5 | 23 KB | ###################################################################################### | 100%
binutils_impl_linux- | 5.2 MB | ###################################################################################### | 100%

5. Error: RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue.

Full error spew:

INFO:Loading decapoda-research_llama-65b-hf...
Traceback (most recent call last):
File "/home/user/oobabooga_linux/text-generation-webui/server.py", line 919, in <module>
shared.model, shared.tokenizer = load_model(shared.model_name)
File "/home/user/oobabooga_linux/text-generation-webui/modules/models.py", line 117, in load_model
model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained
return model_class.from_pretrained(
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2624, in from_pretrained
init_contexts = [deepspeed.zero.Init(config_dict_or_path=deepspeed_config())] + init_contexts
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 762, in __init__
self.param_swapper = AsyncPartitionedParameterSwapper(_ds_config, self.dtype)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py", line 40, in __init__
aio_op = AsyncIOBuilder().load(verbose=False)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
return self.jit_load(verbose)
File "/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 449, in jit_load
raise RuntimeError(
RuntimeError: Unable to JIT load the async_io op due to it not being compatible due to hardware/software issue. None

Solution: I don't know; this is where I am stuck. #1037 suggests that I just need to 'apt install libaio-dev', but I've done that and it doesn't help.

I'm still stuck and cannot for the life of me get the --deepspeed option working on 22.04. I'd truly appreciate any help. Thanks to all in advance!

[AlpsAficionado]

Here is the output of 'ds_report' in my working conda environment in the VM.

DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [YES] ...... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/torch']
torch version .................... 2.0.0
deepspeed install path ........... ['/home/user/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

[lucasjinreal]

Just same error...