Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error occured when pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./ #87

Open
AromaticJ opened this issue May 27, 2020 · 2 comments
Open

error occured when pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./ #87

AromaticJ opened this issue May 27, 2020 · 2 comments

Comments

@AromaticJ
Copy link

AromaticJ commented May 27, 2020 •
edited
Loading

Hi, I was stuck when pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
I used your approach as bellow:
Install NVIDIA-Apex package https://github.com/NVIDIA/apex
$ git clone https://github.com/NVIDIA/apex
$ cd apex
$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Revise the "GANet_deep.py":
add import apex
change all BatchNorm2d and BatchNorm3d to apex.parallel.SyncBatchNorm

My envs:
Ubuntu16.04
CUDA10.1
python3.7
pytorch1.5

error occured as:

(pt) root@d050e71b73df://lane_detect/code/GANet-master/apex# pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-lrlurs2y
Created temporary directory: /tmp/pip-req-tracker-pknxdh68
Initialized build tracking at /tmp/pip-req-tracker-pknxdh68
Created build tracker: /tmp/pip-req-tracker-pknxdh68
Entered build tracker: /tmp/pip-req-tracker-pknxdh68
Created temporary directory: /tmp/pip-install-u01wtkmx
Processing /lane_detect/code/GANet-master/apex
Created temporary directory: /tmp/pip-req-build-yvhrei6z
Added file:///lane_detect/code/GANet-master/apex to build tracker '/tmp/pip-req-tracker-pknxdh68'
Running setup.py (path:/tmp/pip-req-build-yvhrei6z/setup.py) egg_info for package from file:///lane_detect/code/GANet-master/apex
Running command python setup.py egg_info
torch.version = 1.5.0
running egg_info
creating /tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info
writing /tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/top_level.txt
writing manifest file '/tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/SOURCES.txt'
writing manifest file '/tmp/pip-req-build-yvhrei6z/pip-egg-info/apex.egg-info/SOURCES.txt'
/tmp/pip-req-build-yvhrei6z/setup.py:46: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
Source in /tmp/pip-req-build-yvhrei6z has version 0.1, which satisfies requirement apex==0.1 from file:///lane_detect/code/GANet-master/apex
Removed apex==0.1 from file:///lane_detect/code/GANet-master/apex from build tracker '/tmp/pip-req-tracker-pknxdh68'
Skipping wheel build for apex, due to binaries being disabled for it.
Installing collected packages: apex
Created temporary directory: /tmp/pip-record-bk_i7amf
Running command /root/anaconda3/envs/pt/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"'; file='"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-bk_i7amf/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/envs/pt/include/python3.7m/apex
torch.version = 1.5.0
/tmp/pip-req-build-yvhrei6z/setup.py:46: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
from /usr/local/cuda/bin

running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/apex
copying apex/__init__.py -> build/lib.linux-x86_64-3.7/apex
creating build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/__init__.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization
creating build/lib.linux-x86_64-3.7/apex/pyprof
copying apex/pyprof/__init__.py -> build/lib.linux-x86_64-3.7/apex/pyprof
creating build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_initialize.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/__version__.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp
copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp
creating build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/__init__.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils
creating build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN
copying apex/RNN/__init__.py -> build/lib.linux-x86_64-3.7/apex/RNN
creating build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/__init__.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel
copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel
creating build/lib.linux-x86_64-3.7/apex/contrib
copying apex/contrib/__init__.py -> build/lib.linux-x86_64-3.7/apex/contrib
creating build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/__init__.py -> build/lib.linux-x86_64-3.7/apex/normalization
copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization
creating build/lib.linux-x86_64-3.7/apex/mlp
copying apex/mlp/__init__.py -> build/lib.linux-x86_64-3.7/apex/mlp
copying apex/mlp/mlp.py -> build/lib.linux-x86_64-3.7/apex/mlp
creating build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/__init__.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-3.7/apex/optimizers
copying apex/optimizers/fused_adagrad.py -> build/lib.linux-x86_64-3.7/apex/optimizers
creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
copying apex/multi_tensor_apply/__init__.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply
creating build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/usage.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pointwise.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/pooling.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/linear.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/__main__.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/embedding.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/blas.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/randomSample.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/data.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/recurrentCell.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/dropout.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/prof.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/activation.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/optim.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/__init__.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/convert.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/output.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/index_slice_join_mutate.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/softmax.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/base.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/misc.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/utility.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/reduction.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/normalization.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/loss.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
copying apex/pyprof/prof/conv.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof
creating build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/nvmarker.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
copying apex/pyprof/nvtx/__init__.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx
creating build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/db.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/kernel.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/__main__.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/__init__.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/parse.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
copying apex/pyprof/parse/nvvp.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse
creating build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/__init__.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists
creating build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/batch_norm.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
copying apex/contrib/groupbn/__init__.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn
creating build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/__init__.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
copying apex/contrib/xentropy/softmax_xentropy.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy
creating build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam_v3.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/distributed_fused_adam_v2.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_lamb.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/__init__.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
copying apex/contrib/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers
creating build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/__init__.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/self_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
copying apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn
running build_ext
building 'apex_C' extension
creating /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7
creating /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc
Emitting ninja build file /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o.d -pthread -B /root/anaconda3/envs/pt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/flatten_unflatten.cpp -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /tmp/pip-req-build-yvhrei6z/csrc/flatten_unflatten.cpp:2:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/utils/tensor_flatten.h: In member function ‘at::DeprecatedTypeProperties& torch::utils::TensorGroup::type()’:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/utils/tensor_flatten.h:36:28: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
     return tensors[0].type();
                            ^
In file included from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:11:0,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
                 from /tmp/pip-req-build-yvhrei6z/csrc/flatten_unflatten.cpp:1:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:262:30: note: declared here
   DeprecatedTypeProperties & type() const {
                              ^~~~
g++ -pthread -shared -B /root/anaconda3/envs/pt/compiler_compat -L/root/anaconda3/envs/pt/lib -Wl,-rpath=/root/anaconda3/envs/pt/lib -Wl,--no-as-needed -Wl,--sysroot=/ /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -L/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so
building 'amp_C' extension
Emitting ninja build file /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adam.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adam.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adam.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adam.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adam.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adam.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[2/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_l2norm_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_l2norm_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_l2norm_kernel.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_l2norm_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_l2norm_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_l2norm_kernel.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[3/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adagrad.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adagrad.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adagrad.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adagrad.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_adagrad.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_adagrad.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[4/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_axpby_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_axpby_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_axpby_kernel.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_axpby_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_axpby_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_axpby_kernel.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[5/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_sgd_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_sgd_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_sgd_kernel.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[6/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_scale_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_scale_kernel.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_scale_kernel.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_scale_kernel.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[7/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[8/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_novograd.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_novograd.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_novograd.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_novograd.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_novograd.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_novograd.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[9/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_2.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_2.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_2.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_2.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_2.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_2.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[10/11] /usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_1.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_1.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
FAILED: /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_1.o
/usr/local/cuda/bin/nvcc -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_1.cu -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/multi_tensor_lamb_stage_1.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
In file included from /tmp/pip-req-build-yvhrei6z/csrc/multi_tensor_lamb_stage_1.cu:3:0:
/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/ATen/cuda/CUDAContext.h:6:10: fatal error: cusparse.h: No such file or directory
 #include <cusparse.h>
          ^~~~~~~~~~~~
compilation terminated.
[11/11] c++ -MMD -MF /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/amp_C_frontend.o.d -pthread -B /root/anaconda3/envs/pt/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/TH -I/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/root/anaconda3/envs/pt/include/python3.7m -c -c /tmp/pip-req-build-yvhrei6z/csrc/amp_C_frontend.cpp -o /tmp/pip-req-build-yvhrei6z/build/temp.linux-x86_64-3.7/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1400, in _run_ninja_build
    check=True)
  File "/root/anaconda3/envs/pt/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-yvhrei6z/setup.py", line 314, in <module>
    extras_require=extras,
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 580, in build_extensions
    build_ext.build_extensions(self)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/root/anaconda3/envs/pt/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1413, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
Running setup.py install for apex ... error

Cleaning up...
Removing source in /tmp/pip-req-build-yvhrei6z
Removed build tracker: '/tmp/pip-req-tracker-pknxdh68'
ERROR: Command errored out with exit status 1: /root/anaconda3/envs/pt/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"'; file='"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-bk_i7amf/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/envs/pt/include/python3.7m/apex Check the logs for full command output.
Exception information:
Traceback (most recent call last):
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run
use_user_site=options.use_user_site,
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/req/init.py", line 71, in install_given_reqs
**kwargs
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 829, in install
scheme=scheme,
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/operations/install/legacy.py", line 72, in install
cwd=install_req.unpacked_source_directory,
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 275, in runner
spinner=spinner,
File "/root/anaconda3/envs/pt/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /root/anaconda3/envs/pt/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"'; file='"'"'/tmp/pip-req-build-yvhrei6z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-bk_i7amf/install-record.txt --single-version-externally-managed --compile --install-headers /root/anaconda3/envs/pt/include/python3.7m/apex Check the logs for full command output.
1 location(s) to search for versions of pip:
@zyl1336110861
Copy link

Hi, has your question been resolved?

@zyl1336110861
Copy link

Thanks to the issue here NVIDIA/apex#850, I just solved the problem.

  1. git clone https://github.com/NVIDIA/apex
  2. cd apex
  3. git checkout f3a960f80244cf9e80558ab30f7f7e8cbf03c0a0
  4. pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
    my environment:
    gcc 5.4.0
    cuda 10.0
    python 3.6.8
    pytorch 1.1.0
    ubuntu 16.04

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AromaticJ @zyl1336110861