RuntimeError: cublas runtime error : unknown error at /tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCBlas.cu:390

[Jhaprince]

Hi @HimariO , @nikhilbchilwant

First of thanks a lot for this great work and making this code open source.

I am trying to reproduce your results over the hatefulmemedataset provided, but getting stuck because of following error. Will you please help me out with this

Traceback (most recent call last):
File "train_meme_itm.py", line 802, in
run_main_train(args)
File "train_meme_itm.py", line 705, in run_main_train
main(args)
File "train_meme_itm.py", line 470, in main
loss = model(batch, compute_loss=True)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/apex/amp/_initialize.py", line 177, in new_fwd
**applier(kwargs, input_caster))
File "/src/model_villa/vqa.py", line 43, in forward
output_all_encoded_layers=False)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/src/model_villa/model.py", line 385, in forward
output_all_encoded_layers=output_all_encoded_layers)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/src/model_villa/model.py", line 295, in forward
hidden_states = layer_module(hidden_states, attention_mask)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/src/model_villa/layer.py", line 167, in forward
attention_output = self.attention(hidden_states, attention_mask)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/src/model_villa/layer.py", line 125, in forward
self_output = self.self(input_tensor, attention_mask)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/src/model_villa/layer.py", line 85, in forward
attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
RuntimeError: cublas runtime error : unknown error at /tmp/pip-req-build-l1dtn3mo/aten/src/THC/THCBlas.cu:390