CUDA error with CrabNet call

[sgbaird]

CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`
  File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/src/matsci_opt_benchmarks/crabnet_hyperparameter/core.py", line 219, in matbench_metric_calculator
    cb.fit(train_df=train_df)
  File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/src/matsci_opt_benchmarks/crabnet_hyperparameter/core.py", line 104, in evaluate
    results = matbench_metric_calculator(parameters, dummy=dummy)
  File "/uufs/chpc.utah.edu/common/home/sparks-thermoelectric/matsci-opt-benchmarks/scripts/crabnet_hyperparameter/crabnet_hyperparameter_submitit.py", line 147, in <module>
    evaluate(parameter_batch_sets[0][0], dummy=True)
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`