[BUG] Fail to run dp train with 2.2.8 (but success with 2.2.7)

[link89]

Bug summary

Get error the following error when run dp train on a GPU node

error: libdevice not found at ./libdevice.10.bc
2024-02-01 22:45:42.137246: W tensorflow/core/framework/op_kernel.cc:1827] UNKNOWN: JIT compilation failed.
2024-02-01 22:45:42.137338: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 17091566501094239012
Traceback (most recent call last):
  File "/public/groups/ai4ec/libs/conda/deepmd/2.2.8/gpu/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1402, in _do_call
    return fn(*args)
  File "/public/groups/ai4ec/libs/conda/deepmd/2.2.8/gpu/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1385, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/public/groups/ai4ec/libs/conda/deepmd/2.2.8/gpu/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1478, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) UNKNOWN: JIT compilation failed.
         [[{{node ExponentialDecay/Floor}}]]
         [[ExponentialDecay/_163]]
  (1) UNKNOWN: JIT compilation failed.
         [[{{node ExponentialDecay/Floor}}]]
0 successful operations.
0 derived errors ignored.

It works well with 2.2.7. Currently our work around is to use 2.2.7 to train model and run LAMMPS with deepmd 2.2.8.

DeePMD-kit Version

2.2.8

TensorFlow Version

2.14.0

How did you download the software?

Conda

Input Files, Running Commands, Error Log, etc.

deepmd-2.2.8.log.txt

Steps to Reproduce

Install deepmd with the following command

CONDA_OVERRIDE_CUDA="11.8" conda create -p deepmd/2.2.8/gpu deepmd-kit=2.2.8=cuda118py310*  lammps horovod -c conda-forge 

And then run dp train

Further Information, Files, and Links

No response

[njzjz]

Upstream xref:

• Cannot find libdevice in TF 2.11 + compilation fails without ptxas  conda-forge/tensorflow-feedstock#296
• nvvm contents end up in $PREFIX instead of $PREFIX/nvvm conda-forge/cudatoolkit-feedstock#96

[link89]

What's the suggested version of tensorflow to use with deepmd 2.2.8?

[biglinn]

When I perform dogen 'run' with deepmd-kit 2.2.8 and tensorflow 2.15.0, there is a similar problem about tensorflow.

[njzjz]

When I perform dogen 'run' with deepmd-kit 2.2.8 and tensorflow 2.15.0, there is a similar problem about tensorflow.

The problem on TF 2.15 should have been fixed in conda-forge/tensorflow-feedstock#370. It is valuable to know whether the new build still has the problem.

[biglinn]

When I perform dogen 'run' with deepmd-kit 2.2.8 and tensorflow 2.15.0, there is a similar problem about tensorflow.

The problem on TF 2.15 should have been fixed in conda-forge/tensorflow-feedstock#370. It is valuable to know whether the new build still has the problem.

Ok, I will take a try. Thanks!
In addition, I find that deepmd-kit 2.2.7 and tensorflow 2.9.0 can perform very good.

[link89]

@njzjz deepmd/2.2.9 still have this issue. One has to run pip install nvidia-cuda-nvcc-cu11 to workaround. I guess this issue is not fixed yet.

[njzjz]

deepmd/2.2.9 still have this issue.

@link89 the issue is not related to deepmd-kit.