RapidCFD errors with CUDA 11.2, 11.3, 11.4 and 11.5

[TonkomoLLC]

Hello,
In case anyone else tries to compile RapidCFD with one of the more recent (as of Sept 2021) verisons of CUDA.... On my machine (stock Ubuntu 20.04 - meaning gcc 9.3.0 and any other default toolchain items),

• CUDA 11.1 works perfectly

• CUDA 11.2 works normally when no function objects are utilized. However, using the functionObject "libForces.so" gives a "parallel_for" issue first described in looping over mesh cells within a solvers #87:

  terminate called after throwing an instance of 'thrust::system::system_error'
    what():  parallel_for failed: cudaErrorInvalidDeviceFunction: invalid device function
  Aborted (core dumped)
  

  Presently I do not know the reason why this function broke between CUDA 11.1 and 11.2. There are no compilation errors with CUDA 11.2, including no compilation errors for libforces.so.

  For avoidance of doubt, the libForces functionObject is the only item that I tested with CUDA 11.2 that gave an error. Note that my tests are not comprehensive of every feature and solver in RapidCFD, so other issues may be lurking.

• CUDA 11.3 has a compilation issue with src/meshTools,

  /opt/RapidCFD-dev/src/OpenFOAM/lnInclude/PointHit.H(89): error: identifier "Foam::Vector<double> ::zero" is  undefined in device code
  

  Despite this error, interestingly, libOpenFOAM.so compiles correctly (PointHit.H is located in the src/OpenFOAM directory).

Here is a workaround.

In PointHit.H, replace line 89:

            hitPoint_(vector::zero),

with

            hitPoint_(vector(0,0,0)),

The code then compiles and except for the libforces issue, it seems to run OK.

• CUDA 11.4 has the exact same issue as CUDA 11.3.

• CUDA 11.5 gives another error when compiling the dynamic mesh library:

SolidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C:104:6: error: reference to ‘const_iterator’ is ambiguous
  104 |     forAllConstIter(dictionary, dynamicMeshCoeffs_, iter)
  
  solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C:104:59: error: ‘iter’ was not declared in this scope
  104 |     forAllConstIter(dictionary, dynamicMeshCoeffs_, iter)

For now if dynamicMesh is not needed, CUDA 11.5 can be used if the compilation of the dynamic mesh library is disabled by commenting out `src/Allwmake' as follows:

# wmake $makeType dynamicFvMesh

It is also possible to compile the dynamicMesh library by just commenting out the for loop starting at line 104 in multiSolidBodyMotionFvMesh.C, as per the attached. Of course, the feature commented out will not work, but if not needed for your application, RapidCFD will compile.

multiSolidBodyMotionFvMesh.txt

My apologies for not proposing code fixes, but I thought I would at least report my findings in case anyone else tries to compile RapidCFD with CUDA >= 11.2.

Best regards,

Eric

[sleeperss]

simple fix for dynamicMesh :

git diff src/dynamicFvMesh/solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
diff --git a/src/dynamicFvMesh/solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C b/src/dynamicFvMesh/solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
index 83026d6..6092b0e
--- a/src/dynamicFvMesh/solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
+++ b/src/dynamicFvMesh/solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
@@ -101,7 +101,8 @@ Foam::multiSolidBodyMotionFvMesh::multiSolidBodyMotionFvMesh(const IOobject& io)
     gpuPointIDs_.setSize(pointIDs_.size());
     label zoneI = 0;

-    forAllConstIter(dictionary, dynamicMeshCoeffs_, iter)
+    using IteratorType = UILList<DLListBase, entry>;
+    forAllConstIter(IteratorType, dynamicMeshCoeffs_, iter)
     {
         if (iter().isDict())
         {

Moreover I also have had to add this change to whos preconised by @TonkomoLLC :

diff --git a/src/OpenFOAM/fields/Fields/vectorField/vectorField.C b/src/OpenFOAM/fields/Fields/vectorField/vectorField.C
index 08337fa..44862d5 100644
--- a/src/OpenFOAM/fields/Fields/vectorField/vectorField.C
+++ b/src/OpenFOAM/fields/Fields/vectorField/vectorField.C
@@ -13,6 +13,8 @@ template class gpuList<vector>;
 template class gpuField<vector>;

 template Ostream& operator<< <vector>(Ostream&, const gpuList<vector>&);
+template Ostream& operator<< <int>(Ostream&, const gpuList<int>&);
+template Ostream& operator<< <double>(Ostream&, const gpuList<double>&);
 template Istream& operator>> <vector>(Istream&, gpuList<vector>&);

 template<>