feature: parallel solve subspace diagonalization in dav_subspace #5549

pxlxingliang · 2024-11-21T02:44:24Z

Add function Diago_HS_para() to diago the H/S by genelpa or scalapack. The function will perform 2D block distribution of data, call elpa/scalapack for solution, and collect eigenvectors to rank0.
Add a unified function interface pxxxgvx_diag() for calling scalapack to solve the generalized eigenvalue problem with double/complex/float/complex.
Add a new INPUT parameter diag_subspace_method, which support to set the subspace diagonalization method in dav_subspace to use elpa or scalapack to solve.

Reminder

Have you linked an issue with this pull request?
Have you added adequate unit tests and/or case tests for your pull request?
Have you noticed possible changes of behavior below or in the linked issue?
Have you explained the changes of codes in core modules of ESolver, HSolver, ElecState, Hamilt, Operator or Psi? (ignore if not applicable)

Linked Issue

Fix #5480

Unit Tests and/or Case Tests for my changes

A unit test is added for each new feature or bug fix.

What's changed?

Example: My changes might affect the performance of the application under certain conditions, and I have tested the impact on various scenarios...

Any changes of core modules? (ignore if not applicable)

Example: I have added a new virtual function in the esolver base class in order to ...

into diag

…into diag

mohanchen · 2024-11-22T03:23:35Z

docs/advanced/input_files/input-main.md

- **Default**: 4
+- **Default**: 4 
+
+### diag_subspace_method


I would suggest using "diag_subspace" and delete "_method"

mohanchen · 2024-11-22T03:24:16Z

source/module_base/scalapack_connector.h

+		const int* n, float* A, const int* ia, const int* ja, const int*desca, float* B, const int* ib, const int* jb, const int*descb,
+		const float* vl, const float* vu, const int* il, const int* iu,
+		const float* abstol, int* m, int* nz, float* w, const float*orfac, float* Z, const int* iz, const int* jz, const int*descz,
+		float* work, int* lwork, int*iwork, int*liwork, int* ifail, int*iclustr, float*gap, int* info);


at least leave a blank line between two functions

mohanchen · 2024-11-22T03:24:41Z

source/module_hsolver/diag_hs_para.cpp

+                 const int diag_subspace_method,
+                 const int block_size)
+{
+    int myrank;


initialize the variable

mohanchen · 2024-11-22T03:25:33Z

source/module_hsolver/diag_hs_para.cpp

+
+template <typename T>
+void Diago_HS_para(
+                T* h, 


I would suggset using clang-format before you pr a new code, here T is not aligned with the following variables

mohanchen · 2024-11-22T03:26:52Z

source/module_hsolver/diago_pxxxgvx.cpp

+		work, lwork,rwork, lrwork, iwork, liwork, ifail, iclustr, gap, info);
+}
+
+// post processing for pdsygvx/pzhegvx/pdsygvx/pzhegvx


explain the functions of pdsygvx/pzhegvx/pdsygvx/pzhegvx

mohanchen · 2024-11-22T03:27:16Z

source/module_hsolver/diago_pxxxgvx.cpp

+                      typename GetTypeReal<T>::type *const ekb,
+                      T *const wfc_2d)
+{
+    int nprow, npcol, myprow, mypcol;


initialize one variable for a line

mohanchen · 2024-11-22T03:27:30Z

source/module_hsolver/diago_pxxxgvx.cpp

+        int ndim_global = desc[2];
+        const char jobz = 'V', range = 'I', uplo = 'U';
+        const int itype = 1, il = 1, iu = nbands, one = 1;
+        int M = 0, NZ = 0, lwork = -1, lrwork = -1, liwork = -1, info = 0;


one line for a variable

mohanchen · 2024-11-22T03:27:52Z

source/module_hsolver/diago_pxxxgvx.cpp

+}
+
+// template instantiation
+template void pxxxgvx_diag(const int *const desc,


explain the functions of the four templates

haozhihan · 2024-11-22T03:50:43Z

Diago_HS_para implements parallel solution for the subspace process in dav-subspace. If this is efficient, can we just fix one solution method to reduce the burden of users?

haozhihan · 2024-11-22T04:11:09Z

I have a brief insight regarding this PR:

This process involves a transformation of parallel strategy.

from basis parallelism to 2D block parallelism
from 2D block parallelism to basis parallelism

Can this transformation of parallel strategy be more general?

If the H and S matrices of LCAO are solved by iterative method (like cg, dav, and so on, usually used for plane wave basis), it will also involve almost the same transformation of parallel strategy.

pxlxingliang · 2024-11-22T04:31:14Z

Diago_HS_para implements parallel solution for the subspace process in dav-subspace. If this is efficient, can we just fix one solution method to reduce the burden of users?

The parallel diagonization is not always more efficient, it is related to the system size, parallel cores, and efficiency of parallel communication, etc. I have done some tests in #5480.

pxlxingliang · 2024-11-22T04:42:04Z

I have a brief insight regarding this PR:

This process involves a transformation of parallel strategy.

from basis parallelism to 2D block parallelism

from 2D block parallelism to basis parallelism

Can this transformation of parallel strategy be more general?

If the H and S matrices of LCAO are solved by iterative method (like cg, dav, and so on, usually used for plane wave basis), it will also involve almost the same transformation of parallel strategy.

The transformation of different 2D block distribution can be realized easily by call the scalapack function Cpigemr2d() (a uniform interface for different data type is here: https://github.com/deepmodeling/abacus-develop/blob/develop/source/module_base/scalapack_connector.h#L158). While the transformation for basis parallelism is strongly related to the self-defined class in ABACUS (like psi?), may be a class function of psi to do the transformation is better.

Cstandardlib · 2024-11-22T05:41:50Z

source/module_hsolver/diago_pxxxgvx.h

+ */
+
+template <typename T>
+void pxxxgvx_diag(const int* const desc,


Does p stand for "parameter"?

pxxxgvx is the wrapper of pdsygvx_/pzhegvx_/pdsygvx_/pzhegvx_ in scalapack

pxxxgvx is the wrapper of pdsygvx_/pzhegvx_/pdsygvx_/pzhegvx_ in scalapack

I get it. It is "Parallel"

into diag

pxlxingliang · 2024-11-25T02:45:28Z

This feature use parallel_2d to deal with the parallel of H/S, but now parallel_2d is only compiled with ENABLE_LCAO=ON. I suggest to move parallel_2d.cpp to module_base to be a basic class.

@mohanchen, @jinzx10, @kirk0830, what's your suggestions?

root and others added 5 commits November 21, 2024 10:33

feature: parallel solve subspace diagonalization in dav_subspace

200a9cf

[pre-commit.ci lite] apply automatic fixes

2d528b1

Merge branch 'develop' of https://github.com/deepmodeling/abacus-develop

287b4ae

into diag

fix Makefile

2c3bb1b

Merge branch 'diag' of https://github.com/pxlxingliang/abacus-develop …

4127219

…into diag

mohanchen requested review from haozhihan and Cstandardlib November 22, 2024 03:22

mohanchen reviewed Nov 22, 2024

View reviewed changes

mohanchen added the Diago Issues related to diagonalizaiton methods label Nov 22, 2024

Cstandardlib reviewed Nov 22, 2024

View reviewed changes

root added 5 commits November 22, 2024 18:29

fix ut

2cf60e2

fix

d729dd5

fix

7bb7414

Merge branch 'develop' of https://github.com/deepmodeling/abacus-develop

2f491de

into diag

fix test

68cb71c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: parallel solve subspace diagonalization in dav_subspace #5549

feature: parallel solve subspace diagonalization in dav_subspace #5549

pxlxingliang commented Nov 21, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

mohanchen Nov 22, 2024

haozhihan commented Nov 22, 2024

haozhihan commented Nov 22, 2024

pxlxingliang commented Nov 22, 2024

pxlxingliang commented Nov 22, 2024 •

edited

Loading

Cstandardlib Nov 22, 2024

pxlxingliang Nov 22, 2024

Cstandardlib Nov 24, 2024

pxlxingliang commented Nov 25, 2024

feature: parallel solve subspace diagonalization in dav_subspace #5549

Are you sure you want to change the base?

feature: parallel solve subspace diagonalization in dav_subspace #5549

Conversation

pxlxingliang commented Nov 21, 2024

Reminder

Linked Issue

Unit Tests and/or Case Tests for my changes

What's changed?

Any changes of core modules? (ignore if not applicable)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haozhihan commented Nov 22, 2024

haozhihan commented Nov 22, 2024

pxlxingliang commented Nov 22, 2024

pxlxingliang commented Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pxlxingliang commented Nov 25, 2024

pxlxingliang commented Nov 22, 2024 •

edited

Loading