Bs mmerge #7

pittlerf · 2017-02-14T15:24:35Z

Hi,
I have put together the input files, and corrected an error in the smearing routine for the scalar field correlation functions. The openmpi version is not included yet.

Cheers

Feri

kostrzewa · 2017-02-15T09:50:18Z

@pittlerf @urbach @Marcogarofalo
I will only be able to start reviewing this next week (Feb 20th), but there are a number of other things which I need to treat with higher priority right now.

kostrzewa

This is fine to go in once the requested changes have been made. I would prefer to see the contractions modularised because 4000 lines is a heck of a long source file...

kostrzewa · 2017-03-22T15:59:28Z

su3.h

@@ -276,7 +276,7 @@ _sse_store(r);
  (r).c1 -= I * (s).c1;				\
  (r).c2 -= I * (s).c2;

-#define complex_times_vector(r,c,s)		\
+#define _complex_times_vector(r,c,s)		\


could you remove the redefinition of this in line 688?

kostrzewa · 2017-03-22T16:02:45Z

read_input.l

+    mu03_BSM = c;
+    if(myverbose != 0) printf("  BSM parameter mu03 set to %f\n", mu03_BSM);
+  }
+  {SPC}*propagatorsonthefly{EQL}{DIGIT}+ {


we usually follow the convention of yes and no for boolean inputs

kostrzewa · 2017-03-22T16:20:30Z

solver/Makefile.in

@@ -40,7 +40,7 @@ libsolver_TARGETS = bicgstab_complex gmres incr_eigcg eigcg restart_X ortho \
                    generate_dfl_subspace dfl_projector \
                    cg_mms_tm cg_mms_tm_nd solver_field sumr mixed_cg_her index_jd \
                    dirac_operator_eigenvectors	spectral_proj \
-                    jdher_su3vect cg_her_su3vect eigenvalues_Jacobi
+                    jdher_su3vect cg_her_su3vect eigenvalues_Jacobi eigenvalues_krylov


I cannot build the code right now because eigenvalues_krylov.c is missing

pittlerf · 2017-03-23T13:06:42Z

Hi,

I have set up the yes / no input parameters and I created a new library for the contractions and put the necessary functions there. In this way the main executable is only 627 line.

Cheers

Feri

kostrzewa

Thanks a lot, this is looking excellent now. Would you like to claim ownership of your additions?

A remaining issue, I think, is that there is no Makefile.in for the new subdirectory, would you agree?

About the initialisations, I would only worry if these are a significant overhead, if they are not, we are ready to pull this in.

kostrzewa · 2017-03-23T13:16:58Z

contractions/contractions_FP.h

@@ -0,0 +1,32 @@
+/***********************************************************************
+ *
+ * Copyright (C) 2015 Mario Schroeck


This should be "Ferenc Pittler" :)

kostrzewa · 2017-03-23T13:17:22Z

contractions/contractions_FP.c

@@ -0,0 +1,2381 @@
+/***********************************************************************
+ *
+ * Copyright (C) 2009 Carsten Urbach


clearly, you are the author :)

kostrzewa · 2017-03-23T13:20:24Z

test_DslashBSM2.c

@@ -380,10 +380,12 @@ if( strcmp(scalar_input_filename, "create_random_scalarfield") == 0 ) {
 		printf("\n# square norm of the source: ||w||^2 = %e\n\n", squarenorm);
 		fflush(stdout);
 	}
-
+//initialize BSM2f operator
+ init_D_psi_BSM2f();


out of interest, what is the overhead of initialising and freeing this one per gauge configuration?

If it's a tiny fraction of total time, then this can remain, of course.

pittlerf · 2017-03-23T13:46:35Z

Hi,

Thanks for noting me, I claimed the authorship and added a Makefile.in for the contractions directory.

Cheers

Feri

kostrzewa · 2017-03-23T16:28:51Z

I think there might be a small issue when the local lattice volume is just two lattice sites:

$ mpirun -np 16 ./test_DslashBSM2
parameter rho_BSM set to 1.000000
parameter eta_BSM set to 1.000000
parameter  m0_BSM set to 0.000000
# Creating the following cartesian grid for a 4 dimensional parallelisation:
# 2 x 2 x 2 x 2
# The code was compiled with -D_GAUGE_COPY
# The code was compiled for non-blocking MPI calls (spinor and gauge)

# The number of processes is 16 
# The lattice size is 4 x 4 x 4 x 4
# The local lattice size is 2 x 2 x 2 x 2
# Initialising rectangular gauge action stuff
# The lattice is correctly mapped by the index arrays

# The computed plaquette value is 1.218448e-01.
Testing inversion

# square norm of the source: ||w||^2 = 1.228800e+04

cg_her_bi: Inversion took 0.162697 seconds for 42 iterations
Operator inversion time: t_F = 0.162984 sec 

time 0.0200288296
Operator application time: t_F = 0.020029 sec 

# || D_F w ||^2 = 8.4426846639613839e+04
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
Check consistency of D and D^dagger
< D_F w, v >        = 1.3171860303588308e+04 + I*(-5.7667488605880299e+03)
Operator dagger application time: t_F = 0.001365 sec 	 
< w, D_F^dagger v > = 1.3171860303588310e+04 + I*(-5.7667488605880308e+03)

| < D_F w, v > - < w, D_F^dagger v > | = 2.0336919783401661e-12

kostrzewa · 2017-03-28T15:01:30Z

@pittlerf were you able to understand why particular parallelisations seem to produce errors at the 1e-12 level?

pittlerf · 2017-03-28T15:03:29Z

I will check it now.

kostrzewa · 2017-04-24T16:06:16Z

Hi Feri, you keep adding changes to this without seemingly addressing any of the points in this review. Should I just close this then and we forget about it?

kostrzewa · 2017-04-24T16:07:17Z

In particular, there was a version of this which was okay to pull in, then you added lots of new stuff and now I have no idea what is what...

kostrzewa · 2017-03-28T15:02:00Z

contractions_BSM.c

@@ -469,7 +469,7 @@ int main(int argc, char *argv[]){
                 }//end of loop for spinor and color source degrees of freedom
               }
               if (g_cart_id == 0){
-                    snprintf(contractions_fname,200,"bsmcontractions.%.4d.%d.%.8d",nstore, src_idx, iscalar);
+                    snprintf(contractions_fname,200,"bsmcontractions.%.4d.%d.%.8d",nstore, isample, iscalar);


isample is only for stochastic sources

pittlerf · 2017-04-24T20:54:17Z

Hi,
Of course I can remove invert_save and using the value of propagatorsonthefly_BSM in op_invert. For the other thing I saw that it is within a loop index-ed by isample, that was my only reason to include it. Besides, the whole code is now written for source index zero only. Actually I recently changed the format of the output (was asked by Petros).
Feri

pittlerf · 2017-04-24T21:16:32Z

Hi, I checked the contractions code using parallelization with local lattice dimension 222*2. I got the same results. I divided actually a 4**4 lattice in each direction to two MPI threads, and checked the output file for the contraction routine. I use the propagator on the fly option, so the parallelization of the operator was also tested.

kostrzewa · 2017-04-25T07:39:01Z

Hi, I checked the contractions code using parallelization with local lattice dimension 222*2. I got the same results. I divided actually a 4**4 lattice in each direction to two MPI threads, and checked the output file for the contraction routine. I use the propagator on the fly option, so the parallelization of the operator was also tested.

Yes, but the problem is not at the contraction level and there it very likely depends on the parameters whether an issue will or will not show up. However, at the operator level (in test_Dslash), it seems that there is something going wrong and it would be good to understand why that is. It may well be that 1e-12 is to be expected, but then: can we trust residuals coming from an operator that has differences at the 1e-12 level?

kostrzewa · 2017-04-25T08:17:59Z

@pittlerf

Of course I can remove invert_save and using the value of propagatorsonthefly_BSM in op_invert. For the other thing I saw that it is within a loop index-ed by isample, that was my only reason to include it. Besides, the whole code is now written for source index zero only. Actually I recently changed the format of the output (was asked by Petros).

It seems to me that what is necessary is simply a change in the way the BSM inversion is handled in op_invert() and at the BSM operator level. The operator struct has pointers for four propagators, which is clearly insufficient for a full two-flavour inversion (and even more insufficient when also the inverse of the conjugate operator is required). As a result, extending the operator struct is probably the way to go. One could, for instance, add a pointer to an array of spinor fields. For the two-flavour operators, this could then be set to point to some allocated memory for four sets of four e/o propagators (16 volume/2 spinor fields) to store the down-source M and M^dagger propagators as well ass the up-source M and M^dagger propagators. After the call to inverter(), these can be cleanly extracted in the contraction code, without adding further auxiliary fields or additional functions which replicate lots of code.

pittlerf · 2017-04-25T09:56:50Z

Hi,
I have printed out the whole vector using different parallelization and I found aggreement upto 20 digits (I plot exactly that many digits). I do not actually now how MPI_ALLREDUCE works, but that can be the only difference. I tested, when I use the kahan summation the error will be twice as large when I use 2 MPI process. But even this linear scaling fails for large number of processes, for example when I use 16 the error will only 4 times larger.
Feri

check qphix input parameters and set some defaults in the interface

…mergetest conflict as a result of the data type change for the clover field

…r selection logic when QPhiX is not used for a monomial

…ed to the QPhiX interface

…cessor macro

First stab at QPhiX in HMC.

… square_and_max)

…oken, as are the qphix base classes and the wrapper functions for the full-spinor operators. This is due to github issue etmc#404 which should be fixed if we ever need the tests again (this should be done at some point, especially if we want to get the QPhiX interface tested under Travis)

Fix GH#400, DDalphaAMG interface should use correct TM_USE_OMP prepro…

…d make use of ternary operators in computing signs, removing static declaration from static const int sign

add some FIXMEs from our discussion on the 4th of March, 2021

…ng the non-local interaction for the scalar field to F(x+-mu)-F(x)

… i nthe old operator

… (non-kappa normalization)

…so in the rest we have to take into account 3.5r0

…e of csw we actually add 1+0.5*csw*clover instead of 0.5*(1+csw*clover), after the modification code still passes Marco's test

synch up with quda_work and quda_work_add_actions branches

kostrzewa · 2021-10-03T10:31:58Z

contractions_BSM.c

+                 if ( (vectorcurrentcurrent_BSM == 1 ) || ( axialcurrentcurrent_BSM == 1 )){
+                   for(src_idx = 0; src_idx < 12; src_idx++ )
+                   {
+                      snprintf(prop_fname,200,"bsm2prop.%.4d.%.2d.%02d.%.8d.inverted",nstore, T_global-1, src_idx, iscalar);


@pittlerf What does T_global-1 encode here in the isample position?

Hi, yeah, sorry yes, I just thought tat in the case of point sources that we use (just one at (0,0,0)), this might be the place, but of coarse, it should have a standalone place.

kostrzewa requested changes Mar 22, 2017

View reviewed changes

kostrzewa requested changes Mar 23, 2017

View reviewed changes

kostrzewa approved these changes Mar 23, 2017

View reviewed changes

kostrzewa requested changes Apr 24, 2017

View reviewed changes

kostrzewa pushed a commit that referenced this pull request Apr 27, 2017

Merge pull request #7 from kostrzewa/qphix_devel_check_params

22c4580

check qphix input parameters and set some defaults in the interface

kostrzewa and others added 12 commits October 10, 2017 17:42

Merge remote-tracking branch 'etmc/qphix_devel' into qphix_devel_hmc_…

e85ed71

…mergetest conflict as a result of the data type change for the clover field

monomial_solve: add some forgotten 'else' statements to fix the solve…

68aa946

…r selection logic when QPhiX is not used for a monomial

work on QPhiX interface documentation

bbfa182

complete documentation about CGMMS solver and how the shifts are pass…

9f184df

…ed to the QPhiX interface

Correct Bug in Reversibility chec

58e562e

Fix GH#400, DDalphaAMG interface should use correct TM_USE_OMP prepro…

9d7f6f0

…cessor macro

Merge pull request etmc#385 from kostrzewa/qphix_devel_hmc

783716d

First stab at QPhiX in HMC.

Add full clover-improved operator

15c6edb

Add function square_and_max for spinor

c6d8d46

Add min and relative norm to square_and_minmax function (renamed from…

1b2977f

… square_and_max)

Merge pull request etmc#402 from kostrzewa/fix_DDalphaAMG_tm_use_omp

cdc25f9

Fix GH#400, DDalphaAMG interface should use correct TM_USE_OMP prepro…

pittlerf and others added 28 commits December 7, 2020 14:02

correcting prepare source

e0759dc

correcting op_write and invert

75caa53

remove clover from the test operator for only wilson dirac

0c3a22e

adding missing header files

b663fb0

correcting bug in D_psi_BSM3

9273fd9

loading unit scalar field for the inversion

777a9af

removing D_psi_BSM3f.c, removing unnecessary update_backward_gauge an…

b5580ea

…d make use of ternary operators in computing signs, removing static declaration from static const int sign

separate if conditions for the BSM operator in line 167-171

7fe67e9

remove kappa_BSM

f07a9ed

remove typo

de1f1dd

remove warnings

a04128e

correcting sign of usual wilson term

1a33cd4

remove not used routine

fdd75ac

Merge branch 'BSMmerge' of github.com:pittlerf/tmLQCD into HEAD

6b3497b

add some FIXMEs from our discussion on the 4th of March, 2021

746ad87

optimise Fabsadd by storing 'c*\sum_\nu \varphi^2' in a const variable

7eb47a2

Merge pull request #3 from kostrzewa/BSMmerge

4945e5f

add some FIXMEs from our discussion on the 4th of March, 2021

Separating the test routine which should work for a general r0_BSM

267c174

setting the sign of the derivative term (1+-)gamma_mu to -1, correcti…

bb3259b

…ng the non-local interaction for the scalar field to F(x+-mu)-F(x)

correct normalization of the clover term

c8b2092

including a factor of 0.5 multiplying c_sw

50a4d9c

scalar local and nonlocal term taking into account with the same sign

c339525

scalar local and nonlocal term taking into account with the same sign…

2f2faad

… i nthe old operator

adding factor of three coming fromm the r term also the wilson dslash…

824b83e

… (non-kappa normalization)

updating the tests

c99e385

correcting bug: in the clover term only 0.5r0 is taken into account, …

0d66279

…so in the rest we have to take into account 3.5r0

when c_sw=0 we add 1+mu instead of 0.5*(1+mu), because at finite valu…

7ab33d2

…e of csw we actually add 1+0.5*csw*clover instead of 0.5*(1+csw*clover), after the modification code still passes Marco's test

reading non-trivial scalars for DP1 P1

29fcccd

pittlerf pushed a commit to pittlerf/tmLQCD that referenced this pull request Sep 10, 2021

Merge pull request kostrzewa#7 from etmc/quda_work_hmc_more_debugging

63cc201

synch up with quda_work and quda_work_add_actions branches

kostrzewa reviewed Oct 3, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bs mmerge #7

Bs mmerge #7

pittlerf commented Feb 14, 2017

kostrzewa commented Feb 15, 2017

kostrzewa left a comment

kostrzewa Mar 22, 2017

kostrzewa Mar 22, 2017

kostrzewa Mar 22, 2017

pittlerf commented Mar 23, 2017

kostrzewa left a comment •

edited

Loading

kostrzewa Mar 23, 2017

kostrzewa Mar 23, 2017

kostrzewa Mar 23, 2017

kostrzewa Mar 23, 2017

pittlerf commented Mar 23, 2017

kostrzewa commented Mar 23, 2017

kostrzewa commented Mar 28, 2017

pittlerf commented Mar 28, 2017

kostrzewa commented Apr 24, 2017

kostrzewa commented Apr 24, 2017

kostrzewa Mar 28, 2017

pittlerf commented Apr 24, 2017

pittlerf commented Apr 24, 2017

kostrzewa commented Apr 25, 2017

kostrzewa commented Apr 25, 2017

pittlerf commented Apr 25, 2017

kostrzewa Oct 3, 2021 •

edited

Loading

pittlerf Oct 3, 2021

Bs mmerge #7

Are you sure you want to change the base?

Bs mmerge #7

Conversation

pittlerf commented Feb 14, 2017

kostrzewa commented Feb 15, 2017

kostrzewa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pittlerf commented Mar 23, 2017

kostrzewa left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pittlerf commented Mar 23, 2017

kostrzewa commented Mar 23, 2017

kostrzewa commented Mar 28, 2017

pittlerf commented Mar 28, 2017

kostrzewa commented Apr 24, 2017

kostrzewa commented Apr 24, 2017

Choose a reason for hiding this comment

pittlerf commented Apr 24, 2017

pittlerf commented Apr 24, 2017

kostrzewa commented Apr 25, 2017

kostrzewa commented Apr 25, 2017

pittlerf commented Apr 25, 2017

kostrzewa Oct 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kostrzewa left a comment •

edited

Loading

kostrzewa Oct 3, 2021 •

edited

Loading