Add kokkos and its simtbx/LS49 tests to Azure build #728

phyy-nx · 2022-01-25T23:47:54Z

Co-authored-by: Felix Wittwer [email protected]
Co-authored-by: Billy K. Poon [email protected]
Co-authored-by: Nicholas Sauter [email protected]

nksauter · 2022-01-26T00:52:28Z

Actually the Kokkos build would only be expected to work with std c++ >= 11 and probably not with Python 2.7. Is it possible to readjust for that?

bkpoon · 2022-01-26T01:47:56Z

The XFEL CI tests are only in Python 3 and have C++11 enabled.

But the flag is for configure.py, not boostrap.py. The change should be

--config-flags="--enable_kokkos"

I just updated the commit and adjusted the formatting.

bkpoon · 2022-01-26T18:32:58Z

On macOS, it looks like the same error that @mewall had. The gpu extension is not built since --enable_cuda is not provided, but something needs it?

On linux, the KOKKOS_CXXFLAGS has echo as the first item. It should not be there.

bkpoon · 2022-01-26T18:35:16Z

Wait, kokkos is not enabled for macOS.

cctbx_project/simtbx/SConscript

Lines 37 to 39 in 7125da0

    
           # only build kokkos on linux and if kokkos is enabled 
        
           if sys.platform.startswith('linux') and env_etc.enable_kokkos: 
        
             env_simtbx.SConscript("kokkos/SConscript",exports={ 'env' : env_simtbx })

bkpoon · 2022-01-26T18:38:57Z

@JBlaschke, would #675 help with getting rid of echo?

phyy-nx · 2022-01-26T18:40:03Z

The echo is coming from here:

cctbx_project/simtbx/kokkos/SConscript

Lines 139 to 141 in 7125da0

    
           kokkos_cxxflags = subprocess.check_output( 
        
             ['make', '-f', 'Makefile.kokkos', 'print-cxx-flags'], 
        
             cwd=os.environ['KOKKOS_PATH'])

I looked into the Makefile.kokkos and there's lots of calls to echo, wrapped in shell commands. Maybe those aren't working on Azure? That's why I tried to print os.envrion['SHELL'], but it appears that's not set on Azure.

phyy-nx · 2022-01-26T18:42:37Z

Contents of kokkos_cxxflags right before it used:

['echo', '"-std=c++14', '-march=core-avx2', '-mtune=core-avx2', '-fopenmp', '-I./', '-I/__w/1/modules/kokkos/core/src', '-I/__w/1/modules/kokkos/containers/src', '-I/__w/1/modules/kokkos/algorithms/src"']

bkpoon · 2022-01-27T05:50:40Z

.azure-pipelines/xfel/unix-conda-build.yml

@@ -100,6 +111,9 @@ steps:
    chmod +x $(Pipeline.Workspace)/modules/xfel_regression/merging_test_data/merge_thermo.csh
    export OMP_NUM_THREADS=4
    libtbx.run_tests_parallel module=uc_metrics module=simtbx module=xfel_regression module=LS49 nproc=4
+    echo "DEBUG"
+    cat mp4k/rank_0*.err
+    echo "DEBUG2"


These debugging lines will cause this step to always pass since the last command will always run correctly.

We were going to set OMP_NUM_THREADS to 2 to get rid of the OpenMP warning and I'm testing a change that enables kokkos on macOS.

bkpoon · 2022-01-27T19:04:59Z

Now there are 4 remaining test failures

2022-01-27T18:59:52.3610355Z libtbx.python "/__w/1/modules/LS49/adse13_187/cyto_batch.py" N_total=1 test_pixel_congruency=True mosaic_method=double_random mosaic_spread_samples=50 write_output=False test_without_mpi=True log.outdir=mp1k nxmx_local_data=/global/cfs/cdirs/m3562/der/master_files/run_000795.JF07T32V01_master.h5 context=kokkos_gpu [FAIL] 2.2s
2022-01-27T18:59:52.3611179Z   Time:  2.22
2022-01-27T18:59:52.3611567Z   Return code: 1
2022-01-27T18:59:52.3611860Z   OKs: 0
2022-01-27T18:59:52.3612564Z libtbx.python "/__w/1/modules/LS49/adse13_187/tst_multipanel_argchk.py" N_total=1 mosaic_spread_samples=50 test_without_mpi=True log.outdir=mp2k nxmx_local_data=/global/cfs/cdirs/m3562/der/master_files/run_000795.JF07T32V01_master.h5 context=kokkos_gpu [FAIL] 1.9s
2022-01-27T18:59:52.3613376Z   Time:  1.87
2022-01-27T18:59:52.3613671Z   Return code: 1
2022-01-27T18:59:52.3614293Z   OKs: 0
2022-01-27T18:59:52.3615077Z libtbx.python "/__w/1/modules/LS49/adse13_187/tst_write_file_action.py" N_total=1 mosaic_method=double_random mosaic_spread_samples=50 test_without_mpi=True log.outdir=mp3k write_output=False nxmx_local_data=/global/cfs/cdirs/m3562/der/master_files/run_000795.JF07T32V01_master.h5 context=kokkos_gpu [FAIL] 1.8s
2022-01-27T18:59:52.3615871Z   Time:  1.84
2022-01-27T18:59:52.3616154Z   Return code: 1
2022-01-27T18:59:52.3616549Z   OKs: 0
2022-01-27T18:59:52.3618254Z libtbx.python "/__w/1/modules/LS49/adse13_187/tst_write_file_action.py" N_total=1 write_output=True write_experimental_data=True mosaic_spread_samples=62 test_without_mpi=True log.outdir=mp4k nxmx_local_data=/global/cfs/cdirs/m3562/der/master_files/run_000795.JF07T32V01_master.h5 mask_file=/global/cfs/cdirs/m3562/nks/adse13_187/13_221/event_648.mask context=kokkos_gpu [FAIL] 1.9s
2022-01-27T18:59:52.3619147Z   Time:  1.91
2022-01-27T18:59:52.3619446Z   Return code: 1
2022-01-27T18:59:52.3619829Z   OKs: 0

The change for using kokkos on macOS has not been committed yet.

bkpoon · 2022-03-14T18:27:24Z

/azp run

azure-pipelines · 2022-03-14T18:27:44Z

Azure Pipelines successfully started running 3 pipeline(s).

bkpoon · 2022-03-15T22:21:01Z

/azp run "XFEL CI"

azure-pipelines · 2022-03-15T22:21:07Z

No pipelines are associated with this pull request.

bkpoon · 2022-03-15T22:21:40Z

/azp run XFEL CI

azure-pipelines · 2022-03-15T22:21:50Z

Azure Pipelines successfully started running 1 pipeline(s).

bkpoon · 2022-03-17T00:51:03Z

/azp run XFEL CI

azure-pipelines · 2022-03-17T00:51:15Z

Azure Pipelines successfully started running 1 pipeline(s).

bkpoon · 2022-03-17T08:03:00Z

/azp run XFEL CI

azure-pipelines · 2022-03-17T08:03:11Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-authored-by: Felix Wittwer <[email protected]> Co-authored-by: Billy K. Poon <[email protected]> Co-authored-by: Nicholas Sauter <[email protected]>

Seems a bug in Makefile.kokkos print-cxx-flags on Azure

This reverts commit e4f8b77.

This reverts commit 52eb8d0.

This reverts commit 3a1607c.

bkpoon force-pushed the kokkos_azure branch from 66b69ce to 8234f91 Compare January 26, 2022 01:47

phyy-nx force-pushed the kokkos_azure branch from 59b96b5 to 60edf3d Compare January 27, 2022 00:13

bkpoon reviewed Jan 27, 2022

View reviewed changes

bkpoon force-pushed the kokkos_azure branch from f292ae7 to 62ccfb2 Compare February 4, 2022 20:31

bkpoon force-pushed the kokkos_azure branch from ad42a19 to 8558e62 Compare March 14, 2022 17:31

bkpoon force-pushed the kokkos_azure branch from 4b60df4 to 52eb8d0 Compare March 17, 2022 18:19

phyy-nx and others added 6 commits March 30, 2022 06:46

Add kokkos and its simtbx/LS49 tests to Azure build

797367f

Co-authored-by: Felix Wittwer <[email protected]> Co-authored-by: Billy K. Poon <[email protected]> Co-authored-by: Nicholas Sauter <[email protected]>

Debugging prints

c3985d1

Remove echo from kokkos SConscript

6c8e028

Seems a bug in Makefile.kokkos print-cxx-flags on Azure

Typo

3c78da9

More debugging

752529a

Remove debugging output

60290ec

bkpoon added 10 commits March 30, 2022 06:46

Use 2 threads

0c87eff

CI: start building kokkos-kernels for XFEL CI

00e260c

Test lunus branch

3b082ce

Find libgcc_s.so.1

8a642ab

Revert "Find libgcc_s.so.1"

ea3067e

This reverts commit e4f8b77.

Enable kokkos on macOS

02565c1

Enable OpenMP on macOS

4e0c07c

Revert "Enable OpenMP on macOS"

6bce39f

This reverts commit 52eb8d0.

Revert "Enable kokkos on macOS"

47d8ce0

This reverts commit 3a1607c.

Only run kokkos tests on linux

d04f8e1

bkpoon force-pushed the kokkos_azure branch from 52eb8d0 to d04f8e1 Compare March 30, 2022 13:48

bkpoon merged commit 03272eb into master Mar 30, 2022

bkpoon deleted the kokkos_azure branch March 30, 2022 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kokkos and its simtbx/LS49 tests to Azure build #728

Add kokkos and its simtbx/LS49 tests to Azure build #728

phyy-nx commented Jan 25, 2022

nksauter commented Jan 26, 2022

bkpoon commented Jan 26, 2022 •

edited

Loading

bkpoon commented Jan 26, 2022 •

edited

Loading

bkpoon commented Jan 26, 2022

bkpoon commented Jan 26, 2022

phyy-nx commented Jan 26, 2022

phyy-nx commented Jan 26, 2022

bkpoon Jan 27, 2022 •

edited

Loading

bkpoon Jan 27, 2022

bkpoon commented Jan 27, 2022

bkpoon commented Mar 14, 2022

azure-pipelines bot commented Mar 14, 2022

bkpoon commented Mar 15, 2022

azure-pipelines bot commented Mar 15, 2022

bkpoon commented Mar 15, 2022

azure-pipelines bot commented Mar 15, 2022

bkpoon commented Mar 17, 2022

azure-pipelines bot commented Mar 17, 2022

bkpoon commented Mar 17, 2022

azure-pipelines bot commented Mar 17, 2022

Add kokkos and its simtbx/LS49 tests to Azure build #728

Add kokkos and its simtbx/LS49 tests to Azure build #728

Conversation

phyy-nx commented Jan 25, 2022

nksauter commented Jan 26, 2022

bkpoon commented Jan 26, 2022 • edited Loading

bkpoon commented Jan 26, 2022 • edited Loading

bkpoon commented Jan 26, 2022

bkpoon commented Jan 26, 2022

phyy-nx commented Jan 26, 2022

phyy-nx commented Jan 26, 2022

bkpoon Jan 27, 2022 • edited Loading

Choose a reason for hiding this comment

bkpoon Jan 27, 2022

Choose a reason for hiding this comment

bkpoon commented Jan 27, 2022

bkpoon commented Mar 14, 2022

azure-pipelines bot commented Mar 14, 2022

bkpoon commented Mar 15, 2022

azure-pipelines bot commented Mar 15, 2022

bkpoon commented Mar 15, 2022

azure-pipelines bot commented Mar 15, 2022

bkpoon commented Mar 17, 2022

azure-pipelines bot commented Mar 17, 2022

bkpoon commented Mar 17, 2022

azure-pipelines bot commented Mar 17, 2022

bkpoon commented Jan 26, 2022 •

edited

Loading

bkpoon commented Jan 26, 2022 •

edited

Loading

bkpoon Jan 27, 2022 •

edited

Loading