diff --git a/CHANGES.md b/CHANGES.md index 7b7e78a0da..665eff135e 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,3 +1,20 @@ +# 24.09 + + * Code clean-ups / clang-tidy (#2942, #2949) + + * update the `hse_convergence` readme to reflect current convergence + (#2946) + + * update the `bubble_convergence` plotting script (#2947) + + * new Frontier scaling numbers (#2948) + + * more GPU error printing (@3944) + + * science problem updates: `flame_wave` (#2943) + + * documentation updates (#2939) + # 24.08 * lazy QueueReduction has been enabled for the timing diagnostics diff --git a/Docs/source/faq.rst b/Docs/source/faq.rst index 2d806224a2..2b979926ba 100644 --- a/Docs/source/faq.rst +++ b/Docs/source/faq.rst @@ -17,31 +17,46 @@ Compiling There are 2 things you can do to check what’s happening. First, inspect the directories in ``VPATH_LOCATIONS``. This can be done via: - :: + .. prompt:: bash - make print-VPATH_LOCATIONS + make print-VPATH_LOCATIONS Next, ask make to tell you where it is finding each of the source files. This is done through a script ``find_files_vpath.py`` that is hooked into Castro’s build system. You can run this as: - :: + .. prompt:: bash - make file_locations + make file_locations At the end of the report, it will list any files it cannot find in - the vpath. Some of these are to be expected (like ``extern.f90`` - and ``buildInfo.cpp``—these are written at compile-time. But any - other missing files need to be investigated. + the vpath. Some of these are to be expected (like + ``buildInfo.cpp``—these are written at compile-time). But any other + missing files need to be investigated. + +#. *I put a copy of one of the header files (e.g. ``problem_tagging.H``) + in my problem setup but it does not seem to recognized / used by + the build system. Why doesn't my executable use my custom version + of the header?* + + This is likely due to compiler caching / ccache. You need to + clear the cache and the build: + + .. prompt:: bash + + ccache -C + make clean + + Then rebuild and it should be recognized. #. *I’m still having trouble compiling. How can I find out what all of the make variables are set to?* Use: - :: + .. prompt:: bash - make help + make help This will tell you the value of all the compilers and their options. @@ -104,7 +119,7 @@ Debugging Given a MultiFab ``mf``, you can dump out the state as: - :: + .. code:: c++ print_state(mf, IntVect(AMREX_D_DECL(10, 20, 30))); @@ -119,7 +134,7 @@ Debugging You can simply output a FAB to ``std::cout``. Imagine that you are in an MFIter loop, with a MultiFab ``mf``: - :: + .. code:: c++ S = FArrayBox& mf[mfi]; std::cout << S << std::endl; @@ -143,9 +158,9 @@ Profiling When you run, a file named ``gmon.out`` will be produced. This can be processed with gprof by running: - :: + .. prompt:: bash - gprof exec-name + gprof exec-name where *exec-name* is the name of the executable. More detailed line-by-line information can be obtained by passing the -l @@ -159,9 +174,9 @@ Managing Runs Create a file called ``dump_and_continue``, e.g., as: - :: + .. prompt:: bash - touch dump_and_continue + touch dump_and_continue This will force the code to output a checkpoint file that can be used to restart. Other options are ``plot_and_continue`` to output @@ -193,9 +208,9 @@ Managing Runs The build information (including git hashes, modules, EoS, network, etc.) can be displayed by running the executable as - :: + .. prompt:: bash - ./Castro.exe --describe + ./Castro.exe --describe .. _ch:faq:vis: diff --git a/Exec/gravity_tests/hse_convergence/README.md b/Exec/gravity_tests/hse_convergence/README.md index fa639f4708..fab88d7bd7 100644 --- a/Exec/gravity_tests/hse_convergence/README.md +++ b/Exec/gravity_tests/hse_convergence/README.md @@ -7,29 +7,34 @@ in the plotfiles. To run this problem, use one of the convergence scripts: - * ``convergence_plm.sh`` : + * `convergence_plm.sh` : - this runs CTU + PLM using the default HSE BCs and default - use_pslope, then with reflect BCs, then without use_pslope, and - finally runs with reflect instead of HSE BCs. + this runs CTU + PLM using: + 1. the default HSE BCs and `use_pslope` + 2. the HSE BCs with reflection and `use_pslope` + 3. reflect BCs instead of HSE BCs without `use_pslope` + 4. reflect BCs with `use_pslope` - These tests show that the best results come from HSE BCs + reflect vel + These tests show that the best results (by far) come from + `use_pslope=1` and reflecting BCs * convergence_ppm.sh : this runs CTU + PPM in a similar set of configurations as PLM above - (with one additional one: grav_source_type = 4) + 1. the default HSE BCs + 2. HSE BCs with reflection + 3. reflecting BCs + 4. reflecting BCs with `use_pslope` - These tests show that the best results come from HSE BCs + reflect vel + These tests show that the best results (by far) come from + reflecting BCs with `use_pslope=1`, just like the PLM case. * convergence_sdc.sh : - this uses the TRUE_SDC integration, first with SDC-2 + PLM and reflecting BCs, - the SDC-2 + PPM and reflecting BCs, then the same but HSE BCs, and finally - SDC-4 + reflect + this uses the TRUE_SDC integration, first with SDC-2 + PLM and + reflecting BCs, the SDC-2 + PPM and reflecting BCs, then the same + but HSE BCs, and finally SDC-4 + reflect These tests show that the PLM + reflect (which uses the well-balanced use_pslope) and the SDC-4 + reflect give the lowest - errors and expected (or better) convergence: - - + errors and expected (or better) convergence. diff --git a/Exec/gravity_tests/hse_convergence/convergence_plm.sh b/Exec/gravity_tests/hse_convergence/convergence_plm.sh index 285cfed67d..3d12427ad5 100755 --- a/Exec/gravity_tests/hse_convergence/convergence_plm.sh +++ b/Exec/gravity_tests/hse_convergence/convergence_plm.sh @@ -58,43 +58,15 @@ pfile=`ls -t | grep -i hse_512_plt | head -1` fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} -## plm + hse reflect + no pslope - -ofile=plm-hsereflect-nopslope.converge.out - -RUNPARAMS=" -castro.ppm_type=0 -castro.use_pslope=0 -castro.hse_interp_temp=1 -castro.hse_reflect_vels=1 -""" - -${EXEC} inputs.ppm.64 ${RUNPARAMS} >& 64.out -pfile=`ls -t | grep -i hse_64_plt | head -1` -fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel > ${ofile} - -${EXEC} inputs.ppm.128 ${RUNPARAMS} >& 128.out -pfile=`ls -t | grep -i hse_128_plt | head -1` -fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} - -${EXEC} inputs.ppm.256 ${RUNPARAMS} >& 256.out -pfile=`ls -t | grep -i hse_256_plt | head -1` -fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} - -${EXEC} inputs.ppm.512 ${RUNPARAMS} >& 512.out -pfile=`ls -t | grep -i hse_512_plt | head -1` -fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} - - -## plm + reflect +## plm + reflect + nopslope -ofile=plm-reflect.converge.out +ofile=plm-reflect-nopslope.converge.out RUNPARAMS=" castro.ppm_type=0 -castro.use_pslope=1 castro.lo_bc=3 castro.hi_bc=3 +castro.use_pslope=0 """ ${EXEC} inputs.ppm.64 ${RUNPARAMS} >& 64.out @@ -114,16 +86,15 @@ pfile=`ls -t | grep -i hse_512_plt | head -1` fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} +## plm + reflect + pslope -## plm + reflect + nopslope - -ofile=plm-reflect-nopslope.converge.out +ofile=plm-reflect-pslope.converge.out RUNPARAMS=" castro.ppm_type=0 castro.lo_bc=3 castro.hi_bc=3 -castro.use_pslope=0 +castro.use_pslope=1 """ ${EXEC} inputs.ppm.64 ${RUNPARAMS} >& 64.out diff --git a/Exec/gravity_tests/hse_convergence/convergence_ppm.sh b/Exec/gravity_tests/hse_convergence/convergence_ppm.sh index ff6e2c7620..27b4bb86ef 100755 --- a/Exec/gravity_tests/hse_convergence/convergence_ppm.sh +++ b/Exec/gravity_tests/hse_convergence/convergence_ppm.sh @@ -50,12 +50,13 @@ pfile=`ls -t | grep -i hse_512_plt | head -1` fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} -## ppm + grav_source_type = 4 +## ppm + reflect -ofile=ppm-grav4.converge.out +ofile=ppm-reflect.converge.out RUNPARAMS=" -castro.grav_source_type=4 +castro.lo_bc=3 +castro.hi_bc=3 """ ${EXEC} inputs.ppm.64 ${RUNPARAMS} >& 64.out @@ -75,13 +76,14 @@ pfile=`ls -t | grep -i hse_512_plt | head -1` fextrema.gnu.ex -v magvel ${pfile} | grep -i magvel >> ${ofile} -## ppm + reflect +## ppm + reflect + pslope -ofile=ppm-reflect.converge.out +ofile=ppm-reflect-pslope.converge.out RUNPARAMS=" castro.lo_bc=3 castro.hi_bc=3 +castro.use_pslope=1 """ ${EXEC} inputs.ppm.64 ${RUNPARAMS} >& 64.out diff --git a/Exec/hydro_tests/gresho_vortex/problem_initialize_state_data.H b/Exec/hydro_tests/gresho_vortex/problem_initialize_state_data.H index adc5a4c050..f87fb240af 100644 --- a/Exec/hydro_tests/gresho_vortex/problem_initialize_state_data.H +++ b/Exec/hydro_tests/gresho_vortex/problem_initialize_state_data.H @@ -73,7 +73,7 @@ void problem_initialize_state_data (int i, int j, int k, } u_tot += u_phi; - reint += p/(gamma_const - 1.0_rt); + reint += p/(eos_rp::eos_gamma - 1.0_rt); } } } diff --git a/Exec/radiation_tests/Rad2Tshock/problem_initialize.H b/Exec/radiation_tests/Rad2Tshock/problem_initialize.H index 1405a5d6c2..4cffa04c3a 100644 --- a/Exec/radiation_tests/Rad2Tshock/problem_initialize.H +++ b/Exec/radiation_tests/Rad2Tshock/problem_initialize.H @@ -14,8 +14,8 @@ void problem_initialize () eos_state.rho = problem::rho0; eos_state.T = problem::T0; - for (int n = 0; n < NumSpec; n++) { - eos_state.xn[n] = 0.0_rt; + for (auto & X : eos_state.xn) { + X = 0.0_rt; } eos_state.xn[0] = 1.0_rt; diff --git a/Exec/radiation_tests/Rad2Tshock/problem_initialize_rad_data.H b/Exec/radiation_tests/Rad2Tshock/problem_initialize_rad_data.H index 860d2dcd60..87b4721254 100644 --- a/Exec/radiation_tests/Rad2Tshock/problem_initialize_rad_data.H +++ b/Exec/radiation_tests/Rad2Tshock/problem_initialize_rad_data.H @@ -12,6 +12,9 @@ void problem_initialize_rad_data (int i, int j, int k, const GeometryData& geomdata) { + amrex::ignore_unused(nugroup); + amrex::ignore_unused(dnugroup); + const Real* dx = geomdata.CellSize(); const Real* problo = geomdata.ProbLo(); diff --git a/Exec/radiation_tests/Rad2Tshock/problem_initialize_state_data.H b/Exec/radiation_tests/Rad2Tshock/problem_initialize_state_data.H index 96fe4057e0..c5e67920a5 100644 --- a/Exec/radiation_tests/Rad2Tshock/problem_initialize_state_data.H +++ b/Exec/radiation_tests/Rad2Tshock/problem_initialize_state_data.H @@ -16,7 +16,7 @@ void problem_initialize_state_data (int i, int j, int k, // Provides the simulation to be run in the x,y,or z direction // where length direction is the length side in a square prism - Real length_cell; + Real length_cell{}; if (problem::idir == 1) { length_cell = problo[0] + dx[0] * (static_cast(i) + 0.5_rt); } else if (problem::idir == 2) { diff --git a/Exec/reacting_tests/bubble_convergence/analysis/slice_multi.py b/Exec/reacting_tests/bubble_convergence/analysis/slice_multi.py index d05bd7fba9..2f8c1e16b8 100755 --- a/Exec/reacting_tests/bubble_convergence/analysis/slice_multi.py +++ b/Exec/reacting_tests/bubble_convergence/analysis/slice_multi.py @@ -1,5 +1,8 @@ #!/usr/bin/env python3 +import matplotlib +matplotlib.use('agg') + import os import sys import yt @@ -26,35 +29,37 @@ fig = plt.figure() fig.set_size_inches(12.0, 9.0) -grid = ImageGrid(fig, 111, nrows_ncols=(2, 2), axes_pad=0.75, cbar_pad="2%", +grid = ImageGrid(fig, 111, nrows_ncols=(2, 2), + axes_pad=0.75, cbar_pad="2%", label_mode="L", cbar_mode="each") -fields = ["Temp", "magvel", "X(C12)", "rho_enuc"] +fields = ["Temp", "magvel", "X(C12)", "enuc"] for i, f in enumerate(fields): - sp = yt.SlicePlot(ds, "z", f, center=[xctr, yctr, 0.0], width=[L_x, L_y, 0.0], fontsize="12") + sp = yt.SlicePlot(ds, "z", f, center=[xctr, yctr, 0.0*cm], + width=[L_x, L_y, 0.0*cm], fontsize="12") sp.set_buff_size((2000,2000)) if f == "X(C12)": sp.set_log(f, True) - sp.set_cmap(f, "plasma") - sp.set_zlim(f, 1.e-8, 2.e-4) + sp.set_cmap(f, "magma") + sp.set_zlim(f, 1.e-8, 1.e-4) elif f == "magvel": sp.set_log(f, False) #sp.set_zlim(f, 1.e-3, 2.5e-2) - sp.set_cmap(f, "magma") + sp.set_cmap(f, "cividis") elif f == "Temp": - sp.set_log(f, False) - #sp.set_zlim(f, 1.e-3, 2.5e-2) - - elif f == "rho_enuc": sp.set_log(f, True) + sp.set_zlim(f, 5.e7, 2.e8) + + elif f == "enuc": + sp.set_log(f, True, linthresh=1.e11) + sp.set_zlim(f, 1.e11, 1.e14) sp.set_cmap(f, "plasma") - #sp.set_zlim(f, 1.e-3, 2.5e-2) sp.set_axes_unit("cm") @@ -71,5 +76,4 @@ fig.set_size_inches(8.0, 8.0) plt.tight_layout() -plt.savefig("{}_slice.pdf".format(os.path.basename(plotfile))) - +plt.savefig("{}_slice.png".format(os.path.basename(plotfile))) diff --git a/Exec/science/flame_wave/inputs_H_He/inputs.H_He.nonsquare.static.1000Hz.pslope.cool b/Exec/science/flame_wave/inputs_H_He/inputs.H_He.nonsquare.static.1000Hz.pslope.cool new file mode 100644 index 0000000000..5f440f7d36 --- /dev/null +++ b/Exec/science/flame_wave/inputs_H_He/inputs.H_He.nonsquare.static.1000Hz.pslope.cool @@ -0,0 +1,168 @@ +# ------------------ INPUTS TO MAIN PROGRAM ------------------- +max_step = 250000000 +stop_time = 3.0 + +# PROBLEM SIZE & GEOMETRY +geometry.is_periodic = 0 0 +geometry.coord_sys = 1 # 0 => cart, 1 => RZ 2=>spherical +geometry.prob_lo = 0 0 +geometry.prob_hi = 9.8304e4 2.4576e4 +amr.n_cell = 384 192 + +# >>>>>>>>>>>>> BC FLAGS <<<<<<<<<<<<<<<< +# 0 = Interior 3 = Symmetry +# 1 = Inflow 4 = SlipWall +# 2 = Outflow 5 = NoSlipWall +# >>>>>>>>>>>>> BC FLAGS <<<<<<<<<<<<<<<< +castro.lo_bc = 3 3 +castro.hi_bc = 2 2 + +castro.fill_ambient_bc = 1 +castro.ambient_fill_dir = 1 +castro.ambient_outflow_vel = 1 + +castro.allow_non_unit_aspect_zones = 1 + +castro.domain_is_plane_parallel = 1 + +# WHICH PHYSICS +castro.do_hydro = 1 +castro.do_react = 1 +castro.do_rotation = 1 +castro.do_grav = 1 +castro.do_sponge = 1 + +castro.small_temp = 1.e6 +castro.small_dens = 1.e-5 + +castro.ppm_type = 1 +castro.grav_source_type = 2 +castro.use_pslope = 1 +castro.pslope_cutoff_density = 1.e4 + +gravity.gravity_type = ConstantGrav +gravity.const_grav = -1.5e14 + +castro.rotational_period = 0.001 +castro.rotation_include_centrifugal = 0 + +castro.diffuse_temp = 1 +castro.diffuse_cutoff_density_hi = 5.e4 +castro.diffuse_cutoff_density = 2.e4 + +castro.diffuse_cond_scale_fac = 1.0 + +castro.react_rho_min = 1.e2 +castro.react_rho_max = 5.e6 + +castro.react_T_min = 6.e7 + +castro.sponge_upper_density = 1.e2 +castro.sponge_lower_density = 1.e0 +castro.sponge_timescale = 1.e-7 + +# GPU options +castro.hydro_memory_footprint_ratio = 3 + +# TIME STEP CONTROL +castro.cfl = 0.8 # cfl number for hyperbolic system +castro.init_shrink = 0.1 # scale back initial timestep +castro.change_max = 1.1 # max time step growth + +castro.use_retry = 1 +castro.max_subcycles = 16 + +castro.retry_small_density_cutoff = 10.0 + +castro.abundance_failure_tolerance = 0.1 +castro.abundance_failure_rho_cutoff = 1.0 + +# DIAGNOSTICS & VERBOSITY +castro.sum_interval = 100 # timesteps between computing mass +castro.v = 1 # verbosity in Castro.cpp +amr.v = 1 # verbosity in Amr.cpp +amr.run_log = run_log +amr.run_log_terse = run_log_terse + +# REFINEMENT / REGRIDDING +amr.max_level = 2 # maximum level number allowed +amr.ref_ratio = 4 2 2 2 # refinement ratio +amr.regrid_int = 0 # static grids ftw +amr.blocking_factor = 32 # block factor in grid generation +amr.max_grid_size = 128 +amr.n_error_buf = 2 2 2 2 # number of buffer cells in error est + +# CHECKPOINT FILES +amr.check_file = flame_wave_chk # root name of checkpoint file +amr.check_int = 1000 # number of timesteps between checkpoints + +amr.checkpoint_files_output = 1 + +# PLOTFILES +amr.plot_file = flame_wave_H_He_plt # root name of plotfile +amr.plot_per = 2.e-3 # number of seconds between plotfiles +amr.derive_plot_vars = ALL +amr.plot_files_output = 1 + +amr.small_plot_file = flame_wave_H_He_smallplt # root name of plotfile +amr.small_plot_per = 2.e-4 # number of seconds between plotfiles +amr.small_plot_vars = density Temp +amr.derive_small_plot_vars = abar x_velocity y_velocity z_velocity X(H1) X(He4) X(ash) enuc + +# write plotfiles in single-precision +fab.format = NATIVE_32 + +amr.file_name_digits = 7 # pad step number with zeros if needed + +# don't write plotfiles when a stop is requested with dump_and_stop +amr.write_plotfile_with_checkpoint = 0 +castro.output_at_completion = 0 + +# problem initialization + +problem.dtemp = 1.2e9 # added to T_hi in the hot region +problem.x_half_max = 2.56e4 +problem.x_half_width = 2048.e0 + +problem.dens_base = 3.43e6 + +problem.T_star = 2.e8 +problem.T_hi = 2.e8 +problem.T_lo = 8.e6 + +problem.H_star = 2000.e0 +problem.atm_delta = 50.0 + +# helper values for initial composition +other_frac = 0.01 +h_frac = 0.10 + +problem.fuel1_name = "hydrogen-1" +problem.fuel1_frac = h_frac + +problem.fuel2_name = "helium-4" +problem.fuel2_frac = "1.0 - h_frac - other_frac" + +problem.fuel3_name = "oxygen-14" +problem.fuel3_frac = "0.35 * other_frac" + +problem.fuel4_name = "oxygen-15" +problem.fuel4_frac = "0.65 * other_frac" + +problem.ash1_name = "nickel-56" +problem.ash1_frac = 1.0 + +problem.low_density_cutoff = 1.e-4 + +problem.tag_by_density = 0 +problem.refine_height = 3600.0 +problem.max_base_tagging_level = 3 + +# Microphysics + +integrator.rtol_spec = 1.e-5 +integrator.atol_spec = 1.e-5 + +integrator.use_burn_retry = 1 + +network.use_tables = 1 diff --git a/Exec/science/flame_wave/scaling/frontier/frontier-scaling-2024-08-21.txt b/Exec/science/flame_wave/scaling/frontier/frontier-scaling-2024-08-21.txt new file mode 100644 index 0000000000..660b7af7b9 --- /dev/null +++ b/Exec/science/flame_wave/scaling/frontier/frontier-scaling-2024-08-21.txt @@ -0,0 +1,39 @@ +# new scaling numbers for the 3D XRB +# using the same inputs.He.25cm.static.1000Hz as previously +# modules: +# +# module load PrgEnv-gnu +# module load cray-mpich/8.1.28 +# module load craype-accel-amd-gfx90a +# module load amd-mixed/6.1.3 +# module unload darshan-runtime +# +# build info: +# +# EOS: /ccs/home/zingale/Microphysics/EOS/helmholtz +# NETWORK: /ccs/home/zingale/Microphysics/networks/iso7 +# CONDUCTIVITY: /ccs/home/zingale/Microphysics/conductivity/stellar +# INTEGRATOR: VODE +# SCREENING: screen5 +# +# Castro git describe: 24.08-3-g15327db6b +# AMReX git describe: 24.08-25-g6dcaa1223 +# Microphysics git describe: 24.08-2-g8ce3375a + +# nodes rocm mag_grid_size avg time / std dev +# step + 48 6.1.3 128 59.0711 0.2525 + 64 6.1.3 128 42.6938 0.285659 + 128 6.1.3 128 24.5353 1.36496 + 256 6.1.3 128 13.3647 0.108731 + 512 6.1.3 128 7.88166 0.0856889 +1024 6.1.3 128 5.54221 0.0979851 +2048 6.1.3 128 4.55679 0.0528629 + + +# note that the 2048 run uses a blocking factor of 16) + +# in contrast to the previous run, we've disabled all inlining with +# ROCm to get around some compiler bugs, so that might explain some +# slight slowdown here. + diff --git a/Exec/science/wdmerger/Prob.cpp b/Exec/science/wdmerger/Prob.cpp index ac6a41eceb..ba28a87a18 100644 --- a/Exec/science/wdmerger/Prob.cpp +++ b/Exec/science/wdmerger/Prob.cpp @@ -604,7 +604,7 @@ Castro::update_relaxation(Real time, Real dt) { const Real ldt = new_time - old_time; - force[lev].reset(new MultiFab(getLevel(lev).grids, getLevel(lev).dmap, NUM_STATE, 0)); + force[lev] = std::make_unique(getLevel(lev).grids, getLevel(lev).dmap, NUM_STATE, 0); force[lev]->setVal(0.0); MultiFab& S_new = getLevel(lev).get_new_data(State_Type); diff --git a/Exec/science/wdmerger/Problem.H b/Exec/science/wdmerger/Problem.H index db48b224e7..8696516ab6 100644 --- a/Exec/science/wdmerger/Problem.H +++ b/Exec/science/wdmerger/Problem.H @@ -27,11 +27,13 @@ void volInBoundary (amrex::Real time, amrex::Real& vol_p, amrex::Real& vol_s, am // Computes standard dot product of two three-vectors. +static amrex::Real dot_product(const amrex::Real a[], const amrex::Real b[]); // Computes norm of a three vector. +static amrex::Real norm(const amrex::Real a[]); // Problem post-initialization routine. @@ -48,6 +50,7 @@ void problem_post_timestep(); // Write out the git hashes for the various parts of the code. +static void writeGitHashes(std::ostream& log); // Update relaxation process. diff --git a/Exec/science/wdmerger/problem_checkpoint.H b/Exec/science/wdmerger/problem_checkpoint.H index 1ddee5e049..9ee3b16846 100644 --- a/Exec/science/wdmerger/problem_checkpoint.H +++ b/Exec/science/wdmerger/problem_checkpoint.H @@ -7,7 +7,7 @@ #include AMREX_INLINE -void problem_checkpoint (std::string checkpoint_dir) +void problem_checkpoint (const std::string& checkpoint_dir) { std::ofstream com; com.open(checkpoint_dir + "/COM", std::ios::out); diff --git a/Exec/science/wdmerger/problem_restart.H b/Exec/science/wdmerger/problem_restart.H index 2246732b64..3acbe99165 100644 --- a/Exec/science/wdmerger/problem_restart.H +++ b/Exec/science/wdmerger/problem_restart.H @@ -5,7 +5,7 @@ #include AMREX_INLINE -void problem_restart (std::string checkpoint_dir) +void problem_restart (const std::string& checkpoint_dir) { std::ifstream com; com.open(checkpoint_dir + "/COM", std::ios::in); diff --git a/Exec/science/wdmerger/wdmerger_util.cpp b/Exec/science/wdmerger/wdmerger_util.cpp index 65b09f89e5..6cb481b5e4 100644 --- a/Exec/science/wdmerger/wdmerger_util.cpp +++ b/Exec/science/wdmerger/wdmerger_util.cpp @@ -658,7 +658,7 @@ void binary_setup () Real v_P_r, v_S_r, v_P_phi, v_S_phi; - kepler_third_law(problem::radius_P, problem::mass_P, problem::radius_S, problem::mass_S, + kepler_third_law(problem::radius_P, problem::mass_P, problem::radius_S, problem::mass_S, // NOLINT(readability-suspicious-call-argument) castro::rotational_period, problem::orbital_eccentricity, problem::orbital_angle, problem::a, problem::r_P_initial, problem::r_S_initial, v_P_r, v_S_r, v_P_phi, v_S_phi); diff --git a/Source/driver/Castro.cpp b/Source/driver/Castro.cpp index 283cdf1e11..d3fe48b979 100644 --- a/Source/driver/Castro.cpp +++ b/Source/driver/Castro.cpp @@ -566,7 +566,7 @@ Castro::read_params () // in Amr::InitAmr(), right before the ParmParse checks, so if the user opts to // override our overriding, they can do so. - Amr::setComputeNewDtOnRegrid(1); + Amr::setComputeNewDtOnRegrid(true); // Read in custom refinement scheme. @@ -737,7 +737,7 @@ Castro::Castro (Amr& papa, } radiation->regrid(level, grids, dmap); - rad_solver.reset(new RadSolve(parent, level, grids, dmap)); + rad_solver = std::make_unique(parent, level, grids, dmap); } #endif @@ -751,7 +751,7 @@ Castro::Castro (Amr& papa, Castro::~Castro () // NOLINT(modernize-use-equals-default) { #ifdef RADIATION - if (radiation != 0) { + if (radiation != nullptr) { //radiation->cleanup(level); radiation->close(level); } @@ -3125,6 +3125,9 @@ Castro::normalize_species (MultiFab& S_new, int ng) X > 1.0_rt + castro::abundance_failure_tolerance) { #ifndef AMREX_USE_GPU std::cout << "(i, j, k) = " << i << " " << j << " " << k << " " << ", X[" << n << "] = " << X << " (density here is: " << u(i,j,k,URHO) << ")" << std::endl; +#elif defined(ALLOW_GPU_PRINTF) + AMREX_DEVICE_PRINTF("(i, j, k) = %d %d %d, X[%d] = %g (density here is: %g)\n", + i, j, k, n, X, u(i,j,k,URHO)); #endif } } diff --git a/Source/gravity/Gravity.cpp b/Source/gravity/Gravity.cpp index 18eebefbfc..c5f7bac162 100644 --- a/Source/gravity/Gravity.cpp +++ b/Source/gravity/Gravity.cpp @@ -2777,12 +2777,12 @@ Gravity::fill_direct_sum_BCs(int crse_level, int fine_level, const Vector::max()); BL_ASSERT(nPtsYZ <= std::numeric_limits::max()); - ParallelDescriptor::ReduceRealSum(bcXYLo.dataPtr(), nPtsXY); - ParallelDescriptor::ReduceRealSum(bcXYHi.dataPtr(), nPtsXY); - ParallelDescriptor::ReduceRealSum(bcXZLo.dataPtr(), nPtsXZ); - ParallelDescriptor::ReduceRealSum(bcXZHi.dataPtr(), nPtsXZ); - ParallelDescriptor::ReduceRealSum(bcYZLo.dataPtr(), nPtsYZ); - ParallelDescriptor::ReduceRealSum(bcYZHi.dataPtr(), nPtsYZ); + ParallelDescriptor::ReduceRealSum(bcXYLo.dataPtr(), static_cast(nPtsXY)); + ParallelDescriptor::ReduceRealSum(bcXYHi.dataPtr(), static_cast(nPtsXY)); + ParallelDescriptor::ReduceRealSum(bcXZLo.dataPtr(), static_cast(nPtsXZ)); + ParallelDescriptor::ReduceRealSum(bcXZHi.dataPtr(), static_cast(nPtsXZ)); + ParallelDescriptor::ReduceRealSum(bcYZLo.dataPtr(), static_cast(nPtsYZ)); + ParallelDescriptor::ReduceRealSum(bcYZHi.dataPtr(), static_cast(nPtsYZ)); #ifdef _OPENMP #pragma omp parallel diff --git a/Source/gravity/binary.H b/Source/gravity/binary.H index 684d875139..1d2438b7d4 100644 --- a/Source/gravity/binary.H +++ b/Source/gravity/binary.H @@ -67,7 +67,7 @@ void lagrange_iterate (Real& r, Real mass_1, Real mass_2, Real r1, Real r2, Real const Real tolerance = 1.0e-8_rt; const int max_iters = 200; - Real rm, rp; + Real rm{}, rp{}; if (r_min == 0.0_rt && r_max == 0.0_rt) { amrex::Abort("Lagrange point iteration must have at least one non-zero bound provided."); diff --git a/Source/radiation/HypreABec.H b/Source/radiation/HypreABec.H index 10cc5e96f6..b41f641427 100644 --- a/Source/radiation/HypreABec.H +++ b/Source/radiation/HypreABec.H @@ -33,6 +33,10 @@ class HypreABec { ~HypreABec(); + HypreABec(const HypreABec& a) = delete; + HypreABec(const HypreABec&& a) = delete; + HypreABec& operator= (const HypreABec& a) = delete; + HypreABec& operator= (const HypreABec&& a) = delete; /// /// @param v @@ -48,10 +52,10 @@ class HypreABec { /// void setScalars(amrex::Real alpha, amrex::Real beta); - amrex::Real getAlpha() const { + [[nodiscard]] amrex::Real getAlpha() const { return alpha; } - amrex::Real getBeta() const { + [[nodiscard]] amrex::Real getBeta() const { return beta; } diff --git a/Source/radiation/HypreExtMultiABec.H b/Source/radiation/HypreExtMultiABec.H index 911b2ca657..ee217fc1e1 100644 --- a/Source/radiation/HypreExtMultiABec.H +++ b/Source/radiation/HypreExtMultiABec.H @@ -23,12 +23,16 @@ class HypreExtMultiABec : public HypreMultiABec { a2coefs(fine_level+1), ccoefs( fine_level+1), d1coefs(fine_level+1), - d2coefs(fine_level+1), - alpha2(0.0), gamma(0.0), delta1(0.0), delta2(0.0) - { - } + d2coefs(fine_level+1) + {} + + ~HypreExtMultiABec() override; + + HypreExtMultiABec(const HypreExtMultiABec& src) = delete; + HypreExtMultiABec(const HypreExtMultiABec&& src) = delete; - ~HypreExtMultiABec(); + HypreExtMultiABec& operator= (const HypreExtMultiABec& src) = delete; + HypreExtMultiABec& operator= (const HypreExtMultiABec&& src) = delete; amrex::Real& a2Multiplier() { return alpha2; @@ -109,12 +113,12 @@ class HypreExtMultiABec : public HypreMultiABec { amrex::MultiFab& dest, int icomp, amrex::MultiFab& rhs, ///< will not be altered - BC_Mode inhom); + BC_Mode inhom) override; void loadLevelVectorB(int level, amrex::MultiFab& rhs, ///< will not be altered - BC_Mode inhom); + BC_Mode inhom) override; - void loadMatrix(); ///< once all level coeffs and scalars have been set + void loadMatrix() override; ///< once all level coeffs and scalars have been set /// @@ -134,7 +138,7 @@ class HypreExtMultiABec : public HypreMultiABec { amrex::Vector > > ccoefs; ///< face-based, 2 component amrex::Vector > > d1coefs; ///< cell-based but directional amrex::Vector > > d2coefs; ///< face-based - amrex::Real alpha2, gamma, delta1, delta2; ///< multipliers for the above + amrex::Real alpha2{}, gamma{}, delta1{}, delta2{}; ///< multipliers for the above }; #endif diff --git a/Source/radiation/HypreMultiABec.H b/Source/radiation/HypreMultiABec.H index 0a39aec47f..4aec6e89b9 100644 --- a/Source/radiation/HypreMultiABec.H +++ b/Source/radiation/HypreMultiABec.H @@ -21,38 +21,34 @@ class AuxVar { class Connex { public: - Connex() { - other = NULL; - } + Connex() : + other(nullptr) + {} /// /// @param p /// @param r /// - Connex(AuxVar* p, amrex::Real r) { - val = r; - other = p; - } + Connex(AuxVar* p, amrex::Real r) : + val(r), other(p) + {} /// /// @param lev /// @param iv /// @param r /// - Connex(int lev, const amrex::IntVect& iv, amrex::Real r) { - val = r; - index = iv; - level = lev; - other = NULL; - } + Connex(int lev, const amrex::IntVect& iv, amrex::Real r) : + val(r), index(iv), level(lev), other(nullptr) + {} /// /// @param c /// - bool same_target(const Connex& c) { - return ((other != NULL) + [[nodiscard]] bool same_target(const Connex& c) const { + return ((other != nullptr) ? (other == c.other) - : (c.other == NULL && level == c.level && index == c.index)); + : (c.other == nullptr && level == c.level && index == c.index)); } amrex::Real val; @@ -63,8 +59,7 @@ class AuxVar { public: - AuxVar() : secondary_flag(0) { - } + AuxVar() = default; /// @@ -72,7 +67,7 @@ class AuxVar { /// @param r /// void push(AuxVar* p, amrex::Real r) { - a.push_back(Connex(p,r)); + a.emplace_back(p, r); } @@ -82,7 +77,7 @@ class AuxVar { /// @param r /// void push(int lev, const amrex::IntVect& iv, amrex::Real r) { - a.push_back(Connex(lev,iv,r)); + a.emplace_back(lev, iv, r); } @@ -95,7 +90,7 @@ class AuxVar { /// @param p->secondary_flag /// if (p->secondary_flag == 0) { // don't count the same secondary twice - a.push_back(Connex(p,1.0)); + a.emplace_back(p, 1.0); p->secondary_flag = 1; } } @@ -104,7 +99,7 @@ class AuxVar { return a.empty(); } - bool secondary() { + [[nodiscard]] bool secondary() const { return secondary_flag; } @@ -128,7 +123,7 @@ class AuxVar { protected: std::list a; - int secondary_flag; + int secondary_flag{}; }; /// @@ -149,7 +144,7 @@ class AuxVarBox { /// @param bx /// AuxVarBox(const amrex::Box& bx) : domain(bx) { - int numpts = domain.numPts(); + auto numpts = domain.numPts(); dptr = new AuxVar[numpts]; } @@ -157,13 +152,19 @@ class AuxVarBox { delete[] dptr; } + AuxVarBox(const AuxVarBox& src) = delete; + AuxVarBox(const AuxVarBox&& src) = delete; + + AuxVarBox& operator= (const AuxVarBox& src) = delete; + AuxVarBox& operator= (const AuxVarBox&& src) = delete; + AuxVar& operator()(const amrex::IntVect& p) { BL_ASSERT(!(dptr == 0)); BL_ASSERT(domain.contains(p)); return dptr[domain.index(p)]; } - const amrex::Box& box() const { + [[nodiscard]] const amrex::Box& box() const { return domain; } @@ -346,7 +347,9 @@ class CrseBndryAuxVar : public BndryAuxVarBase { /// @param ori /// @param i /// - int size (const amrex::Orientation ori, int i) const { return aux[ori][i].size(); } + int size (const amrex::Orientation ori, int i) const { + return static_cast(aux[ori][i].size()); + } AuxVarBox& operator() (const amrex::Orientation ori, int i, int j) { return *aux[ori][i][j]; @@ -408,6 +411,11 @@ class HypreMultiABec { virtual ~HypreMultiABec(); + HypreMultiABec(const HypreMultiABec& src) = delete; + HypreMultiABec(const HypreMultiABec&& src) = delete; + + HypreMultiABec operator=(const HypreMultiABec& src) = delete; + HypreMultiABec operator=(const HypreMultiABec&& src) = delete; /// /// @param level @@ -422,10 +430,10 @@ class HypreMultiABec { const amrex::DistributionMapping& _dmap, amrex::IntVect _crse_ratio); - int crseLevel() { + [[nodiscard]] int crseLevel() const { return crse_level; } - int fineLevel() { + [[nodiscard]] int fineLevel() const { return fine_level; } @@ -501,10 +509,10 @@ class HypreMultiABec { /// void setScalars(amrex::Real alpha, amrex::Real beta); - amrex::Real getAlpha() const { + [[nodiscard]] amrex::Real getAlpha() const { return alpha; } - amrex::Real getBeta() const { + [[nodiscard]] amrex::Real getBeta() const { return beta; } diff --git a/Source/radiation/MGRadBndry.H b/Source/radiation/MGRadBndry.H index 8b922323d4..a565e2eb43 100644 --- a/Source/radiation/MGRadBndry.H +++ b/Source/radiation/MGRadBndry.H @@ -22,31 +22,28 @@ public: const int _ngroups, const amrex::Geometry& _geom); - ~MGRadBndry(); - - /// /// @param phys_bc /// @param geom /// @param ratio /// - virtual void setBndryConds(const amrex::BCRec& phys_bc, - const amrex::Geometry& geom, amrex::IntVect& ratio); + void setBndryConds(const amrex::BCRec& phys_bc, + const amrex::Geometry& geom, amrex::IntVect& ratio) override; /// /// @param bc /// @param phys_bc_mode /// - virtual void setBndryFluxConds(const amrex::BCRec& bc, - const BC_Mode phys_bc_mode = Inhomogeneous_BC); + void setBndryFluxConds(const amrex::BCRec& bc, + const BC_Mode phys_bc_mode = Inhomogeneous_BC) override; /// /// @param _face /// - virtual int mixedBndry(const amrex::Orientation& _face) const { - return (bcflag[_face] > 1) ? 1 : 0; + int mixedBndry(const amrex::Orientation& _face) const override { + return (bcflag[_face] > 1) ? 1 : 0; } @@ -99,7 +96,7 @@ public: NGBndry* operator()(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, int _ngroups, - const amrex::Geometry& _geom) const { + const amrex::Geometry& _geom) const override { /// /// @param _grids diff --git a/Source/radiation/MGRadBndry.cpp b/Source/radiation/MGRadBndry.cpp index 73c4abf41a..32ab3824c9 100644 --- a/Source/radiation/MGRadBndry.cpp +++ b/Source/radiation/MGRadBndry.cpp @@ -58,10 +58,6 @@ MGRadBndry::MGRadBndry(const BoxArray& _grids, } } -MGRadBndry::~MGRadBndry() -{ -} - void MGRadBndry::init(const int _ngroups) { // obsolete implementation of the Marshak boundary condition requires diff --git a/Source/radiation/NGBndry.H b/Source/radiation/NGBndry.H index 0fccdaa3c3..e90c1faef2 100644 --- a/Source/radiation/NGBndry.H +++ b/Source/radiation/NGBndry.H @@ -22,16 +22,10 @@ class NGBndry : public RadInterpBndryData public: NGBndry(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, int _ncomp, const amrex::Geometry& _geom) : - -/// -/// @param _grids -/// @param _dmap -/// @param _ncomp -/// @param _geom -/// RadInterpBndryData(_grids,_dmap,_ncomp,_geom) { } + /// /// @param bc /// @param phys_bc_mode @@ -51,7 +45,7 @@ public: /// /// @param _face /// - virtual int mixedBndry(const amrex::Orientation& _face) const { + virtual int mixedBndry(const amrex::Orientation& /* _face */) const { return 0; } @@ -62,13 +56,6 @@ protected: /// amrex::Vector< std::unique_ptr > > bctypearray[2*AMREX_SPACEDIM]; - -/// -/// @param src -/// -private: - NGBndry(const NGBndry& src); - NGBndry& operator=(const NGBndry& src); }; /// diff --git a/Source/radiation/RadBndry.H b/Source/radiation/RadBndry.H index 2f137f4e72..4781ee6c11 100644 --- a/Source/radiation/RadBndry.H +++ b/Source/radiation/RadBndry.H @@ -28,30 +28,28 @@ public: RadBndry(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, const amrex::Geometry& _geom, amrex::Real bv); - ~RadBndry(); - /// /// @param phys_bc /// @param geom /// @param ratio /// - virtual void setBndryConds(const amrex::BCRec& phys_bc, - const amrex::Geometry& geom, amrex::IntVect& ratio); + void setBndryConds(const amrex::BCRec& phys_bc, + const amrex::Geometry& geom, amrex::IntVect& ratio) override; /// /// @param bc /// @param phys_bc_mode /// - virtual void setBndryFluxConds(const amrex::BCRec& bc, - const BC_Mode phys_bc_mode = Inhomogeneous_BC); + void setBndryFluxConds(const amrex::BCRec& bc, + const BC_Mode phys_bc_mode = Inhomogeneous_BC) override; /// /// @param _face /// - virtual int mixedBndry(const amrex::Orientation& _face) const { + int mixedBndry(const amrex::Orientation& _face) const override { return (bcflag[_face] > 1) ? 1 : 0; } @@ -110,8 +108,8 @@ public: /// NGBndry* operator()(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, - int _ncomp, - const amrex::Geometry& _geom) const { + int /* _ncomp */, + const amrex::Geometry& _geom) const override { return new RadBndry(_grids, _dmap, _geom); } }; diff --git a/Source/radiation/RadBndry.cpp b/Source/radiation/RadBndry.cpp index 3635f8a512..aebb5b39fc 100644 --- a/Source/radiation/RadBndry.cpp +++ b/Source/radiation/RadBndry.cpp @@ -65,10 +65,6 @@ RadBndry::RadBndry(const BoxArray& _grids, const DistributionMapping& _dmap, setBndryValues(bv); } -RadBndry::~RadBndry() -{ -} - void RadBndry::init() { // obsolete implementation of the Marshak boundary condition requires diff --git a/Source/radiation/RadSolve.H b/Source/radiation/RadSolve.H index d85c6dee43..2fb694ab03 100644 --- a/Source/radiation/RadSolve.H +++ b/Source/radiation/RadSolve.H @@ -29,7 +29,6 @@ class RadSolve { RadSolve (amrex::Amr* Parent, int level, const amrex::BoxArray& grids, const amrex::DistributionMapping& dmap); - ~RadSolve () {} /// /// query runtime parameters diff --git a/Source/radiation/Radiation.H b/Source/radiation/Radiation.H index 58be11061a..7aff7e5bc5 100644 --- a/Source/radiation/Radiation.H +++ b/Source/radiation/Radiation.H @@ -111,8 +111,6 @@ public: /// @param restart /// Radiation(amrex::Amr* Parent, class Castro* castro, int restart = 0); - ~Radiation() { } - /// /// @param level @@ -193,11 +191,11 @@ public: return delta_T_rat_level[lev]; } - amrex::Real deltaEnergyTol() { + [[nodiscard]] amrex::Real deltaEnergyTol() const { return delta_e_rat_dt_tol; } - amrex::Real deltaTTol() { + [[nodiscard]] amrex::Real deltaTTol() const { return delta_T_rat_dt_tol; } diff --git a/Source/radiation/_interpbndry/RadBndryData.H b/Source/radiation/_interpbndry/RadBndryData.H index 3df1041532..c58eab3d9b 100644 --- a/Source/radiation/_interpbndry/RadBndryData.H +++ b/Source/radiation/_interpbndry/RadBndryData.H @@ -94,11 +94,25 @@ public: //@ManMemo: administrative functions //@ManDoc: default constructor RadBndryData() : amrex::BndryRegister() {}; + + ~RadBndryData() = default; + + // + // Disabled! + // +//@ManDoc: copy constructor + RadBndryData(const RadBndryData& src) = delete; +//@ManDoc: copy operator + RadBndryData& operator = (const RadBndryData& src) = delete; +//@ManDoc: move constructor + RadBndryData(const RadBndryData&& src) = delete; +//@ManDoc: move operator + RadBndryData& operator = (const RadBndryData&& src) = delete; + //@ManDoc: constructor specifying number of components and box of physical domain (cell-centered) RadBndryData(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, int _ncomp, const ProxyGeometry& geom); -//@ManDoc: destructor - virtual ~RadBndryData(); + //@ManDoc: allocate bndry fabs along given face void define(const amrex::BoxArray& _grids, const amrex::DistributionMapping& _dmap, int _ncomp, const ProxyGeometry& geom); @@ -168,15 +182,6 @@ public: } -private: - // - // Disabled! - // -//@ManDoc: copy constructor - RadBndryData(const RadBndryData& src); -//@ManDoc: copy operator - RadBndryData& operator = (const RadBndryData& src); }; #endif - diff --git a/Source/radiation/_interpbndry/RadBndryData.cpp b/Source/radiation/_interpbndry/RadBndryData.cpp index 850a36e797..0439f707dc 100644 --- a/Source/radiation/_interpbndry/RadBndryData.cpp +++ b/Source/radiation/_interpbndry/RadBndryData.cpp @@ -32,10 +32,6 @@ RadBndryData::RadBndryData(const BoxArray& _grids, const DistributionMapping& _d // (*this) = src; // } -RadBndryData::~RadBndryData() -{ -} - std::ostream& operator << (std::ostream& os, const RadBndryData &mgb) { const BoxArray& grds = mgb.boxes(); @@ -129,4 +125,3 @@ RadBndryData::define(const BoxArray& _grids, const DistributionMapping& _dmap, } } } - diff --git a/Source/rotation/Castro_rotation.cpp b/Source/rotation/Castro_rotation.cpp index 9c98f878ca..67cbf15384 100644 --- a/Source/rotation/Castro_rotation.cpp +++ b/Source/rotation/Castro_rotation.cpp @@ -91,13 +91,13 @@ Castro::construct_new_rotation_source(MultiFab& source, MultiFab& state_old, Mul { const int IOProc = ParallelDescriptor::IOProcessorNumber(); Real run_time = ParallelDescriptor::second() - strt_time; - + Real llevel = level; #ifdef BL_LAZY Lazy::QueueReduction( [=] () mutable { #endif ParallelDescriptor::ReduceRealMax(run_time,IOProc); - amrex::Print() << "Castro::construct_new_rotation_source() time = " << run_time << " on level " << level << "\n" << "\n"; + amrex::Print() << "Castro::construct_new_rotation_source() time = " << run_time << " on level " << llevel << "\n" << "\n"; #ifdef BL_LAZY }); #endif diff --git a/external/Microphysics b/external/Microphysics index 14b8b0e317..3961b439ba 160000 --- a/external/Microphysics +++ b/external/Microphysics @@ -1 +1 @@ -Subproject commit 14b8b0e3173041968943d4bbac2c4803a33abceb +Subproject commit 3961b439ba5c8193975018300e4c23a09b533206 diff --git a/external/amrex b/external/amrex index ac5dde35b6..74127d6d8f 160000 --- a/external/amrex +++ b/external/amrex @@ -1 +1 @@ -Subproject commit ac5dde35b6c10f5d91e289edeff218bde84878a4 +Subproject commit 74127d6d8fa83f922069a25e7ef9f153aa73f68c