Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove the true-SDC tolerance runtime parameters #2571

Merged
merged 27 commits into from
Sep 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
32f09b5
move SDC-specific routiens into Source/sdc
zingale Sep 18, 2023
8795437
Merge branch 'development' of github.com:amrex-astro/Castro into deve…
zingale Sep 18, 2023
30862ec
move the Newton solver into its own header
zingale Sep 18, 2023
fcf55d9
start removing density from the reacting system
zingale Sep 18, 2023
1ebc26d
the Newton stuff compiles now
zingale Sep 18, 2023
27f8b32
this builds and runs
zingale Sep 18, 2023
426b621
fix codespell
zingale Sep 18, 2023
c3a4c40
Merge branch 'development' into sdc_cleaning_III
zingale Sep 19, 2023
2298fce
some vode work
zingale Sep 19, 2023
243e09b
Merge branch 'development' into sdc_cleaning
zingale Sep 19, 2023
af5f653
Merge branch 'sdc_cleaning' into sdc_cleaning_II
zingale Sep 19, 2023
a16e3bb
Merge branch 'sdc_cleaning_II' into sdc_cleaning_III
zingale Sep 19, 2023
074590d
some more cleaning
zingale Sep 19, 2023
0ca5370
Merge branch 'development' into sdc_cleaning_II
zingale Sep 19, 2023
7097f1c
Merge branch 'sdc_cleaning_II' into sdc_cleaning_III
zingale Sep 19, 2023
e19d77b
I think this is how it should be now for VODE
zingale Sep 19, 2023
06919ca
Merge branch 'development' into sdc_cleaning_III
zingale Sep 21, 2023
893c1cb
fix
zingale Sep 21, 2023
f9be052
fix
zingale Sep 21, 2023
bbaf405
Merge branch 'development' into sdc_cleaning_III
zingale Sep 21, 2023
5a7b886
Merge branch 'development' into sdc_cleaning_III
zingale Sep 21, 2023
8cfb5be
update benchmark
zingale Sep 21, 2023
3b7d2f0
fix loop
zingale Sep 22, 2023
af1ad1c
remove sdc-specific tolerances
zingale Sep 22, 2023
e47d141
Merge branch 'development' into sdc_cleaning_IV
zingale Sep 22, 2023
94b65e0
Merge branch 'development' into sdc_cleaning_IV
zingale Sep 22, 2023
feb0039
fix tols
zingale Sep 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 6 additions & 16 deletions Docs/source/sdc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,23 +42,13 @@ The options that affect the nonlinear solve are:
* 3 : use VODE for the first iteration and then Newton for the
subsequent iterations.

In all cases, the type of Jacobian (analytic or numerical) is determined by
``integrator.jacobian``.

* ``sdc_solver_tol_dens`` : the relative error on the density in solving the nonlinear system.

* ``sdc_solver_tol_spec`` : the relative error on the partial densities, :math:`(\rho X_k)`
in the nonlinear solve.

* ``sdc_solver_tol_ener`` : the relative error of the energy in the nonlinear solve.
The tolerances for both the Newton and VODE solver are controlled by
the usual Microphysics parameters: ``integrator.rtol_spec``,
``integrator.atol_spec``, ``integrator.rtol_enuc``,
``integrator.atol_enuc``.

* ``sdc_solver_atol`` : the absolute error in the mass fractions during the nonlinear solve.

* ``sdc_solver_relax_factor`` : the factor by which to relax the
tolerances (i.e. increase them) for earlier iterations. We reach
the desired tolerances on the final iteration.

* ``sdc_solve_for_rhoe`` : whether we solve the system in terms of :math:`(\rho e)` or :math:`(\rho E)`.
In all cases, the type of Jacobian (analytic or numerical) is determined by
``integrator.jacobian``.



Expand Down
9 changes: 5 additions & 4 deletions Exec/reacting_tests/bubble_convergence/inputs_2d.128
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1

castro.grav_source_type = 2
Expand Down Expand Up @@ -100,3 +96,8 @@ integrator.atol_spec = 1.e-10
network.small_x = 1.e-10

integrator.rtol_enuc = 1.e-10

integrator.rtol_spec = 1.e-10
integrator.rtol_enuc = 1.e-5
integrator.atol_spec = 1.e-10
integrator.atol_enuc = 1.e-10
9 changes: 4 additions & 5 deletions Exec/reacting_tests/bubble_convergence/inputs_2d.256
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1

castro.grav_source_type = 2
Expand Down Expand Up @@ -99,4 +95,7 @@ integrator.atol_spec = 1.e-10

network.small_x = 1.e-10

integrator.rtol_enuc = 1.e-10
integrator.rtol_spec = 1.e-10
integrator.rtol_enuc = 1.e-5
integrator.atol_spec = 1.e-10
integrator.atol_enuc = 1.e-10
8 changes: 2 additions & 6 deletions Exec/reacting_tests/bubble_convergence/inputs_2d.32
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1

castro.grav_source_type = 2
Expand Down Expand Up @@ -92,7 +88,7 @@ amr.refine.tempgrad.field_name = Temp

integrator.rtol_spec = 1.e-10
integrator.atol_spec = 1.e-10
integrator.rtol_enuc = 1.e-5
integrator.atol_enuc = 1.e-10

network.small_x = 1.e-10

integrator.rtol_enuc = 1.e-10
8 changes: 2 additions & 6 deletions Exec/reacting_tests/bubble_convergence/inputs_2d.512
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1

castro.grav_source_type = 2
Expand Down Expand Up @@ -97,7 +93,7 @@ amr.refine.tempgrad.field_name = Temp

integrator.rtol_spec = 1.e-10
integrator.atol_spec = 1.e-10
integrator.rtol_enuc = 1.e-5
integrator.atol_enuc = 1.e-10

network.small_x = 1.e-10

integrator.rtol_enuc = 1.e-10
8 changes: 2 additions & 6 deletions Exec/reacting_tests/bubble_convergence/inputs_2d.64
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1

castro.grav_source_type = 2
Expand Down Expand Up @@ -96,7 +92,7 @@ amr.refine.tempgrad.field_name = Temp

integrator.rtol_spec = 1.e-10
integrator.atol_spec = 1.e-10
integrator.rtol_enuc = 1.e-5
integrator.atol_enuc = 1.e-10

network.small_x = 1.e-10

integrator.rtol_enuc = 1.e-10
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1"

srun ${CASTRO_EXEC} inputs_2d.64 ${RUNPARAMS} amr.plot_file=bubble_64_sdc4_plt &> 64.out
Expand Down
3 changes: 0 additions & 3 deletions Exec/reacting_tests/reacting_convergence/convergence_sdc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ castro.sdc_order=2
castro.time_integration_method=2
castro.ppm_type=0
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_ener=1.e-10
castro.sdc_solver=1
castro.use_retry=0"

Expand Down
3 changes: 0 additions & 3 deletions Exec/reacting_tests/reacting_convergence/convergence_sdc4.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_ener=1.e-10
castro.sdc_solver=1
castro.use_retry=0"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,8 @@ amr.refine.denerr.field_name = density

network.small_x = 1.e-10

integrator.atol_spec = 1.e-12
integrator.rtol_spec = 1.e-6
integrator.atol_spec = 1.e-10
integrator.rtol_enuc = 1.e-6
integrator.atol_enuc = 1.e-10

Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@ castro.time_integration_method=2
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_spec=1.e-10
castro.sdc_solver_tol_dens=1.e-10
castro.sdc_solver_tol_ener=1.e-5
castro.sdc_solver_atol=1.e-10
castro.sdc_solver=1"

srun ${CASTRO_EXEC} inputs.64 amr.plot_file=react_converge_64_sdc4_plt ${RUNPARAMS}
Expand Down
10 changes: 6 additions & 4 deletions Exec/science/Detonation/inputs-det-x.sdc.test
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,6 @@ castro.time_integration_method = 2
castro.sdc_order = 4

castro.sdc_solver = 2
castro.sdc_solver_tol_dens = 1.e-5
castro.sdc_solver_tol_spec = 1.e-5
castro.sdc_solver_tol_ener = 1.e-5
castro.sdc_solver_atol = 1.e-10

castro.ppm_type = 0

Expand Down Expand Up @@ -101,4 +97,10 @@ amr.refine.tempgrad.field_name = Temp
# Microphysics

integrator.call_eos_in_rhs = 1

integrator.rtol_spec = 1.e-5
integrator.rtol_enuc = 1.e-5
integrator.atol_spec = 1.e-10
integrator.atol_enuc = 1.e-10

integrator.jacobian = 1
11 changes: 4 additions & 7 deletions Exec/science/Detonation/nse_runs/inputs.template.true_sdc
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,6 @@ castro.sdc_quadrature = @@SDC_QUADRATURE@@
castro.limit_fourth_order=1
castro.use_reconstructed_gamma1=1
castro.sdc_solve_for_rhoe=1
castro.sdc_solver_tol_dens=1.e-8
castro.sdc_solver_tol_spec=1.e-8
castro.sdc_solver_tol_ener=1.e-8
castro.sdc_solver_atol = 1.e-8
castro.sdc_solver=2


Expand Down Expand Up @@ -111,9 +107,10 @@ amr.refine.tempgrad.field_name = Temp

integrator.call_eos_in_rhs = 1

integrator.rtol_spec = 1.e-6
integrator.atol_spec = 1.e-6
integrator.rtol_enuc = 1.e-6
integrator.rtol_spec = 1.e-8
integrator.atol_spec = 1.e-8
integrator.rtol_enuc = 1.e-8
integrator.atol_enuc = 1.e-8

integrator.jacobian = 2
integrator.ode_max_steps = 1500000
10 changes: 4 additions & 6 deletions Exec/science/flame/inputs.1d.sdc
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,6 @@ castro.time_integration_method = 2
castro.sdc_order = 4

castro.sdc_solve_for_rhoe = 1
castro.sdc_solver_tol_dens = 1.e-10
castro.sdc_solver_tol_spec = 1.e-10
castro.sdc_solver_tol_ener = 1.e-6
castro.sdc_solver_atol = 1.e-10
castro.sdc_solver = 1
castro.sdc_solver_relax_factor = 1

Expand Down Expand Up @@ -107,8 +103,10 @@ amr.refine.dengrad.field_name = density

# Microphysics

integrator.rtol_spec = 1.e-8
integrator.atol_spec = 1.e-8
integrator.rtol_spec = 1.e-10
integrator.atol_spec = 1.e-10
integrator.rtol_enuc = 1.e-6
integrator.atol_enuc = 1.e-10

network.small_x = 1.e-10

Expand Down
16 changes: 0 additions & 16 deletions Source/driver/_cpp_parameters
Original file line number Diff line number Diff line change
Expand Up @@ -267,22 +267,6 @@ sdc_extra int 0
# which SDC nonlinear solver to use? 1 = Newton, 2 = VODE, 3 = VODE for first iter
sdc_solver int 1

# relative tolerance for the nonlinear solve on rho X_k with SDC
sdc_solver_tol_spec Real 1.e-6

# relative tolerance for the nonlinear solve on (rho e) or (rho E) with SDC
sdc_solver_tol_ener Real 1.e-6

# absolute tolerance for species with SDC (this will be multiplied by
# the current rho in the zone to define the absolute tolerance on (rho
# X)).
sdc_solver_atol Real 1.e-10

# factor by which to reduce the SDC solver tol for each iteration before the last
# (e.g., for iteration k out of kmax iterations, the tol is
# :math:`\epsilon/f^{(k_\mathrm{max} - k)}`
sdc_solver_relax_factor Real 1.0

# do we solve for (rho e) or (rho E) in the SDC nonlinear solve?
sdc_solve_for_rhoe int 1

Expand Down
14 changes: 4 additions & 10 deletions Source/sdc/sdc_newton_solve.H
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,6 @@ f_sdc_jac(const Real dt_m,

// get the Jacobian.

/// First we get the dR/dU expression.
// Instead of the decomposition into dw/dU and dR/dw
// written out in the original paper Appendix A, we instead use the
// form from the simplified-SDC paper (Zingale et al. 2022). Note:
Expand Down Expand Up @@ -157,12 +156,6 @@ sdc_newton_solve(const Real dt_m,

ierr = NEWTON_SUCCESS;

// the tolerance we are solving to may depend on the
// iteration
Real relax_fac = std::pow(sdc_solver_relax_factor, sdc_order - sdc_iteration - 1);
Real tol_spec = sdc_solver_tol_spec * relax_fac;
Real tol_ener = sdc_solver_tol_ener * relax_fac;

// update the density and momenta for this zone -- they don't react

U_new[URHO] = U_old[URHO] + dt_m * C[URHO];
Expand Down Expand Up @@ -264,10 +257,11 @@ sdc_newton_solve(const Real dt_m,
// multiply by density to get a partial density limit

for (int n = 0; n < NumSpec; ++n) {
eps_tot(1 + n) = tol_spec * std::abs(U_react(1 + n)) +
sdc_solver_atol * std::abs(U_new[URHO]);
eps_tot(1 + n) = integrator_rp::rtol_spec * std::abs(U_react(1 + n)) +
integrator_rp::atol_spec * std::abs(U_new[URHO]);
}
eps_tot(NumSpec+1) = tol_ener * std::abs(U_react(NumSpec+1)) + sdc_solver_atol;
eps_tot(NumSpec+1) = integrator_rp::rtol_enuc * std::abs(U_react(NumSpec+1)) +
integrator_rp::atol_enuc;

// compute the norm of the weighted error, where the
// weights are 1/eps_tot
Expand Down
14 changes: 4 additions & 10 deletions Source/sdc/vode_rhs_true_sdc.H
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,6 @@ sdc_vode_solve(const Real dt_m,

#if (INTEGRATOR == 0)

// The tolerance we are solving to may depend on the
// iteration
auto relax_fac = std::pow(sdc_solver_relax_factor, sdc_order - sdc_iteration - 1);
auto tol_spec = sdc_solver_tol_spec * relax_fac;
auto tol_ener = sdc_solver_tol_ener * relax_fac;

// Update the momenta for this zone -- they don't react

for (int n = 0; n < 3; ++n) {
Expand Down Expand Up @@ -207,12 +201,12 @@ sdc_vode_solve(const Real dt_m,
dvode_state.jacobian_type = integrator_rp::jacobian;

// relative tolerances
dvode_state.rtol_spec = tol_spec;
dvode_state.rtol_enuc = tol_ener;
dvode_state.rtol_spec = integrator_rp::rtol_spec;
dvode_state.rtol_enuc = integrator_rp::rtol_enuc;

// absolute tolerances
dvode_state.atol_spec = sdc_solver_atol * U_old[URHO]; // this way, atol is the minimum x
dvode_state.atol_enuc = sdc_solver_atol * U_old[UEINT];
dvode_state.atol_spec = integrator_rp::atol_spec * U_old[URHO]; // this way, atol is the minimum x
dvode_state.atol_enuc = integrator_rp::atol_enuc * U_old[UEINT];

int istate = dvode(burn_state, dvode_state);

Expand Down