Notes on Reproducibility/Determinism

[alwinm]

I intend this post to be mostly educative and wiki-like.

I am not a very precise person--for now when I say reproducibility and determinism I really mean three questions:

"can we run Cholla twice on the same machine with the same binary on the same inputs and get the same outputs."
"can we run Cholla from a restart and get the same outputs as if we ran Cholla without a restart interrupt"
"can we run Cholla with a different number of ranks and get the same output"

Any failure of the three, in my opinion, indicates one of these:

• A bug
• An unavoidable feature
• A design choice to sacrifice one of the three

The latter two are a bit of a spectrum, as some design choices to avoid sacrificing one of the three may be so costly that it would be deemed impossible, hence unavoidable.

For now, I document the three major sources of indeterminism I am aware of:

atomicAdd.

Not all atomicAdds cause this, but for any kernel with multiple atomicAdds to the same address, the atomicAdds may occur in indeterminate order, which creates indeterminism because floating point operations are not associative (order matters).

• Density_CIC_gpu (multiple close-by particles may atomicAdd to the same cell). Ameliorate by preventing particle overlap.
• Feedback_CIC_gpu (multiple close-by stellar particles may atomicAdd to the same cell). Ameliorate by preventing particle overlap.

PRNG

Although possible to massage PRNGs to be fully deterministic, in many cases they have not been, because it can be costly in terms of developer time to do this massage.

• Feedback_CIC_gpu (poisson random supernovae).
• Write_Rotated_Projection_HDF5 (randomize cell centers slightly to prevent aliasing). Only indeterminate when changing number of ranks.

Read_Grid

Some physics modules may need to save additional state for a restart to be fully consistent with a continuous run.

• Gravity requires a previous timestep and previous potential (potential_1_h, potential_1_d) to extrapolate half step potential.
• Feedback_CIC_gpu (for now) requires PRNG state

[alwinm]

Note: neighboring average slow cells data race?

[bcaddy]

For the PRNG: could we just have it print out the seed it used? Then at least we can rerun that exact simulation if we want.

[alwinm]

Any changes in the number of ranks you use will cause differences which is not strictly necessary but it's nice when two simulations using a different number of ranks have the same output. PRNG state did not move across ranks but particles do.

In the form I've put into the PR and talked out with Orlando, a given particle ID at a given n_step with given seed will always produce the same random number. This also removes the requirement that the feedback PRNG saves state.