Proper support for run!(simulation, pickup=true) with OceanSeaIceModel

[glwagner]

@NoraLoose wrote:

I run this notebook, but with checkpointing

ocean.output_writers[:checkpointer] = Checkpointer(ocean.model; schedule=TimeInterval(10days), prefix="model_checkpoint")

This saves a series of checkpoint files in the working directory, with the latest being model_checkpoint_iteration13920.jld2.

To pick up from this checkpoint, I re-run the same notebook up to the end of the section "Set up output writers" and then try a bunch of different things: First,

run!(simulation, pickup=true)

which gives me the error

No checkpointers found: cannot pickup simulation!

even though I run the notebook from the same working directory. Next, I try

run!(simulation, pickup="./model_checkpoint_iteration13920.jld2")

which gives me the error

type OceanSeaIceModel has no field timestepper

Originally posted by @NoraLoose in #302

[glwagner]

The interface for checkpointing is generalized slightly in this Oceananigans PR: CliMA/Oceananigans.jl#4003

Once that is merged, we should be able to extend set! for Simulation{<:OceanSeaIceModel} and thereby correctly pick up from checkpoints.

[glwagner]

We also may want to extend Checkpointer to work with OceanSeaIceModel:

https://github.com/CliMA/Oceananigans.jl/blob/0b6e88e90982cdb3a92d7afeca6eaa307b35a13a/src/OutputWriters/checkpointer.jl#L64-L71

I think it would be a good exercise to try to do this, which might expose some changes that are also needed in Oceananigans.OutputWriters.

[NoraLoose]

I'm not sure if this is relevant to the discussion, but currently

latest_checkpoint = "model_checkpoint_iteration13920.jld2"
set!(ocean.model, latest_checkpoint)

triggers the warning

┌ Warning: Tendencies for η do not exist in checkpoint and could not be restored.
└ @ Oceananigans.OutputWriters /pscratch/sd/n/nloose/depot/packages/Oceananigans/ohMpu/src/OutputWriters/checkpointer.jl:254

[glwagner]

Actually, I think was a bug/inconsistency in the checkpointer. On 0.95.3 we used fields(model) to figure out which fields to save:

https://github.com/CliMA/Oceananigans.jl/blob/87ecfedf9a045d7bb5733eb6be1dddf2f75200ec/src/OutputWriters/checkpointer.jl#L167

but then prognostic_fields(model) to load them back:

https://github.com/CliMA/Oceananigans.jl/blob/87ecfedf9a045d7bb5733eb6be1dddf2f75200ec/src/OutputWriters/checkpointer.jl#L210

The distinction between fields and prognostic_fields is kind of annoying (we should probably fix that), but the easier fix is to use prognostic_fields(model) when writing output. I think this change was made on main since there I find

https://github.com/CliMA/Oceananigans.jl/blob/da6ee8745fda308d75a7d8388a47992ebb7f1cc7/src/OutputWriters/checkpointer.jl#L187

@NoraLoose possibly you can try re-running your problem (unfortunately this also requires regenerating the checkpointing data) on main to see if the issue is fixed.

Apparently, we don't test whether the free surface is checkpointed correctly. Maybe we should add a test like that to Oceananigans.

[NoraLoose]

This is great - thanks @glwagner!