Add automated benchmarks

[GeorgeR227]

This PR adds scripts to be able to run automated Decapodes benchmarks using SLURM. Close #255.

[GeorgeR227]

Example table produced by this pipeline:

float_type	code_target	resolution	Setup(s)	Mesh(s)	Simulate(s)	Solve(s)
Float32	CPUTarget	1	0.00442026	2.06637	0.0048096	0.407688
Float32	CPUTarget	1	0.00606146	1.25604	0.00592313	0.539786
Float32	CPUTarget	2	0.00604858	0.0633645	0.00154441	0.129122
Float32	CPUTarget	2	0.00476156	0.0485201	0.00120713	0.0989334
Float32	CPUTarget	5	0.00601547	0.00268157	0.00026683	0.0207069
Float32	CPUTarget	5	0.00491691	0.00213588	0.000213745	0.0166049
Float64	CPUTarget	1	0.00578402	2.60628	0.00817521	0.582806
Float64	CPUTarget	1	0.00421326	1.29076	0.00487088	0.423234
Float64	CPUTarget	2	0.00604492	0.0653029	0.00152405	0.120193
Float64	CPUTarget	2	0.00606543	0.0722557	0.0016161	0.130485
Float64	CPUTarget	5	0.00578138	0.00266238	0.000281898	0.0200672
Float64	CPUTarget	5	0.0065762	0.00273011	0.000276936	0.022008

[GeorgeR227]

Tables now include more info, plus fixed bug where the times were getting put in incorrect columns:

Task ID	float_type	code_target	resolution	Setup	Mesh	Simulate	Solve	Steps	Steps/Second
3	Float32	CPUTarget	1	0.00425298	0.400341	0.00475522	0.808238	1554	1922
6	Float32	CPUTarget	1	0.00433072	0.410321	0.00477151	0.840614	1554	1848
2	Float32	CPUTarget	2	0.00409975	0.097131	0.00121753	0.0544297	401	7367
5	Float32	CPUTarget	2	0.00429304	0.0966397	0.0012154	0.0554229	401	7235
1	Float32	CPUTarget	5	0.00414652	0.0156887	0.000215182	0.00250875	78	31091
4	Float32	CPUTarget	5	0.00415102	0.0157135	0.000207889	0.00235587	78	33108
12	Float64	CPUTarget	1	0.00469358	0.463062	0.00565343	1.32324	1549	1170
9	Float64	CPUTarget	1	0.00574905	0.488389	0.00629864	1.24648	1549	1242
11	Float64	CPUTarget	2	0.00573913	0.118075	0.00158401	0.0829361	395	4762
8	Float64	CPUTarget	2	0.00537613	0.117056	0.00159936	0.0816467	395	4837
10	Float64	CPUTarget	5	0.00558345	0.0196298	0.000277577	0.00262849	72	27392
7	Float64	CPUTarget	5	0.00542299	0.0189401	0.000276971	0.00265752	72	27092

[GeorgeR227]

I've just integrated DrWatson.jl into the benchmarking suite. It's ability to handle different input parameters and keep track of file directories is awesome but it seems to have trouble with reading in the data from the saved benchmark table correctly. DrWatson really seems to prefer working with JLD2 but BenchmarkTools saves its results as JSON and also seems to include a lot of extra information.

[jpfairbanks]

I followed these instructions and got this error message

(Decapodes) pkg> instantiate

julia> @quickactivate :benchmarks
ERROR: ArgumentError: Package benchmarks not found in current path.
- Run `import Pkg; Pkg.add("benchmarks")` to install the benchmarks package.
Stacktrace:
 [1] macro expansion
   @ Base ./loading.jl:1766 [inlined]
 [2] macro expansion
   @ Base ./lock.jl:267 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1747
 [4] #invoke_in_world#3
   @ Base ./essentials.jl:921 [inlined]
 [5] invoke_in_world
   @ Base ./essentials.jl:918 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1740
 [7] macro expansion
   @ ~/.julia/packages/DrWatson/qmLuV/src/project_setup.jl:213 [inlined]
 [8] top-level scope
   @ REPL[6]:1

[jpfairbanks]

(Decapodes) pkg> dev ./benchmarks fixed it. although I got a Circular dependency warning. so now i can't precompile Decapodes and benchmarks.

[jpfairbanks]

it seems like the instructions should be

1. Clone the Decapodes repo
2. cd to the Decapodes/benchmarks directory
3. run julia>]dev ..
4. run julia> include("main.jl")

and this main.jl should run all the benchmarks that are configured in the main_config.toml file. This way it is more like docs builds than like an interactive tool.

[jpfairbanks]

maybe we want main.jl to define a function benchmark(name::String, config="main_config.toml") that runs a simulation with the name and benchmark(config="main_config.toml") that runs all the configured benchmarks

[GeorgeR227]

Yeah, for the main.jl I can add that. Actually a lot of the code there can probably refactored.

I understand the confusion with the DrWatson instructions, the @quickactivate works by searching upwards for a toml name with benchmark so running it from Decapodes won't work, as you've seen. However, running it from within the benchmarks dir should work. I'll add an instruction asking the user to cd into benchmarks.

[jpfairbanks]

If you don't call this the error message is quite cryptic. maybe the main.jl script should check that the configs exist and if not, then automatically call config_generate for you?

[GeorgeR227]

If we do this, we should just always call config_generate to avoid stale configs. I'd like to update the that cryptic error, was the error you referenced the, "No configuration information found for $sim_name"?

[jpfairbanks]

Yeah that was the cryptic error. I feel like a makefile approach of conditional compilation is such a good fit for what you are doing. If the individual configs don't exists, then you need to regenerate them. But I think we can get rid of that step entirely by a later comment

[jpfairbanks]

Instead of relying on functions with particular names, we could make a struct in Decapodes that is SimulationConfig that has these 4 functions as fields and then constructors can provide default implementations of these fields if it makes sense to go with a generic implementation.

That would give users a way of encapsulating a simulation into a reusable chunk for their benchmarks, which would live outside this repo.

[GeorgeR227]

Yeah, that sounds good. This struct can live in the src and be brought into scope with the @quickactive :benchmarks. I'll also add a function field that takes in the file config information and organizes it into something the simulation can use throughout.

[jpfairbanks]

What does exp_pro stand for? A more informative name would be helpful here.

[GeorgeR227]

exp_pro was a default folder provided by DrWatson, meant for "Data from processing experiments" per their docs. But it's fair enough to consider that we don't need to actually use these folders.

[jpfairbanks]

Now that I understand that:

1. each simulation still needs code for its set up that needs to go in a folder
2. the simulations could in theory have different dependencies (because they are in their own folder)
3. there is a generate_configs script that is copying this data into individual config tomls for each simulation.

It makes sense to me that these config tomls actually live 1 per folder and specify all the configurations of the same simulation that you want to run. But the global main_config.toml is actually just a list of all the configurations that you want to run. Much like in the docs, docs/main.jl has a big list of all the pages in the docs to tell you which markdown files to include.

There is also the possibility to do a makefile with a rule like

for every folder f in benchmarks/src/$f there is a target benchmarks/results/$f/run.out and in order to make that target you need the dependencies benchmarks/src/$f/main.jl and benchmarks/src/$f/config.toml

and then you could do

> make all

and have all the benchmarks run. Or run a specific benchmark with

> make benchmarks/results/f/run.out

A make rule that depends on all the run.out targets could then be used to make the final summary tables in HTML format.

[GeorgeR227]

The idea with having different configs per architecture was that a user might want different configurations for each. For example, gpu sims could include larger mesh sizes or longer times. I think this flexibility is nice to have.

I like your idea of the role of the main_config.toml since the data processing is the same, we just change where we read the data. Plus, if we keep separate tomls for each architecture then the user can choose to run just cpu benchmarks or just gpu benchmarks.

Just thinking through the pipeline, the user can just provide their main_config.toml which lists all sims/arches to run. main.jl can then read the main config and auto-run the config generation that looks for the user provided toml for each of these sims and generates a list of configs for tasks. It'll then basically just run the benchmarks as it does now.

We can implement the make all feature by scanning the src dir for valid physics folders and then running through the same pipeline as above.

[GeorgeR227]

As for post-processing, the idea now is to collect all the data from the benchmarks into .jld2 files that can be collected into dataframes using DrWatson. I've provided a basic post-processing step in scripts/post_processing/default_out.jl but we can add more scripts for different post-processing as we find the need for it. This could include deciding to output to markdown or HTML or whatever else.

[lukem12345]

I don't see that post processing script. Did you commit it?

[GeorgeR227]

Yeah that's my bad, I just pushed those.

[jpfairbanks]

The idea with having different configs per architecture was that a user might want different configurations for each. For example, gpu sims could include larger mesh sizes or longer times. I think this flexibility is nice to have.

Yeah, I think you should be able to have multiple configurations in the same folder and then the main_config.toml should list out the ones you want to run.

So you could have heat.cpu and heat.cuda and heat.cuda-massive all in heat/config.toml and then in benchmarks/config.toml only list out the ones you want to run.