Benchmarking and improving HMC/NUTS

[yebai]

I suggest that we use this issue as an umbrella page for tracking improvements of benchmarking performances (memory, running time). For every new commit, we could write a small julia script to automatically (see GitHub.jl) post a new comment to this issue containing updated runtime information.

Related issues:

• Improving the speed of single program run Improving the speed of single program run Turing.jl#294
• Using function type instead of Function may improve efficiency Using function type instead of Function may improve efficiency Turing.jl#300
• Beyond vectorisation? Beyond vectorisation? Turing.jl#283
• Use vec/parallelisation/GPU to speed up link/invlink Use vec/parallelisation/GPU to speed up link/invlink Turing.jl#314
• ForwardDiff.gradient is slow ForwardDiff.gradient is slow Turing.jl#234

[yebai]

Model	Turing	Stan	Ratio
Mixture-of-Categorical	21.31	5.93	3.59
Bernoulli Model	3.52	0.02	175.88
Binormal (sampling from the prior)	3.31	0.04	82.67
School 8	2.27	0.1	22.69
LDA	84.27	3.12	27.01
Simple Gaussian Model	3.1	0.02	154.89
LDA-vec	18.79	3.12	6.02
Kid	14.62	1.46	10.01

PS. perhaps we should link back to the commit as well.

[xukai92]

Link to Benchmark spread sheet

[xukai92]

I wonder if there is a GitHub service like Travis does this.

[yebai]

It's possible to perform these benchmarking using Travis, by running each task in a separate stage. For example

• stage: LDA, run PR and master, compute runtime ratio
• stage: Mixture: run PR and master, compute runtime ratio
• ...

All these runtime ratios can be automatically posted into PR comments (Mohamed already implemented this in TuringBenchmarking), and saved in a separate repo.

Fully supporting this idea still requires some work. I think we can solve this issue after VarInfo is merged.

[xukai92]

I'm aware of that. Just wonder if there is an off-the-shelf GitHub service like that.

[yebai]

Just wonder if there is an off-the-shelf GitHub service like that.

This feature will be added to TuringBenchmarks.

[storopoli]

Guys do you need help to transfer Travis stuff to GitHub Actions?