Benchmark workflows #398

mothran · 2023-02-23T00:21:03Z

Adding workflow to support basic benchmark'ing on PR and saving benchmark results to a github page.

TODO

Save criterion diff reports between the PR branch and main to the benchmarks repo pages (Cargo-criterion tool support benchmark-action/github-action-benchmark#145 potential fix here, or we customize the action itself)
Add report diffs into the comment created on the PR with charts.

.github/workflows/bench.yml

Created a Criterion compare workflow for PRs and kept the github pages workflow for all pushes to main

github-actions · 2023-02-24T16:58:55Z

Benchmark for Linux-cuda} `91d5b44`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	843.8±7.52ms 37.9 KElem/sec	N/A	N/A
fib/100/run	473.5±4.22ms 67.5 KElem/sec	N/A	N/A
fib/200/proof	849.8±8.09ms 37.6 KElem/sec	N/A	N/A
fib/200/run	470.5±4.12ms 67.9 KElem/sec	N/A	N/A

github-actions · 2023-02-24T17:01:42Z

Benchmark for Linux-default} `91d5b44`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.6±0.03s	4.6±0.02s	0.00%
fib/100/run	1784.9±6.67ms 17.9 KElem/sec	N/A	N/A
fib/200/proof	4.6±0.02s	4.6±0.05s	0.00%
fib/200/run	1787.5±6.90ms 17.9 KElem/sec	N/A	N/A

github-actions · 2023-02-24T17:08:34Z

Benchmark for macOS-default} `91d5b44`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.0±0.04s	4.0±0.02s	0.00%
fib/100/run	1073.3±16.62ms 29.8 KElem/sec	N/A	N/A
fib/200/proof	4.0±0.03s	4.1±0.04s	+2.50%
fib/200/run	1071.5±9.84ms 29.8 KElem/sec	N/A	N/A

github-actions · 2023-02-24T17:09:12Z

Benchmark for macOS-metal} `91d5b44`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	2.9±0.02s 11.2 KElem/sec	N/A	N/A
fib/100/run	1076.5±4.01ms 29.7 KElem/sec	N/A	N/A
fib/200/proof	2.9±0.01s 11.2 KElem/sec	N/A	N/A
fib/200/run	1078.7±4.61ms 29.6 KElem/sec	N/A	N/A

github-actions · 2023-02-24T20:04:35Z

Benchmark for Linux-default `689d57c`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	1046.2±5.43ms 30.5 KElem/sec	N/A	N/A
fib/100/run	365.7±56.73ms 87.4 KElem/sec	N/A	N/A
fib/200/proof	1640.1±358.34ms 19.5 KElem/sec	N/A	N/A
fib/200/run	354.5±15.88ms 90.1 KElem/sec	N/A	N/A

github-actions · 2023-02-24T20:13:16Z

Benchmark for Linux-cuda `689d57c`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	852.3±11.13ms 37.5 KElem/sec	N/A	N/A
fib/100/run	475.0±4.81ms 67.3 KElem/sec	N/A	N/A
fib/200/proof	849.8±7.01ms 37.6 KElem/sec	N/A	N/A
fib/200/run	476.2±3.97ms 67.1 KElem/sec	N/A	N/A

github-actions · 2023-02-24T20:21:39Z

Benchmark for macOS-default `689d57c`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.1±0.07s	4.1±0.05s	0.00%
fib/100/run	1073.0±11.67ms 29.8 KElem/sec	N/A	N/A
fib/200/proof	4.2±0.02s	4.2±0.02s	0.00%
fib/200/run	1066.7±5.73ms 30.0 KElem/sec	N/A	N/A

github-actions · 2023-02-24T20:22:55Z

Benchmark for macOS-metal `689d57c`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	2.8±0.01s 11.2 KElem/sec	N/A	N/A
fib/100/run	1072.1±6.08ms 29.8 KElem/sec	N/A	N/A
fib/200/proof	2.8±0.01s 11.2 KElem/sec	N/A	N/A
fib/200/run	1076.4±6.55ms 29.7 KElem/sec	N/A	N/A

github-actions · 2023-02-24T21:00:33Z

Benchmark for Linux-cuda `bb9d31b`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	845.2±8.67ms 37.8 KElem/sec	N/A	N/A
fib/100/run	472.5±4.58ms 67.6 KElem/sec	N/A	N/A
fib/200/proof	854.2±12.17ms 37.4 KElem/sec	N/A	N/A
fib/200/run	474.9±2.39ms 67.3 KElem/sec	N/A	N/A

github-actions · 2023-02-24T21:01:23Z

Benchmark for macOS-metal `bb9d31b`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	2.8±0.01s 11.3 KElem/sec	N/A	N/A
fib/100/run	1074.6±7.24ms 29.7 KElem/sec	N/A	N/A
fib/200/proof	2.8±0.01s 11.3 KElem/sec	N/A	N/A
fib/200/run	1077.8±8.90ms 29.6 KElem/sec	N/A	N/A

github-actions · 2023-02-24T21:01:45Z

Benchmark for macOS-default `bb9d31b`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.1±0.04s	4.1±0.04s	0.00%
fib/100/run	1068.4±7.03ms 29.9 KElem/sec	N/A	N/A
fib/200/proof	4.2±0.03s	4.2±0.05s	0.00%
fib/200/run	1066.7±5.41ms 30.0 KElem/sec	N/A	N/A

github-actions · 2023-02-24T21:10:57Z

Benchmark for Linux-default `bb9d31b`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.6±0.04s	4.6±0.03s	0.00%
fib/100/run	1777.3±2.38ms 18.0 KElem/sec	N/A	N/A
fib/200/proof	4.6±0.03s	4.6±0.03s	0.00%
fib/200/run	1775.9±2.02ms 18.0 KElem/sec	N/A	N/A

flaub

Nice!!

github-actions · 2023-02-24T21:46:36Z

Benchmark for Linux-default `73f03f1`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	1468.8±312.96ms	1541.4±206.40ms	+4.94%
fib/100/run	357.8±24.78ms	369.9±44.26ms	+3.38%
fib/200/proof	1429.7±234.35ms	1734.5±317.36ms	+21.32%
fib/200/run	350.1±9.68ms	393.6±58.05ms	+12.43%

github-actions · 2023-02-24T21:50:03Z

Benchmark for Linux-cuda `73f03f1`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	843.3±5.93ms	844.6±8.17ms	+0.15%
fib/100/run	468.1±3.56ms	472.2±3.93ms	+0.88%
fib/200/proof	850.8±6.67ms	851.3±6.91ms	+0.06%
fib/200/run	471.5±3.32ms	473.1±5.22ms	+0.34%

github-actions · 2023-02-24T21:59:27Z

Benchmark for macOS-metal `73f03f1`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	2.8±0.01s	2.8±0.02s	0.00%
fib/100/run	1074.5±2.80ms	1074.6±3.52ms	+0.01%
fib/200/proof	2.8±0.01s	2.8±0.01s	0.00%
fib/200/run	1075.3±5.68ms	1076.6±4.37ms	+0.12%

github-actions · 2023-02-24T21:59:58Z

Benchmark for macOS-default `73f03f1`

Click to view benchmark

Test	Base	PR	%
fib/100/proof	4.0±0.06s	4.0±0.05s	0.00%
fib/100/run	1069.1±14.38ms	1079.0±14.14ms	+0.93%
fib/200/proof	4.1±0.05s	4.1±0.07s	0.00%
fib/200/run	1071.7±9.93ms	1074.7±12.14ms	+0.28%

Added first rev of benchmark workflow

0619e4a

mothran requested a review from flaub February 23, 2023 00:21

Fixed matrix.arch typo

00e84df

flaub reviewed Feb 23, 2023

View reviewed changes

.github/workflows/bench.yml Outdated Show resolved Hide resolved

mothran added 7 commits February 22, 2023 18:35

Update repo name

1feb076

Drops token PERMS and moved gh-action-bench to v1

1568356

Broke up benchmarks into two workflows

b816027

Created a Criterion compare workflow for PRs and kept the github pages workflow for all pushes to main

Merge branch 'main' into parker/benchmarks

c608165

Added required branchName to workflow

458ba6b

corrected benchmark name

3951f6a

test change

333cc41

mothran added 2 commits February 24, 2023 10:42

Fixed trailing }

a4b814a

Added PR trigger for testing

0e9b54d

Removed the PR trigger for bench_trends

29d7b3a

mothran marked this pull request as ready for review February 24, 2023 21:18

flaub approved these changes Feb 24, 2023

View reviewed changes

Test commit - testing PR check workflow updates

50ab831

flaub assigned mothran Feb 24, 2023

mothran merged commit d6e4d91 into main Feb 24, 2023

mothran deleted the parker/benchmarks branch February 24, 2023 23:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark workflows #398

Benchmark workflows #398

mothran commented Feb 23, 2023 •

edited

Loading

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

flaub left a comment

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

github-actions bot commented Feb 24, 2023

Benchmark workflows #398

Benchmark workflows #398

Conversation

mothran commented Feb 23, 2023 • edited Loading

TODO

github-actions bot commented Feb 24, 2023

Benchmark for Linux-cuda} 91d5b44

github-actions bot commented Feb 24, 2023

Benchmark for Linux-default} 91d5b44

github-actions bot commented Feb 24, 2023

Benchmark for macOS-default} 91d5b44

github-actions bot commented Feb 24, 2023

Benchmark for macOS-metal} 91d5b44

github-actions bot commented Feb 24, 2023

Benchmark for Linux-default 689d57c

github-actions bot commented Feb 24, 2023

Benchmark for Linux-cuda 689d57c

github-actions bot commented Feb 24, 2023

Benchmark for macOS-default 689d57c

github-actions bot commented Feb 24, 2023

Benchmark for macOS-metal 689d57c

github-actions bot commented Feb 24, 2023

Benchmark for Linux-cuda bb9d31b

github-actions bot commented Feb 24, 2023

Benchmark for macOS-metal bb9d31b

github-actions bot commented Feb 24, 2023

Benchmark for macOS-default bb9d31b

github-actions bot commented Feb 24, 2023

Benchmark for Linux-default bb9d31b

flaub left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 24, 2023

Benchmark for Linux-default 73f03f1

github-actions bot commented Feb 24, 2023

Benchmark for Linux-cuda 73f03f1

github-actions bot commented Feb 24, 2023

Benchmark for macOS-metal 73f03f1

github-actions bot commented Feb 24, 2023

Benchmark for macOS-default 73f03f1

mothran commented Feb 23, 2023 •

edited

Loading

Benchmark for Linux-cuda} `91d5b44`

Benchmark for Linux-default} `91d5b44`

Benchmark for macOS-default} `91d5b44`

Benchmark for macOS-metal} `91d5b44`

Benchmark for Linux-default `689d57c`

Benchmark for Linux-cuda `689d57c`

Benchmark for macOS-default `689d57c`

Benchmark for macOS-metal `689d57c`

Benchmark for Linux-cuda `bb9d31b`

Benchmark for macOS-metal `bb9d31b`

Benchmark for macOS-default `bb9d31b`

Benchmark for Linux-default `bb9d31b`

Benchmark for Linux-default `73f03f1`

Benchmark for Linux-cuda `73f03f1`

Benchmark for macOS-metal `73f03f1`

Benchmark for macOS-default `73f03f1`