Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extract multiple metrics from golang benchmarks #119

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ jobs:
with:
node-version: 16
- uses: actions/setup-go@v1
with:
go-version: "1.17.6"
- uses: actions/cache@v1
with:
path: ~/.npm
Expand All @@ -63,7 +65,7 @@ jobs:
cp ./dev/bench/data.js before_data.js
git checkout -
- name: Run benchmark
run: cd examples/go && go test -bench 'BenchmarkFib' | tee output.txt
run: cd examples/go && go test ./fib_test.go ./fib.go -bench 'BenchmarkFib' | tee output.txt
- name: Store benchmark result
uses: ./
with:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ jobs:
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v1
with:
go-version: "1.17.6"
Comment on lines 17 to +19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we depend on a specific version of Go for this feature to work? Maybe it's worth upgrading the actions/setup-go to v4 and using go-version: 'stable'?

Suggested change
- uses: actions/setup-go@v1
with:
go-version: "1.17.6"
- uses: actions/setup-go@v4
with:
go-version: "stable"

- name: Run benchmark
run: cd examples/go && go test -bench 'BenchmarkFib' | tee output.txt

Expand Down
2 changes: 2 additions & 0 deletions examples/go/fib_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ func BenchmarkFib10(b *testing.B) {
for i := 0; i < b.N; i++ {
var _ = Fib(10)
}
b.ReportMetric(3.0, "auxMetricUnits")
}

func BenchmarkFib20(b *testing.B) {
for i := 0; i < b.N; i++ {
var _ = Fib(20)
}
b.ReportMetric(4.0, "auxMetricUnits")
}
Comment on lines 8 to 19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would actually suggest creating a third benchmark function that reports additional metrics instead of modifying the existing ones

Suggested change
for i := 0; i < b.N; i++ {
var _ = Fib(10)
}
b.ReportMetric(3.0, "auxMetricUnits")
}
func BenchmarkFib20(b *testing.B) {
for i := 0; i < b.N; i++ {
var _ = Fib(20)
}
b.ReportMetric(4.0, "auxMetricUnits")
}
for i := 0; i < b.N; i++ {
var _ = Fib(10)
}
}
func BenchmarkFib20(b *testing.B) {
for i := 0; i < b.N; i++ {
var _ = Fib(20)
}
}
func BenchmarkFib20WithAuxMetric(b *testing.B) {
for i := 0; i < b.N; i++ {
var _ = Fib(20)
}
b.ReportMetric(4.0, "auxMetricUnits")
}

25 changes: 17 additions & 8 deletions src/extract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,13 @@ function extractGoResult(output: string): BenchmarkResult[] {
// Example:
// BenchmarkFib20-8 30000 41653 ns/op
// BenchmarkDoWithConfigurer1-8 30000000 42.3 ns/op
const reExtract = /^(Benchmark\w+(?:\/?[\w()$%^&*-]*?)*?)(-\d+)?\s+(\d+)\s+([0-9.]+)\s+(.+)$/;
// Example if someone has used the ReportMetric function to add additional metrics to each benchmark:
// BenchmarkThing-16 1 95258906556 ns/op 64.02 UnitsForMeasure2 31.13 UnitsForMeasure3

// reference, "Proposal: Go Benchmark Data Format": https://go.googlesource.com/proposal/+/master/design/14313-benchmark-format.md
// "A benchmark result line has the general form: <name> <iterations> <value> <unit> [<value> <unit>...]"
// "The fields are separated by runs of space characters (as defined by unicode.IsSpace), so the line can be parsed with strings.Fields. The line must have an even number of fields, and at least four."
const reExtract = /^(Benchmark\w+(?:\/?[\w()$%^&*-]*?)*?)(-\d+)?\s+(\d+)\s+(.+)$/;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've changed this regexp quite significantly. Could you explain a bit why is this so? I thought the metrics might appear at the end of the currently handled cases so this shorter regexp is a little surprising. If you've found a way to simplify it then great but a little explanation would go a long way :)

Additionally, could we use named capture groups (link so it's easier to understand the parts of the regular expression


for (const line of lines) {
const m = line.match(reExtract);
Expand All @@ -356,15 +362,18 @@ function extractGoResult(output: string): BenchmarkResult[] {
const name = m[1];
const procs = m[2] !== undefined ? m[2].slice(1) : null;
const times = m[3];
const value = parseFloat(m[4]);
const unit = m[5];
const remainder = m[4];

let extra = `${times} times`;
if (procs !== null) {
extra += `\n${procs} procs`;
const pieces = remainder.split(/[ \t]+/);
for (let i = 0; i < pieces.length; i = i + 2) {
let extra = `${times} times`.replace(/\s\s+/g, ' ');
if (procs !== null) {
extra += `\n${procs} procs`;
}
const value = parseFloat(pieces[i]);
const unit = pieces[i + 1];
ret.push({ name, value, unit, extra });
}

ret.push({ name, value, unit, extra });
}

return ret;
Expand Down
3 changes: 3 additions & 0 deletions test/data/extract/go_output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,8 @@ BenchmarkFib20 30000 40537.123 ns/op
BenchmarkFib/my_tabled_benchmark_-_10-8 5000000 325 ns/op
BenchmarkFib/my_tabled_benchmark_-_20 30000 40537.123 ns/op
BenchmarkFib/my/tabled/benchmark_-_20 30001 40537.456 ns/op
BenchmarkFib11-16 4587789 262.7 ns/op 3.000 auxMetricUnits
BenchmarkFib22-16 37653 31915 ns/op 4.000 auxMetricUnits
PASS
PASS
ok _/Users/rhayasd/Develop/github.com/benchmark-action/github-action-benchmark/examples/go 3.614s
24 changes: 24 additions & 0 deletions test/extract.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,30 @@ describe('extractResult()', function () {
value: 40537.456,
extra: '30001 times',
},
{
name: 'BenchmarkFib11',
unit: 'ns/op',
value: 262.7,
extra: '4587789 times\n16 procs',
},
{
name: 'BenchmarkFib11',
unit: 'auxMetricUnits',
value: 3,
extra: '4587789 times\n16 procs',
},
{
name: 'BenchmarkFib22',
unit: 'ns/op',
value: 31915,
extra: '37653 times\n16 procs',
},
{
name: 'BenchmarkFib22',
unit: 'auxMetricUnits',
value: 4,
extra: '37653 times\n16 procs',
},
],
},
{
Expand Down