Implement a custom benchmark runner

[jack-weilage]

What is the problem this feature would solve?

Javascript currently doesn't have a "perfect" benchmark runner. Many solutions either require extensive configuration or don't accept any at all. Many are unmaintained or inaccurate.

Bun could jump in (much like deno did) and plug this gap.

What is the feature you are proposing to solve the problem?

A bench and baseline method could be added to bun:test, extending the pattern of test. bench could be called in *.test.{js|ts}, just like test.

import { sleepSync } from 'bun'
import { describe, test, bench, baseline } from 'bun:test'

test('1 + 1 = 2', () => {
  expect(1 + 1).toBe(2)
})
describe('performance', () => {
  bench('wait 1ms', () => {
    sleepSync(1)
  })
  baseline('wait 5ms', () => {
    sleepSync(5)
  })
  bench('wait 10ms', () => {
    sleepSync(10)
  })
})

When executed (via bun test), the above code could output this (just a mockup):

bun test vX.X.X (hash)

✓ 1 + 1 = 2 [0.12ms]
⏱ performance > wait 1ms   100,000,000 ops/sec (5x faster than wait 5ms)
⏱ performance > wait 5ms   20,000,000 ops/sec  (baseline)
⏱ performance > wait 10ms  10,000,000 ops/sec  (2x slower than wait 5ms)

If this would clutter test files or be too difficult to implement (how would benchmarks be grouped?), bench and baseline could be only allowed in *.bench.{js,ts}.

What alternatives have you considered?

mitata: Inaccurate and impossible to configure.
deno bench: Requires deno to use.
vitest bench: Seems overkill to use an entire testing library, just for its bench runner.
a custom runner: I've written one, but it's currently blocked by #5709.

[MrRedaL]

I'm okay