CHANGELOG.md

Unreleased

v1.20.1 - 02 Apr 2024

• fix release script

v1.20.0 - 02 Apr 2024

• fix Rust benchmarks not comparing to baseline (#235)
• feat Comment on PR and auto update comment (#223)

v1.19.3 - 02 Feb 2024

• fix ratio is NaN when previous value is 0. Now, print 1 when both values are 0 and +-∞ when divisor is 0 (#222)
• fix action hangs in some cases for go fiber benchmarks (#225)

v1.19.2 - 26 Jan 2024

• fix markdown rendering for summary is broken (#218)

v1.19.1 - 25 Jan 2024

• fix improve flaky CI runs (#215)
• fix write with retry fails with the separate repository (#216)

v1.19.0 - 25 Jan 2024

• docs Add description for skip-fetch-gh-pages (#180)
• fix Mismatch input in action.yml (#191)
• fix Update Manifest.toml to fix julia test failure (#210)
• chore update to node 20 (#208)
• chore update actions/* (#212)
• fix summary-always and gh-repository don't work together (#214)

v1.18.0 - 07 Jul 2023

• feat getServerUrl refers to the GITHUB_SERVER_URL environment variable (#169)
• feat extract multiple metrics from Golang benchmarks (#177)
• fix getCommitFromGitHubAPIRequest to refer to GITHUB_API_URL (#171)
• chore Remove unreachable code from extract.ts (#153)

v1.17.0 - 26 Apr 2023

• feat support for JMH parameters (as separate charts) (#161)
• feat enable user to specify the ref being tested (#163)
• feat allow more characters in Golang bench outputs (#131)

v1.16.2 - 08 Feb 2023

• Fix use commit.id over commit object (#155)

v1.16.1 - 06 Feb 2023

• Fix action.yml missing summary-always input

v1.16.0 - 05 Feb 2023

• Feat Support pr summary for benchmark output (#138)

v1.15.0 - 03 Nov 2022

• Feat Add support for Java via JMH (#134)
• Chore Update @actions/core, @actions/exec and @actions/io to the latest version (#137)

v1.14.0 - 28 May 2022

• Feat Added benchmark luau support (#123)
• Chore Bump minimist from 1.2.5 to 1.2.6 (#114)
• Feat Implement deploy to another repository (#112)

v1.13.0 - 17 Feb 2022

• Feat: Updated urls to support GHES (#104)
• Feat: Add support for BenchmarkDotNet (#109)
• Chore Bump node-fetch from 2.6.6 to 2.6.7 (#107)

v1.12.0 - 28 Jan 2022

• Feat: Support private repositories (#105)
• Chore Bump action runner to node v16 (#106)

v1.11.3 - 31 Dec 2021

• Fix: Fix trailing whitespace characters in cargo benchmarks (#97)

v1.11.2 - 28 Dec 2021

• Fix: Added option to use Rust benchmark names with spaces (#94)

v1.11.1 - 04 Dec 2021

• Fix: Fix/go tabled benchmarks (#32)
• New: Support BenchmarkTools.jl in Julia (#89)
• Improve: Update several dependencies including TypeScript v4.5.2
• Improve: Use Jest for unit testing

v1.10.0 - 28 Oct 2021

• New: Allow user defined custom benchmarks (#81)

v1.9.0 - 12 Oct 2021

• Fix: manual and scheduled runs (#74)

v1.8.1 - 10 Jun 2020

• Fix: Allow / in cargo bench benchmark name (#26)
• New: Add an example with Criterion.rs for Rust projects

Changes

v1.8.0 - 17 Mar 2020

• New: Added comment-always option to leave a comment of benchmarking results at the commit always. Thanks @pksunkara
• New: Added save-data-file option to skip saving data file. Setting false to this value is useful when you don't want to update Git repository. Thanks @pksunkara
• Improve: +/- is now replaced with ±
• Improve: Better formatting for floating point numbers

Changes

v1.7.1 - 23 Feb 2020

• Fix: Benchmark output parser could not parse \r\n as newline correctly (#16)
• Improve: Prefer @actions/github.GitHub wrapper to @octokit/rest.Octokit

Changes

v1.7.0 - 21 Jan 2020

• New: Add Catch2 support. Please read the example for more details. Thanks @bernedom
• Fix: Deploying to GitHub Pages did not work when checking out the repository with actions/checkout@v2
• Improve: Update several dependencies including @actions/* packages

Changes

v1.6.7 - 01 Jan 2020

• Fix: Extracting the benchmark result value from go test -bench did not assume float numbers (Fixed #5)
• Fix: Running this action on pull_request event caused an error since head_commit payload is not set at the event. In the case, now this action tries to extract the commit information from pull_request payload

Changes

v1.6.6 - 23 Dec 2019

• Fix: Parse floating numbers in the benchmark results from Benchmark.js. (Thanks @Bnaya)

Changes

v1.6.5 - 19 Dec 2019

• Fix: Titles are set to empty in auto-generated default index.html. To apply this fix, please remove current index.html in your GitHub Pages branch and run this action again
• Fix: Skip fetching GitHub Pages branch before switching to the branch when skip-fetch-gh-pages is set to true
• Improve: Explicitly note no action output from this action in README.md

Changes

v1.6.4 - 16 Dec 2019

• Fix: Supported actions/checkout@v2
• Improve: Refactored index.html automatically generated when it does not exist
• Improve: Update dependencies (actions/github v2)

Changes

v1.6.3 - 08 Dec 2019

• Improve: Tweak number of retries for more robust automatic git push

Changes

v1.6.2 - 07 Dec 2019

• Fix: Retry logic for git push did not work properly since stderr output was not included in error message

Changes

v1.6.1 - 07 Dec 2019

• Fix: Time unit of mean time in pytest benchmark results were always sec. Now time units are converted to msec, usec and nsec if necessary
• Fix: Detecting rejection by remote on git push was not sufficient
• Improve: Add a small link at right bottom of dashboard page to show this action provided the page
• Improve: Showed at least 1 significant digit for threshold float values like 2.0
• Improve: Updated dependencies

Changes

v1.6.0 - 04 Dec 2019

• New: fail-threshold input was added. Format is the same as alert-threshold, but you can give different thresholds to sending a commit comment and making the workflow fail by giving different value to fail-threshold from alert-threshold. This value is optional. If omitted, fail-threshold value is the same as alert-threshold
• Improve: Retry logic was improved on git push failed due to remote branch updates after git pull. Now this action retries entire process to update gh-pages branch when the remote rejected automatic git push. Previously this action tried to rebase the local onto the remote but it sometimes failed due to conflicts

Changes

v1.5.0 - 30 Nov 2019

• New: Added max-items-in-chart input was added to limit the number of data points in a graph chart.
• New: Supported Google C++ Benchmark Framework for C++ projects. Please check the example project and the example workflow to know the setup
• Fix: Fix the order of graphs in the default index.html. To apply this fix, please remove index.html in your GitHub Pages branch and run your benchmark workflow again
• Improve: Use the actions marketplace URL for the link to this action in commit comment
• Improve: Updated dependencies
• Dev: Added Many tests for checking the updates on a new benchmark result
• Dev: Changed directory structure. Sources are now put in src/ directory

Changes

v1.4.0 - 23 Nov 2019

• New: external-data-json-path input was added to support to put benchmark data externally rather than Git branch
  • By using this input and actions/cache, you no longer need to use Git branch for this action if you only want performance alerts. Benchmark data is stored as workflow cache.
  • By this input, minimal setup for this action is much easier. Please read 'How to use' section in README.md.

Changes

v1.3.2 - 23 Nov 2019

• Improve: Styles in alert commit comment were improved
• Fix: When benchmark name (with name input) contained spaces, URL for the workflow which detected performance regression was broken

Changes

v1.3.1 - 21 Nov 2019

• Fix: git push sometimes failed in the situation where prepush hook is set and runs unexpectedly. Now git push is run with --no-verify for pushing auto generated commit to remote.

Changes

v1.3.0 - 21 Nov 2019

• New: Alert feature was added 🎉
  • With this feature enabled, you can get alert commit comment or make workflow fail when possible performance regression is detected like this
  • comment-on-alert input was added to enable commit comment on alert. github-token input is necessary as well to use GitHub API. Unlike deploying GitHub Pages, secrets.GITHUB_TOKEN is sufficient for this purpose (if you don't use GitHub Pages). The input is set to false by default.
  • fail-on-alert input was added to mark running workflow fail on alert. The input is set to false by default.
  • alert-threshold input was added to specify the threshold to check alerts. When current result gets worse than previous exceeding the threshold. Value is ratio such as "200%". For example, when benchmark gets result 230 ns/iter and previous one was 100ns/iter, it means 230% worse and an alert will happen.
  • Please read documentation for setup
• New: alert-comment-cc-users input was added to specify users mentioned in an alert commit comment so that they can easily notice it via GitHub notification
• New: skip-fetch-gh-pages input was added to skip git pull which is automatically executed on public repo or when you set github-token on private repo.
• Improve: E2E checks on CI were added
• Improve: Updated dependencies

Changes

v1.2.0 - 17 Nov 2019

• New: Support pytest-benchmark for Python projects which use pytest
  • Benchmark value is how long one iteration takes (seconds/iter)
• Improve: Show more extra data in tooltip which are specific to tools
  • Go
    • Iterations
    • Number of CPUs used
  • Benchmark.js
    • Number of samples
  • pytest-benchmark
    • Mean time
    • Number of rounds

For reflecting the extra data improvement, please refresh your index.html. Remove current index.html in GitHub Pages branch and push the change to remote, then re-run your benchmark workflow.

Changes

v1.1.4 - 16 Nov 2019

• Improve: Title styles in default index.html which is generated when no index.html is in your GitHub Pages branch. If you want to update your index.html to the latest, please remove it and push to remote at first then re-run your workflow which will invoke github-action-benchmark
• Improve: More metadata in action.yml. Now icon and its color are set.

Changes

v1.1.3 - 16 Nov 2019

• Fix: Retry failed when no Git user config is provided. Ensure to give bot user info to each git command invocations

Changes

v1.1.2 - 16 Nov 2019

• Improve: Added retry for git push. When remote GitHub Pages branch is updated after the current workflow had fetched the branch, git push will fail because the remote branch is not up-to-date. In the case this action will try to rebase onto the latest remote by git pull --rebase and git push again. This is useful when your multiple workflows may be trying to push GitHub Pages branch at the same timing. auto-push input must be set to true for this.
• Fix: Description for auto-push was missing in action.yml

Changes

v1.1.1 - 14 Nov 2019

• Improve: More strict check for auto-push input. Now the value must be one of true, false (default value is false)

Changes

v1.1.0 - 14 Nov 2019

• New: Added auto-push input
  • If this value is set to true, this action pushes GitHub Pages branch to remote automatically. You no longer need to push the branch by yourself.
  • Below github-token input must be set for this
  • This input is optional. You can still push the branch by yourself if you want
  • Please read documentation for more details
• New: Added github-token input
  • For doing some operations which requires GitHub API token, this input is necessary
    • pull from remote branch when your repository is private
    • push to remote branch
    • deploy and trigger GitHub Pages build
  • This input is optional. When you do none of above operations, this input is not necessary
• README.md was updated to avoid the issue on public repository (#1)

e.g.

- name: Store benchmark result
  uses: rhysd/github-action-benchmark@v1
  with:
    name: My Project Go Benchmark
    tool: 'go'
    output-file-path: output.txt
    github-token: ${{ secrets.PERSONAL_GITHUB_TOKEN }}
    auto-push: true

Note that you need to make a personal access token for deploying GitHub Pages from GitHub Action workflow. Please read RADME.md for more details.

Changes

v1.0.2 - 10 Nov 2019

First release 🎉

Please read documentation for getting started:

https://github.com/benchmark-action/github-action-benchmark#readme

Changes