Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New feature: Supplement historic benchmark results manually #99

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitpod.dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
FROM gitpod/workspace-full:latest

RUN bash -c ". .nvm/nvm.sh && nvm install 12 && nvm use 12 && nvm alias default 12"

RUN echo "nvm use default &>/dev/null" >> ~/.bashrc.d/51-nvm-fix
7 changes: 7 additions & 0 deletions .gitpod.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
image:
file: .gitpod.Dockerfile
tasks:
- init: npm install && npm run build
vscode:
extensions:
- dbaeumer.vscode-eslint
37 changes: 37 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,43 @@ If the amplitude is not acceptable, please prepare a stable environment to run b
GitHub action supports [self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners).


### Supplement historic benchmark results manually

If you have started to collect benchmark results at a certain stage of your project but now also would like to compare these results to older historic values, this tool also allows you to check in these historic data at a later point in time so they will appear in the GitHub Pages graphs as well:

1. Create the historic data yourself, and append the hash of the relevant commit to its name (e.g., `benchmark-data-e1d6443a1e89280d0ade9bf2d81050f17c50189b.json`).
2. Clone & install this repository:
```shell
git clone https://github.com/benchmark-action/github-action-benchmark.git
cd github-action-benchmark
npm install && npm build
```
3. In your repository folder, create a file that contains the **full** configuration of your job configuration. You can retrieve this config from the job log of any prior workflow run under the `with:` key in line 2 of the step log. For instance, create a file named `config.yml` that looks like this:
<details><summary><code>config.yml</code></summary><pre><code>tool: customSmallerIsBetter
output-file-path: benchmark-data.json
auto-push: true
gh-pages-branch: gh-pages
comment-always: false
fail-on-alert: true
name: Benchmark
benchmark-data-dir-path: dev/bench
skip-fetch-gh-pages: false
save-data-file: true
comment-on-alert: false
alert-threshold: 200%
github-token: '<your_token>'</code></pre>
</details>
For auto-push, you need to generate a new token and insert it above as GitHub will obfuscate it in the log.
4. For each historic commit, check in the benchmark data by running the following:
```shell
GITHUB_REPOSITORY=<owner>/<repo> node /path/to/github-action-benchmark/dist/src/index_local.js config.yml <commit-sha>
```
Assuming that you have prepared all historic `benchmark-data-<sha>.json` files in your repo, this one-liner will automatically check in all of them for you:
```shell
ls | grep -oP '(?<=^benchmark-data-)\w+(?=\.json$)' | tr '\n' '\0' | GITHUB_REPOSITORY=LinqLover/squeak-tracedebugger xargs -0 -n1 node /path/to/github-action-benchmark/dist/src/index_local.js config.yml
```


### Customizing the benchmarks result page

This action creates the default `index.html` in the directory specified with `benchmark-data-dir-path`
Expand Down
Loading