LHA benchmark bot

[felixhekhorn]

I'd like to have a "LHA benchmark bot" that runs upon adding a label (e.g. run-lha-benchmark) and that runs all LHA benchmarks, since we need to conserve them. There is no need to run them on every commit, but I'd like to run at the end of each PR.

The idea is, of course, stolen from nnpdf.

For the moment I asked people expli - citly, but this could be automized ...

[alecandido]

All the benchmarks run can be automated with very little effort. The only problem was how to evaluate the output (when it should raise an error/warning).

I don't know what you mean by "bot", but if you mean "like fitbot" that is only a workflow, like ours, simply triggered by a label event. We can do it as well, or even do something better.
Still, the "end of a PR" is not an event, just because it is ill-defined. Instead of the label, you can trigger by the button coming with workflow_dispatch, it is just the same, and the straight-forward way to get a button (no magic label to know, it is the officially documented button).

[felixhekhorn]

• Indeed something like the "fitbot" ...
• also a button is just fine, and yes, exactly because it is ill-defined I'd like to keep it manual
• and the output can for now be just the usual print by ekomark that has to be validated by a user

[felixhekhorn]

a workflow can expose some assets after he has run, right?

[alecandido]

If you acknowledge the impossibility to fail, the alternative is to make a report, like vp ones: you need to compare to something, otherwise is difficult to evaluate, but I believe you can compare against PR base.

So, you need a workflow that:

• trigger on workflow_dispatch
• launch two jobs:
  1. checkout PR branch, and run benchmark runners
  2. checkout PR base, and run benchmark runners (this one can be cached with actions/cache)
• each job upload the results as an artifact
• a third job is fired when the other two are completed, download the artifacts, produce a comparison, and reupload it as an artifact
• after that the job (or fourth one) post a message on the PR with the link to the artifact for the comparison

[alecandido]

Artifacts for the branch benchmark result can have a small retention period (couple of days), while the other two you keep for a longer time (the base for caching purpose, the report for checking). But you still don't need both of them after PR merging, and if PR continues for long, it is worth rerunning (at least you will have to rebase)

[alecandido]

Actions for artifacts are:

• https://github.com/actions/upload-artifact
• https://github.com/actions/download-artifact

Action for posting on PR:

• the one used by Juan in fitbot: https://github.com/unsplash/comment-on-pr
• a more canonical way: https://stackoverflow.com/a/64126737/8653979

[felixhekhorn]

all this is v3 of what I'm thinking about 🙃 ... in the first stage I don't need a comparison, I know we have to match 0.0xy (I'm only thinking about LHA atm) ... so a simple print is sufficient and that could be done even without any artifact

[alecandido]

At least upload the single output as an artifact, scrolling the logs is a pain
(and you can also include the db, that can be used with the navigator)

[felixhekhorn]

Closed via #227