Rust benchmarks not comparing to baseline

[rcrisanti]

It seems like for rust benchmarks the benchmark names are not being found from the previous data, even though it exists in the data.js file. You can see this even in this repo's own workflows. If you check the logs of a run of .github/workflows/ci.yml in the "Run Rust benchmark example" job at the end of the "Store benchmark result" step you will see in the generated table that the "Previous" & "Ratio" columns are empty. (For example here.)

I haven't been able to figure out why, because it seems like the data exists in the data.js file, and it is working fine for other tools. I have reproduced this in one of the repos I work on, where we have a pytest-benchmark job that works fine, but a cargo bench job that does not.

This doesn't raise an error, but it won't block a PR even if the new benchmark is over the threshold.

[ktrz]

Thanks @rcrisanti for pointing that out. I'll investigate and let you know when I figure out the reason for this and hopefully get it fixed

[ktrz]

Hi @rcrisanti

I was able to fix the issue in PR #235
The problem was that both Rust and Criterion.rs benchmarks were posting results under the same name. I separated the Criterion.rs benchmark results and fixed the data on the gh-pages. Now the data are compared properly as you can see here

[rcrisanti]

Ah, I see. I had the same issue in my repo where the rust & python benchmarks had the same default name of "Benchmark". Changing them so they had different names seems to have fixed it for me! Thanks for looking into it and explaining the reasoning.

You may want to clarify this in the README. I had originally kept this as the default because it says "This value must be identical across all benchmarks in your repository", but it really should be different for each tool you use, even if they are in the same repo.