Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproduction/refresh of these results #22

Open
3 tasks
ml-evs opened this issue Nov 8, 2024 · 0 comments
Open
3 tasks

Reproduction/refresh of these results #22

ml-evs opened this issue Nov 8, 2024 · 0 comments
Assignees

Comments

@ml-evs
Copy link
Member

ml-evs commented Nov 8, 2024

After discussions with @VicTrqt, @SurgeArrester, @yqdleiyi and others, we think it would be a good idea to reproduce these results a few years on with the latest additions to MODNet. I'm raising this issue to keep me honest and actually do it...

Whilst there have not been huge developments in MODNet since our first submission, we have been actively using it in "production" and need to investigate:

  • how well our new, fast subset of matminer works on the original tasks
  • how well the GA hyperparameter opt works (I think the defaults will need to be updated for the larger tasks here)
  • generally how easily these tasks can be performed via the MODNet API now

Hopefully this will lead to improvements in MODNet generally.

I'm not sure if matbench is still being actively maintained, but we could even consider a new submission, depending on the results. There have been some queries raised about the leaderboard results that we should also keep in mind, e.g., materialsproject/matbench#262. I think we're in a better place to really share precisely the scripts used to run these benchmarks this time around, as before they were mostly tied with figure generation for our benchmarking paper, with matbench as an afterthought.

@ml-evs ml-evs self-assigned this Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant