Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically save best model during training #44

Open
chriscyyeung opened this issue May 11, 2023 · 1 comment
Open

Automatically save best model during training #44

chriscyyeung opened this issue May 11, 2023 · 1 comment

Comments

@chriscyyeung
Copy link
Collaborator

I think instead of (or maybe in addition to) saving after a certain number of epochs, we can save the best model based on the validation loss?

@ungi
Copy link
Member

ungi commented May 11, 2023

Finding the best model is not as simple as saving the one that has minimum loss function. Sometimes when training for extremely long (thousands of epochs), the model can learn a better (more general) representation of the data without decreasing the loss function. I couldn't find where I learned this, but it was related to this theory: https://medium.com/@MITIBMLab/estimating-information-flow-in-deep-neural-networks-b2a77bdda7a7

Saving the model regularly is generally a good practice. We could add an option like "model_save_frequency". E.g. if it's 5 then the model would be saved after every 5 epochs using names like model_005, model_010, etc. And we could save on the wandb report all the metrics for each saved model.

I also had positive experience in the past training for a few hundred more epochs after it seemed like the metrics did not improve.

chriscyyeung added a commit that referenced this issue May 18, 2023
chriscyyeung added a commit that referenced this issue May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants