Save the hyperparameters to the checkpoint

[awaelchli]

Saves a hyperparameters.yaml file to the checkpoint. Feel free to discuss the naming on this PR.

This will also be used to store the prompt template so that it can be dynamically loaded from the checkpoint after finetuning.

Part of #1005

[rasbt]

I wonder if we should add a model_dir or so to the hyperameter file. This could be

model_dir = out_dir/final

[awaelchli]

@rasbt It is important that this file contains exactly the arguments that were passed into the script, to capture exactly how it was called. It's not the same as meta data.

It's not immediately clear to me how saving the path of the checkpoint where it is saved would be useful. Because that would become wrong once you move the checkpoint somewhere else.

[rasbt]

It's not immediately clear to me how saving the path of the checkpoint where it is saved would be useful. Because that would become wrong once you move the checkpoint somewhere else.

Ok that's fair. Never mind it was a silly idea.