Load config from checkpoint_dir

[Andrei-Aksionov]

Hi there 👋

As was discussed in #721, this PR adds functionality to first try to load lit_config.json file from checkpoint_dir and, if it doesn't exist, try to load a config from lit_gpt/config.py by matching the name of the checkpoint_dir.

The only pivot from the discussion is that instead of adding such functionality into utils.py file, this PR extends Config class, so we don't have a problem with different versions of the class (e.g. LoRA version of Config).

[carmocca]

LGTM! I don't think any tutorials need an update.

[carmocca]

On second thought, why would any script other than prepare_redpajama use this? In all the other scripts that were changed, we want to load a directory containing existing weights, so the new function is unnecessary and the scripts will still fail later if the weights do not exit. Shouldn't this only modify prepare_redpajama.py?

[Andrei-Aksionov]

My intent was to have a uniformity across the whole repo when it comes to loading lit_config.json.
At the same time I agree, it adds some level of ambiguity.

P.S. Is there a chance that weights exist, but not lit_config.json?

[carmocca]

No unless the user made changes or moved files. For the scripts that are meant to load weights, I prefer to have this fail with a missing file error instead of loading a config that could not match the weights file