Prepare repo for release

[epwalsh]

• consolidate Python configs into pyproject.toml, other clean up #353
• remove data team's stuff #357
• Code cleanup: move torch-related util functions out of util.py and into their own module. (Separate torch utils #375)
• Code cleanup: split up components in model.py into submodules. (Small simplifications #377)
• Fix stale links in README, scripts cleanup  #359

[Muennighoff]

@epwalsh Is it fine if I tackle the two remaining items in a PR? 🙂

[epwalsh]

@Muennighoff that would be great, go ahead.

[Muennighoff]

Great - did the first one here: #375

For splitting the components, I was thinking of moving the Attention & MLP out of OlmoBlock, so there'd be sth like class Attention & class MLP. To make it work with the other Blocks, I will probably also have class LlamaAttention, class LlamaMLP similar to how there are different LayerNorm classes that inherit from the same base. Is that what you were thinking of as well? @epwalsh

[epwalsh]

@Muennighoff separating out attention certainly makes sense from an API perspective. I think we used to have it that way, actually, but then our different block implementations became highly coupled with how attention was implemented, so we did away with that.

There might be a clean way to decouple them again, but that makes me a bit nervous because at this point we need to make sure any code changes to the model do not change:

1. The fully qualified parameter names (i.e. model.state_dict() keys should remain the same).
2. How FSDP shards things, since that could impact our ability to restart existing runs. Adding new submodules could affect that.

[Muennighoff]

I see, maybe it's not worth splitting them out then? Are there other parts you want to change to be releasable?

I find the current modeling code not that bad, but I would maybe clean up some of the comments and simplify a few things without changing the logic.

[epwalsh]

@Muennighoff let's try to avoid code changes unless there's low-hanging fruit that won't have side-affects on training. I think it's still worth reorganizing + doing some clean up that you're suggesting.

[Muennighoff]

Should we close this issue? I think the 5 points are now addressed, but let me know if not & I can work on it :)

[epwalsh]

I think we're good!