Weight decay for token embeddings

[MasterSkepticista]

Hi,

Why did we not exclude embedding weights from weight decay?

build-nanogpt/train_gpt2.py

Lines 185 to 186 in 6104ab1

	decay_params = [p for n, p in param_dict.items() if p.dim() >= 2]
	nodecay_params = [p for n, p in param_dict.items() if p.dim() < 2]

Re: karpathy/minGPT#24 (comment)

This may also explain divergence in longer runs (?)