Label smoothing in training

[melihyilmaz]

After encountering the NaN outputs from the model half way through training in a few runs, I experimented with minimal label smoothing when calculating the training loss as a mitigation strategy. I was able redo the same training runs with the same setup significantly longer without encountering NaNs and with similar performance metrics compared to the original runs.

Loss calculation is only impacted by smoothing during training steps, i.e. not validation, and I tentatively added the minimal label smoothing factor as the default option.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (235420f) 89.43% compared to head (5e58b65) 89.47%.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #261      +/-   ##
==========================================
+ Coverage   89.43%   89.47%   +0.03%     
==========================================
  Files          12       12              
  Lines         909      912       +3     
==========================================
+ Hits          813      816       +3     
  Misses         96       96              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wfondrie]

This looks good to me aside from needing a unit test.

[wfondrie]

This looks good to me!