Separate optimization for trunk and embedder

[lfleck]

Hi guys,
your examples (e.g. TwoStreamMetricLoss) show separate optimizers for trunk and embedder.
Is that just for the sake of explanation how to deal with trunk and embedder in general, or is there a design decision behind? In the case of the latter, do you have any references for that? Most literature I've seen so far trains them jointly (e.g. SimCLR). Many thanks!

[KevinMusgrave]

It is common in metric learning papers to use a different learning rate for the trunk and embedder.

• The trunk is usually a model pretrained on Imagenet, so they use a lower learning rate on that part.
• The embedder is usually just a single fully connected layer that is randomly initialized, so they use a higher learning rate on that part.

This CVPR 2020 paper mentions using a 10 times lower learning rate on the trunk model: