weight according to initial loss scale

[smilesun]

for each regularization loss, before training start, calculate the ratio of this loss term with respect to the task loss (cross entropy for classiifcaiton)

e.g. regularization loss for DANN is

DomainLab/domainlab/models/model_dann.py

Line 108 in 1df808a

return [lc_d], [self.alpha]

.

However, we do not need to care about this line above, but rather,

DomainLab/domainlab/models/a_model.py

Line 52 in 1df808a

loss_reg = self.list_inner_product(list_loss, list_multiplier)

This issue is to make a feature for DomainLab such that for each loss term, we have a reference ratio of this loss term over the task loss, then the user input gamma_reg will be interpreted as a multiplication factor only on top of this ratio, i.e.
list_multiplier[j]*ratio[j], where j is the index for the loss term in question, ratio is something we have to calculate before training starts, here:

DomainLab/domainlab/algos/trainers/train_basic.py

Line 22 in 1df808a

def before_tr(self):

we probalby need to define something like self.list_reg_loss_over_task_loss_ratio in

DomainLab/domainlab/models/a_model.py

Line 17 in 1df808a

def __init__(self):

then before

DomainLab/domainlab/models/a_model.py

Line 52 in 1df808a

loss_reg = self.list_inner_product(list_loss, list_multiplier)

we do new_list_multiplier= [mtuple[0]*mtuple[1] for mtuple in zip(self.list_reg_loss_over_task_loss_ratio, list_multiplier)]

[smilesun]

working on a pr #863