Gap between influence and real loss difference

[HAoYifei996]

Hi, I am trying to perform a leave-one-out retraining to compare the difference between influence computed by calc_influence_single here and actual loss difference by computing two times of loss before and after retrain. I use calc_loss here to compute loss. Also, I scale the influence by len(trainset). Surprisingly I found big difference between computed influence and real loss difference after retrain. For a random pick test_idx=10, train_idx_to_remove = 609, I'm getting the following result:

actual_loss_diffs:  tensor(0.7848, device='cuda:0', grad_fn=<SubBackward0>)
predict_loss_diffs:  -0.003923072204589844

which doesn't seem very relevant to me.

Thanks in advance for any kind suggestions!

[HAoYifei996]

So far I think it may have something to do with the net in the example is only trained with 10 epochs, which does not achieve close to global optimum, but I'm not sure.