Question about SelfSupervisedLoss

[andreafuschino]

Hi Kevin! first of all thanks for the contribution, nice repository. I want to ask you about the new SelfSupervisedLoss. Could you explain in general how it works (if I understand well, the labels are generated randomly) and how the augmented data should be generated. It would be interesting and useful to have a code example with this new feature. Thanks in advance!

[KevinMusgrave]

It assumes that:

• ref_emb[i] is an augmented version of embeddings[i].
• ref_emb[i] is the only augmented version of embeddings[i] in the batch.

Here's how it's implemented:

pytorch-metric-learning/src/pytorch_metric_learning/losses/self_supervised_loss.py

Lines 48 to 62 in c38c07c

	def forward(self, embeddings, ref_emb):
	"""
	embeddings: representations of the original set of inputs
	ref_emb: representations of an augmentation of the inputs.
	*args: variable length argument list, where each argument
	is an additional representation of an augmented version of the input.
	i.e. ref_emb2, ref_emb3, ...
	"""
	labels = torch.arange(embeddings.shape[0])
	return self.loss(
	embeddings=embeddings,
	labels=labels,
	ref_emb=ref_emb,
	ref_labels=labels.clone(),
	)

Pseudo-code example of how you can use it:
https://github.com/KevinMusgrave/pytorch-metric-learning#using-loss-functions-for-unsupervised--self-supervised-learning

I've added "example for SelfSupervisedLoss" to my todo list: #633

[carbocation]

I think that this is implied in all of the comments and documentation so far, but just to be sure: each i in range(len(ref_emb)) will be treated as the positive once? In other words, if the batch shape is (4, 10), then in some sense there will be 4 iterations, and in the first i=0 will be the positive and i=1...3 will be the negatives. Then i=1 will be positive (etc)?

Just double checking because the alternative could be that only the 0th will be positive and all others will be treated as the negative. But I don't think that the documentation (or my review of the code) indicate that this is the case. Rather, I think that the former (each batch index gets its turn as the index for the positive pair) is how things seem to work.

[KevinMusgrave]

The way losses are calculated depends on the loss function. For NTXentLoss, a loss is computed for each positive pair, and the final loss is the average. Index i=0 might be part of many positive pairs, and index i=1 might not be part of any positive pairs.

Here are some additional comments that might help for NTXentLoss:

• [Question] Understanding terminology reference, embedding #600 (comment)
• NT-Xent loss with multiple positive pairs #606 (comment)

If you are using the SelfSupervisedLoss wrapper with NTXentLoss, then every index is part of exactly 1 positive pair. In this case, you are correct that each index basically gets a "turn" in computing the loss.