Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CausalLanguageModeling masking error on last item only condition #762

Closed
sungho-ham opened this issue Jan 4, 2024 · 1 comment
Closed

Comments

@sungho-ham
Copy link
Contributor

sungho-ham commented Jan 4, 2024

Description

When using clm for masking, it generates wrong masking schema.
It can be checked by following simple code.
In case of length 2 input, there should not be difference between last item only and all items conditions.
However, there is difference between them.

import torch
from transformers4rec.torch import masking

def get_masking_info(train_on_last:bool):
    item_ids = torch.tensor([[1, 2, 0], ])
    mask = masking.CausalLanguageModeling(hidden_size=10, train_on_last_item_seq_only=train_on_last)
    masking_info = mask.compute_masked_targets(item_ids, training=True)
    return masking_info

print(get_masking_info(False))
print(get_masking_info(True))
MaskingInfo(schema=tensor([[ True, False, False]]), targets=tensor([[2, 0, 0]]))
MaskingInfo(schema=tensor([[ True,  True, False]]), targets=tensor([[2, 0, 0]]))  -> schema shoule be [ True,  True, False]

Related Code

mask_labels = item_ids != self.padding_idx

I think following code is correct:
mask_labels = labels != self.padding_idx

@sungho-ham
Copy link
Contributor Author

sungho-ham commented Jan 4, 2024

I've found my suggestion was wrong and already fixed.
#723 (review)
However, from what I understand, the results is not correct. I reported it again according to the bug reporting format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant