Fix the CLM performance mismatch between evaluation and manual inference

[sararb]

Fixes #719

Goals ⚽

The current CLM strategy is as follows

The main issues of the current implementation are:

• We mask useful past information when we evaluate on the last item only ==> This is wrong as past information is a valuable context for predictions.
• At inference, the padded positions are represented with 0-embeddings while during training these positions are replaced with trainable [MASK] embeddings. ==> We should have the same training, evaluation, and inference representation strategy.

Implementation Details 🚧

• Updated the class CausalLanguageModeling to:

  • Replace padded positions with [MASK] embeddings at inference.
  • During the evaluation of the last item, I defined the label_mask as the padding mask information (to keep information about actual past items).

• I ran the t4r_paper_repro script using 5 days of ecomrees46 dataset and these are the results:

**CLM run using main branch (w/o fix)**
Recall@10 of manually masked test data = 0.3915199603272998
eval//next-item/recall_at_10 0.25108

**Fix the evaluation on last item by not masking the past information**
eval_/next-item/recall_at_10 = 0.2980385422706604
Recall@10 of manually masked test data = 0.29812381188527975

Testing Details 🔍

• Changed test_mask_only_last_item_for_eval to get target mask information from lm.masked_targets
• Changed test_sequential_tabular_features_ignore_masking as the inference mode of CLM is changing the inputs by replacing 0 padded positions with [MASK] embeddings

Future work

• The [MASK] embeddings is not used to generate the next-item prediction scores and this raises a question about whether we should remove it and just use 0-embedding to represent padded positions.
  ==> We need to re-run T4Rec paper experiments without [MASK] variable and check how the evaluation results are impacted.

[github-actions]

Documentation preview

https://nvidia-merlin.github.io/Transformers4Rec/review/pr-723

[gabrielspmoreira]

Good catch!

[gabrielspmoreira]

Great job @sararb