Discrepancy between my evaluation results and README for MNLI in evaluation.py

[TinaChen95]

Hi, I'm running evaluation.py on MNLI as described in the README, but I'm getting different results compared to what's displayed there. I'm using Google Colab for this, and you can find my notebook here: https://colab.research.google.com/drive/1UahAOTIwALfEC_DXE11mVOp5iSgwHoYH?usp=sharing

When I run evaluation.py, it shows the following results:
Task: mnli
Model path: ../CoFi-MNLI-s95
Model size: 4330279
Sparsity: 0.949
Accuracy: 0.091
Seconds/example: 0.000561

However, in the README file, the results for the same evaluation are different:
Task: MNLI
Model path: princeton-nlp/CoFi-MNLI-s95
Model size: 4920106
Sparsity: 0.943
mnli/acc: 0.8055
Seconds/example: 0.010151

I need help figuring out why there's a discrepancy between my results and what's described in the README. I've tried to follow the instructions in the README as closely as possible, but I may have missed something. Thank you for any assistance you can provide.

[xiamengzhou]

this is interesting, what is the model path ../CoFi-MNLI-s95?

[gaishun]

Have you solved this problem? I encountered the same problem.

[xiamengzhou]

It seems to be an issue with transformers' versions. It should be compatible with transformers==4.17.0 and datasets==1.14.0 but might not work with versions beyond.

[SHUSHENGQIGUI]

It seems to be an issue with transformers' versions. It should be compatible with transformers==4.17.0 and datasets==1.14.0 but might not work with versions beyond.
but that setting is always conficted with the huggingface version. can i check out your python env? I am crazy about this issue. please