Mask CLS at the end of seq

[gpucce]

@yiren-jiran thanks for finding the bug, in #549 it looks like I made a mistake in creating the cls mask.

This PR should be the fix, will have to look how much changing this affects models performance.

[yiren-jian]

I believe fixing this issue requires re-training CoCa models.

However, it's not supervised to me that the current trained models working well: [CLS] token was still able to "see" all the relevant preceding text, but not itself.

Fixing the bug without re-training models would hurt retrieval/classification performance, I guess.

[gpucce]

Indeed, likely needs retraining we this change.

[gpucce]

@rwightman what would be the best way to address this? Does it make sense to retrain the model?

[rom1504]

my opinion is

1. do not change the default
2. do small scale experiment to see if the fixes help
3. if they don't help, have a comment explaining why things are how they are and don't change the default
4. if they do help significantly, plan to retrain

[rom1504]

but first fix the CI here

[gpucce]

@rom1504, @iejMac it appears to be an issue with caching older test run times or smth similar, removing

open_clip/.github/workflows/ci.yml

Lines 96 to 98 in fb72f4d

	touch .test_durations
	cp .test_durations durations_1
	mv .test_durations durations_2

seems to fix it, however I am not super confident about github ci,
if you or someone else who is can check it would probably help

[gpucce]

@rom1504 @iejMac so I found that pluggy was not versioned, adding it to the requirements-test.txt file fixed the issue in the appropriate way I believe.

Would still be useful to have someone that knows more about pytest confirm this

[gpucce]

@rom1504 sorry for several tags, I think it was a caching issue but honestly not sure how it got fixed

[gpucce]

Moved to #551