is the res_tangent_cft file representing the reported result in the paper?

[zichaow]

Hi ! I was trying to reproduce the bpref result in your paper by running trec_eval with the res_tangent_cft file in this repo. The results were a bit off from the reported Tangent-CFT bpref in the paper... Am I doing something wrong?

what I did:
trec_eval -l3 judge.dat res_tangent_cft | grep -v "docno" | grep -E "(^P|bpref)" for full bpref and

trec_eval -l1 judge.dat res_tangent_cft | grep -v "docno" | grep -E "(^P|bpref)" for partial bpref.

The results I got were 0.5685 for full bpref and 0.6824 for partial bpref. In the paper those numbers are 0.6 and 0.71.

Thanks in advance for your help and clarification !

[zichaow]

i think it might be because of the eval file. got another one and can reproduce. thans!

[w32zhong]

@moonlightlane Hi, which eval file you are talking about? Can you share it here? I have the same question here.

[zichaow]

@moonlightlane Hi, which eval file you are talking about? Can you share it here? I have the same question here.

@t-k- Hi, sorry for the late reply; This is the eval file I think reproduced the TangentCFT results:
https://drive.google.com/file/d/1DhwDgu7C7IwHDQ4_Xqb-18uxhnTWKYUX/view?usp=sharing

[w32zhong]

@moonlightlane

Thanks! I am able to reproduce the results from using that judgment file, however, I still have trouble generating the same level scores from my own run. Any idea?

[zichaow]

I think there might be some mismatch of the labels/ids. for example, line 5 in the file I shared is `Duckworth–Lewis_method:2` while in the tangent-cft repo `judge.dat` file it is `DuckworthLewis_method:2`. Not sure if this causes the mismatch in the results...

…

On Mon, Aug 9, 2021 at 9:34 PM Wei ***@***.***> wrote: @moonlightlane <https://github.com/moonlightlane> Thanks! I am able to reproduce the results from using that judgment file, however, I still have trouble generating the same level scores from my own run. Any idea? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBQSITFOINQ4ZXXYDHXQCTT4CF2DANCNFSM45IPR7YA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

-- Jack Wang Rice University, Houston, TX Hanszen College Class of 2016 Electrical Engineering, PhD student zw16.web.rice.edu

[w32zhong]

@moonlightlane yes I did get a version of qrels from the task organizer, but I still run into trouble to reproduce the results. The code in this repo seems broken actually, so I also come up with a PR: #6

[MingchangLi]

@w32zhong hi, have you generated the same level scores from your own run? I cant reproduce paper results in my own run.

[w32zhong]

Never reproduced Tangent CFT published scores. @lishuaiCN