You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some datasets contain document relevance scores or at least a sorted order of ground truth relevant documents, for example HotpotQA. In order to evaluate rankers and retrievers regarding their ability to not only distinguish relevant from non-relevant documents but also to sort relevant documents by their relevance, we need a new component DocumentNDCGEvaluator. The currently available DocumentMRREvaluator considers relevancy of documents to be binary.
The text was updated successfully, but these errors were encountered:
Hello @srini047 thanks for the initiative and for reaching out. Unfortunately it's too late this time as we already started working on it, sorry. I hope you find another issue in our contributions wanted list that you would like to contribute to. Thank you! 🙏
Some datasets contain document relevance scores or at least a sorted order of ground truth relevant documents, for example HotpotQA. In order to evaluate rankers and retrievers regarding their ability to not only distinguish relevant from non-relevant documents but also to sort relevant documents by their relevance, we need a new component DocumentNDCGEvaluator. The currently available DocumentMRREvaluator considers relevancy of documents to be binary.
The text was updated successfully, but these errors were encountered: