Results:
- The results of the evaluation using various parameters is attached in the csv files in this release.
- The Pass@1 scores of the models for both the English and Hinglish prompts is detailed in
Pass@1_data.csv. - The IRT latency - (2 Parameter IRT model) scores of the models (English and Hinglish versions of models taken in the same analysis) are present in
IRT_results.csv - The binary matrix result (0 for incorrect and 1 for correct) of the evaluated solutions to all the problems in the dataset by each of the models is contained in
binary_matrix.txt.
Note : The zip file containing these is attached below.