Evaluations in backend

[mmabrouk]

No description provided.

[mmabrouk]

@MohammedMaaz @aakrem @aybruhm

Comparison view:

• Currently, each input/row is displayed twice. We need to map the evaluation scenarios from the multiple runs. This should ideally be done in the backend. Here is a simple Python algorithm for how this could be implemented:

from collections import defaultdict

list1 = [('a', 1), ('b', 2)]
list2 = [('a', 10), ('b', 2)]
list3 = [('a', 3), ('b', 2), ('c', 5)]

merged_dict = defaultdict(list)
for lst in [list1, list2, list3]:
    for k, v in lst:
        merged_dict[k].append(v)

• Input columns are displayed twice. They should be shown only once.
• The order of the columns should be: Input/Expected output/Output var1/Output var2/Results var1/Results var2.

Evaluation view:

• Remove the id column.
• There's no option to shut down failed evaluations. We need a method to stop them.
• There's no way to determine if an evaluation has failed or if an evaluation scenario has failed.
• There's no option to rerun an evaluation.

Evaluators view:

• Exact match should be available from the start. This should probably be handled in the backend. This also applies to all evaluators that don't need configuration.