Support for LIFO / FIFO setting on judgement queue

[sauterl]

In some cases, it would be desirable for evaluation organisers to influence how answers are judged, e.g. whether simply the new submissions are added to the end of the queue or to the front (i.e. judged earlier).
This could be a setting on the task run.

[sauterl]

Hint: We will include this as a parameter for judgement-target typed tasks.

[sauterl]

We have implemented this for the time being in such a way as that the JUDGEMENT or VOTE target options now come with a configuration key order, which accepts LIFO (case insensitive) to produce a LIFO queue of judgements.
However, this is per task and the dequeue endpoints still operate on the first open validator.

An additional desirable outcome could be that always the newest (based on time) should be judged in the UI.
I am not sure on how we would achieve this with the current architecture, as we would have to not only heavily alter the implemented JudgementValidators, but also track across multiple ones, which submission came in last.

@lucaro any thoughts on this one?

[lucaro]

I'd argue that rather than (or in addition to?) changing the insertion order per validator, it is probably more useful to give a priority flag to the validators. This could be any number; the higher, the more important. The mechanism that selects the validator would then not just use the least recent, but the least recent with the highest number.

[sauterl]

Fair enough, I implemnted this with e033a43 by adding the configuration parameter JUDGEMENT.priority which expects an integer. Higher value represents higher priority.

Evaluation administrators have to set the priority on each task template.

[lucaro]

Do they have to set the priority, or can they set the priority? I guess there should be a reasonable default, no?

[sauterl]

Let's adjust this once more:

The default is the index in the task list (as opposed to the 0 currently implemented).

However, I'd suggest that then a run parameter could influence whether higher priority value results in being considered first or last.
Do you agree?

[lucaro]

Why would the default not be 0? Tasks don't need to be run in the order they are in the template. In most cases, tasks will not have any relative priority. If they do, you can actively boost a task by giving it a higher priority.

[sauterl]

Just to clarify: I was contemplating whether we should switch from the currently implemented default (0) to a more sophisticated default, which would be the index of the task template.
The current default's semantic is such that tasks which are started earlier, are queued earlier (according to stable sorting), except they have a higher priority number.
Per task, we now have the opportunity to specify whether LIFO or FIFO should be applied.
In summary, this is the behaviour before the change (expempt LIFO-per-task and priority across tasks).

The semantic of the proposed default whould have been as follows:
The default priority would be either the task template or something clever based on the task run's start time. This way, the default could have been influenced more by administrators.
However, based on the discussion here I refrain from implementing this.

Still, the priority system currently is based on the task template. And might not reflect the order of the tasks as they were run during the evaluation.

[lucaro]

Just to clarify: I was contemplating whether we should switch from the currently implemented default (0) to a more sophisticated default, which would be the index of the task template.

No, I would not do this. I would explicitly want to keep the 0 default. The order in which the tasks are actually used can be completely independent of the order in the template, so relying on this will only cause confusion. If the evaluation administrator wants to deviate from the order in the template, they presumably have a reason.