create datasetBlockages collection + block datasets

[severo]

We apply rate limiting on the jobs, based on the total duration in a window (see #2279 (comment)).

Follows #2931

[lhoestq]

Looks all good to me, though I'm concerned that it might slow down the next_waiting_job query if there are blocked datasets with thousands of their jobs in the queue (it will have to iterate over them all to filter them and get back a job from an unblocked dataset).

Anyway if it's rhe case it can be fixed by defining a target_start_time that is equal to created_at by default and that can be modified to later values (e.g. now + 1h) if a dataset is blocked

[severo]

it might slow down the next_waiting_job query if there are blocked datasets with thousands of their jobs in the queue (it will have to iterate over them all to filter them and get back a job from an unblocked dataset).

yes, I thought the same: I just changed to remove the short jobs (they should not affect too much anyway) + store as int, not float

[AndreaFrancis]

LGTM, but I would suggest creating or adding some tests with memory limits to ensure that it won't affect too much the next_waiting_job function.

[severo]

Hmmm I think it will have nearly no impact, since the collection of blocked datasets shouldn't have more than <10 entries, if everything works well. The only "expensive" operation we add is when we compute the sum of the durations (that I optimized a bit with #2933 (comment)), but it's run on job termination, not on job start.

[severo]

The CI error is due to issues on the Hub. Merging