What this PR does:

Currently, each tenant has single FIFO request queue in the query path, where queriers assigned for the tenant will pick up next request from the queue.

Having a single FIFO queue results in situations where

1. Slow queries exhaust all queriers and their concurrency
2. Fast (or more important) queries get queued up, waiting for next querier to pick them up
3. If the slow queries took 30 seconds, then the fast (or more important) queries now have 30 seconds added to their latency
4. If the slow queries timed out, then the fast (or more important) queries are also likely to timeout in the queue (without even reaching querier)

This PR suggests to introduce new "query priority" for each tenant that does the following:

1. Able to assign priorities to queriers that meet the query attributes (regex, start time, end time)
2. When query priority is enabled, schedulers use priority queue that orders the requests by priority
3. You can also have "reserved" queriers that only pick requests with a certain priority. This is useful when you want to guarantee certain number of queriers available to handle high priority requests at any given time.

Here's an example configuration:

  <tenant-id>:
    query_priority:
      enabled: true
      default_priority: 0
      priorities:
        - priority: 3
          reserved_queriers: 1
          query_attributes:
            - regex: "special_query"
        - priority: 2
          query_attributes:
            - regex: "max"
              time_window:
                start: 2h
                end: 0h
        - priority: 1
          query_attributes:
            - time_window:
                start: 24h

Which issue(s) this PR fixes:
n/a

End-to-end test result:

There were 4 things to verify from the end-to-end test:

• Confirm high priority requests get dequeued more than lower priority requests
• Confirm reserved queriers do not pick up low priority requests
• Confirm the tenant limit config change gets picked up by query frontend and scheduler without the pods restarting
• Check memory and cpu usage

In short, all of them were verified successfully with some cool remarks:

• P50 and avg latency droped significantly, as my test was assigning higher priority to ingester queries (data within 24h)
• Number of timeouts decreased, at the cost of more 429s and higher queue length (queue length increased with low queries not handled in time, which can be addressed by scaling up queriers in real world scenario)
• Memory usage of QFE is better when the query priority is enabled
• Memory usage of scheduler is slightly worse when the query priority is enabled, but negligible

Image 1: Metrics with new labels showing smooth transition between FIFO and priority queue.

Image 2: Change in QPS per status code and latency.

Image 3: Change in CPU and memory usage.

Image 4: One reserved querier waiting for priority 3 query that was never enqueued.

Test setup:

Queries ran continuously:

"query?query=max({__name__=~'metric_[0-4]'})&time=0h
"query?query=avg({__name__=~'metric_[1-5]'})&time=-12h
"query?query=min({__name__=~'metric_[2-6]'})&time=-32h

Query priority config used:

    query_priority:
      enabled: true
      default_priority: 0
      priorities:
        - priority: 3
          reserved_queriers: 1
          query_attributes:
            - regex: "special_query"
        - priority: 2
          query_attributes:
            - regex: "max"
              time_window:
                start: 2h
                end: 0h
        - priority: 1
          query_attributes:
            - time_window:
                start: 24h

Timeline:

14:00 | Applied new image that contains this PR
14:50 | Enabled query priority config
15:45 | Disabled query priority config without removing the priority definitions

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]