Mimir Query Engine

[jhesketh]

We're building a new query engine to improve the peak memory consumption of queriers.

The work has already progressed and is fairly mature. A good description of the new engine is found here:
https://github.com/grafana/mimir/blob/main/pkg/streamingpromql/README.md

This issue is to track the PRs and large number of changes involved. It will be an index instead of maintaining a list of PRs in the CHANGELOG.

Main / Unreleased

• MQE: Add support for stddev #9367
• MQE: Add tests for annotations on avg #9368
• MQE: add support for comparison operations #9398
• MQE: remove obsolete feature flags #9399
• MQE: Tidy up test case #9403
• MQE: Add helper for idx calcs #9417
• MQE: Add extra comparison op edge case tests #9418
• MQE: Add support for stdvar #9419
• MQE: Add support for increase function #9420
• MQE: fix incorrect query results or "found duplicate series for the match group" errors when binary operation has unsorted labels in on #9482
• MQE: separate feature flag for binary comparison operations #9504
• MQE: include more information in query stats log lines #9505
• MQE: add support for and #9507
• MQE: fix panic in rate or increase over native histograms with changing schema #9518
• MQE: only reduce resolution of delta in rate and increase once #9531
• MQE: Add support for label_replace #9532
• MQE: add support for unless #9533
• MQE: fix issue where rate or increase can return multiple series with the same labels #9553
• MQE: introduce series metadata function type #9558
• MQE: don't reuse FloatHistogram instances for successive points in instant vector selectors #9588
• MQE: add support for or #9589
• MQE: Refactor functions over instant vectors to support scalar args and add clamp function #9639
• MQE: Add support for clamp_min and clamp_max #9641
• MQE: Fix iterating over scalar values in Clamp #9642
• MQE: Add support for round #9651
• MQE: subqueries #9664
• MQE: add support for unary addition #9681
• MQE: create slices with capacity for number of steps remaining, not all possible steps #9717
• MQE: improve performance of subqueries when the parent query is a range query with many steps #9719
• MQE: Add support for changes function #9724
• MQE: fix or where both sides have series with the same labels #9874
• MQE: Add support for histogram_quantile #9929
• MQE: fix issue where subqueries could return series with no points #9998
• MQE: create slices with capacity for number of steps remaining, not all possible steps for functions over range vectors #10007
• MQE: mangle native histograms in returned slices to catch use-after-return bugs + fix issues found #10010
• MQE: Add support for histogram_avg #10046
• MQE: Ensure scalar operators close their inner operators #10047
• MQE: Add support for histogram_fraction #10048
• MQE debt: use time range to determine argument value #10050
• MQE: add support for group_left and group_right (aka many-to-one and one-to-many matching) #10119
• MQE: Add support for histogram_stddev function #10139
• MQE: eliminate loops in BucketedPool #10199
• MQE: don't select more data than strictly necessary #10209
• MQE: Add support for histogram_stdvar #10211
• MQE: track number of processed samples in each query #10232
• MQE: Fix panic in loading too many samples #10261

2.14.0

• Mimir query engine: Add PromQL engine test cases from Prometheus #8422
• Mimir query engine: add support for basic trigonometric and mathematical functions #8430
• Mimir query engine: add HTTP header to force use of Prometheus' engine #8454
• Mimir query engine: update description for -querier.enable-promql-engine-fallback CLI flag #8455
• Streaming engine: Native histogram aggregations #8360
• MQE: Refactor aggregations #8490
• MQE: Histogram range vectors #8508
• MQE: Add support for rate on native histograms #8577
• Mimir query engine: annotations #8660
• Mimir query engine: add support for aggregations with without, and resolve potential for hash collisions #8671
• Mimir query engine: avoid copying native histograms where possible #8677
• MQE: Refactor range functions to allow functions other than rate #8747
• Mimir query engine: fix issue where rate() over native histograms could panic or return incorrect results #8850
• Mimir query engine: add feature toggle for binary operations #8872
• MQE: Add NH support to binary operations #8838
• Mimir query engine: sync upstream test cases #8911
• Mimir query engine: split LimitingPool #8909
• MQE: Use kahan in sum aggregation #8923
• Mimir query engine: add native histogram custom bucket-related annotations #8924
• Mimir query engine: add support for count_over_time, present_over_time and last_over_time #8925
• MQE: Enable binary operations by default #8932
• Mimir query engine: improve performance of binary operations #8933
• Mimir query engine: add support for min_over_time, max_over_time, sum_over_time and avg_over_time #8934
• Mimir query engine: add support for offset in instant and range vector selectors #8962
• Mimir query engine: pass enabled feature toggles to engine #8986
• Mimir query engine: emit annotations for binary operations between incompatible native histograms #8993
• Mimir query engine: add support for scalar constants and pi() #8995
• MQE: Add support for more aggregation functions #9008
• Mimir query engine: add support for vector() and scalar() #9017
• Mimir query engine: add tool to find disabled but supported test cases #9018
• Mimir query engine: correctly handle functions that produce multiple series with the same labels #9019
• Mimir query engine: fix multiple org IDs present error when MQE is used to evaluate a cross-tenant query #9120
• MQE: Panic when memory isn't managed as we expect #9121
• Mimir query engine: Introduce FunctionOverInstantVector type #9136
• Mimir query engine: fix issue where unsupported aggregations that take a parameter do not trigger fallback #9139
• Mimir query engine: add support for binary operations between instant vectors and scalars #9140
• MQE: Fix native histogram bugs #9145
• Mimir query engine: fix panic in min and max over series containing only histograms #9191
• MQE: Also apply workaround for shared span slices to range vectors #9192
• Mimir query engine: expand TestConcurrentQueries to include testing queries with float samples #9194
• Mimir query engine: add support for unary negation #9196
• MQE: Add support for avg aggregation #9201
• MQE: Handle min/max NaN's same as promql #9212
• MQE: Remove workaround copying spans #9225
• Mimir query engine: fix issue where sum and avg results over native histograms can be corrupted #9260
• MQE: Native histogram lookback should re-use existing histograms #9272
• Mimir query engine: binary operations between two scalars #9277
• Mimir query engine: enable supported upstream test cases #9278
• Mimir query engine: extract query time range values to their own type #9280
• MQE: Add tests for various mixed data and edge cases #9281
• MQE: Add support for count aggregation #9342
• MQE: Add support for group aggregation  #9343
• MQE: fix rate() over native histograms where first point in range is a counter reset #9371
• MQE: Add support for resets function #9859
• MQE: Add support for deriv function #9858

2.13.0

• First iteration of streaming PromQL engine #7693
• Streaming PromQL engine: optionally fall back to Prometheus' engine if query is not supported #7898
• Streaming PromQL engine: extract linked list of series type #7899
• Streaming PromQL engine: add support for range vector selectors as the top-level expression #8023
• Streaming PromQL engine: sort series in matrix at end #8058
• Streaming PromQL engine: binary arithmetic operations with one-to-one matching #8096
• Streaming PromQL engine: native histograms #8121
• Mimir query engine: support query cancellation and timeout #8197
• Mimir query engine: add memory consumption per query limit #8230
• Mimir query engine: finish reorganisation started in #8230 #8247
• Mimir query engine: report estimated peak memory consumption as a metric and in traces #8270
• Mimir query engine: improve performance for case where right side of binary operation has fewer series than the left side #8276
• Mimir query engine: track active queries #8277
• Mimir query engine: rename metrics and CLI flag values to match new name #8291
• Mimir query engine: report queries rejected due to hitting the memory consumption limit in the cortex_querier_queries_rejected_total metric #8303
• Mimir query engine: extend binary operation tests to assert on error messages #8340
• Mimir Query Engine: Add instant-vector functions #8256
• Mimir query engine: complete renaming from #8256 #8348