From d8c593e259a37ed7c44a9a4c81826cd58a2d8a95 Mon Sep 17 00:00:00 2001 From: Yuri Nikolic Date: Mon, 18 Mar 2024 17:48:03 +0100 Subject: [PATCH] Move -querier.minimize-ingester-requests from experimental to advanced Signed-off-by: Yuri Nikolic --- CHANGELOG.md | 6 ++++-- cmd/mimir/config-descriptor.json | 2 +- cmd/mimir/help-all.txt.tmpl | 2 +- docs/sources/mimir/configure/about-versioning.md | 1 - .../mimir/configure/configuration-parameters/index.md | 2 +- docs/sources/mimir/release-notes/v2.12.md | 5 ++++- pkg/querier/querier.go | 2 +- 7 files changed, 12 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c9954ef5be7..c8a79246cd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ * `prometheus_sd_updates_total` renamed to `cortex_prometheus_sd_updates_total` * `prometheus_sd_refresh_failures_total` renamed to `cortex_prometheus_sd_refresh_failures_total` * `prometheus_sd_refresh_duration_seconds` renamed to `cortex_prometheus_sd_refresh_duration_seconds` -* [CHANGE] Query-frontend: the default value for `-query-frontend.not-running-timeout` has been changed from 0 (disabled) to 2s. The configuration option has also been moved from "experimental" to "advanced". #7126 +* [CHANGE] Query-frontend: the default value for `-query-frontend.not-running-timeout` has been changed from 0 (disabled) to 2s. The configuration option has also been moved from "experimental" to "advanced". #7127 * [CHANGE] Store-gateway: to reduce disk contention on HDDs the default value for `blocks-storage.bucket-store.tenant-sync-concurrency` has been changed from `10` to `1` and the default value for `blocks-storage.bucket-store.block-sync-concurrency` has been changed from `20` to `4`. #7136 * [CHANGE] Store-gateway: Remove deprecated CLI flags `-blocks-storage.bucket-store.index-header-lazy-loading-enabled` and `-blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout` and their corresponding YAML settings. Instead, use `-blocks-storage.bucket-store.index-header.lazy-loading-enabled` and `-blocks-storage.bucket-store.index-header.lazy-loading-idle-timeout`. #7521 * [CHANGE] Store-gateway: Mark experimental CLI flag `-blocks-storage.bucket-store.index-header.lazy-loading-concurrency` and its corresponding YAML settings as advanced. #7521 @@ -49,9 +49,10 @@ * [CHANGE] The configuration option `-querier.max-query-into-future` has been deprecated and will be removed in Mimir 2.14. #7496 * [CHANGE] Distributor: the metric `cortex_distributor_sample_delay_seconds` has been deprecated and will be removed in Mimir 2.14. #7516 * [CHANGE] Query-frontend: The deprecated YAML setting `frontend.cache_unaligned_requests` has been moved to `limits.cache_unaligned_requests`. #7519 +* [CHANGE] Querier: the CLI flag `-querier.minimize-ingester-requests` has been moved from "experimental" to "advanced". #7638 * [FEATURE] Introduce `-server.log-source-ips-full` option to log all IPs from `Forwarded`, `X-Real-IP`, `X-Forwarded-For` headers. #7250 * [FEATURE] Introduce `-tenant-federation.max-tenants` option to limit the max number of tenants allowed for requests when federation is enabled. #6959 -* [FEATURE] Cardinality API: added a new `count_method` parameter which enables counting active label values. #7085 +* [FEATURE] Cardinality API: added a new `count_method` parameter which enables counting active label names. #7085 * [FEATURE] Querier / query-frontend: added `-querier.promql-experimental-functions-enabled` CLI flag (and respective YAML config option) to enable experimental PromQL functions. The experimental functions introduced are: `mad_over_time()`, `sort_by_label()` and `sort_by_label_desc()`. #7057 * [FEATURE] Alertmanager API: added `-alertmanager.grafana-alertmanager-compatibility-enabled` CLI flag (and respective YAML config option) to enable an experimental API endpoints that support the migration of the Grafana Alertmanager. #7057 * [FEATURE] Alertmanager: Added `-alertmanager.utf8-strict-mode-enabled` to control support for any UTF-8 character as part of Alertmanager configuration/API matchers and labels. It's default value is set to `false`. #6898 @@ -121,6 +122,7 @@ * [BUGFIX] promql: Fix Range selectors with an @ modifier are wrongly scoped in range queries. #7475 * [BUGFIX] Fix metadata API using wrong JSON field names. #7475 * [BUGFIX] Ruler: fix native histogram recording rule result corruption. #7552 +* [BUGFIX] Querier: fix HTTP status code translations for remote read requests. Previously, remote-read had conflicting behaviours: when returning samples all internal errors were translated to HTTP 400; when returning chunks all internal errors were translated to HTTP 500. #7487 ### Mixin diff --git a/cmd/mimir/config-descriptor.json b/cmd/mimir/config-descriptor.json index 5e1b4cab641..9e772385770 100644 --- a/cmd/mimir/config-descriptor.json +++ b/cmd/mimir/config-descriptor.json @@ -1863,7 +1863,7 @@ "fieldDefaultValue": true, "fieldFlag": "querier.minimize-ingester-requests", "fieldType": "boolean", - "fieldCategory": "experimental" + "fieldCategory": "advanced" }, { "kind": "field", diff --git a/cmd/mimir/help-all.txt.tmpl b/cmd/mimir/help-all.txt.tmpl index 9a6b3a1312f..3bab726ebd8 100644 --- a/cmd/mimir/help-all.txt.tmpl +++ b/cmd/mimir/help-all.txt.tmpl @@ -1714,7 +1714,7 @@ Usage of ./cmd/mimir/mimir: -querier.max-samples int Maximum number of samples a single query can load into memory. This config option should be set on query-frontend too when query sharding is enabled. (default 50000000) -querier.minimize-ingester-requests - [experimental] If true, when querying ingesters, only the minimum required ingesters required to reach quorum will be queried initially, with other ingesters queried only if needed due to failures from the initial set of ingesters. Enabling this option reduces resource consumption for the happy path at the cost of increased latency for the unhappy path. (default true) + If true, when querying ingesters, only the minimum required ingesters required to reach quorum will be queried initially, with other ingesters queried only if needed due to failures from the initial set of ingesters. Enabling this option reduces resource consumption for the happy path at the cost of increased latency for the unhappy path. (default true) -querier.minimize-ingester-requests-hedging-delay duration Delay before initiating requests to further ingesters when request minimization is enabled and the initially selected set of ingesters have not all responded. Ignored if -querier.minimize-ingester-requests is not enabled. (default 3s) -querier.prefer-streaming-chunks-from-ingesters diff --git a/docs/sources/mimir/configure/about-versioning.md b/docs/sources/mimir/configure/about-versioning.md index 83f1f39223d..04171800fae 100644 --- a/docs/sources/mimir/configure/about-versioning.md +++ b/docs/sources/mimir/configure/about-versioning.md @@ -122,7 +122,6 @@ The following features are currently experimental: - Use of Redis cache backend (`-blocks-storage.bucket-store.metadata-cache.backend=redis`) - Streaming chunks from ingester to querier (`-querier.prefer-streaming-chunks-from-ingesters`) - Streaming chunks from store-gateway to querier (`-querier.prefer-streaming-chunks-from-store-gateways`, `-querier.streaming-chunks-per-store-gateway-buffer-size`) - - Ingester query request minimisation (`-querier.minimize-ingester-requests`) - Limiting queries based on the estimated number of chunks that will be used (`-querier.max-estimated-fetched-chunks-per-query-multiplier`) - Max concurrency for tenant federated queries (`-tenant-federation.max-concurrent`) - Maximum response size for active series queries (`-querier.active-series-results-max-size-bytes`) diff --git a/docs/sources/mimir/configure/configuration-parameters/index.md b/docs/sources/mimir/configure/configuration-parameters/index.md index 87578891e99..cffd910f52f 100644 --- a/docs/sources/mimir/configure/configuration-parameters/index.md +++ b/docs/sources/mimir/configure/configuration-parameters/index.md @@ -1313,7 +1313,7 @@ store_gateway_client: # CLI flag: -querier.streaming-chunks-per-store-gateway-buffer-size [streaming_chunks_per_store_gateway_series_buffer_size: | default = 256] -# (experimental) If true, when querying ingesters, only the minimum required +# (advanced) If true, when querying ingesters, only the minimum required # ingesters required to reach quorum will be queried initially, with other # ingesters queried only if needed due to failures from the initial set of # ingesters. Enabling this option reduces resource consumption for the happy diff --git a/docs/sources/mimir/release-notes/v2.12.md b/docs/sources/mimir/release-notes/v2.12.md index f7af16a9cb4..c91154b7f90 100644 --- a/docs/sources/mimir/release-notes/v2.12.md +++ b/docs/sources/mimir/release-notes/v2.12.md @@ -48,9 +48,12 @@ Additionally, the following previously experimental features are now considered - **The maximum number of concurrent index header loads across all tenants**, configurable via the `-blocks-storage.bucket-store.index-header.lazy-loading-concurrency` CLI flag on store-gateways. It defaults to `4`. -- **The maximum time to wait for the query-frontend to become ready before rejecting requests**, configurable via the `-query-frontend.not-running-timeout` CLI flags on query-frontends. +- **The maximum time to wait for the query-frontend to become ready before rejecting requests**, configurable via the `-query-frontend.not-running-timeout` CLI flag on query-frontends. It now defaults to `2s`. +- **The CLI flag that allows queriers to reduce pressure on ingesters** by initially querying only the minimum set of ingesters required to reach quorum, `-querier.minimize-ingester-requests`. + It is now enabled by default. + - **Spread-minimizing token-related CLI flags**: `-ingester.ring.token-generation-strategy`, `-ingester.ring.spread-minimizing-zones` and `-ingester.ring.spread-minimizing-join-ring-in-order`. You can read more about this feature [in our blog post](https://grafana.com/blog/2024/03/07/how-we-improved-ingester-load-balancing-in-grafana-mimir-with-spread-minimizing-tokens/). diff --git a/pkg/querier/querier.go b/pkg/querier/querier.go index e7303aede75..ba0ca37da3f 100644 --- a/pkg/querier/querier.go +++ b/pkg/querier/querier.go @@ -53,7 +53,7 @@ type Config struct { PreferAvailabilityZone string `yaml:"prefer_availability_zone" category:"experimental" doc:"hidden"` StreamingChunksPerIngesterSeriesBufferSize uint64 `yaml:"streaming_chunks_per_ingester_series_buffer_size" category:"advanced"` StreamingChunksPerStoreGatewaySeriesBufferSize uint64 `yaml:"streaming_chunks_per_store_gateway_series_buffer_size" category:"experimental"` - MinimizeIngesterRequests bool `yaml:"minimize_ingester_requests" category:"experimental"` // Enabled by default as of Mimir 2.11, remove altogether in 2.12. + MinimizeIngesterRequests bool `yaml:"minimize_ingester_requests" category:"advanced"` MinimiseIngesterRequestsHedgingDelay time.Duration `yaml:"minimize_ingester_requests_hedging_delay" category:"advanced"` // PromQL engine config.