Skip to content

Commit

Permalink
Fixing review findings
Browse files Browse the repository at this point in the history
Signed-off-by: Yuri Nikolic <[email protected]>
  • Loading branch information
duricanikolic committed Mar 12, 2024
1 parent 33c0dc6 commit 6e98ab9
Showing 1 changed file with 48 additions and 46 deletions.
94 changes: 48 additions & 46 deletions docs/sources/mimir/release-notes/v2.12.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,6 @@ For the complete list of changes, refer to the [CHANGELOG](https://github.com/gr

## Features and enhancements

- **Added `mimir-distroless` container image, built upon a `distroless` image** (currently, it is `gcr.io/distroless/static-debian12`).
This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

- **Added support to only count series that are considered _active_** through the Cardinality API endpoint `/api/v1/cardinality/label_names` by passing the `count_method` parameter.
If set to `active` it counts only series that are considered _active_ according to the `-ingester.active-series-metrics-idle-timeout` flag setting rather than counting all in-memory series.

Expand All @@ -33,64 +29,30 @@ For the complete list of changes, refer to the [CHANGELOG](https://github.com/gr
If this computation fails, the `cortex_bucket_index_compaction_jobs_errors_total` metric is updated instead.
The estimated number of compaction jobs is also shown in **Top tenants**, **Tenants**, and **Compactor** dashboards.

- **Added `mimir-distroless` container image built upon a `distroless` image** (`gcr.io/distroless/static-debian12`).
This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

- **The number of pre-allocated workers used to forward push requests to the ingesters**, configurable via the `-distributor.reusable-ingester-push-workers` CLI flag on distributors.
It now defaults to `2000`.
Note that this is a performance optimization, and not a limiting feature.
If not enough workers available, a new goroutine will be spawned.
If not enough workers available, new goroutines will be spawned.

- **The number of gRPC server workers used to serve the requests**, configurable via the `-server.grpc.num-workers` CLI flag.
It now defaults to `100`.
Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
When all workers are busy, a new goroutine will be spawned.
If not enough workers are available, new goroutines will be spawned.

- **The maximum number of concurrent index header loads across all tenants**, configurable via the `-blocks-storage.bucket-store.index-header.lazy-loading-concurrency` CLI flag on store-gateways.
It defaults to `4`.

- **The maximum time to wait for the query-frontend to become ready before rejecting requests**, configurable via the `-query-frontend.not-running-timeout` CLI flags on query-frontends.
It now defaults to `2s`.

- **Spread-minimizing token-related CLI flags**: `-ingester.ring.token-generation-strategy`, `-ingester.ring.spread-minimizing-zones` and `-ingester.ring.spread-minimizing-join-ring-in-order`.

## Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

- **The maximum number of tenant IDs that may be for a federated query** can be configured via the `-tenant-federation.max-tenants` CLI flag on query-frontends.
By default, it's `0`, meaning that the limit is disabled.

- **Sharding of active series queries can be enabled** via the `-query-frontend.shard-active-series-queries` CLI flag on query-frontends.

- **Timely head compaction** can be enabled via the `-blocks-storage.tsdb.timely-head-compaction-enabled` on ingesters.
If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.

- **Streaming of responses from querier to query-frontend** can be enabled via the `-querier.response-streaming-enabled` CLI flag on queriers.
This is currently supported only for responses from the `/api/v1/cardinality/active_series` endpoint.

- **The maximum response size for active series queries**, in bytes, can be set via the `-querier.active-series-results-max-size-bytes` CLI flag on queriers.

- **Metric relabeling on a per-tenant basis can be forcefully disabled** via the `-distributor.metric-relabeling-enabled` CLI flag on rulers.
Metrics relabeling is enabled by default.

- **Query Queue Load Balancing by Query Component**. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
This feature is recommended to be enabled.
The following CLI flags must be set to true in order to be in effect:

- `-query-frontend.additional-query-queue-dimensions-enabled` on the query-frontend.
- `-query-scheduler.additional-query-queue-dimensions-enabled` on the query-scheduler.

- **Owned series tracking in ingesters** can be enabled via the `-ingester.track-ingester-owned-series` CLI flag.
When enabled, ingesters will track the number of in-memory series that still map to the ingester based on the ring state.
These counts are more reactive to ring and shard changes than in-memory series, and can be used when enforcing tenant series limits by enabling the `-ingester.use-ingester-owned-series-for-limits` CLI flag.
This feature requires [zone-aware replication]({{< relref "../configure/configure-zone-aware-replication" >}}) to be enabled, and the replication factor to be equal to the number of zones.

### Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently.
Refer to the [Grafana Mimir Helm chart documentation](/docs/helm-charts/mimir-distributed/latest/).
You can read more about this feature [in our blog post](https://grafana.com/blog/2024/03/07/how-we-improved-ingester-load-balancing-in-grafana-mimir-with-spread-minimizing-tokens/).

## Important changes

Expand Down Expand Up @@ -167,6 +129,41 @@ The following metrics are removed or deprecated:
- `cortex_bucket_store_blocks_loaded_by_duration` has been removed.
- `cortex_distributor_sample_delay_seconds` has been deprecated and will be removed in Mimir 2.14.

## Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

- **The maximum number of tenant IDs that may be for a federated query** can be configured via the `-tenant-federation.max-tenants` CLI flag on query-frontends.
By default, it's `0`, meaning that the limit is disabled.

- **Sharding of active series queries can be enabled** via the `-query-frontend.shard-active-series-queries` CLI flag on query-frontends.

- **Timely head compaction** can be enabled via the `-blocks-storage.tsdb.timely-head-compaction-enabled` on ingesters.
If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.

- **Streaming of responses from querier to query-frontend** can be enabled via the `-querier.response-streaming-enabled` CLI flag on queriers.
This is currently supported only for responses from the `/api/v1/cardinality/active_series` endpoint.

- **The maximum response size for active series queries**, in bytes, can be set via the `-querier.active-series-results-max-size-bytes` CLI flag on queriers.

- **Metric relabeling on a per-tenant basis can be forcefully disabled** via the `-distributor.metric-relabeling-enabled` CLI flag on rulers.
Metrics relabeling is enabled by default.

- **Query Queue Load Balancing by Query Component**. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
This feature is recommended to be enabled.
The following CLI flags must be set to true in order to be in effect:

- `-query-frontend.additional-query-queue-dimensions-enabled` on the query-frontend.
- `-query-scheduler.additional-query-queue-dimensions-enabled` on the query-scheduler.

- **Owned series tracking in ingesters** can be enabled via the `-ingester.track-ingester-owned-series` CLI flag.
When enabled, ingesters will track the number of in-memory series that still map to the ingester based on the ring state.
These counts are more reactive to ring and shard changes than in-memory series, and can be used when enforcing tenant series limits by enabling the `-ingester.use-ingester-owned-series-for-limits` CLI flag.
This feature requires [zone-aware replication](https://grafana.com/docs/mimir/latest/configure/configure-zone-aware-replication/) to be enabled, and the replication factor to be equal to the number of zones.

## Bug fixes

- Distributor: fixed an issue where `-distributor.metric-relabeling-enabled` could cause distributors to panic.
Expand All @@ -191,3 +188,8 @@ The following metrics are removed or deprecated:
- Ruler: requests to remote querier when server's response exceeds its configured max payload size aren't retried anymore.
- Ruler: fixed a regression that caused client errors to be tracked in `cortex_ruler_write_requests_failed_total` metric.
- Ruler: fixed an issue with recording rule result being corruption due to an usage of a bad native histogram pointer.

### Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm charts are released independently.
Refer to the [Grafana Mimir Helm chart documentation](/docs/helm-charts/mimir-distributed/latest/).

0 comments on commit 6e98ab9

Please sign in to comment.