Skip to content

Commit

Permalink
Runbook: clarify MimirIngesterReachingSeriesLimit errors and retries
Browse files Browse the repository at this point in the history
  • Loading branch information
bboreham committed Sep 25, 2024
1 parent 82c47c8 commit b54c094
Showing 1 changed file with 9 additions and 1 deletion.
10 changes: 9 additions & 1 deletion docs/sources/mimir/manage/mimir-runbooks/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,15 @@ If nothing obvious from the above, check for increased load:

### MimirIngesterReachingSeriesLimit

This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit. Once the limit is reached, writes to the ingester will fail (5xx) for new series, while appending samples to existing ones will continue to succeed.
This alert fires when the `max_series` per ingester instance limit is enabled and the actual number of in-memory series in an ingester is reaching the limit.
The threshold is set at 80%, to give some chance to react before the limit is reached.
Once the limit is reached, writes to the ingester will fail for new series. Appending samples to existing ones will continue to succeed.

Note that the error responses sent back to the sender are classed as "server error" (5xx), which should result in a retry by the sender.
While this situation continues, these retries will stall the flow of data, and newer data will queue up on the sender.
If the condition is cleared in a short time, service can be restored with no data loss.

This is different to what happens when the `max_global_series_per_user` is exceeded, which is considered a "client error" (4xx) where excess data is discarded.

In case of **emergency**:

Expand Down

0 comments on commit b54c094

Please sign in to comment.