Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingester: promote series limit using own series to stable #9496

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
* `-ruler.query-frontend.grpc-client-config.grpc-compression=s2`
* [FEATURE] Alertmanager: limit added for maximum size of the Grafana configuration (`-alertmanager.max-config-size-bytes`). #9402
* [FEATURE] Ingester: Experimental support for ingesting out-of-order native histograms. This is disabled by default and can be enabled by setting `-ingester.ooo-native-histograms-ingestion-enabled` to `true`. #7175
* [FEATURE] Ingester: Promote user series limits base on owned series to stable. Enable `-ingester.use-ingester-owned-series-for-limits` and `-ingester.track-ingester-owned-series` by default. #9496
* [ENHANCEMENT] Ruler: Support `exclude_alerts` parameter in `<prometheus-http-prefix>/api/v1/rules` endpoint. #9300
* [ENHANCEMENT] Distributor: add a metric to track tenants who are sending newlines in their label values called `cortex_distributor_label_values_with_newlines_total`. #9400

Expand Down
13 changes: 5 additions & 8 deletions cmd/mimir/config-descriptor.json
Original file line number Diff line number Diff line change
Expand Up @@ -3387,21 +3387,19 @@
"required": false,
"desc": "When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldDefaultValue": true,
"fieldFlag": "ingester.use-ingester-owned-series-for-limits",
"fieldType": "boolean",
"fieldCategory": "experimental"
"fieldType": "boolean"
},
{
"kind": "field",
"name": "track_ingester_owned_series",
"required": false,
"desc": "This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldDefaultValue": true,
"fieldFlag": "ingester.track-ingester-owned-series",
"fieldType": "boolean",
"fieldCategory": "experimental"
"fieldType": "boolean"
},
{
"kind": "field",
Expand All @@ -3411,8 +3409,7 @@
"fieldValue": null,
"fieldDefaultValue": 15000000000,
"fieldFlag": "ingester.owned-series-update-interval",
"fieldType": "duration",
"fieldCategory": "experimental"
"fieldType": "duration"
},
{
"kind": "block",
Expand Down
6 changes: 3 additions & 3 deletions cmd/mimir/help-all.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -1492,7 +1492,7 @@ Usage of ./cmd/mimir/mimir:
-ingester.out-of-order-time-window duration
[experimental] Non-zero value enables out-of-order support for most recent samples that are within the time window in relation to the TSDB's maximum time, i.e., within [db.maxTime-timeWindow, db.maxTime]). The ingester will need more memory as a factor of rate of out-of-order samples being ingested and the number of series that are getting out-of-order samples. If query falls into this window, cached results will use value from -query-frontend.results-cache-ttl-for-out-of-order-time-window option to specify TTL for resulting cache entry.
-ingester.owned-series-update-interval duration
[experimental] How often to check for ring changes and possibly recompute owned series as a result of detected change. (default 15s)
How often to check for ring changes and possibly recompute owned series as a result of detected change. (default 15s)
-ingester.partition-ring.consul.acl-token string
ACL Token used to interact with Consul.
-ingester.partition-ring.consul.cas-retry-delay duration
Expand Down Expand Up @@ -1680,11 +1680,11 @@ Usage of ./cmd/mimir/mimir:
-ingester.stream-chunks-when-using-blocks
Stream chunks from ingesters to queriers. (default true)
-ingester.track-ingester-owned-series
[experimental] This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled.
This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled. (default true)
-ingester.tsdb-config-update-period duration
[experimental] Period with which to update the per-tenant TSDB configuration. (default 15s)
-ingester.use-ingester-owned-series-for-limits
[experimental] When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit.
When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit. (default true)
-log.format string
Output log messages in the given format. Valid formats: [logfmt, json] (default "logfmt")
-log.level value
Expand Down
6 changes: 6 additions & 0 deletions cmd/mimir/help.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -455,6 +455,8 @@ Usage of ./cmd/mimir/mimir:
The maximum number of in-memory series per metric name, across the cluster before replication. 0 to disable.
-ingester.max-global-series-per-user int
The maximum number of in-memory series per tenant, across the cluster before replication. 0 to disable. (default 150000)
-ingester.owned-series-update-interval duration
How often to check for ring changes and possibly recompute owned series as a result of detected change. (default 15s)
-ingester.partition-ring.consul.hostname string
Hostname and port of Consul. (default "localhost:8500")
-ingester.partition-ring.delete-inactive-partition-after duration
Expand Down Expand Up @@ -487,6 +489,10 @@ Usage of ./cmd/mimir/mimir:
File path where tokens are stored. If empty, tokens are not stored at shutdown and restored at startup. Must be empty if -ingester.ring.token-generation-strategy is set to "spread-minimizing".
-ingester.ring.zone-awareness-enabled
True to enable the zone-awareness and replicate ingested samples across different availability zones. This option needs be set on ingesters, distributors, queriers, and rulers when running in microservices mode.
-ingester.track-ingester-owned-series
This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled. (default true)
-ingester.use-ingester-owned-series-for-limits
When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit. (default true)
-log.format string
Output log messages in the given format. Valid formats: [logfmt, json] (default "logfmt")
-log.level value
Expand Down
17 changes: 8 additions & 9 deletions docs/sources/mimir/configure/configuration-parameters/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -1251,19 +1251,18 @@ instance_limits:
# CLI flag: -ingester.error-sample-rate
[error_sample_rate: <int> | default = 10]

# (experimental) When enabled, only series currently owned by ingester according
# to the ring are used when checking user per-tenant series limit.
# When enabled, only series currently owned by ingester according to the ring
# are used when checking user per-tenant series limit.
# CLI flag: -ingester.use-ingester-owned-series-for-limits
[use_ingester_owned_series_for_limits: <boolean> | default = false]
[use_ingester_owned_series_for_limits: <boolean> | default = true]

# (experimental) This option enables tracking of ingester-owned series based on
# ring state, even if -ingester.use-ingester-owned-series-for-limits is
# disabled.
# This option enables tracking of ingester-owned series based on ring state,
# even if -ingester.use-ingester-owned-series-for-limits is disabled.
# CLI flag: -ingester.track-ingester-owned-series
[track_ingester_owned_series: <boolean> | default = false]
[track_ingester_owned_series: <boolean> | default = true]

# (experimental) How often to check for ring changes and possibly recompute
# owned series as a result of detected change.
# How often to check for ring changes and possibly recompute owned series as a
# result of detected change.
# CLI flag: -ingester.owned-series-update-interval
[owned_series_update_interval: <duration> | default = 15s]

Expand Down
10 changes: 5 additions & 5 deletions pkg/ingester/ingester.go
Original file line number Diff line number Diff line change
Expand Up @@ -202,9 +202,9 @@ type Config struct {

ErrorSampleRate int64 `yaml:"error_sample_rate" json:"error_sample_rate" category:"advanced"`

UseIngesterOwnedSeriesForLimits bool `yaml:"use_ingester_owned_series_for_limits" category:"experimental"`
UpdateIngesterOwnedSeries bool `yaml:"track_ingester_owned_series" category:"experimental"`
OwnedSeriesUpdateInterval time.Duration `yaml:"owned_series_update_interval" category:"experimental"`
UseIngesterOwnedSeriesForLimits bool `yaml:"use_ingester_owned_series_for_limits"`
UpdateIngesterOwnedSeries bool `yaml:"track_ingester_owned_series"`
OwnedSeriesUpdateInterval time.Duration `yaml:"owned_series_update_interval"`

PushCircuitBreaker CircuitBreakerConfig `yaml:"push_circuit_breaker"`
ReadCircuitBreaker CircuitBreakerConfig `yaml:"read_circuit_breaker"`
Expand Down Expand Up @@ -236,8 +236,8 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet, logger log.Logger) {
f.Uint64Var(&cfg.ReadPathMemoryUtilizationLimit, "ingester.read-path-memory-utilization-limit", 0, "Memory limit, in bytes, for CPU/memory utilization based read request limiting. Use 0 to disable it.")
f.BoolVar(&cfg.LogUtilizationBasedLimiterCPUSamples, "ingester.log-utilization-based-limiter-cpu-samples", false, "Enable logging of utilization based limiter CPU samples.")
f.Int64Var(&cfg.ErrorSampleRate, "ingester.error-sample-rate", 10, "Each error will be logged once in this many times. Use 0 to log all of them.")
f.BoolVar(&cfg.UseIngesterOwnedSeriesForLimits, "ingester.use-ingester-owned-series-for-limits", false, "When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit.")
f.BoolVar(&cfg.UpdateIngesterOwnedSeries, "ingester.track-ingester-owned-series", false, "This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled.")
f.BoolVar(&cfg.UseIngesterOwnedSeriesForLimits, "ingester.use-ingester-owned-series-for-limits", true, "When enabled, only series currently owned by ingester according to the ring are used when checking user per-tenant series limit.")
f.BoolVar(&cfg.UpdateIngesterOwnedSeries, "ingester.track-ingester-owned-series", true, "This option enables tracking of ingester-owned series based on ring state, even if -ingester.use-ingester-owned-series-for-limits is disabled.")
f.DurationVar(&cfg.OwnedSeriesUpdateInterval, "ingester.owned-series-update-interval", 15*time.Second, "How often to check for ring changes and possibly recompute owned series as a result of detected change.")
f.BoolVar(&cfg.PushGrpcMethodEnabled, "ingester.push-grpc-method-enabled", true, "Enables Push gRPC method on ingester. Can be only disabled when using ingest-storage to make sure ingesters only receive data from Kafka.")

Expand Down
Loading