[BUG] AKS Managed prometheus - Big discrepancy between portal metrics and prometheus metrics

[grzesuav]

Describe the bug
There are few metrics:

• Active time series in Azure portal
• Active time series % utilization in Azure portal
• scrape_samples_scraped from prometheus - which is the number of samples the target exposed.

I cannot correlate the first two (from azure portal) with prometheus one.

To Reproduce

Expected behavior

Those metrics should be in line, there is no other way currently to see the number of metrics in each job other than scrape_samples_scraped metric.

Which should be trusted ?

Additional context

Related #4159

[grzesuav]

Continuation of topic started in #4689 @vishiy @aritraghosh

[vishiy]

@grzesuav - how is scrape_samples_scraped equal to time-series count ? one is samples scraped (not ingested, as many could be dropped due to relabelings) and another is unique time-series (not samples).
To compare samples scraped with per minute AMW ingestion quota usage -
You should try -

sum(sum_over_time(scrape_samples_post_metric_relabeling [1m] ))

and compare with the metrics chart in the portal (for the samples/min ingested metric and not time-series).

[grzesuav]

I use this query and have this plot for last 7 days. I cannot explain the drop for controlplane-apiserver job as there were no activity on our side with any of the config

here is the azure portal metric

[grzesuav]

@vishiy

1. how I can check cardinality of particular metric to disable it ?
2. how I can check what is being throttled by AKS Prometheus ?

[grzesuav]

Also while extending the query per cluster and for wider timeline- it seems something changed on 20.12 which caused significant drop of metrics in centralus. Was there any fix on aks/managed prometheus side applied around that date ?

[grzesuav]

After looking at historical data

1. November -
2. December -

it seems that mine minimalIngestionProfile setting was ignored since ~12 of November till mid December, and now is keep being ignored in one cluster. It seems like configmap setting keeps getting ignored similar to #4689

So after all flagging the quota here was a red herring, as the reason for increased ingestion was not respecting minimalIngestionProfile setting from configmap.

At least this is my current working theory.