Feature: Allow overide of "instance" label

[nhatfield]

Can you allow the ability to overide the instance label? This should give users the ability to push a single metric name, for many instances. For example:

server 1: metric.test=1
server 2: metric.test=2

to allow prometheus to map both servers using the same metric name. This is to mimic the default behavior of node exporter allowing the user to select a query in grafana, metric.test and the values should show up from both server1 and server2

[beorn7]

Not sure I understand your question completely.

Have you read the section about the job and the instance labels? If you configure your Prometheus server to scrape the Pushgateway with honor_labels: true, the instance label pushed to the Pushgateway will prevail (or, if you have not pushed an instance label, there won't be any instance label after ingestion).

[nhatfield]

@beorn7 i think this is exactly what i need.... would that allow me to send in metrics on the same name but with different instance tags so I can query a single metric in grafana and see all of the "instances" reporting that metric?. Also how do I set this flag when i start the docker container? I dont see it in the doc

Thanks

[beorn7]

I'll convert this to a discussion in the "Q&A" section, because that's a better fit. (It's not a feature request, after all…) See you on the other side!

[beorn7]

The honor_labels: true is not a command line flag, but part of your scrape config. And it's the recommended way of scraping a Pushgateway anyway.

The scrape config should look similar to this:

scrape_configs:
- job_name: 'pushgateway'  
  honor_labels: true  
  static_configs:  
  - targets: ['pushgateway.example.org:9091']

[nhatfield]

I see, is this config on the prometheus configs? or is it the pushgateway configs?

Thanks for your help

[nhatfield]

Looks like prometheus side since the target is pushgateway! I'll give this a shot and update

[nhatfield]

I'm going to mark this as the answer since it definitely honors the label instead of making it an exported_instance label but, it does not solve the actual problem. @beorn7

What Im expecting, is the default behavior seen when prometheus scrapes multiple servers and they all have the same metric names with different values. Take CPU usage for example:

Here we have 1 metric node_cpu_seconds_total that is reported by many servers being scraped by prometheus, and we can see all servers reporting this metric on a single widget WITHOUT adding multiple queries...

However on the prometheus push gateway, non-unique metric id's are overwritten when another server/instance sends in that same metric with a different value. For example:

There are actually several instances reporting this metric above, but I can only see one of them in the graph. Also, if you look in prometheus push gateway ui, I can only see one of the servers (usually the last server that is processed for metric data)

This metric is being reported by several servers! Is there a way to solve this problem?

Thanks

[nhatfield]

Hmmmm, so it looks like the problem still remains. Maybe this is something Im doing wrong @beorn7 . Check this out....

My prometheus .yml config looks like this:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['prometheus:9090']

  - job_name: 'pushgateway'
    honor_labels: true
    static_configs:
    - targets: ['prometheus_pushgateway:9091']
    ```
    
    Im collecting metrics from multiple `instances` and I tag the instance names like this:
    
    ```
        cat <<EOF | curl -s --data-binary @- http://${prometheus_pg_host}:${prometheus_pg_port}/metrics/job/${project}
# HELP ${project}_${name}_block_height helium block height
# TYPE ${project}_${name}_block_height gauge
${project}_${name}_block_height{instance="${name}"} ${height}
EOF

project=test
name="server1" # these names are supplied by the metric collector and populated
name="server2" # these names are supplied by the metric collector and populated

What is happening is, instead of adding both labels to the metrics sent, prometheus gateway just overwrites the existing values from server1 with server2's metrics.

Essentially, I would expect to see 2 entries here 1 from instance=server1 and another from instance=server2

[nhatfield]

@beorn7 Never mind I got it working.. the mistake was I set the honor_labels under pushgateway job but i was creating a job in pushgateway called hnt_miner =) Awesome, thank you for your help...

[nhatfield]

NVM @beorn7 I spoke too soon. It looked like it was working before but now that I'm testing it out again it isn't working. I think the first one showed up earlier because I pushed the metrics up (which accepted the results from server 1) prometheus scraped it. Then i fixed the config and pushed the second one up. After trying this again with more metrics, prometheus push gateway is back to overwriting metrics sent with non-unique metric names with one another. IE:

example_metric_count {instance=server1} 0
example_metric_count {instance=server2} 1

which results in example_metric_count = 1 and the other is never seen.

[nhatfield]

EUREKA!!!! Ok now I understand what was wrong. It might be a good idea to make the documentation a little more clearer for idiots like me. The problem I was having is that I was not separating the labels from jobs . I was doing an all or nothing approach. Here's what I mean:

The WRONG WAY:

    cat <<EOF | curl -s --data-binary @- http://${prometheus_pg_host}:${prometheus_pg_port}/metrics/job/${project}.    <<< Notice the project here
# HELP ${project}_block_height helium block height
# TYPE ${project}_block_height gauge
${project}_block_height{instance="${name}"} ${height}

CORRECT WAY:

    cat <<EOF | curl -s --data-binary @- http://${prometheus_pg_host}:${prometheus_pg_port}/metrics/job/${project}_${name//-/_}  <<< Uniquely identify the job name but leave the metrics below as the same names
# HELP ${project}_block_height helium block height
# TYPE ${project}_block_height gauge
${project}_block_height{instance="${name}"} ${height}

This allows you to send the same metric name but have different instances/servers reporting the metric!! HUZZAH!! Thanks @beorn7 for your patience and guidance on honor_labels.

For anyone seeing this, to be clear my problem was NOT about honor_labels . Instead it was because I was not uniquely naming the job names in order for prometheus to scrape same metric names with different values from difference instances / servers. Take a real close look at: https://github.com/prometheus/pushgateway#url for more information. It's all about the job name!!

Thanks