Librato showing dyno name instead of app name in source property

[jamischarles]

I believe this was working a few weeks ago. If that's incorrect, please accept my apologies.

Use case:

I'm draining several of my heroku apps into my l2met instance, and forwarding those on to librato.

Problem:
I'm trying to look at the p95 service time across several apps.

When I try to separate the apps in this graph, the sources show dyno names instead of app names. I expect to see the app name there. That's how it's supposed to work, right?

[ryandotsmith]

As it happens, this is sort of a strange issue. Previously, l2met would pull the hostname from the router log lines. This was removed in #96 as it caused major problems for Heroku SAAS apps that used sub-domains for their customers. (e.g. bmw.mycorp.com, apple.mycorp.com) This removed the ability to look at the app's router data in aggregate. And looking at the router's log lines, we have no way of determining what the app is. Consider the case when the app has many top level domains.

The current behavior in l2met does not use a source for Heroku Router log lines. However, there is a workaround. You can use source-prefix in your drain url to prefix all metric sources with a particular string. This will affect all metrics processed from the drain. For example, if you had an app named foo, you could append the following query parameter on your log drain url:

https://[email protected]/logs?source-prefix=foo

This will cause all metrics to have a source value prefix with foo. E.g. Your runtime metrics would have a source of foo.web.1 and your router logs would have a source of foo.

[ryandotsmith]

More information on drain prefixes can be found here: https://github.com/ryandotsmith/l2met/wiki/Usage#prefix

[ryandotsmith]

I also noticed that the Heroku router has web.X in the dyno= field. We should modify the Heroku router to read that field and use it as the source. In conjunction with the source prefix, you could have router metrics with sources identical to the runtime metrics.

[nextmat]

👍 that increased consistency would be quite helpful

[jamischarles]

@ryandotsmith That worked well. Thx. You mind if I start an FAQ section and have this be the first item?

[ryandotsmith]

@jamischarles that would be awesome. Want to make a FAQ page in the Wiki?

[ryandotsmith]

I am going to take a stab at implementation today.

[aseemk]

We should modify the Heroku router to read that field and use it as the source. In conjunction with the source prefix, you could have router metrics with sources identical to the runtime metrics.

We just upgraded to this new l2met scheme and ran into this inconsistency as well. Big +1!

[aseemk]

Question: we just started logging custom metrics from our own app code, e.g. measure#pdf_generation_time=1234ms, and I'm just seeing that the source for those is simply our source-prefix, e.g. production, and not the full dyno name, e.g. production.worker.5. Is this expected?

I'm guessing so, since we aren't logging source=worker.5. But our code within Heroku doesn't know which dyno it is, and we'd love to correlate our own custom metrics with Heroku's dyno runtime metrics (e.g. RAM usage for production.worker.5).

Would it be possible for l2met to use the dyno name as the source for custom metrics too, not just router metrics?