Snapstats Collector not properly handling connection errors to remote graphite server

[Dieterbe]

not sure if this is the correct repo, but last night I got paged for our grafanalabs alert [Alerting] Disabled Snap Tasks - task-agent-2 alert. the chart did not show any data .

looking at the snap tasks I noticed that the Snapstats Collector had this "last error":

Error: dial tcp xxx.xxx.xxx.xxx:2003: i/o timeout

so it seems to me this task needs to try harder to reconnect to the graphite server after it encounters a timeout, connection drop, etc. it should buffer data and try to flush later.
i presume the source code for that task lives in this repo but i'm not sure.

in the meantime I will change the "no data" condition to result in "no data" instead of "alerting".
we can change it back when this is fixed.

[woodsaj]

The collector plugins dont send metrics to TSDB's/ that is the job of publisher plugins. In this case that would be the snap-plugin-publisher-graphite

[Dieterbe]

intelsdi-x/snap-plugin-publisher-graphite#47