DC/OS Metrics API - Shows No Cassandra Metrics

[viavokal]

Hello,

Here's a description of a syndrome currently experienced in our environments, which I hope maybe you have experience or a word of advice on how to mitigate:

We are using dcos-cassandra-service for our DC/OS Managed Cassandra service.
We have one DC/OS cluster running on 1.9.1 which has Cassandra metrics available from it via DC/OS metrics API.
Another DC/OS cluster we are using, running on 1.9.0 with the same configuration, shows no Cassandra related metrics when queried using the (DCOS) CLI or API.
Initially, we thought of upgrading the 1.9.0 env to match the 1.9.1 and hope this will resolve the missing Cassandra metrics issue.
Eventually, we've upgraded it to 1.9.2 after finding out it has a fix to another issue we're experiencing in the 1.9.1 environment, which is that its API is returning only a partial changing metrics list upon each and every request (DCOS-16350)
Thing is, even after testing on the 1.9.2 environment, there are still no Cassandra metrics available via the DCOS Metrics API.
When dumping statsd received packets upon its UDP port, it appears that Cassandra is reporting its metrics properly on all environments (whether if the Metrics API is reporting those metrics or not, 1.9.0 - 1.9.2)
Apparently, something else is blocking the API / Mesos metrics module from exposing those metrics onwards to be later reported via the CLI/API.

Would appreciate your support,
Avi K

[viavokal]

Apparently both 1.9.0 and 1.9.2 for some reason are using dcos-metrics version 1.1.0 while 1.9.1 is using the 1.9.1-2 version, extracted from mesos-slave logs:

(1.9.2):

Aug 07 12:47:09 **** mesos-agent[2514]: I0807 12:47:09.864226  2533 http.cpp:307] HTTP GET for /slave(1)/state from ****:51128 with User-Agent='dcos-metrics/1.1.0-76-g1b4e013'

(1.9.0):

Aug 07 13:10:26 **** mesos-agent[1927]: I0807 13:10:26.942142  1933 http.cpp:307] HTTP GET for /slave(1)/state from ****:39236 with User-Agent='dcos-metrics/1.1.0'

(1.9.1):

Aug 07 12:43:18 ma-10-21-3-**** mesos-agent[3524]: I0807 12:43:18.295280  3542 http.cpp:307] HTTP GET for /slave(1)/state from ****:54464 with User-Agent='dcos-metrics/1.9.1-2-gcca85f3'

[triclambert]

This repo is deprecated and will be archived in one week. Please see the latest version of Cassandra or DSE for DC/OS:

https://docs.mesosphere.com/service-docs/cassandra/
https://docs.mesosphere.com/service-docs/dse/ (enterprise-only)