metrics.html.md.erb

---
title: PCF Healthwatch Metrics
owner: PCF Healthwatch
---

This topic lists derived and generated metrics provided by Pivotal Cloud Foundry (PCF) Healthwatch. The metrics are used to monitor the health and performance of the PCF platform.

##<a id='cli'></a>Cloud Foundry CLI Health

The Cloud Foundry command line interface (CLI) enables developers to create and manage PCF apps. PCF Healthwatch executes a continuous test suite validating the core app developer functions of the CLI.

See the table below for information on metrics related to Cloud Foundry CLI smoke tests.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Can push</td>
  <td><code>healthwatch.health.check.cliCommand.push</code> and <code>healthwatch.health.check.cliCommand.push.timeout</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass, <code>0</code> = fail, or <code>-1</code> = test did not run</td>
</tr>
<tr>
  <td>Can start</td>
  <td><code>healthwatch.health.check.cliCommand.start</code> and <code>healthwatch.health.check.cliCommand.start.timeout</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass, <code>0</code> = fail, or <code>-1</code> = test did not run</td>
</tr>
<tr>
  <td>Receiving logs</td>
  <td><code>healthwatch.health.check.cliCommand.logs</code> and <code>healthwatch.health.check.cliCommand.logs.timeout</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass, <code>0</code> = fail, or <code>-1</code> test did not run</td>
</tr>
<tr>
  <td>Can stop</td>
  <td><code>healthwatch.health.check.cliCommand.stop</code> and <code>healthwatch.health.check.cliCommand.stop.timeout</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass, <code>0</code> = fail, or <code>-1</code> test did not run</td>
</tr>
<tr>
  <td>Can delete</td>
  <td><code>healthwatch.health.check.cliCommand.delete</code> and <code>healthwatch.health.check.cliCommand.delete.timeout</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass, <code>0</code> = fail, or <code>-1</code> test did not run</td>
</tr>
<tr>
  <td>Test app push time</td>
  <td><code>healthwatch.health.check.cliCommand.pushTime</code></td>
  <td>5 min</td>
  <td>Time in ms</td>
</tr>
<tr>
  <td>Overall smoke test battery result</td>
  <td><code>healthwatch.health.check.cliCommand.success</code></td>
  <td>5 min</td>
  <td><code>1</code> = pass or <code>0</code> = fail</td>
</tr>
<tr>
  <td>Overall smoke test battery run time</td>
  <td><code>healthwatch.health.check.cliCommand.duration</code></td>
  <td>5 min</td>
  <td>Time in ms</td>
</tr>
</table>

<p class="note"><strong>Note</strong>: Timeout metrics are written only when a timeout occurs. Their value is always zero.</p>

##<a id='opsman'></a>Ops Manager Health

Issues with Ops Manager health can impact an operator's ability to perform an upgrade or to rescale the PCF platform when necessary. Therefore, it is recommended to continuously monitor Ops Manager availability. PCF Healthwatch executes this check as a part of its test suite.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Ops Manager availability</td>
  <td><code>healthwatch.health.check.OpsMan.available</code></td>
  <td>1 min</td>
  <td><code>1</code> = pass or <code>0</code> = fail</td>
</tr>
</table>

##<a id='appsman'></a>Apps Manager Health

These metrics help users to monitor app availability and responsiveness. PCF Healthwatch uses Apps Manager as a canary app and continuously checks its health.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Apps Manager availability</td>
  <td><code>healthwatch.health.check.AppsMan.available</code></td>
  <td>1 min</td>
  <td><code>1</code> = pass or <code>0</code> = fail</td>
</tr>
<tr>
  <td>Aps Manager response time</td>
  <td><code>healthwatch.health.check.AppsMan.responseTime</code></td>
  <td>1 min</td>
  <td>Time in ms</td>
</tr>
</table>

##<a id='bosh-director'></a>BOSH Director Health

Losing the BOSH Director does not significantly impact the experience of PCF end users. However, this issue means a loss of resiliency for BOSH-managed VMs. It is recommended to continuously monitor the BOSH Director health. PCF Healthwatch executes this check as a part of its test suite.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>BOSH Director health</td>
  <td><code>healthwatch.health.check.bosh.director.success</code> and <code>healthwatch.health.check.bosh.director.timeout</code></td>
  <td>10 min</td>
  <td><code>1</code> = pass or <code>0</code> = fail</td>
</tr>
</table>

<p class="note"><strong>Note</strong>: The timeout metric is written if a deploy or delete task takes more than 10 minutes.</p>

##<a id='loss-rates'></a>Loss Rates

This section lists metrics used to monitor Loggregator, the PCF component responsible for logging.

###<a id='firehose-loss'></a>Firehose Loss Rate

This metric calculation is recommended for automating and monitoring platform scaling. Two versions of the metric (per minute and per hour) are used to monitor the Loggregator Firehose.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Firehose loss rate</td>
  <td><code>healthwatch.Firehose.LossRate.1H</code> and <code>healthwatch.Firehose.LossRate.1M</code></td>
  <td>Loss rate per minute and per hour</td>
</tr>
</table>

###<a id='adapter-loss'></a>Adapter Loss Rate

This metric calculation is recommended for automating and monitoring platform scaling. The metric is used to monitor the Scalable Syslog feature of Loggregator.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Adapter loss rate (syslog drain performance)</td>
  <td><code>healthwatch.ScalableSyslog.Adapter.LossRate.1M</code></td>
  <td>Loss rate per minute</td>
</tr>
</table>

###<a id='rlp-loss'></a>Reverse Log Proxy Loss Rate

This metric calculation is recommended for automating and monitoring platform scaling. The metric is used to monitor the Scalable Syslog feature of Loggregator.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Reverse Log Proxy loss rate (syslog drain performance)</td>
  <td><code>healthwatch.ScalableSyslog.RLP.LossRate.1M</code></td>
  <td>Loss rate per minute</td>
</tr>
</table>

##<a id='capacity'></a>Capacity Metrics

This section lists metrics used to monitor the total percentage of available memory, disk, and cell container capacity.

###<a id='memory'></a>Available Memory

This metric calculation is recommended for automating and monitoring platform scaling.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Available memory</td>
  <td><code>healthwatch.Diego.TotalPercentageAvailableMemoryCapacity.5M</code></td>
  <td>Percentage of available memory (averaged over 5 min)</td>
</tr>
</table>

###<a id='disk'></a>Available Disk

This metric calculation is recommended for automating and monitoring platform scaling.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Available disk</td>
  <td><code>healthwatch.Diego.TotalPercentageAvailableDiskCapacity.5M</code></td>
  <td>Percentage of available disk (averaged over 5 min)</td>
</tr>
</table>

###<a id='cell-container'></a>Available Cell Container Capacity

This metric calculation is recommended for automating and monitoring platform scaling.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Description</th>
</tr>
<tr>
  <td>Available cell container capacity</td>
  <td><code>healthwatch.Diego.TotalPercentageAvailableContainerCapacity.5M</code></td>
  <td>Percentage of available cell container capacity (averaged over 5 min)</td>
</tr>
</table>

##<a id='bosh-deployment'></a>BOSH Deployment Occurrence

Monitoring BOSH deployment occurrence adds context to related data, such as VM (job) health.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>BOSH deployment occurrence</td>
  <td><code>healthwatch.bosh.deployment</code></td>
  <td>30 sec</td>
  <td><code>1</code> = a running deployment or <code>0</code> = not a running deployment</td>
</tr>
</table>

##<a id='other-metrics'></a>Other Platform Metrics

This section lists existing platform metrics used by PCF Healthwatch. For more information about these metrics, see [Key Performance Indicators](https://docs.pivotal.io/pivotalcf/1-11/monitoring/kpi.html) and [Key Capacity Scaling Indicators](https://docs.pivotal.io/pivotalcf/1-11/monitoring/key-cap-scaling.html).

###<a id='job-health'></a>Job Health

The Job Health metric is used for every VM in the CF deployment. This does not include additional deployments, such as MySQL or Redis.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Job health</td>
  <td><code>system.healthy</code></td>
  <td>1 min</td>
  <td><code>1</code> = the system is healthy or <code>0</code> = the system is not healthy</td>
</tr>
</table>

###<a id='job-vitals'></a>Job Vitals

The Job Vitals metrics are written for core ERT jobs. This does not include additional deployments, such as MySQL or Redis.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>CPU utilization</td>
  <td><code>system.cpu.user</code></td>
  <td>1 min</td>
  <td>Percentage of CPU used</td>
</tr>
<tr>
  <td>Memory utilization</td>
  <td><code>system.mem.percent</code></td>
  <td>1 min</td>
  <td>Percentage of system memory used</td>
</tr>
<tr>
  <td>Disk utilization</td>
  <td><code>system.disk.system.percent</code></td>
  <td>1 min</td>
  <td>Percentage of system disk used</td>
</tr>
<tr>
  <td>Persistent disk utilization</td>
  <td><code>system.disk.persistent.percent</code></td>
  <td>1 min</td>
  <td>Percentage of persistent disk used</td>
</tr>
<tr>
  <td>Ephemeral disk utilization</td>
  <td><code>system.disk.ephemeral.percent</code></td>
  <td>1 min</td>
  <td>Percentage of ephemeral disk used</td>
</tr>
</table>

###<a id='kpis-capacity'></a>Diego Cell Capacity

The Diego Cell Capacity metrics are used to monitor the amount of memory and disk available for a Diego cell.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Available memory</td>
  <td><code>rep.CapacityRemainingMemory</code></td>
  <td>1 min</td>
  <td>Amount of memory (MiB) available for a Diego cell to allocate to containers</td>
</tr>
<tr>
  <td>Available disk</td>
  <td><code>rep.CapacityRemainingDisk</code></td>
  <td>1 min</td>
  <td>Amount of disk (MiB) available for a Diego cell to allocate to containers</td>
</tr>
</table>

###<a id='app-instances'></a>Application Instances

The Application Instances metrics are used to monitor the health of application instances (AIs). For more information about the lifecycle of an app container and crash events, see [Crash Events](https://docs.pivotal.io/pivotalcf/1-11/devguide/deploy-apps/app-lifecycle.html#crash-events).

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Current running AIs and change in running AIs</td>
  <td><code>bbs.LRPsRunning</code></td>
  <td>During events</td>
  <td>Total number of LRP instances running on Diego cells</td>
</tr>
<tr>
  <td>Crashed AIs</td>
  <td><code>bbs.CrashedActualLRPs</code></td>
  <td>30 sec</td>
  <td>Total number of LRP instances that have crashed in a deployment</td>
</tr>
</table>

###<a id='logging'></a>Logging Performance

The Loggregator Firehose throughput metric is used to monitor PCF logging performance.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Firehose throughput</td>
  <td><code>DopplerServer.listeners.totalReceivedMessageCount</code></td>
  <td>5 sec</td>
  <td>Total number of messages received across all Doppler listeners</td>
</tr>
</table>

###<a id='router'></a>Router

The Router metrics are used to monitor the health and performance of the Gorouter.

<table class="nice">
<tr>
  <th>Test</th>
  <th>Metric</th>
  <th>Frequency</th>
  <th>Description</th>
</tr>
<tr>
  <td>Router throughput</td>
  <td><code>gorouter.total_requests</code></td>
  <td>5 sec</td>
  <td>Lifetime number of requests completed by the Gorouter VM</td>
</tr>
<tr>
  <td>Router latency</td>
  <td><code>gorouter.latency</code></td>
  <td>Emitted per Gorouter request</td>
  <td>Time (ms) the Gorouter takes to handle requests to its app endpoints</td>
</tr>
<tr>
  <td>Router jobs CPU</td>
  <td><code>system.cpu.user</code></td>
  <td>1 min</td>
  <td>Percentage of CPU used by the Gorouter job</td>
</tr>
<tr>
  <td>502 bad gateways</td>
  <td><code>gorouter.bad_gateways</code></td>
  <td>5 sec</td>
  <td>Lifetime number of bad gateways, or 502 responses, from the Gorouter itself</td>
</tr>
</table>