Plotting Netperf CDFs

The netperf benchmark outputs the latency data that it collects as a JSON encoded histogram. With a small python script to parse and combine the histograms and an R script to plot the resulting data file, it's easy to plot CDFs to compare latency data across different dimensions.

If, for example, you wanted to run netperf across several different zones, you could run PKB with something like the following configuration:

netperf:
  flags:
    machine_type: n1-standard-2
    ip_addresses: INTERNAL
    netperf_benchmarks: TCP_RR
    netperf_histogram_buckets: 1000
    cloud: GCP
  flag_matrix_defs:
    multi_zone:
      zones: [
          us-west1-a, us-west1-a, us-west1-a, us-west1-a, us-west1-a,
          europe-north1-a, europe-north1-a, europe-north1-a, europe-north1-a, europe-north1-a,
          us-west2-a, us-west2-a, us-west2-a, us-west2-a, us-west2-a
      ]

With the following command line:

./pkb.py --benchmark_config_file=netperf_config.yml --benchmarks=netperf --flag_matrix=multi_zone --run_processes=15

This would run 15 different copies of the netperf benchmark and put all resulting samples in a single results file.

With the following script, you can parse that results file and output the data in a format that makes it easy to plot:

#!/usr/bin/env python2.7

import collections
import json
import re
import sys


def main():
  if len(sys.argv) != 2:
      print "usage: %s samples_file.json" % sys.argv[0]
      sys.exit(1)
  latency_histogram_by_zone = collections.defaultdict(
          lambda : collections.defaultdict(int))
  total_samples = collections.defaultdict(int)
  with open(sys.argv[1]) as samples_file:
    for line in samples_file.readlines():
      sample = json.loads(line)
      if sample['metric'] == 'TCP_RR_Latency_Histogram':
        labels = sample['labels']
        zone = re.search(r'\|sending_zone:(.*?)\|', labels).group(1)
        histogram = json.loads(
                re.search(r'\|histogram:(.*?)\|', labels).group(1))
        for bucket, count in histogram.iteritems():
          latency_histogram_by_zone[zone][float(bucket)] += int(count)
          total_samples[zone] += int(count)
  for zone, histogram in latency_histogram_by_zone.iteritems():
    running_count = 0
    for bucket in sorted(histogram):
      running_count += histogram[bucket]
      percentile = 100.0 * running_count / total_samples[zone]
      print ','.join((zone, str(bucket), str(percentile)))


if __name__ == "__main__":
    main()

With an even smaller R script, you can plot the resulting data file:

library(ggplot2)
df <- read.csv('zone_cdf.csv', sep=',', col.names=c('zone','latency','percentile'))
ggplot(data=df, aes(x=latency, y=percentile, group=zone)) + geom_line(aes(color=zone)) + xlim(0, 200) + xlab('latency (us)')
ggsave('zone_cdf.png')