Supported input metric data formats

The harness receives metric data from the binary or app running on device. Depending on the framework and the backend platform, the collected data format may be different. The harness supports a few metric data formats. If the binary outputs those formats, there may not a need to change the harness. Instead, the existing data parsing format can be reused.

To specify the data converter to use, you can add a field converter to the model field of the meta data file (json file).

All converters are inherited from DataConverterBase. They should implement two methods:

• collect: collect the data from raw strings.
• convert: convert the data to the format that the harness understands.

Below are some of the data formats that are supported by the harness.

json_converter

The json_converter takes a string with multiple lines, or a list of strings with each string a single line. Each line or each entry is a json string. One json string reports one metric for one entity. If multiple metrics are reported for one entity, they are reported separately.

The format of the json string is as follows:

Option 1:

{"type": <type>, "metric": <metric>, "unit": <unit>, "value": <value>}

Option 2:

{"type": <type>, "metric": <metric>, "unit": <unit>, "info_string": <info_string>}

Option 3:

{
  "type": <type>, 
  "metric": <metric>, 
  "unit": <unit>, 
  "num_runs": <num of reported runs>,
  "summary": {
    "p0": <p0 value of the all runs>,
    "p10": <p10 value of all runs>,
    "p50": <p50 value of all runs>,
    "p90": <p90 value of all runs>,
    "p100": <p100 value of all runs>,
    "mean": <average value of all runs>,
    "stdev": <standard deviation of all runs>,
    "MAD": <mean absolute deviation of all runs>,
  }
}

The meaning of each field is as follows:

• type: a unique identifier. It must be NET to indicate the metric is for an entire model. It can also be an operator name. It can be other meaningful identifier for your purpose. Entries with the same type refer to the same entity.
• metric: a string to indicate the metric. Usually type and metric uniquely identifies the entry in one benchmark run.
• unit: a string indicating the unit of the metric.
• value (optional): a float number indicating the collected metric in one benchmark iteration.
• info_string (optional): a string that provides more information on the metric or model. In different iterations of one benchmark run, its content must be identical.
• num_runs (optional): the number of iterations in one benchmark run.
• summary (optional): the summary statistics for one benchmark run. The field values are described in the above example. All fields are optional. The value is passed to the harness without change. The harness does not try to calculate the statistics.

The binary may run multiple times and collect the same data each time. The number of iterations is specified in the field iter in the json file. The harness relies on the number of times the combined field type of value NET and the field value to determine whether the pre-determined number of iterations are satisfied. Thus it is important to use NET for the model level metric.

json_with_identifier_converter

json_with_identifier_converter takes input data with multiple lines. Each line may or may not be a line with the metric. The lines with metric contain identifier specified by the framework. The content after the identifier is expected to be a valid json string. The content before the identifier can be anything. This converter is useful if the output is adb logcat from Android. The output of the benchmark metrics may be intermixed with logs from other processes on the Android system.

Except the above difference, the parser is the same as json_converter.