added compacted MLPerf inference v3.1 results in the Collective Mind …

…format for the MLCommons CK playground

CONTRIBUTING.md

@@ -6,4 +6,8 @@ Generally we encourage people to become a MLCommons member if they wish to contr
 
 Regardless of if you are a member, your organization needs to sign the MLCommons CLA. Please fill out this [CLA sign up form](https://forms.gle/Ew1KkBVpyeJDuRw67) form to get started.
 
-MLCommons project work is tracked with issue trackers and pull requests. Modify the project in your own fork and issue a pull request once you want other developers to take a look at what you have done and discuss the proposed changes. Ensure that cla-bot and other checks pass for your Pull requests.
+MLCommons project work is tracked with issue trackers and pull requests. Modify the project in your own fork and issue a pull request once you want other developers to take a look at what you have done and discuss the proposed changes. Ensure that cla-bot and other checks pass for your Pull requests.
+
+
+* [Grigori Fursin](https://cKnowledge.org/gfursin)
+* [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh)

README.md

@@ -1,33 +1,34 @@
-#  MLPerf™ Inference Results in the MLCommons CM format
+#  MLPerf Benchmark Results in the MLCommons CM format
 
-This is the repository containing aggregated results of the [MLPerf™ Inference benchmark]( https://github.com/mlcommons/inference )
-and the [TinyMLPerf benchmark](https://github.com/mlcommons/tiny) in the [MLCommons CM format](https://github.com/mlcommons/ck)
-for the [Collective Knowledge Playground](https://x.cKnowledge.org)
+This repository contains compacted and aggregated results of the [MLPerf Inference benchmark]( https://github.com/mlcommons/inference ),
+[MLPerf Training benchmark]() and [TinyMLPerf benchmark](https://github.com/mlcommons/tiny) 
+in the compact [MLCommons Collective Mind format](https://github.com/mlcommons/ck)
+for the [MLCommons CK Playground](https://x.cKnowledge.org)
 being developed by the [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce).
 
-The goal is to make it easier for the community to analyze MLPerf inference results, 
+The goal is to make it easier for the community to analyze MLPerf results, 
 add derived metrics such as performance/Watt and constraints,
-and link reproducibility reports as shown in these examples:
+generate graphs, prepare reports and link reproducibility reports as shown in these examples:
 * [Power efficiency to compare Qualcomm, Nvidia and Sima.ai devices](https://cKnowledge.org/mlcommons-mlperf-inference-gui-derived-metrics-and-conditions)
 * [Reproducibility report for Nvidia Orin](https://access.cknowledge.org/playground/?action=experiments&name=mlperf-inference--v3.0--edge--closed--image-classification--offline&result_uid=3751b230c800434a)
 
-# How to import raw MLPerf results to CK/CM format
+## How to import raw MLPerf results to CK/CM format
 
 Install [MLCommons CM framework](https://github.com/mlcommons/ck/blob/master/docs/installation.md).
 
-## MLPerf inference benchmark results
+### MLPerf inference benchmark results
 
 Follow this [README](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/import-mlperf-inference-to-experiment/README-extra.md) from the related CM automations script.
 
 You can see aggregated results [here](https://access.cknowledge.org/playground/?action=experiments&tags=mlperf-inference,all).
 
-## TinyMLPerf benchmark results
+### TinyMLPerf benchmark results
 
 Follow this [README](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/import-mlperf-tiny-to-experiment/README-extra.md) from the related CM automations script.
 
 You can see aggregated results [here](https://access.cknowledge.org/playground/?action=experiments&tags=mlperf-tiny,all).
 
-## MLPerf training benchmark results
+### MLPerf training benchmark results
 
 Follow this [README](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/import-mlperf-training-to-experiment/README-extra.md) from the related CM automations script.
 
@@ -38,30 +39,31 @@ You can see aggregated results [here](https://access.cknowledge.org/playground/?
 
 ### Using your own Python script
 
-You can use this repository to fix, update and improve experimental results
+You can use this repository to analyze, reuse, update and improve MLPerf results
+compact
 by calculating and adding derived metrics (performance/watt)
-or links to reproducibility reports that will be visible in a CK Playground GUI.
+or links to reproducibility reports that will be visible at the [MLCommons CK playground](https://access.cknowledge.org/playground/?action=experiments).
 
 Install [MLCommons CM framework](https://github.com/mlcommons/ck/blob/master/docs/installation.md).
 
 Pull CM repository with automation recipes and with MLPerf results in the CM format:
 ```bash
 cm pull repo mlcommons@ck
-cm pull repo mlcommons@cm_inference_results
+cm pull repo mlcommons@ck_mlperf_results
 ```
 
-Find CM entries with MLPerf v3.0 experiments from CMD:
+Find CM entries with MLPerf inference v3.1 experiments from CMD:
 ```bash
-cm find experiment --tags=mlperf-inference,v3.0
+cm find experiment --tags=mlperf-inference,v3.1
 ```
 
-Find CM entries with MLPerf v3.0 experiments from Python:
+Find CM entries with MLPerf inference v3.1 experiments from Python:
 ```python
 import cmind
 
 r = cmind.access({'action':'find',
                   'automation':'experiment,a0a2d123ef064bcb',
-                  'tags':'mlperf-inference,v3.0'})
+                  'tags':'mlperf-inference,v3.1'})
 
 if r['return']>0: cmind.error(r)
 
@@ -81,7 +83,23 @@ cm run script "process mlperf-inference results" --experiment_tags=mlperf-infere
 ```
 
 
-# Contact us
 
-This project is maintained by the [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce).
+
+## Copyright
+
+2021-2023 [MLCommons](https://mlcommons.org)
+
+## License
+
+[Apache 2.0](LICENSE.md)
+
+## Project coordinators
+
+[Grigori Fursin](https://cKnowledge.org/gfursin) and [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh).
+
+## Contact us
+
+This project is maintained by the [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce),
+[cTuning foundation](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org).
+
 Join our [Discord server](https://discord.gg/JjWNWXKxwT) to ask questions, provide your feedback and participate in further developments.

cmr.yaml

@@ -1,3 +1,3 @@
-alias: mlcommons@cm_inference_results
+alias: mlcommons@ck_mlperf_results
 git: true
 uid: 6fbf42fa292142da

experiment/mlperf-inference--all--datacenter--closed--image-classification--multistream/2023-04-06.00-36-37.103508/cm-result.json

@@ -1,4 +1,106 @@
 [
+  {
+    "Accuracy": 76.076,
+    "Accuracy_div_100": 0.76076,
+    "Availability": "available",
+    "Division": "closed",
+    "Location": "closed/NVIDIA/results/L4x1_TRT/resnet50/MultiStream",
+    "MlperfModel": "resnet",
+    "Model": "resnet50",
+    "Organization": "NVIDIA",
+    "Platform": "L4x1_TRT",
+    "Result": 0.846827,
+    "Result_Units": "Latency (ms)",
+    "Scenario": "MultiStream",
+    "SystemName": "ASROCKRACK 1U1G-MILAN (1x L4, TensorRT)",
+    "SystemType": "datacenter,edge",
+    "Units": "Latency (ms)",
+    "accelerator_model_name": "NVIDIA L4",
+    "accelerators_per_node": 1,
+    "compliance": 1,
+    "errors": 0,
+    "framework": "TensorRT 9.0.0, CUDA 12.2",
+    "git_url": "https://github.com/mlcommons/inference_results_v3.1",
+    "has_power": false,
+    "host_processor_core_count": 16,
+    "host_processor_model_name": "AMD EPYC 7313P 16-Core Processor",
+    "host_processors_per_node": 2,
+    "inferred": 0,
+    "notes": "",
+    "number_of_nodes": 1,
+    "operating_system": "Ubuntu 20.04.4",
+    "uid": "e844b9ea278f400a",
+    "url": "https://github.com/mlcommons/inference_results_v3.1/tree/master/closed/NVIDIA/results/L4x1_TRT/resnet50/MultiStream",
+    "version": "v3.1"
+  },
+  {
+    "Accuracy": 76.094,
+    "Accuracy_div_100": 0.76094,
+    "Availability": "available",
+    "Division": "closed",
+    "Location": "closed/CTuning/results/gcp_g2.standard.4-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream",
+    "MlperfModel": "resnet",
+    "Model": "resnet50",
+    "Organization": "CTuning",
+    "Platform": "gcp_g2.standard.4-nvidia_original-gpu-tensorrt-vdefault-default_config",
+    "Result": 0.994519,
+    "Result_Units": "Latency (ms)",
+    "Scenario": "MultiStream",
+    "SystemName": "Google Cloud Platform (g2.standard.4)",
+    "SystemType": "datacenter,edge",
+    "Units": "Latency (ms)",
+    "accelerator_model_name": "NVIDIA L4",
+    "accelerators_per_node": 1,
+    "compliance": 1,
+    "errors": 0,
+    "framework": "Nvidia inference implementation with CM API, TensorRT v8.6.1.6",
+    "git_url": "https://github.com/mlcommons/inference_results_v3.1",
+    "has_power": false,
+    "host_processor_core_count": 2,
+    "host_processor_model_name": "Intel(R) Xeon(R) CPU @ 2.20GHz",
+    "host_processors_per_node": 1,
+    "inferred": 0,
+    "notes": "Powered by MLCommons CM automation language and CK playground. ",
+    "number_of_nodes": 1,
+    "operating_system": "Ubuntu 22.04 (linux-5.19.0-1030-gcp-glibc2.35)",
+    "uid": "d471167e561a4640",
+    "url": "https://github.com/mlcommons/inference_results_v3.1/tree/master/closed/CTuning/results/gcp_g2.standard.4-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream",
+    "version": "v3.1"
+  },
+  {
+    "Accuracy": 76.078,
+    "Accuracy_div_100": 0.76078,
+    "Availability": "available",
+    "Division": "closed",
+    "Location": "closed/Dell/results/R750xa_A100_PCIE_80GBx4_TRT/resnet50/MultiStream",
+    "MlperfModel": "resnet",
+    "Model": "resnet50",
+    "Organization": "Dell",
+    "Platform": "R750xa_A100_PCIE_80GBx4_TRT",
+    "Result": 0.701063,
+    "Result_Units": "Latency (ms)",
+    "Scenario": "MultiStream",
+    "SystemName": "Dell PowerEdge R750xa (4x A100-PCIe-80GB, TensorRT)",
+    "SystemType": "datacenter,edge",
+    "Units": "Latency (ms)",
+    "accelerator_model_name": "NVIDIA A100-PCIe-80GB",
+    "accelerators_per_node": 4,
+    "compliance": 1,
+    "errors": 0,
+    "framework": "TensorRT 9.0.0, CUDA 12.2",
+    "git_url": "https://github.com/mlcommons/inference_results_v3.1",
+    "has_power": false,
+    "host_processor_core_count": 64,
+    "host_processor_model_name": "Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz",
+    "host_processors_per_node": 2,
+    "inferred": 0,
+    "notes": "",
+    "number_of_nodes": 1,
+    "operating_system": "Rocky Linux 9.1",
+    "uid": "ccc4659daeaa4f4e",
+    "url": "https://github.com/mlcommons/inference_results_v3.1/tree/master/closed/Dell/results/R750xa_A100_PCIE_80GBx4_TRT/resnet50/MultiStream",
+    "version": "v3.1"
+  },
   {
     "Accuracy": 76.148,
     "Accuracy_div_100": 0.76148,