About

Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX) is an educational community project to learn how to run AI, ML and other emerging workloads in the most efficient and cost-effective way across diverse models, data sets, software and hardware.

CK consists of several sub-projects:

• Collective Mind framework (CM) - a very lightweight Python-based framework with minimal dependencies intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware.

  • CM4MLOPS - a collection of portable, extensible and technology-agnostic automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware: see online catalog at CK playground, online MLCommons catalog

  • CM interface to run MLPerf inference benchmarks

  • CM4ABTF - a unified CM interface and automation recipes to run automotive benchmark across different models, data sets, software and hardware from different vendors.

• CMX (the next generation of CM and CM4MLOps) - we are developing the next generation of CM to make it simpler and more flexible based on user feedback. Please follow this project here.

• Collective Knowledge Playground - a unified platform to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results in a reproducible format with CM workflows, and organize public optimization challenges and reproducibility initiatives to co-design more efficient and cost-effiective software and hardware for emerging workloads.

• Artifact Evaluation - automating artifact evaluation and reproducibility initiatives at ML and systems conferences.

License

Apache 2.0

Copyright

• Copyright (c) 2021-2024 MLCommons
• Copyright (c) 2014-2021 cTuning foundation

Author

• Grigori Fursin (FlexAI, cTuning)

Maintainers

• CM/CM4Research/CM4MLPerf-results: Grigori Fursin
• CM4MLOps: Arjun Suresh and Anandhu Sooraj
• CMX (the next generation of CM, CM4MLOps and CM4MLPerf): Grigori Fursin

Citing our project

If you found the CM automation framework helpful, kindly reference this article: [ ArXiv ], [ BibTex ].

To learn more about the motivation behind CK and CM technology, please explore the following presentations:

• "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ ArXiv ]
• ACM REP'23 keynote about the MLCommons CM automation framework: [ slides ]
• ACM TechTalk'21 about Collective Knowledge project: [ YouTube ] [ slides ]
• Journal of Royal Society'20: [ paper ]

CM Documentation

• CM installation GUI
• CM Getting Started Guide and FAQ
  • Common CM interface to run MLPerf inference benchmarks
  • Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23
  • CM automation recipes for MLOps and DevOps
  • Other CM tutorials
• Full documentation
• CM taskforce
• CMX, CM and CK history

Acknowledgments

The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf, CM4Research and CMX) was created by Grigori Fursin and sponsored by cTuning.org, OctoAI and HiPEAC. Grigori donated CK to MLCommons to benefit the community and to advance its development as a collaborative, community-driven effort.

We thank MLCommons, FlexAI and cTuning for supporting this project, as well as our dedicated volunteers and collaborators for their feedback and contributions!