From 1d43909381fe672de05197381d2108d914c7faed Mon Sep 17 00:00:00 2001 From: Joannes Madu <134279516+joannesmadu@users.noreply.github.com> Date: Mon, 14 Oct 2024 09:33:53 +0100 Subject: [PATCH] Requirements research/brid 98 (#10) * directory cleanup * directory cleanup * directory cleanup * directory cleanup * directory cleanup * directory cleanup * sidebar URL polish * sidebar URL polish * sidebar URL polish * sidebar URL polish * sidebar URL polish * sidebar URL polish * sidebar URL polish * sidebar URL relative links restored * sidebar URL relative links amend * sidebar URL - config.yml amend * sidebar URL - config.yml amend * sidebar URL - base URL * sidebar URL - base URL x relative path amend * sidebar URL - base URL x relative path amend * sidebar URL relative path amend * sidebar URL relative path amend * ruby security consolidation * sidebar live links final * image relative links * home nav * relative links across pages * relative links resolved across pages * requirements fleshed out --------- Co-authored-by: Joannes Madu --- mlops_big_picture/mlops_summary.md | 1 - mlops_big_picture/requirements_research.md | 18 ++++++++++++------ mlops_big_picture/versioning.md | 1 - 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/mlops_big_picture/mlops_summary.md b/mlops_big_picture/mlops_summary.md index 1ec740a..cf93975 100644 --- a/mlops_big_picture/mlops_summary.md +++ b/mlops_big_picture/mlops_summary.md @@ -13,7 +13,6 @@ MLOps is a set of practices that aims to unify the release cycle for machine lea Each page under "MLOps: The Big Picture" has been designed to address the research, implementation and troubleshooting regarding each component of the team's MLOps pipeline. The components, which are reflective of the typical order in which data flows through an MLOps pipeline, comprise: MLOps Pipeline Data FLow - diff --git a/mlops_big_picture/requirements_research.md b/mlops_big_picture/requirements_research.md index a9df50b..2c430dd 100644 --- a/mlops_big_picture/requirements_research.md +++ b/mlops_big_picture/requirements_research.md @@ -5,11 +5,19 @@ title: BridgeAI MLOps Knowledge Hub ## Where did the journey start? How were the design decisions made across the pipeline for each component? - +Our journey towards the creation of an end-to-end MLOps pipeline began as an extension of our [AI Adoption Assessment](https://iuk.ktn-uk.org/opportunities/ai-adoption-assessment-toolkit-from-digital-catapult/){:target="_blank"} initiative completed in collaboration with BridgeAI. We believed that in order to make the implementation of MLOps as viable as possible for users, we would need to create an end-to-end pipeline for users that they could adjust to suit their requirements and scope, rather than simply provide instructions on how to create one. -We decided on different tools for different components of our pipeline such as registry, data versioning and model monitoring by conducting spikes (research) for optimal tools based on our requirements. This research is covered in the pages underneath "MLOps: The Big Picture". +We decided on different tools for different components of our pipeline such as registry, data versioning and model monitoring by conducting research on optimal tools based on our requirements. This research is covered in the pages underneath "MLOps: The Big Picture", and is grounded in the evaluation of sets of tools per component that were suitable for implementing into our pipeline. More information on alternative tools we considered can be found in the [Horizon Scan](./corporate_perspective/prerequisites.html#horizon-scan){:target="_blank"} of this hub. -Some components did not have formal research conducted, and were instead decided on because they are widely used in the industry and therefore have in-depth documentation/community notes. +Some components (being data store, prediction service, monitoring) did not have formal research conducted, and were instead decided on because they are widely used in the industry and therefore have in-depth documentation/community notes. + +Our final tech stack upon deliberating, with links to corresponding pages in this hub: +1. [Data versioning](./mlops_big_picture/versioning.html){:target="_blank"}: DVC +2. [Training pipeline](./mlops_big_picture/DAG.html){:target="_blank"}: Airflow +3. [Model serving](./mlops_big_picture/serving.html){:target="_blank"}: MLflow +4. [Prediction service](./mlops_big_picture/pred_service.html){:target="_blank"}: Swagger +5. [Model monitoring](./mlops_big_picture/monitoring.html){:target="_blank"}: EvidentlyAI +6. [GitOps](./mlops_big_picture/gitops.html){:target="_blank"}: Flux ## What were the requirements for our ML model and the MLOps pipeline? @@ -18,6 +26,4 @@ We wanted to demonstrate an end-to-end, open source, pre-made MLOps pipeline. We Our base requirements: 1. The pipeline should be made of open source tools, to keep it cost-friendly and help with experimentation - -3. We should have a Minimum Viable MLOps pipeline with basic automation - +2. We should have a Minimum Viable MLOps pipeline with basic automation that users can then build on using the information provided in the hub, and other resources linked throughout the hub \ No newline at end of file diff --git a/mlops_big_picture/versioning.md b/mlops_big_picture/versioning.md index 785e8a5..02ccf1a 100644 --- a/mlops_big_picture/versioning.md +++ b/mlops_big_picture/versioning.md @@ -39,7 +39,6 @@ title: BridgeAI MLOps Knowledge Hub An image showing the different versions of data, features and model. Source: [DVC](https://dvc.org/doc/use-cases/versioning-data-and-model-files){:target="_blank"} -