Elyra Component Library - The Component Library for AI, Machine Learning, ETL, and Data Science

TL;DR

• set of re-usable coarse-grained components (just a bunch of code)
• think of tasks, not functions (e.g. read from database, transform data, train model, deploy model, ...)
• write once, runs everywhere (export to Kubeflow, Apache Airflow, Apache Nifi, ...)
• just use python, no other skills required (no Kubeflow component yaml, maven, Java, ...)
• 1st class citizen in JupyterLab and the Elyra Pipeline Editor (creating a low code / no code IDE for data science)
• upstream repository to IBM Watson Studio Pipelines contributed components in IBM Cloud Pak for Data

This is a component library for artificial intelligence, machine learning, "extract, transform, load" processes and data science. The goal is to enable low-code/no-code rapid prototyping by providing ready-made components for various business domains, supporting various computer languages, working on various data flow editors and running on diverse execution engines. To demonstrate its utility, we constructed a workflow composed exclusively of this library's components. To demonstrate the capabilities of this library, we made use of a publicly available Computed Tomography (CT) scans dataset [covidata] and created a deep learning model, which is supposed to classify exams as either COVID-19 positive or negative. The pipeline was built with Elyra's Pipeline Visual Editor, with support for local, Airflow and Kubeflow execution https://arxiv.org/abs/2103.03281.

Low Code / No Code pipeline creation tool for data science

Bring the latest and greatest libraries at the hands of everybody.

AIX360/LIME highlights a poor deep learning covid classification model looking at bones only

Components of this library can be exported as:

1. KubeFlow pipeline components
2. Apache Airflow components
3. Standalone graphical components for the Elyra pipeline editor
4. Standalone components to be run from the command line

Visually create pipelines from notebooks and run everywhere

Each notebook is following a similar format.

1. The first cell contains a description of the component itself.
2. The second cell installs all dependencies using pip3.
3. The third cell imports all dependencies.
4. The fourth cell contains a list of dependencies, input parameters, and return values as Python comments
5. The fifth cell reads the input parameters from environment variables.

Export notebooks and files as runtime components for different engines

To learn more on how this library works in practice, please have a look at the following video

Related work

ploomber orchest

[covidata] Joseph Paul Cohen et al. COVID-19 Image Data Collection: Prospective Predictions Are the Future, arXiv:2006.11988, 2020