Create a single repository structure, image with relevant dependencies for data pipelines

[eoriorda]

No description provided.

[MichaelTiemannOSC]

Related: #201

The above is related because AT PRESENT, virtually all of our data pipelines have library requirements not currently met by any existing default runtime environments. The AICoE tutorial explains how to use Thoth, Sesheta, and other CI/CD tools to build and deploy one's own images into Quay, but that quickly leads to runtime image fragmentation. The Highlander approach is in development, but not yet ready.

From an ARCHITECTURAL perspective, we need to describe how people should write pipelines today (using notebooks and suffering the costs of installing and loading their own modules on a case-by-case basis), as well as where we are going (what basic approach to deciding when to create a new image, maintained by whom, and how a part of the larger data reproducibility story). I think it is unsatisfying to achieve our data reproducibility solution by creating a runtime image manageability problem.

[eoriorda]

Produce a dedicated image pipeline repository . Goal to have an image created in the OSC Quay account . @erikerlandson

[HeatherAck]

Defer to after COP27, create standard image with pre-loaded libraries.

[HeatherAck]

See also: #98

[HeatherAck]

still in progress; need to break out "default" versions of jupyter notebooks (AI and non-AI);
Need DBT directory structure beyond simple use case (@mtiemann - reached out to DBT community for guidance).

• @HeatherAck find related issues and combine create a series [PCAF 14; Need a new template for data pipeline ingestion #232 ]