Development/deployment w/containers

[agladstein]

Discussion on development/deployment workflow w/containers.

[julianpistorius]

@agladstein Does this include running a Jupyter notebook in the container? Or should that be a separate issue?

[agladstein]

I think not.
@j23414 ?

[j23414]

in my case, not a jupyter notebook... I'm running a pipeline of scripts (one bash script which calls perl and R scripts as needed)

[agladstein]

@julianpistorius

[julianpistorius]

OK. That's good. We can create a separate 'Jupyter in Singularity' issue.

[julianpistorius]

@jpcolella & @agladstein - This is what I was thinking about when I mentioned 'sub containers':

https://www.sylabs.io/guides/2.5.1/user-guide/reproducible_scif_apps.html

The Scientific Filesystem (SCIF) provides internal modularity of containers, and it makes it easy for the creator to give the container implied metadata about software. For example, installing a set of libraries, defining environment variables, or adding labels that belong to app foo makes a strong assertion that those dependencies belong to foo . When I run foo, I can be confident that the container is running in this context, meaning with foo's custom environment, and with foo's libraries and executables on the path.

https://sci-f.github.io/

Goals

For the official description of goals, see the specification. In the following sections, we hope to show that SCIF is useful because it allows for:

• flexible, internal modularity where the definition of modularity is entirely based on the needs of the creator and user, and the resulting container reflects that.
• reproducible practices by way of providing portable environments with modular internal contents that are easily discovered.

While SCi-F is not a workflow manager, it follows naturally that the creator of a SCIF app might use these internal variables to have modules internally talk to one another. The user and creator do not need to know the structural specifics of the standard, but only how to reference them.

https://sci-f.github.io/spec-v1

Goals

The Scientific Filesystem (SCIF) will:

• Enable a UA to write tools and software that work with a predictibly structured SCIF.
• Enable the SC to generate discoverable and predicable SCIF that work easily with integrations
• Define a set of environment variables with reasonable defaults that:
  • Can be referenced by other installed applications, ideal for integrations that foster inter-app functionality
  • Reliably determine locations for data, apps, and metadata.
• Define a recipe format that maps to a SCIF, and vice versa.
• Provide basic software to generate and manage SCIF.
  • A single entrypoint that directs to app entrypoints and environments
  • Preview a filesystem from a recipe file
  • Generate a filesystem from a recipe file
  • Generate a recipe file from a SCIF
  • Generate a build specification for a container using a recipe file
  • An API for developers to interact and control a SCIF
• In the context of modular, encapsulated container environment, provide reproducible scientific software that can be shared with confidence of discoverability.