Use custom runtime

[victorlin]

follow-up to #169 (comment)

Context

Fauna is currently available through the Nextstrain managed runtimes (docker-base, conda-base). There are a couple of reasons to consider its removal from those:

1. It is only used for seasonal-flu and avian-flu and there are no plans to incorporate new pathogens during the expansion of Nextstrain's core pathogens.
2. With the current usage, there is no reason to update the code and dependency version constraints. This goes against the grain of the base runtimes, which are frequently kept up to date with latest versions of tools and their dependencies. At some point down the road, dependencies may fail to resolve entirely (e.g. if Fauna is pinned to Pandas v1 and Augur stops supporting that version).

Possible solutions

1. Define, publish, and use a custom Docker image that extends nextstrain/base.
2. Define and use a custom Conda environment that extends nextstrain::nextstrain-base.

Both of these could be pursued to allow flexibility for users.

[victorlin]

This should be considered under a broader proposal: nextstrain/public#7

[joverlee521]

Fauna is not included in the Nextstrain conda runtime (past discussion on this in nextstrain/conda-base#3), which is even more reason to just use a custom Docker image for fauna.

[jameshadfield]

Define, publish, and use a custom Docker image that extends nextstrain/base.

just use a custom Docker image for fauna

In this specific case, if the avian-flu ingest workflow is run in the docker/aws runtimes, i.e. starting within the canonical nextstrain docker container, and a rule needs fauna, what's the plan to do this? Docker-in-docker is discouraged. Running multiple sibling containers and communicating between them is going to require a bunch of work to enable.

If we don't have a plan then I think this issue should be closed (and nextstrain/public#7) and leave it all for workflows-as-programs buildpacks, when we get to that.

[victorlin]

No plan. The current workaround, as noted in nextstrain/public#7:

The ad-hoc approach of defining/creating custom runtimes by extending a base (examples: 1, 2, 3) has been used internally to some extent. The process is quite involved and not ideal for users at large.

After our recent dev discussion, I also think it's appropriate to close these issues for now.