Port old dev docs

book/_toc.yml

@@ -27,6 +27,12 @@ parts:
       - file: plugins/how-to-guides/intro
         sections:
           - file: plugins/how-to-guides/set-up-development-environment
+          - file: plugins/how-to-guides/register-a-plugin
+          - file: plugins/how-to-guides/create-register-method
+          - file: plugins/how-to-guides/create-register-visualizer
+          - file: plugins/how-to-guides/create-register-pipeline
+          - file: plugins/how-to-guides/create-register-transformer
+          - file: plugins/how-to-guides/artifact-collections-as-io
           # - file: plugins/how-to-guides/add-citation
           # - file: plugins/how-to-guides/usage-example
           # - file: plugins/how-to-guides/type-map
@@ -35,6 +41,7 @@ parts:
       - file: plugins/explanations/intro
         sections:
           - file: plugins/explanations/types-of-types
+          - file: plugins/explanations/transformers
       - file: plugins/references/intro
         sections:
           - file: plugins/references/antipatterns
@@ -49,6 +56,8 @@ parts:
     chapters:
       - file: framework/intro
         title: Introduction
+        sections:
+          - file: framework/explanations/architecture
 
   - caption: "CI Internals"
     chapters:
@@ -57,4 +66,6 @@ parts:
 
   - caption: "Back Matter"
     chapters:
-      - file: bibliography
+      - file: back-matter/glossary
+      - file: back-matter/bibliography
+      - file: back-matter/genindex

book/back-matter/genindex.md

@@ -0,0 +1 @@
+# Index

book/back-matter/glossary.md

@@ -0,0 +1,103 @@
+# Glossary
+
+```{glossary}
+Action
+  A generic term to describe a concrete {term}`method`, {term}`visualizer`, or {term}`pipeline`.
+  Actions accept parameters and/or files ({term}`artifacts <Artifact>` or {term}`metadata`) as input, and generate some kind of output.
+
+Archive
+  The directory structure of a QIIME 2 {term}`Result`.
+  Contains *at least* a root directory (named by {term}`UUID`) and a ``VERSION`` file within that directory.
+
+Artifact
+  A QIIME 2 {term}`Result` that contains data to operate on.
+
+Deployment
+  An installation of QIIME 2 as well as zero-or-more {term}`interfaces <Interface>` and {term}`plugins <Plugin>`.
+  The collection of interfaces and plugins in a deployment can be defined by a {term}`distribution` of QIIME 2.
+
+Directory Format
+  A string that represents a particular layout of files and or directories as well as how their contents will be structured.
+
+Distribution
+  A collection of QIIME 2 plugins that are designed to be installed together.
+  These are generally grouped by a theme. For example, the Amplicon Distribution provides a collection of plugins for analysis of microbiome amplicon data, while the Shotgun Distribution provides a collection of plugins for analysis of microbiome shotgun metagenomics data.
+  When a distribution is installed, that particular installation of QIIME 2 is an example of a {term}`deployment`.
+
+Format
+  A string that represents a particular file format.
+
+Framework
+  The engine of orchestration that enables QIIME 2 to function together as a cohesive unit.
+
+Identifier
+  A unique value that denotes an individual sample or feature.
+
+Identity
+  Distinguishes a piece of data. QIIME 2 does not consider a rename (like UNIX ``mv``) to change identity, however re-running a command, would change identity.
+
+Input
+  Data provided to an {term}`action`. Can be an {term}`artifact` or {term}`metadata`.
+
+Interface
+  A user-interface responsible for coordinating user-specified intent into {term}`framework`-driven action.
+
+Metadata
+  Columnar data for annotating additional values to existing data. Operates along Sample IDs or Feature IDs.
+
+Method
+  A method accepts some combination of QIIME 2 {term}`artifacts <Artifact>` and {term}`parameters <Parameter>` as {term}`input`, and produces one or more QIIME 2 artifacts as {term}`output`.
+
+Output
+  Objects returned by an {term}`action`. Can be {term}`artifact(s) <Artifact>` or {term}`visualization(s) <Visualization>`.
+
+Parameter
+  A value that alters the behavior of an {term}`action`.
+
+Payload
+  Data that is meant for primary consumption or interpretation (in contrast to *metadata* which may be useful retrospectively, but is not primarily useful).
+
+Pipeline
+  A pipeline accepts some combination of QIIME 2 {term}`artifacts <Artifact>` and {term}`parameters <Parameter>` as {term}`input`, and produces one or more QIIME 2 {term}`artifacts <Artifact>` and/or {term}`visualizations <Visualization>` as {term}`output`.
+
+Plugin
+  A discrete module that registers some form of additional functionality with the {term}`framework`, including new {term}`methods <Method>`, {term}`visualizers <Visualizer>`, {term}`formats <Format>`, or {term}`transformers <Transformer>`.
+
+Primitive Type
+  A {term}`type` that is used to communicate parameters to an {term}`interface`.  These are predefined by the {term}`framework` and cannot be extended.
+
+Result
+  A generic term for either a {term}`Visualization` or an {term}`Artifact`.
+
+Provenance
+  Data describing how an analysis was performed, captured automatically whenever users perform a QIIME 2 {term}`Action`.
+  Provenance information describes the host system, the computing environment, Actions performed, parameters passed, primary sources cited, and more.
+
+Semantic Type
+  A {term}`type` that is used to classify {term}`artifacts<Artifact>` and how they can be used.
+  These types may be extended by {term}`plugins<Plugin>`.
+
+Transformer
+  A function registered on the {term}`framework` capable of converting data in one {term}`format` into data of another {term}`format`.
+
+Type
+  A term that is used to represent several different ideas in QIIME 2, and which is therefore ambiguous when used on its own.
+  More specific terms are *file type*, *semantic type*, and *data type*. See [](types-of-types) for more information.
+
+UUID
+  Universally Unique IDentifier, in the context of QIIME 2, almost certainly refers to a *Version 4* UUID, which is a randomly generated ID.
+  See this [RFC](https://tools.ietf.org/html/rfc4122) or this [wikipedia entry](https://en.wikipedia.org/wiki/Universally_unique_identifier) for details.
+
+View
+  A particular representation of data. This includes on-disk formats and in-memory data structures (objects).
+
+Visualization
+  A QIIME 2 {term}`Result` that contains an interactive visualization.
+
+Visualization (Type)
+  The {term}`type` of a {term}`visualization`.
+  There are no subtyping relations between this type and any other (it is a singleton) and cannot be extended (because it is a singleton).
+
+Visualizer
+  A visualizer accepts some combination of QIIME 2 {term}`artifacts <Artifact>` and {term}`parameters <Parameter>` as {term}`input`, and produces exactly one {term}`visualization` as {term}`output`.
+```

book/framework/explanations/architecture.md

@@ -0,0 +1,104 @@
+(q2-architecture-overview)=
+# QIIME 2 architecture overview
+
+The goal of this document is to give the reader a high-level understanding of the components of QIIME 2, and how they are inter-related.
+
+
+At the highest level, there are three kinds of components in QIIME 2:
+
+  - The interfaces, which are responsible for translating user intent into action.
+  - The framework, whose behavior and purpose will be described in further detail below.
+  - The plugins, which define *all* domain-specific functionality.
+
+```{figure} ../images/simple_component_diagram.svg
+:name: Box and Arrow diagram of QIIME 2 components
+
+**Box and Arrow diagram of QIIME 2 components.**
+Interfaces only interact with plugins through the framework, which will invoke plugin behavior as needed.
+Solid arrows are direct dependency.
+Dash-dotted arrows are a deferred dependency (via entry-point).
+```
+
+The above diagram illustrates the most important restriction of the architecture.
+Interfaces cannot and *should not* have any particular knowledge about plugins ahead of time.
+Instead they must request that information from the framework, which provides a high-level description of all of the actions available via SDK (Software Development Kit) objects.
+
+At first glance, this restriction may seem onerous.
+However, because interfaces cannot communicate directly with plugins, they also never need to coordinate with them.
+This means interfaces are entirely decoupled from the plugins and more importantly, plugins are always decoupled from interfaces.
+A developer of a plugin, does not need to concern themselves with providing any interface-specific functionality, meaning development of both plugins and interfaces can be done in parallel.
+Only changes in the framework itself require coordination between the other two component types.
+This key constraint coupled with a set of semantically rich SDK objects allows *multiple* kinds of interfaces to be dynamically generated.
+This allows QIIME 2 to adapt its UI to both the audience and the task at hand.
+
+## Detailed Component Diagram
+A more complete version of the above figure is found below:
+
+```{figure} ../images/complex_component_diagram.svg
+:name: Detailed Box and Arrow diagram of QIIME 2 components
+
+**Detailed Box and Arrow diagram of QIIME 2 components.**
+Solid arrows are a direct dependency.
+Dash-dotted arrows are a deferred dependency (via entry-point).
+Dashed rounded boxes surrounding other components indicate a group of like-components.
+The larger gray box indicates a nested component, containing sub-components.
+Text within angle-brackets (`<>`) indicate a Python package/import name.
+```
+
+Here we observe that interfaces use a particular sub-component of the framework called the SDK.
+We also see that one of the interfaces is built into the framework itself (the Artifact API), however it is not any more privileged compared to any of the other interfaces, and none of the other interfaces use it directly.
+
+Looking now at the plugins we see that they use a sub-component of the framework called the Plugin API.
+This is responsible for constructing and registering the relevant SDK objects for use by interfaces.
+We also see that plugins can depend on other plugins.
+
+At this point the rough picture of how an interface uses a plugin can be seen.
+Plugins are loaded by the framework's SDK via an entry-point (more on that later).
+This in turn causes the plugin code to interact with the Plugin API, which constructs SDK objects.
+These SDK objects are then introspected and manipulated by any number of Interfaces.
+
+## Following A Command Through QIIME 2
+
+To get a better idea of where the responsibility of these components starts and ends, we can look at a sequence diagram describing the execution of an action by a user.
+
+```{figure} ../images/action_call_sequence_diagram.svg
+:name: UML Sequence Diagram of an action-call in QIIME 2
+
+**UML Sequence Diagram of an action-call in QIIME 2**
+This diagram is read from top to bottom, which indicates the passage of some non-specific amount of time.
+Components are vertical columns.
+An activated state of a component is indicated by a narrow box.
+Components can perform actions either upon other components, or upon themselves.
+These actions are denoted with a solid arrow pointing at the actor in question.
+The label indicates what action is performed and, when provided, parenthesis indicate some kind of argument that is provided.
+Not all arguments are enumerated for brevity.
+Results of an action are denoted with a dashed arrow and are usually labeled with the result's name.
+```
+
+This figure has four components: a User, an Interface, the Framework, and a Plugin.
+We see first, a User invoking some action with some files and parameters.
+The Interface receives this and is activated.
+It locates the plugin and the action requested from the Framework, receiving SDK objects (not shown).
+Then it loads the provided files as QIIME 2 Artifacts.
+It is then ready to call the action (an SDK object) with the User's artifacts and parameters.
+
+The Framework then provides some input validation (it is much faster to check that the data provided will work for the action requested than to fail halfway through a very long process, requiring the User to start over).
+The Framework then identifies what format the data should be in, and invokes relevant code defined by a Plugin (though not necessarily the same one) for converting that data.
+Finally, with validated input and a compatible format, the data is provided to the plugin to perform whatever task the User intended.
+
+Once finished, the Plugin returns the results and the Framework will again convert that data into a particular format for storage using a Plugin.
+The Framework then writes that data into an archive (as `/data/`) and records what steps just occurred (in `/provenance/`).
+This completed archive is now an artifact and is returned to the Interface.
+The Interface decides to save the artifact to a file and then returns that to the User.
+
+## Summary
+
+In this example we see that the activation of each component is strictly nested.
+It forms a sort of "onion of responsibility" between the component layers.
+We also note that the Interface waits for the task to finish before becoming inactive; there are other modes of calling actions which are asynchronous and can be used instead.
+In either case, we see that each component is successively responsible for less tasks which become more specific as we move to the right.
+
+The end result is:
+  - The Interface need only care about communicating with the User.
+  - The Plugin need only care about manipulating data to some effect.
+  - The Framework concerns itself with coordinating the overall effort and recording the data surrounding the action.