port content from dev.qiime2.org and glossary of q2book

* SQUASH: Add built book to ignore

* SQUASH: Remove some placeholder stuff

* SQUASH: Add tutorial

* SQUASH: Add usage

* SQUASH: Add cache

* SQUASH: Update toc

* SQUASH: Add sphinx extensions to config

* SQUASH: Flesh out usage and cache with old docs

* SQUASH: Forgot about Pool

* SQUASH: Remove redundant #

* SQUASH: Add glossary

* SQUASH: Start writing type-mapping docs

* SQUASH: Todos

* Typo

* Add some placeholder text

* At least for now q2doc doesn't seem to be needed

* remove broken links

* md to rst

---------

Co-authored-by: Greg Caporaso <[email protected]>
Co-authored-by: Greg Caporaso <[email protected]>

.gitignore

@@ -69,3 +69,6 @@ target/
 
 #Editor file
 .vscode
+
+#Jupyter book
+book/_build

book/10-tutorial/10-plugins/10-plugin-setup.rst

@@ -0,0 +1,35 @@
+Instantiating the `qiime2.plugin.Plugin` object
+###############################################
+
+The next step is to instantiate a QIIME 2 ``Plugin`` object. This might look like the following:
+
+.. code-block:: python
+
+   from qiime2.plugin import Plugin
+   import q2_diversity
+
+   plugin = Plugin(
+       name='diversity',
+       version=q2_diversity.__version__,
+       website='https://qiime2.org',
+       user_support_text='https://forum.qiime2.org',
+       package='q2_diversity'
+   )
+
+This will provide QIIME with essential information about your ``Plugin``.
+
+The ``name`` parameter is the name that users will use to access your plugin from within different QIIME 2 interfaces. It should be a "command line friendly" name, so should not contain spaces or punctuation. (Avoiding uppercase characters and using dashes (``-``) instead of underscores (``_``) are preferable in the plugin ``name``, but not required).
+
+``version`` should be the version number of your package (the same that is used in its ``setup.py``).
+
+``website`` should be the page where you'd like end users to refer for more information about your package. Since ``q2-diversity`` doesn't have its own website, we're including the QIIME 2 website here.
+
+``package`` should be the Python package name for your plugin.
+
+While not shown in the previous example, plugin developers can optionally provide the following parameters to ``qiime2.plugin.Plugin``:
+
+* ``citation_text``: free text describing how users should cite the plugin and/or the underlying tools it wraps. If not provided, users are told to cite the ``website``.
+
+* ``user_support_text``: free text describing how users should get help with the plugin (e.g. issue tracker, StackOverflow tag, mailing list, etc.). If not provided, users are referred to the ``website`` for support. ``q2-diversity`` is supported on the QIIME 2 Forum, so we include that URL here. We encourage plugin developers to support their plugins on the QIIME 2 Forum, so you can include that URL as the ``user_support_text`` for your plugin. If you do that, you should get in the habit of monitoring the QIIME 2 Forum for technical support questions.
+
+The ``Plugin`` object can live anywhere in your project, but by convention it will be in a file called ``plugin_setup.py``. For an example, see ``q2_diversity/plugin_setup.py``.

book/10-tutorial/10-plugins/20-registering-actions.rst

@@ -0,0 +1,95 @@
+Registering an Action
+#####################
+
+Once you have functions that you'd like to register as ``Actions`` (i.e., either ``Methods`` or ``Visualizers``), and you've instantiated your ``Plugin`` object, you are ready to register those functions. This will likely be done in the file where the ``Plugin`` object was instantiated, as it will use that instance (which will be referred to as ``plugin`` in the following examples).
+
+Registering a Method
+++++++++++++++++++++
+
+First we'll register a ``Method`` by calling ``plugin.methods.register_function`` as follows:
+
+.. code-block:: python
+
+   from q2_types import (FeatureTable, Frequency, Phylogeny,
+                         Rooted, DistanceMatrix)
+   from qiime2.plugin import Str, Choices, Properties, Metadata
+
+   import q2_diversity
+   import q2_diversity._beta as beta
+
+   plugin.methods.register_function(
+       function=q2_diversity.beta_phylogenetic,
+       inputs={'table': FeatureTable[Frequency],
+               'phylogeny': Phylogeny[Rooted]},
+       parameters={'metric': Str % Choices(beta.phylogenetic_metrics())},
+       outputs=[('distance_matrix', DistanceMatrix % Properties('phylogenetic'))],
+       input_descriptions={
+           'table': ('The feature table containing the samples over which beta '
+                     'diversity should be computed.'),
+           'phylogeny': ('Phylogenetic tree containing tip identifiers that '
+                         'correspond to the feature identifiers in the table. '
+                         'This tree can contain tip ids that are not present in '
+                         'the table, but all feature ids in the table must be '
+                         'present in this tree.')
+       },
+       parameter_descriptions={
+           'metric': 'The beta diversity metric to be computed.'
+       },
+       output_descriptions={'distance_matrix': 'The resulting distance matrix.'},
+       name='Beta diversity (phylogenetic)',
+       description=("Computes a user-specified phylogenetic beta diversity metric"
+                    " for all pairs of samples in a feature table.")
+   )
+
+The values being provided are:
+
+``function``: The function to be registered as a method.
+
+``inputs``: A dictionary indicating the parameter name and its *semantic type*, for each input ``Artifact``. These semantic types differ from the data types that you provided in your `mypy`_ annotation of the input, as semantic types describe the data, where the data types indicate the structure of the data. The currently available semantic types can be viewed by running ``qiime tools import --show-importable-types`` (until merge of `this PR <https://github.com/qiime2/q2cli/pull/291>`_, which will improve on how this information is accessed from the command line). In the example above we're indicating that the ``table`` parameter must be a ``FeatureTable`` of ``Frequency`` (i.e. counts), and that the ``phylogeny`` parameter must be a ``Phylogeny`` that is ``Rooted``.  Notice that the keys in ``inputs`` map directly to the parameter names in ``q2_diversity.beta_phylogenetic``.
+
+``parameters``: A dictionary indicating the parameter name and its *semantic type*, for each input ``Parameter``. These parameters are primitive values (i.e., non-``Artifacts``). In the example above, we're indicating that the ``metric`` should be a string from a specific set (in this case, the set of known phylogenetic beta diversity metrics).
+
+``outputs``: A list of tuples indicating each output name and its semantic type.
+
+``input_descriptions``: A dictionary containing input artifact names and their corresponding descriptions. This information is used by interfaces to instruct users how to use each specific input artifact.
+
+``parameter_descriptions``: A dictionary containing parameter names and their corresponding descriptions. This information is used by interfaces to instruct users how to use each specific input parameter. You should not include any default parameter values in these descriptions, as these will generally be added automatically by an interface.
+
+``output_descriptions``: A dictionary containing output artifact names and their corresponding descriptions. This information is used by interfaces to inform users what each specific output artifact will be.
+
+``name``: A human-readable name for the ``Method``. This may be presented to users in interfaces.
+
+``description``: A human-readable description of the ``Method``. This may be presented to users in interfaces.
+
+Registering a Visualizer
+++++++++++++++++++++++++
+
+Registering ``Visualizers`` is the same as registering ``Methods``, with two exceptions.
+
+First, you call ``plugin.visualizers.register_function`` to register a ``Visualizer``.
+
+Next, you do not provide ``outputs`` or ``output_descriptions`` when making this call, as ``Visualizers``, by definition, only return a single visualization. Since the visualization output path is a required parameter, you do not include this in an ``outputs`` list (it would be the same for every ``Visualizer`` that was ever registered, so it is added automatically).
+
+Registering ``q2_diversity.alpha_group_significance`` as a ``Visualizer`` looks like the following:
+
+.. code-block:: python
+
+   plugin.visualizers.register_function(
+       function=q2_diversity.alpha_group_significance,
+       inputs={'alpha_diversity': SampleData[AlphaDiversity]},
+       parameters={'metadata': Metadata},
+       input_descriptions={
+           'alpha_diversity': 'Vector of alpha diversity values by sample.'
+       },
+       parameter_descriptions={
+           'metadata': 'The sample metadata.'
+       },
+       name='Alpha diversity comparisons',
+       description=("Visually and statistically compare groups of alpha diversity"
+                    " values.")
+   )
+
+Registering a Pipeline
+++++++++++++++++++++++
+
+TODO: put a pipeline registration here

book/20-understanding/10-plugins/10-plugin.md

@@ -1 +1,3 @@
-# The `qiime2.plugin.Plugin` object
+# The `qiime2.plugin.Plugin` object
+
+## TODO: Write dev-docs on this object

book/20-understanding/20-interfaces/10-plugin-manager.md

@@ -1 +1,3 @@
-# # The `qiime2.sdk.PluginManager` object
+# The `qiime2.sdk.PluginManager` object
+
+## TODO: Write dev-docs on this object

book/30-how-to/10-plugins/10-type-mapping.rst

@@ -0,0 +1,49 @@
+Mapping input semantic types to output semantic types
+#####################################################
+
+We use the ``TypeMatch`` object to map input types to output types.
+
+.. code-block:: python
+
+    # Beginning of setup
+    from qiime2.plugin import Plugin, TypeMatch
+    from qiime2.core.testing.type import IntSequence1, IntSequence2
+    from qiime2.core.testing.method import split_ints
+    from qiime2.plugin import Citations
+
+    citations = Citations.load('citations.bib', package='qiime2.core.testing')
+    dummy_plugin = Plugin(
+        name='dummy-plugin',
+        description='Description of dummy plugin.',
+        short_description='Dummy plugin for testing.',
+        version='0.0.0-dev',
+        website='https://github.com/qiime2/qiime2',
+        package='qiime2.core.testing',
+        user_support_text='For help, see https://qiime2.org',
+        citations=[citations['unger1998does'], citations['berry1997flying']]
+    )
+    # End of setup
+
+    T = TypeMatch([IntSequence1, IntSequence2])
+    dummy_plugin.methods.register_function(
+        function=split_ints,
+        inputs={
+            'ints': T
+        },
+        parameters={},
+        outputs={
+            'left': T,
+            'right': T
+        },
+        name='Split sequence of integers in half',
+        description='This method splits a sequence of integers in half, returning '
+                    'the two halves (left and right). If the input sequence\'s '
+                    'length is not evenly divisible by 2, the right half will '
+                    'have one more element than the left.',
+        citations=[
+            citations['witcombe2006sword'], citations['reimers2012response']]
+    )
+
+Create a ``TypeMatch`` object taking a list of the different types you intend to match. By convention, we name this variable T.
+
+Use this object as the type of the inputs and outputs you want to match. In the above example, `ints` can take an IntSequence1 or an IntSequence2. `left` and `right` will both be the same type as `ints`.

book/40-reference/00-intro.md

@@ -1 +1,3 @@
-# Introduction
+# Introduction
+
+This portion of the docs will serve as a quick reference for various QIIME 2 terms and APIs.

book/40-reference/10-api/00-intro.md

@@ -1 +1,3 @@
-# QIIME 2 API reference
+# QIIME 2 API reference
+
+Here you will learn the intended methods of interacting with various parts of the guts of QIIME 2.

book/40-reference/10-api/10-usage/00-index.rst

@@ -0,0 +1,4 @@
+Usage API
+=========
+
+.. automodule:: qiime2.sdk.usage

book/40-reference/10-api/10-usage/10-authors.rst

@@ -0,0 +1,78 @@
+Usage API (for example authors)
+===============================
+.. contents::
+   :local:
+
+This page outlines elements of the Usage API which are used by example authors
+(and overriden by interface drivers) to describe some example situation in the
+framework for documentation, testing, or interface generating purposes.
+
+
+.. currentmodule:: qiime2.sdk.usage
+
+Initializers
+------------
+These methods prepare some data for use in an example.
+
+.. automethod:: Usage.init_artifact
+.. automethod:: Usage.init_metadata
+.. automethod:: Usage.init_format
+
+Importing
+---------
+These methods demonstrate how to import an artifact.
+
+.. automethod:: Usage.import_from_format
+
+Metadata
+--------
+These methods demonstrate how to manipulate metadata.
+
+.. automethod:: Usage.get_metadata_column
+.. automethod:: Usage.view_as_metadata
+.. automethod:: Usage.merge_metadata
+
+.. _usage-annotations:
+
+Annotations
+-----------
+These methods do not return anything, but may be displayed in other ways.
+
+.. automethod:: Usage.comment
+.. automethod:: Usage.help
+.. automethod:: Usage.peek
+
+
+Actions
+-------
+These methods invoke a plugin's action.
+
+.. automethod:: Usage.action
+
+Parameter Objects for :meth:`Usage.action`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+These three classes define a deferred action that should be taken by some
+interface driver.
+
+.. autoclass:: UsageAction
+   :special-members: __init__
+
+.. autoclass:: UsageInputs
+   :special-members: __init__
+
+.. autoclass:: UsageOutputNames
+   :special-members: __init__
+
+
+.. _results-and-assertions:
+
+Results and Assertions
+~~~~~~~~~~~~~~~~~~~~~~
+The outputs of :meth:`qiime2.sdk.usage.Usage.action` are stored in a vanity
+class :class:`qiime2.sdk.usage.UsageOutputs` which contain
+:class:`qiime2.sdk.usage.UsageVariable`'s. Assertions are performed on these
+output variables.
+
+.. autoclass:: UsageOutputs
+.. autoclass:: UsageVariable
+   :members: assert_has_line_matching, assert_output_type

book/40-reference/10-api/10-usage/20-drivers.rst

@@ -0,0 +1,54 @@
+Usage API (for interface drivers)
+=================================
+
+.. this unusual mechanism using automethod prevents double indexing the class
+   while still permitting references to the methods.
+
+.. autoclass:: qiime2.sdk.usage.Usage
+
+   .. automethod:: __init__
+   .. autoattribute:: asynchronous
+   .. autoattribute:: namespace
+   .. automethod:: usage_variable
+   .. automethod:: render
+
+.. autoclass:: qiime2.sdk.usage.UsageVariable
+   :noindex:
+
+   .. automethod:: __init__
+   .. autoattribute:: name
+   .. autoattribute:: var_type
+   .. autoattribute:: value
+   .. autoattribute:: factory
+   .. autoattribute:: use
+   .. autoproperty:: is_deferred
+   .. automethod:: to_interface_name
+   .. automethod:: execute
+   .. automethod:: save
+
+.. autoclass:: qiime2.sdk.usage.UsageAction
+   :noindex:
+
+   .. autoattribute:: plugin_id
+   .. autoattribute:: action_id
+   .. automethod:: get_action
+
+.. autoclass:: qiime2.sdk.usage.UsageInputs
+   :noindex:
+
+   .. automethod:: map_variables
+   .. automethod:: __getitem__
+   .. automethod:: __contains__
+   .. automethod:: keys
+   .. automethod:: values
+   .. automethod:: items
+
+.. autoclass:: qiime2.sdk.usage.UsageOutputNames
+   :noindex:
+
+   .. automethod:: __getitem__
+   .. automethod:: __contains__
+   .. automethod:: keys
+   .. automethod:: values
+   .. automethod:: items
+