Add tutorials

- Add the new tutorial pages along with supporting images.
- Update the reference to demo videos

docs/redirects.yml

@@ -39,3 +39,7 @@
 - type: page
   from_url: /analysis/setup.html
   to_url: /tutorial/setup.html
+
+- type: page
+  from_url: /videos.html
+  to_url: /tutorial/videos.html

docs/src/index.rst

@@ -11,7 +11,22 @@ If something in this documentation is broken or unclear, please `open an issue <
 If you have a specific question, post a note over at the `discussion board <https://discussion.nextstrain.org/>`_ -- we're happy to help!
 
 .. toctree::
-   :maxdepth: 2
+    :maxdepth: 1
+    :titlesonly:
+    :caption: Tutorials
+    :hidden:
+
+    tutorial/intro
+    tutorial/setup
+    tutorial/example-data
+    tutorial/custom-data
+    tutorial/genomic-surveillance
+    tutorial/next-steps
+    tutorial/videos
+
+
+.. toctree::
+   :maxdepth: 1
    :titlesonly:
    :caption: Table of contents
 

docs/src/tutorial/custom-data.rst

@@ -0,0 +1,175 @@
+Run using custom data
+=====================
+
+This tutorial builds on the previous tutorial. Here, we will walk through how to supply your own genomic data and analyze this with the example reference data, which will serve as background context for the new data. We will explain how to get the new data from GISAID, but you could replace this data with your own private sequences if needed.
+
+.. contents:: Table of Contents
+   :local:
+
+Prerequisites
+-------------
+
+1. :doc:`example-data`. This tutorial sets up the command line environment used in the following tutorial.
+2. You have a GISAID account. `Register <https://www.gisaid.org/registration/register/>`__ if you do not have an account yet. However, registration may take a few days. Follow :doc:`alternative data preparation methods <../reference/data-prep/index>` in place of **Curate data from GISAID** if you wish to continue the following tutorial in the meantime.
+
+Setup
+-----
+
+If you are not already there, change directory to the ``ncov`` directory:
+
+   .. code:: text
+
+      cd ncov
+
+and activate the ``nextstrain`` conda environment:
+
+   .. code:: text
+
+      conda activate nextstrain
+
+Curate data from GISAID
+-----------------------
+
+We will retrieve 10 sequences from GISAID's EpiCoV database.
+
+1. Navigate to `GISAID <https://www.gisaid.org/>`__ and select **Login**.
+
+   .. image:: ../images/gisaid-homepage.png
+      :width: 400
+      :alt: GISAID login link
+
+2. Login to your GISAID account.
+
+   .. image:: ../images/gisaid-login.png
+      :width: 200
+      :alt: GISAID login
+
+3. In the top left navigation bar, select **EpiCoV** then **Search**.
+
+   .. image:: ../images/gisaid-epicov-search.png
+      :width: 400
+      :alt: GISAID EpiCoV Search
+
+4. Filter to sequences that pass the following criteria:
+
+   1. Has a complete genome
+   2. Has high coverage
+   3. Has an exact collection date
+
+   .. image:: ../images/gisaid-select-sequences-10-highlighted.png
+      :width: 700
+      :alt: GISAID EpiCoV select first 10 sequences
+
+5. Select the first 10 sequences.
+
+6. Select **Download** in the bottom right of the search results.
+7. Select **Input for the Augur pipeline** as the download format.
+
+   .. image:: ../images/gisaid-augur-pipeline-download.png
+      :width: 400
+      :alt: GISAID EpiCoV download as Input for the Augur pipeline
+
+   .. note::
+
+      You may see different download options, but it is fine as long as **Input for the Augur pipeline** is available.
+
+8. Select **Download**.
+9. Download/move the ``.tar`` file into the ``ncov/data/`` directory.
+10. Extract by opening the downloaded ``.tar`` file in your file explorer. It contains a folder prefixed with ``gisaid_auspice_input_hcov-19_`` containing two files: one ending with ``.metadata.tsv`` and another with ``.sequences.fasta``.
+11. Rename the files as ``custom.metadata.tsv`` and ``custom.sequences.fasta``.
+12. Move the files up to the ``ncov/data/`` directory.
+13. Delete the empty ``gisaid_auspice_input_hcov-19_``-prefixed folder and the ``.tar`` file if it is still there.
+
+.. hint::
+
+   Read :doc:`the full data prep guide <../reference/data-prep/index>` for other ways to curate custom data.
+
+Run the workflow
+----------------
+
+From within the ``ncov/`` directory, run the ``ncov`` workflow using a pre-written ``--configfile``:
+
+.. code:: text
+
+   nextstrain build . --cores all --configfile ncov-tutorial/custom-data.yaml
+
+Break down the command
+~~~~~~~~~~~~~~~~~~~~~~
+
+The workflow can take several minutes to run. While it is running, you can investigate the contents of ``custom-data.yaml`` (comments excluded):
+
+.. code-block:: yaml
+
+   inputs:
+     - name: reference_data
+       metadata: https://data.nextstrain.org/files/ncov/open/reference/metadata.tsv.xz
+       sequences: https://data.nextstrain.org/files/ncov/open/reference/sequences.fasta.xz
+     - name: custom_data
+       metadata: data/custom.metadata.tsv
+       sequences: data/custom.sequences.fasta
+
+   refine:
+     root: "Wuhan-Hu-1/2019"
+
+   builds:
+     custom-build:
+       title: "Build with custom data and example data"
+       subsampling_scheme: all
+       auspice_config: ncov-tutorial/auspice-config-custom-data.json
+
+This is the same as the previous file, with some additions:
+
+1. A second input for the custom data, referencing the metadata and sequences files downloaded from GISAID.
+2. A ``builds`` section that defines one output :term:`docs.nextstrain.org:dataset` using:
+
+   1. A custom name ``custom-build``, which will be used to create the dataset filename, in this case ``auspice/ncov_custom-build.json``.
+   2. A custom title ``Build with custom data and example data``, which will be shown when you visualize the dataset in Auspice.
+   3. :ref:`A pre-defined subsampling scheme <configuration-subsampling>` ``all``, which tells the workflow to skip subsampling.
+   4. An Auspice config file, ``ncov-tutorial/auspice-config-custom-data.json``, which defines various parameters for how the data should be visualized in Auspice. It has the following contents:
+
+      .. code-block:: json
+
+         {
+           "colorings": [
+             {
+               "key": "custom_data",
+               "title": "Custom data",
+               "type": "categorical"
+             }
+           ],
+           "display_defaults": {
+             "color_by": "custom_data"
+           }
+         }
+
+      This JSON does two things:
+
+      1. Create a new coloring ``custom_data`` which reflects a special metadata column generated by the ncov workflow. When there is more than one input, each data input produces a new final metadata column with categorical values ``yes`` or ``no`` representing whether the sequence was from the input.
+      2. Set the default Color By as the new ``custom_data`` coloring.
+
+   .. note ::
+
+      **Build** is a widely used term with various meanings. In the context of the ncov workflow, the ``builds:`` section defines output :term:`datasets <docs.nextstrain.org:dataset>` to be generated by the workflow (i.e. "build" a dataset).
+
+Visualize the results
+---------------------
+
+Run this command to start the :term:`docs.nextstrain.org:Auspice` server, providing ``auspice/`` as the directory containing output dataset files:
+
+.. code:: text
+
+   nextstrain view auspice/
+
+Navigate to http://127.0.0.1:4000/ncov/custom-build. The resulting :term:`docs.nextstrain.org:dataset` should have similar phylogeny to the previous dataset, with additional sequences:
+
+.. figure:: ../images/dataset-custom-data-highlighted.png
+   :alt: Phylogenetic tree from the "custom data" tutorial as visualized in Auspice
+
+
+1. The custom dataset name ``custom-build`` can be seen in the dataset selector, as well as the dataset URL.
+2. The custom dataset title can be seen at the top of the page.
+3. The custom coloring is used by default. You can see which sequences are from the custom data added in this tutorial.
+
+   .. note::
+
+      You may not see all 10 custom sequences - some can be filtered out due to quality checks built into the ncov workflow.

docs/src/tutorial/example-data.rst

@@ -0,0 +1,99 @@
+Run using example data
+======================
+
+The aim of this first tutorial is to introduce our SARS-CoV-2 workflow.
+To do this, we will run the workflow using a small set of reference data which we provide.
+This tutorial leads on to subsequent tutorials where we will walk through more complex scenarios.
+
+.. contents:: Table of Contents
+   :local:
+
+Prerequisites
+-------------
+
+1. :doc:`setup`. These instructions will install all of the software you need to complete this tutorial and others.
+
+Setup
+-----
+
+1. Activate the ``nextstrain`` conda environment:
+
+   .. code:: text
+
+      conda activate nextstrain
+
+2. Change directory to the ``ncov`` directory:
+
+   .. code:: text
+
+      cd ncov
+
+3. Download the example tutorial repository into a new directory ``ncov/ncov-tutorial/``:
+
+   .. code:: text
+
+      git clone https://github.com/nextstrain/ncov-tutorial
+
+Run the workflow
+----------------
+
+From within the ``ncov/`` directory, run the ``ncov`` workflow using a configuration file provided in the tutorial directory:
+
+.. code:: text
+
+   nextstrain build . --cores all --configfile ncov-tutorial/example-data.yaml
+
+Break down the command
+~~~~~~~~~~~~~~~~~~~~~~
+
+The workflow can take several minutes to run. While it is running, you can learn about the parts of this command:
+
+- ``nextstrain build .``
+   - This tells the :term:`docs.nextstrain.org:Nextstrain CLI` to :term:`build <docs.nextstrain.org:build (verb)>` the workflow from ``.``, the current directory. All subsequent command-line parameters are passed to the workflow manager, Snakemake.
+- ``--cores all``
+   - This required Snakemake parameter specifies the number of CPU cores to use (:doc:`more info <snakemake:executing/cli>`).
+- ``--configfile ncov-tutorial/example-data.yaml``
+   - ``--configfile`` is another Snakemake parameter used to configure the ncov workflow.
+   - ``ncov-tutorial/example-data.yaml`` is a YAML file which provides custom workflow configuration including inputs and outputs. Contents with comments excluded:
+
+      .. code-block:: yaml
+
+         inputs:
+           - name: reference_data
+             metadata: https://data.nextstrain.org/files/ncov/open/reference/metadata.tsv.xz
+             sequences: https://data.nextstrain.org/files/ncov/open/reference/sequences.fasta.xz
+
+         refine:
+           root: "Wuhan-Hu-1/2019"
+
+      This provides the workflow with one input named ``reference_data``, which is a small dataset maintained by the Nextstrain team. The metadata and sequences files are downloaded directly from the associated URLs. :doc:`See the complete list of SARS-CoV-2 datasets we provide through data.nextstrain.org <../reference/remote_inputs>`.
+
+      The ``refine`` entry specifies the root sequence for the example GenBank data.
+
+      For more information, see :doc:`../reference/configuration-reference`.
+
+The workflow output produces a new directory ``auspice/`` containing a file ``ncov_default-build.json``, which will be visualized in the following section. The workflow also produces intermediate files in a new ``results/`` directory.
+
+Visualize the results
+---------------------
+
+Run this command to start the :term:`docs.nextstrain.org:Auspice` server, providing ``auspice/`` as the directory containing output dataset files:
+
+.. code:: text
+
+   nextstrain view auspice/
+
+Navigate to http://127.0.0.1:4000/ncov/default-build. The resulting :term:`docs.nextstrain.org:dataset` should show a phylogeny of ~200 sequences:
+
+.. figure:: ../images/dataset-example-data.png
+   :alt: Phylogenetic tree from the "example data" tutorial as visualized in Auspice
+
+To stop the server, press :kbd:`Control-C` on your keyboard.
+
+.. note::
+
+   You can also view the results by dragging the dataset files all at once onto `auspice.us <https://auspice.us>`__:
+
+   - ``auspice/ncov_default-build.json``
+   - ``auspice/ncov_default-build_root-sequence.json``
+   - ``auspice/ncov_default-build_tip-frequencies.json``