Merge pull request #215 from CovertLab/documentation

Documentation

.github/workflows/docs_deploy.yml

@@ -12,18 +12,16 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v2
         with:
-          python-version: "3.10"
+          python-version: "3.11"
       - name: Install dependencies
         run: |
           sudo apt-get install pandoc
           python -m pip install --upgrade pip wheel
           pip install numpy Cython
-          pip install -r requirements.txt
-          python setup.py install
           pip install -r doc/requirements.txt
       - name: Compile Cython components
         run: |
-          make clean compile
+          make compile
       - name: Build documentation
         run: |
           cd doc

.github/workflows/docs_test.yml

@@ -14,18 +14,16 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v2
         with:
-          python-version: "3.10"
+          python-version: "3.11"
       - name: Install dependencies
         run: |
           sudo apt-get install pandoc
           python -m pip install --upgrade pip wheel
           pip install numpy Cython
-          pip install -r requirements.txt
-          python setup.py install
           pip install -r doc/requirements.txt
       - name: Compile Cython components
         run: |
-          make clean compile
+          make compile
       - name: Build documentation
         run: |
           cd doc

.github/workflows/pytest.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.11"]
     steps:
     - uses: actions/checkout@v2
     - name: Setup Python ${{ matrix.python-version }}

.github/workflows/pytest_master.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.10"]
+        python-version: ["3.11"]
     steps:
     - uses: actions/checkout@v2
     - name: Setup Python ${{ matrix.python-version }}

README.md

@@ -1,39 +1,37 @@
-# Vivarium-Ecoli
+# Vivarium *E. coli*
 
 ![vivarium](doc/_static/ecoli_master_topology.png)
 
 ## Background
 
-Vivarium-ecoli is a port of the Covert Lab's 
-[E. coli Whole Cell Model](https://github.com/CovertLab/wcEcoli) 
+Vivarium *E. coli* is a port of the Covert Lab's 
+[E. coli Whole Cell Model](https://github.com/CovertLab/wcEcoli) (wcEcoli)
 to the [Vivarium framework](https://github.com/vivarium-collective/vivarium-core).
 
-Below is a high-level overview of the advantages and WIP features.
+As in wcEcoli, raw experimental data (`reconstruction/ecoli/flat`) is first processed
+by the parameter calculator or Parca (`reconstruction/ecoli/fit_sim_data_1.py`) to calculate 
+model parameters (e.g. transcription probabilities).
 
-The scope of this project is to migrate the Whole Cell Model's processes, and therefore takes 
-wcEcoli's `sim_data` as its starting point in the simulation pipeline.
-`sim_data` is a large configuration object created by the parameter calculator (ParCa). 
-For this reason the `reconstruction/` and `wholecell/utils/` folders have been duplicated 
-here as they are necessary to unpickle the serialized `sim_data` object. If a new `sim_data` 
-object is required to be read, the corresponding wcEcoli folders will have to be synchronized.
+These parameters are used to configure processes (`ecoli/processes`) that are linked together
+into a complete simulation in accordance with the topology generated by
+`ecoli/experiments/ecoli_master_sim.py`.
 
-All state handling (previously handled by Bulk- and UniqueMolecules states/containers/views) 
-and the actual running of the simulation (previously `wholecell.sim.Simulation`) are now 
-handled entirely by Vivarium's core engine and process interface. 
+For more details, refer to the [user guide](https://covertlab.github.io/vivarium-ecoli/index.html).
 
-The new process classes can be found in `ecoli/processes/*` and are linked together using 
-a Vivarium topology that is generated in `ecoli/experiments/ecoli_master_sim.py`.
+## Installation
 
-## Pyenv Installation
+> **Note:** The following instructions assume a Linux or MacOS system. Windows users can
+> attempt to follow the same instructions after setting up 
+> [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).
 
 pyenv lets you install and switch between multiple Python releases and multiple "virtual 
 environments", each with its own pip packages. Using pyenv, create a virtual environment 
-and install Python 3.10.12. For a tutorial on how to create a virtual environment, follow 
+and install Python 3.11.3. For a tutorial on how to create a virtual environment, follow 
 the instructions [here](https://github.com/CovertLab/wcEcoli/blob/master/docs/create-pyenv.md) 
 and stop once you reach the "Create the 'wcEcoli3' python virtual environment" step. Then, 
 run the following command in your terminal:
 
-    $ pyenv virtualenv 3.10.12 viv-ecoli && pyenv local viv-ecoli
+    $ pyenv virtualenv 3.11.3 viv-ecoli && pyenv local viv-ecoli
 
 Update `pip`, `setuptools` and `wheel` to avoid issues with these:
 
@@ -51,6 +49,9 @@ And build the Cython components:
 
     $ make clean compile
 
+In order to save simulation data for downstream analyses, 
+[install and start MongoDB](https://www.mongodb.com/docs/manual/installation/).
+
 ## Running the simulation
 
 To run the simulation, set the `PYTHONPATH` environment variable to the cloned repository and run
@@ -62,7 +63,7 @@ To run the simulation, set the `PYTHONPATH` environment variable to the cloned r
 For details on configuring simulations through either the command-line interface or .json files, 
 see the [Ecoli-master configurations readme](readmes/ecoli_configurations.md).
 
-## wcEcoli Migration
+## Benefits of Vivarium
 
 The main motivation behind the migration is to modularize the processes and allow them to be run 
 in new contexts or reconfigured or included in different biological models. 
@@ -95,14 +96,20 @@ are synchronized between processes.
 
 ## Causality Network
 
-After running a simulation, you can explore the Causality visualization tool (see 
-[CovertLab/causality](https://github.com/CovertLab/causality)) to examine the model's causal links and 
-simulation output correlations.
+> **Note:** Must be a member of the `CovertLab` GitHub organization.
+
+[The Causality visualization tool](https://github.com/CovertLab/causality) can be used to manually examine the model's
+output (molecule counts, reaction fluxes, etc.). First, [run a simulation](#running-the-simulation) ensuring that data is set to be emitted to MongoDB (e.g. `emitter` is `database` in [configuration](readmes/ecoli_configurations.md)). Then, build the Causality network with the following command:
+
+```
+python ecoli/analysis/buildCausalityNetwork.py
+```
 
-## Current state
+Then, clone the Causality repository into the same folder as the vivarium-ecoli repository (e.g. `/dev/causality` and `/dev/vivarium-ecoli`) and follow the installation instructions.
 
-As of September 2021, 
-The sim_data is generated with wcEcoli branch [vivarium-ecoli-52021](https://github.com/CovertLab/wcEcoli/tree/vivarium-ecoli-52021)
+Finally, start the Causality server, which will open an interactive webpage in your web browser:
 
-All effort has been made to translate these processes as faithfully as possible. This means previous 
-behavior is intact, including the partitioning assumption from the original model. 
+```
+cd /dev/causality
+python site/server.py ../vivarium-ecoli/out/seriesOut/
+```

doc/apidoc_templates/module.rst_t

@@ -0,0 +1,5 @@
+{{- "``{}``".format(basename) | heading }}
+.. automodule:: {{ qualname }}
+{%- for option in automodule_options %}
+   :{{ option }}:
+{%- endfor %}

doc/apidoc_templates/package.rst_t

@@ -0,0 +1,49 @@
+{%- macro automodule(modname, options) -%}
+.. automodule:: {{ modname }}
+{%- for option in options %}
+   :{{ option }}:
+{%- endfor %}
+{%- endmacro %}
+
+{%- macro toctree(docnames) -%}
+.. toctree::
+   :titlesonly:
+   :maxdepth: {{ maxdepth }}
+{% for docname in docnames %}
+   {{ docname }}
+{%- endfor %}
+{%- endmacro %}
+
+{{- "``{}``".format(pkgname) | heading }}
+
+{%- if is_namespace %}
+.. py:module:: {{ pkgname }}
+{% endif %}
+
+{%- if modulefirst and not is_namespace %}
+{{ automodule(pkgname, automodule_options) }}
+{% endif %}
+
+{%- if subpackages %}
+{{ toctree(subpackages) }}
+{% endif %}
+
+{%- if submodules %}
+{% if separatemodules %}
+{{ toctree(submodules) }}
+{% else %}
+{%- for submodule in submodules %}
+{% if show_headings %}
+{{- "``{}``".format(submodule) | heading(2) }}
+{% endif %}
+{{ automodule(submodule, automodule_options) }}
+{% endfor %}
+{%- endif %}
+{%- endif %}
+
+{%- if not modulefirst and not is_namespace %}
+Module contents
+---------------
+
+{{ automodule(pkgname, automodule_options) }}
+{% endif %}

doc/apidoc_templates/toc.rst_t

@@ -0,0 +1,8 @@
+{{ header | heading }}
+
+.. toctree::
+   :maxdepth: {{ maxdepth }}
+{% for docname in docnames %}
+   {{ docname }}
+{%- endfor %}
+

doc/conf.py

@@ -93,20 +93,29 @@
         'https://vivarium-core.readthedocs.io/en/latest/',
         None,
     ),
+    'numpy': ('https://numpy.org/doc/stable', None),
+    'unum': ('https://unum.readthedocs.io/en/latest/', None)
 }
 
 
 # -- sphinx.ext.autodoc options --
 autodoc_inherit_docstrings = False
 # The Python dependencies aren't really required for building the docs
 autodoc_mock_imports = [
-    'cobra', 'arrow', 'IPython', 'numba', 'line-profiler', 'iteround',
-    'pytest',
+    'stochastic_arrow', 'numba', 'line-profiler', 'iteround', 'aesara', 'pandas',
     # Runs code on import and fails due to missing solvers.
     'wholecell.utils.modular_fba',
+    # Runs code on import and fails due to missing packages
+    'ecoli.library.parameters',
+    'sympy', 'cv2', 'Bio', 'tqdm', 'cvxpy', 'pymunk', 'skimage', 'dill',
 ]
+# Move typehints from signature into description
+autodoc_typehints = "description"
 # Concatenate class and __init__ docstrings
 autoclass_content = 'both'
+# Remove domain objects (e.g. functions, classes, attributes) from 
+# table of contents
+toc_object_entries = False
 
 def autodoc_skip_member_handler(app, what, name, obj, skip, options):
     if name.startswith('test_'):
@@ -134,16 +143,16 @@ def run_apidoc(_):
     exclude = (
         os.path.join(cur_dir, path) for path in (
             '../ecoli/analysis',
-            '../ecoli/library',
-            '../ecoli/models',
             '../ecoli/plots',
-            '../ecoli/processes/registries.py',
-            '../ecoli/states',
             '../ecoli/experiments/ecoli_master_sim_tests.py',
         )
     )
-    apidoc.main(
-        ['-f', '-e', '-E', '-o', apidoc_dir, module_path, *exclude])
+
+    # Custom templates to only put top-level document titles in 
+    # table of contents
+    template_dir = 'apidoc_templates/'
+    apidoc.main(['-t', template_dir,
+        '-f', '-e', '-E', '-M', '-o', apidoc_dir, module_path, *exclude])
 
 
 objects_to_pprint = {}

doc/index.rst

@@ -1,7 +1,15 @@
 Welcome to Vivarium *E. coli*'s documentation!
 ==============================================
 
-Vivarium *E. coli* project is a port of the `Covert Lab's *E. coli* Whole Cell Model <https://doi.org/10.1128/ecosalplus.ESP-0001-2020>`_ to the `Vivarium framework <https://vivarium-collective.github.io>`_.
+Vivarium *E. coli* project is a port of the |text|_ to the `Vivarium framework <https://vivarium-collective.github.io>`_.
+
+..
+   Comment: We need to use text substitution because ReST does not
+   support nesting italics and hyperlinking
+
+.. _text: https://doi.org/10.1128/ecosalplus.ESP-0001-2020
+
+.. |text| replace:: Covert Lab's *E. coli* Whole Cell Model
 
 .. image:: ./_static/ecoli_master_topology.png
     :width: 100%
@@ -18,9 +26,8 @@ Vivarium *E. coli* project is a port of the `Covert Lab's *E. coli* Whole Cell M
    Experiments <reference/experiments>
    tutorials
    docs
-   reference/index
-
-.. note:: Vivarium *E. coli* has not yet been publicly released.
+   reference/stores
+   API Reference <reference/api/modules>
 
 .. WARNING::
    This documentation is very much a work in progress. It likely

doc/reference/experiments.rst

@@ -5,6 +5,6 @@ Experiments
 .. toctree::
    :maxdepth: 4
 
-   api/ecoli.experiments.colony
-   api/ecoli.experiments.comparison
    api/ecoli.experiments.ecoli_master_sim
+   api/ecoli.experiments.metabolism_redux_sim
+   api/ecoli.experiments.tet_amp_sim

doc/reference/processes.rst

@@ -2,34 +2,8 @@
 Processes
 =========
 
-.. toctree::
-   :maxdepth: 4
+All processes are loctated in the folder :doc:`ecoli/processes <api/ecoli.processes>`.
+Refer to the `this page <https://vivarium-core.readthedocs.io/en/latest/guides/processes.html>`_
+in the `vivarium-core` documentation for further information about processes in general.
+
 
-   api/ecoli.processes.allocator
-   api/ecoli.processes.cell_division
-   api/ecoli.processes.chromosome_replication
-   api/ecoli.processes.chromosome_structure
-   api/ecoli.processes.complexation
-   api/ecoli.processes.equilibrium
-   api/ecoli.processes.mass
-   api/ecoli.processes.metabolism
-   api/ecoli.processes.metabolism_gd
-   api/ecoli.processes.partition
-   api/ecoli.processes.polypeptide_elongation
-   api/ecoli.processes.polypeptide_initiation
-   api/ecoli.processes.protein_degradation
-   api/ecoli.processes.rna_degradation
-   api/ecoli.processes.shape
-   api/ecoli.processes.tf_binding
-   api/ecoli.processes.transcript_elongation
-   api/ecoli.processes.transcript_initiation
-   api/ecoli.processes.two_component_system
-   api/ecoli.processes.chemotaxis.chemoreceptor_cluster
-   api/ecoli.processes.chemotaxis.coarse_motor
-   api/ecoli.processes.chemotaxis.flagella_motor
-   api/ecoli.processes.listeners.mRNA_counts
-   api/ecoli.processes.listeners.mass_listener
-   api/ecoli.processes.membrane.membrane_potential
-   api/ecoli.processes.spatiality.diffusion_network
-   api/ecoli.processes.spatiality.spatial_geometry
-   api/ecoli.processes.stubs.exchange_stub