-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #233 from CovertLab/basic_docs
- Loading branch information
Showing
86 changed files
with
3,557 additions
and
12,694 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # Modified from GitHub Actions template | ||
|
|
||
| name: Workflow | ||
|
|
||
| # Improves reproducibility and speed | ||
| env: | ||
| OPENBLAS_NUM_THREADS: 1 | ||
| OMP_NUM_THREADS: 1 | ||
|
|
||
| on: | ||
| push: | ||
| branches: [master] | ||
| pull_request: | ||
| branches: [master] | ||
|
|
||
| jobs: | ||
| Reproducibility: | ||
| runs-on: macos-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ["3.11"] | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Setup Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v2 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - name: Install dependencies | ||
| run: | | ||
| pip install --upgrade pip wheel | ||
| pip install numpy==1.26.4 | ||
| pip install -r requirements.txt | ||
| - name: Compile Cython components | ||
| run: | | ||
| make clean compile | ||
| - name: Set PYTHONPATH | ||
| run: | | ||
| echo "PYTHONPATH=." >> $GITHUB_ENV | ||
| - name: Test ParCa reproducibility | ||
| run: | | ||
| python runscripts/parca.py -c 3 -o out/parca_1 | ||
| python runscripts/parca.py -c 3 -o out/parca_2 | ||
| python runscripts/debug/compare_pickles.py out/parca_1/kb out/parca_2/kb | ||
| - name: Test simulation reproducibility | ||
| run: | | ||
| python ecoli/experiments/ecoli_master_sim.py \ | ||
| --generations 1 --emitter parquet --emitter_arg out_dir='out' \ | ||
| --experiment_id "parca_1" --daughter_outdir "out/parca_1" \ | ||
| --sim_data_path "out/parca_1/kb/simData.cPickle" --fail_at_total_time & | ||
| python ecoli/experiments/ecoli_master_sim.py \ | ||
| --generations 1 --emitter parquet --emitter_arg out_dir='out' \ | ||
| --experiment_id "parca_2" --daughter_outdir "out/parca_2" \ | ||
| --sim_data_path "out/parca_2/kb/simData.cPickle" --fail_at_total_time | ||
| python runscripts/debug/diff_simouts.py -o "out" "parca_1*" "parca_2*" | ||
| Two-gens: | ||
| runs-on: macos-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ["3.11"] | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - name: Setup Python ${{ matrix.python-version }} | ||
| uses: actions/setup-python@v2 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| - name: Install dependencies | ||
| run: | | ||
| pip install --upgrade pip wheel | ||
| pip install numpy==1.26.4 mypy | ||
| pip install -r requirements.txt | ||
| - name: Install nextflow | ||
| run: | | ||
| curl -s https://get.nextflow.io | bash | ||
| chmod +x nextflow | ||
| echo "PATH=.:$PATH" >> $GITHUB_ENV | ||
| - name: Compile Cython components | ||
| run: | | ||
| make clean compile | ||
| - name: Set PYTHONPATH | ||
| run: | | ||
| echo "PYTHONPATH=." >> $GITHUB_ENV | ||
| - name: Two generations | ||
| run: | | ||
| python runscripts/workflow.py --config ecoli/composites/ecoli_configs/two_generations.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| # Modified from GitHub Actions template | ||
|
|
||
| name: Basic tests and QA | ||
| name: QA | ||
|
|
||
| # Improves reproducibility and speed | ||
| env: | ||
|
|
@@ -35,7 +35,20 @@ jobs: | |
| make clean compile | ||
| - name: Test with pytest | ||
| run: | | ||
| python -m pytest --cov=ecoli --durations=0 | ||
| python -m pytest --cov-report xml:cov.xml --cov=ecoli --cov=reconstruction --cov=wholecell --cov=runscripts --durations=0 | ||
| - name: Code Coverage Report | ||
| uses: irongut/[email protected] | ||
| with: | ||
| filename: cov.xml | ||
| format: markdown | ||
| indicators: true | ||
| output: both | ||
| - name: Add Coverage PR Comment | ||
| uses: marocchino/sticky-pull-request-comment@v2 | ||
| if: github.event_name == 'pull_request' | ||
| with: | ||
| recreate: true | ||
| path: code-coverage-results.md | ||
| Mypy: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| ========== | ||
| Composites | ||
| ========== | ||
|
|
||
| :py:class:`~ecoli.composites.ecoli_master.Ecoli` is a so-called composer | ||
| that is responsible for aggregating all the Processes, Steps, topologies, | ||
| and the flow for the Steps into a unified "composite" model that vivarium-core | ||
| is able to run. Unlike a typical Vivarium composer which simply collects all | ||
| these pieces, the :py:class:`ecoli.composites.ecoli_master.Ecoli` composer | ||
| automatically makes several modifications to these components in order to | ||
| support features unique to our model. | ||
|
|
||
| .. _composite_partitioning: | ||
|
|
||
| -------------------------- | ||
| Partitioning Modifications | ||
| -------------------------- | ||
|
|
||
| As described in :ref:`partitioning`, our model contains many processes that inherit | ||
| from :py:class:`~ecoli.processes.partition.PartitionedProcess` because they require | ||
| special handling to avoid overdrafting bulk molecules. These processes are not directly | ||
| included in the final composite but instead used to parameterize two Steps | ||
| that are included and run with the final model: a | ||
| :py:class:`~ecoli.processes.partition.Requester` and an | ||
| :py:class:`~ecoli.processes.partition.Evolver`. | ||
|
|
||
| The :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_processes_and_steps` | ||
| method of the :py:class:`~ecoli.composites.ecoli_master.Ecoli` composer is responsible | ||
| for creating these two Steps, the :py:class:`~ecoli.processes.allocator.Allocator` steps | ||
| sandwiched between them in each execution layer, and the | ||
| :py:class:`~ecoli.processes.unique_update.UniqueUpdate` Steps the run at the very end | ||
| of each execution layer. It is also responsible for updating the flow to arrange | ||
| these Steps in the order described in :ref:`implementation`. As an end-user, all you | ||
| have to do to add a new partitioned process is ensure that it inherits from | ||
| :py:class:`~ecoli.processes.partition.PartitionedProcess` and is included in the flow | ||
| supplied to :py:class:`~ecoli.composites.ecoli_master.Ecoli` (see :ref:`/experiments.rst` | ||
| for a description of the interface for specifying simulation options). | ||
|
|
||
| The :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_topology` method is responsible | ||
| for creating the topologies for each Requester and Evolver. It does so by first copying the | ||
| topology of the corresponding :py:class:`~ecoli.processes.partition.PartitionedProcess` then | ||
| adding and wiring the following additional ports: | ||
|
|
||
| - ``process``: Wired to store located at ``("process", "process_name")``. Contains the | ||
| instance of :py:class:`~ecoli.processes.partition.PartitionedProcess` that is shared | ||
| by both the Requester and the Evolver. Stored as a single-element tuple because | ||
| vivarium-core has special handling for stores containing naked processes that is not | ||
| applicable here. | ||
| - ``request`` (:py:class:`~ecoli.processes.partition.Requester` only): Wired to store | ||
| located at ``("request", "process_name")``. Contains a single sub-store called ``bulk`` | ||
| that is updated every time the Requester runs with a list of tuples | ||
| ``(Index of bulk molecule in structured Numpy array, count requested)``. | ||
| - ``allocate`` (:py:class:`~ecoli.processes.partition.Evolver` only): Wired to store | ||
| located at ``("allocate", "process_name")``. Contains a single sub-store called ``bulk`` | ||
| that is updated every time the :py:class:`~ecoli.processes.allocator.Allocator` for the | ||
| corresponding execution layer is run with a 1D array of partitioned bulk counts. | ||
| - ``global_time``, ``timestep``, ``next_update_time``: See :ref:`timesteps`. | ||
|
|
||
| ---------------------- | ||
| Division Modifications | ||
| ---------------------- | ||
|
|
||
| vEcoli has a variety of options related to cell division that, if enabled, | ||
| prompt :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_processes_and_steps` | ||
| to add certain Steps and | ||
| :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_topology` to add | ||
| their corresponding topologies to the final composite model. | ||
|
|
||
| - ``divide``: Adds :py:class:`~ecoli.processes.cell_division.Division` when ``True`` | ||
| - ``d_period``: Adds :py:class:`~ecoli.processes.cell_division.MarkDPeriod` when ``True`` | ||
| but only if ``divide`` is ``True`` | ||
| - ``generations``: Adds :py:class:`~ecoli.processes.cell_division.StopAfterDivision` | ||
| when ``True`` but only if ``divide`` is ``True`` | ||
|
|
||
| ----------- | ||
| Log Updates | ||
| ----------- | ||
|
|
||
| For debugging purposes, it may be useful to know exactly what updates are returned | ||
| by each Process/Step in the model at every timestep. The ``log_updates`` boolean | ||
| configuration option adds a ``log_update`` port to each process wired to a store | ||
| located at ``("log_update", "process_name")`` (see | ||
| :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_topology`). It also wraps | ||
| each process using | ||
| :py:func:`~ecoli.library.logging_tools.make_logging_process` to make it write the | ||
| contents of its update to this log update store (see | ||
| :py:meth:`~ecoli.composites.ecoli_master.Ecoli.generate_processes_and_steps`). | ||
|
|
||
| The analysis plots located in :py:mod:`~ecoli.analysis.single.blame` can be used | ||
| to visualize these updates. | ||
|
|
||
| .. warning:: | ||
| This feature should only be turned for debugging purposes and | ||
| only when using the in-memory emitter (``timeseries``). | ||
|
|
||
| ------------- | ||
| Initial State | ||
| ------------- | ||
|
|
||
| The :py:meth:`~ecoli.composites.ecoli_master.Ecoli.initial_state` method is responsible | ||
| for generating the initial state used to populate many of the stores in the simulation | ||
| (see the "Initialization" sub-headings in :ref:`/stores.rst`). | ||
|
|
||
| It also allows users to manually override initial state values and populates the | ||
| ``("process", "process_name")`` stores mentioned in :ref:`composite_partitioning`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.