Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miscellaneous #266

Merged
merged 54 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
85081c2
Use all available Sherlock partitions
thalassemia Dec 3, 2024
92cfb5f
Try different lineage seed for rich media Jenkins test
thalassemia Dec 3, 2024
5cc766d
Go back to running ParCa as separate SLURM job
thalassemia Dec 3, 2024
5152889
Honor lineage seed setting
thalassemia Dec 3, 2024
a1662b0
Automatic unique index generation
thalassemia Dec 4, 2024
bb278a1
Remove full process parallelization because it is currently not pract…
thalassemia Dec 4, 2024
a8156b3
Convert unique molecules back to normal Numpy arrays for JSON
thalassemia Dec 5, 2024
62b17e1
Fix mypy errors
thalassemia Dec 5, 2024
5d48139
Clarify unique indices and fix chromosome segment unique indices
thalassemia Dec 5, 2024
75d1499
Fit RNA deg Km with minimization
thalassemia Dec 5, 2024
7b068a7
64-bit Jax and sweet spot minimization tolerance
thalassemia Dec 6, 2024
42b296d
Start setting up uv install
thalassemia Dec 7, 2024
26e3179
Use uv everywhere, including docs
thalassemia Dec 7, 2024
b724113
Use uv for container images and remove hvplot dependency
thalassemia Dec 7, 2024
9c786d6
Replace Jax with lightweight Autograd
thalassemia Dec 7, 2024
a6ffd1c
Try Debian image with uv
thalassemia Dec 7, 2024
f9962a0
Add project dir for Singularity image
thalassemia Dec 7, 2024
17dcd91
Install git and use gcc
thalassemia Dec 7, 2024
4265a8a
Add temporary flag while waiting for stochastic-arrow 1.1.0
thalassemia Dec 7, 2024
1cbe75a
No persistent cache on Cloud Build
thalassemia Dec 7, 2024
7cbf34e
Clarify ortools constraint
thalassemia Dec 7, 2024
cc0dfd4
No cache for WCM Docker build
thalassemia Dec 7, 2024
ae08c46
Use gcc for WCM image
thalassemia Dec 7, 2024
5b7365c
Fix Altair plot syntax
thalassemia Dec 7, 2024
60987a4
Fix fold change plot
thalassemia Dec 7, 2024
1d7ecc5
WCM image should use non-editable install
thalassemia Dec 7, 2024
8586f58
Make container build scripts sh compliant
thalassemia Dec 7, 2024
a7bb125
Remove requirements.txt now that we use uv
thalassemia Dec 7, 2024
d098ed0
Use Python 3.12 Debian slim base image
thalassemia Dec 7, 2024
ada3416
Fix publish dir substitution
thalassemia Dec 7, 2024
6274a3c
Strip container image of unnecessary files
thalassemia Dec 7, 2024
289ab2e
Add procps to container images
thalassemia Dec 7, 2024
62e4b58
Fix gcloud build substitution
thalassemia Dec 7, 2024
f527fc5
Try Kaniko cache
thalassemia Dec 7, 2024
e79bc23
Remove data dir from package installation
thalassemia Dec 7, 2024
6e809a2
Move murein strand length distribution to flat files
thalassemia Dec 7, 2024
3621487
Try editable install to fix cython modules
thalassemia Dec 7, 2024
c476fa1
Move tetracycline gene FC to flat
thalassemia Dec 7, 2024
ff6db2e
Fix publishDir substitution
thalassemia Dec 7, 2024
9426bc8
Activate Python virtual environment in Singularity containers
thalassemia Dec 7, 2024
19ea32b
Clarify C compilers
thalassemia Dec 7, 2024
c268a23
Force uv to install in container venv
thalassemia Dec 7, 2024
b766b20
Unique indices should be unique between molecule types as well
thalassemia Dec 8, 2024
2b95c00
Add temporary flag to GHA jobs while waiting for stochastic-arrow rel…
thalassemia Dec 8, 2024
5226b8b
Add dev dependencies for QA tests
thalassemia Dec 8, 2024
2308bdf
Remove redundant data
thalassemia Dec 8, 2024
e28883a
Fix CI tests
thalassemia Dec 8, 2024
053d03c
Use cast to fix type check
thalassemia Dec 8, 2024
7c2ff81
Set writable flag after converting to MetadataArray
thalassemia Dec 8, 2024
e759f3d
Handle unique molecules with zero count
thalassemia Dec 8, 2024
daf8283
Use official uv GitHub action
thalassemia Dec 8, 2024
01d3620
Fix pytests
thalassemia Dec 9, 2024
158711a
Regenerate unique indices after merging partial and full mRNA arrays
thalassemia Dec 9, 2024
563235c
Update to latest stochastic-arrow
thalassemia Dec 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Greatly improves performance and reproducibility
OMP_NUM_THREADS=1
8 changes: 8 additions & 0 deletions .gcloudignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!include:.gitignore
notebooks
data
reconstruction/sim_data
reconstruction/ecoli/scripts
.git
.github
doc
22 changes: 9 additions & 13 deletions .github/workflows/docs_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,19 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
sudo apt-get install pandoc
python -m pip install --upgrade pip wheel
pip install numpy==1.26.4
pip install -r doc/requirements.txt
- name: Compile Cython components
run: |
make compile
enable-cache: true
version: "0.5.7"
- name: Install Pandoc
run: sudo apt-get install pandoc
- name: Install model
run: USE_CYTHON=1 uv sync --frozen --extra docs
- name: Build documentation
run: |
cd doc
make html
uv run make html
- name: Deploy documentation to GitHub pages
run: |
cd doc/_build/html
Expand Down
22 changes: 9 additions & 13 deletions .github/workflows/docs_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,16 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
sudo apt-get install pandoc
python -m pip install --upgrade pip wheel
pip install numpy==1.26.4
pip install -r doc/requirements.txt
- name: Compile Cython components
run: |
make compile
enable-cache: true
version: "0.5.7"
- name: Install Pandoc
run: sudo apt-get install pandoc
- name: Install model
run: USE_CYTHON=1 uv sync --frozen --extra docs
- name: Build documentation
run: |
cd doc
make html
uv run make html
65 changes: 19 additions & 46 deletions .github/workflows/pr_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@

name: Workflow

# Improves reproducibility and speed
env:
OPENBLAS_NUM_THREADS: 1
OMP_NUM_THREADS: 1

on:
push:
branches: [master]
Expand All @@ -16,72 +11,50 @@ on:
jobs:
Reproducibility:
runs-on: macos-latest
strategy:
matrix:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install --upgrade pip wheel
pip install numpy==1.26.4
pip install -r requirements.txt
- name: Compile Cython components
run: |
make clean compile
- name: Set PYTHONPATH
run: |
echo "PYTHONPATH=." >> $GITHUB_ENV
enable-cache: true
version: "0.5.7"
- name: Install model
run: USE_CYTHON=1 uv sync --frozen
- name: Test ParCa reproducibility
run: |
python runscripts/parca.py --config ecoli/composites/ecoli_configs/run_parca.json \
uv run --env-file .env runscripts/parca.py --config ecoli/composites/ecoli_configs/run_parca.json \
-c 3 -o out/parca_1
python runscripts/parca.py --config ecoli/composites/ecoli_configs/run_parca.json \
uv run --env-file .env runscripts/parca.py --config ecoli/composites/ecoli_configs/run_parca.json \
-c 3 -o out/parca_2
python runscripts/debug/compare_pickles.py out/parca_1/kb out/parca_2/kb
uv run --env-file .env runscripts/debug/compare_pickles.py out/parca_1/kb out/parca_2/kb
- name: Test simulation reproducibility
run: |
python ecoli/experiments/ecoli_master_sim.py \
uv run --env-file .env ecoli/experiments/ecoli_master_sim.py \
--generations 1 --emitter parquet --emitter_arg out_dir='out' \
--experiment_id "parca_1" --daughter_outdir "out/parca_1" \
--sim_data_path "out/parca_1/kb/simData.cPickle" --fail_at_total_time &
python ecoli/experiments/ecoli_master_sim.py \
uv run --env-file .env ecoli/experiments/ecoli_master_sim.py \
--generations 1 --emitter parquet --emitter_arg out_dir='out' \
--experiment_id "parca_2" --daughter_outdir "out/parca_2" \
--sim_data_path "out/parca_2/kb/simData.cPickle" --fail_at_total_time
python runscripts/debug/diff_simouts.py -o "out" "parca_1*" "parca_2*"
uv run --env-file .env runscripts/debug/diff_simouts.py -o "out" "parca_1*" "parca_2*"
Two-gens:
runs-on: macos-latest
strategy:
matrix:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install --upgrade pip wheel
pip install numpy==1.26.4 mypy
pip install -r requirements.txt
enable-cache: true
version: "0.5.7"
- name: Install model
run: USE_CYTHON=1 uv sync --frozen
- name: Install nextflow edge
run: |
curl -s https://get.nextflow.io | bash
chmod +x nextflow
echo "PATH=.:$PATH" >> $GITHUB_ENV
NXF_EDGE=1 ./nextflow self-update
- name: Compile Cython components
run: |
make clean compile
- name: Set PYTHONPATH
run: |
echo "PYTHONPATH=." >> $GITHUB_ENV
- name: Two generations
run: |
python runscripts/workflow.py --config ecoli/composites/ecoli_configs/two_generations.json
uv run --env-file .env runscripts/workflow.py --config ecoli/composites/ecoli_configs/two_generations.json
71 changes: 21 additions & 50 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,6 @@

name: QA

# Improves reproducibility and speed
env:
OPENBLAS_NUM_THREADS: 1
OMP_NUM_THREADS: 1

on:
push:
branches: [master]
Expand All @@ -16,26 +11,18 @@ on:
jobs:
Pytest:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install --upgrade pip wheel
pip install numpy==1.26.4
pip install -r requirements.txt
- name: Compile Cython components
run: |
make clean compile
enable-cache: true
version: "0.5.7"
- name: Install model
run: USE_CYTHON=1 uv sync --frozen --extra dev
- name: Test with pytest
run: |
python -m pytest --cov-report xml:cov.xml --cov=ecoli --cov=reconstruction --cov=wholecell --cov=runscripts --durations=0
uv run --env-file .env pytest --cov-report xml:cov.xml --cov=ecoli --cov=reconstruction --cov=wholecell --cov=runscripts --durations=0
- name: Code Coverage Report
uses: irongut/[email protected]
with:
Expand All @@ -51,45 +38,29 @@ jobs:
path: code-coverage-results.md
Mypy:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install --upgrade pip wheel
pip install numpy==1.26.4 mypy
pip install -r requirements.txt
- name: Compile Cython components
run: |
make clean compile
enable-cache: true
version: "0.5.7"
- name: Install model
run: USE_CYTHON=1 uv sync --frozen --extra dev
- name: Mypy
run: |
mypy
uv run --env-file .env mypy
Lint:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11"]
steps:
- uses: actions/checkout@v2
- name: Setup Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- name: Install uv
uses: astral-sh/setup-uv@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install --upgrade pip wheel
pip install numpy==1.26.4 ruff
pip install -r requirements.txt
- name: Compile Cython components
run: |
make clean compile
enable-cache: true
version: "0.5.7"
- name: Install model
run: USE_CYTHON=1 uv sync --frozen --extra dev
- name: Ruff
run: |
ruff check doc ecoli migration wholecell runscripts validation reconstruction
uv run --env-file .env ruff check doc ecoli migration wholecell runscripts validation reconstruction
20 changes: 8 additions & 12 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
/build
/dist
/doc/_build
/vivarium_ecoli.egg-info
/vEcoli.egg-info
/Build_sequences.egg-info
/Fast_polymerize_sums.egg-info
/Monte_carlo_complexation.egg-info
Expand Down Expand Up @@ -37,17 +37,12 @@ __pycache__
._*
.python-version
.pytest_cache

# Pipenv files #
################
Pipfile
Pipfile.lock
pyproject.toml
.mypy_cache/
.ruff_cache/

# VS Code files #
#################
/.vscode
/.env

# Generated Files #
###################
Expand All @@ -61,8 +56,9 @@ cache/
##################
.nextflow*
nextflow_temp
trace-*

# Virtual environment #
#######################
.venv/

# Other files #
##################
/notebooks/ck/
/notebooks/ck_example/
21 changes: 0 additions & 21 deletions Makefile

This file was deleted.

Loading
Loading