requirements.txt insufficiently details installing model dependencies

[JonahRileyHuggins]

Hi all,

the requirements.txt file, if used to install the model dependencies, fails on execution.
Example:
pip install -r requirements.txt &> dependencyInstallOutput.txt
dependencyInstallOutput.txt

This is because some packages within the requirements.txt file has dependencies of there own that must be met before install. For instance, AMICI lists that it needs Swig, some form of CBLAS, and C/C++ compilers as dependencies.

This makes it exceedingly difficult to demonstrate installing the model dependencies outside of using the docker container.

One solution I have tried instead is an Anaconda environment.yml file in combination with a .condarc file, which creates a local virtual environment with all the necessary dependencies needed to run the SPARCED model. This however, only works on Linux thus-far.

Are there any potential solutions that could be proposed to enable dependency installation?

[erdemlab]

I personally understand and accept the issue, but that is the very reason we used docker as a solution to all dependency requirements and a solution across operating systems.

[JonahRileyHuggins]

Thanks for the prompt reply! I agree that operating SPARCED through docker locally is a good solution to the dependency issue.

However, HPC environments don’t typically support Docker, as evident by Clemson’s Palmetto Cluster. Apptainer is a decent solution with conversion capabilities for that specific instance. I have written some documentation on converting the docker container to Apptainer, however, I have yet to evaluate the converted container’s efficiency.

Further, I’ve read some brief materials on using git within a docker container, however it seems discouraged and another barrier for those wanting to collaborate on development.

Are we able to circumvent these issues so that new collaborators have an easier point of entry?

[ChocolateCharlie]

Hi all!

Nice to e-meet you @JonahRileyHuggins! I was an intern at the lab in Summer 2022, so things probably changed since then, but I thought maybe my feedback on getting hands-on with installing SPARCED could help?

As I had to install several times an environment for SPARCED, I wrote a small step-by-step installation guide here which relies on an Anaconda environment for a Ubuntu OS. It is very basic as it was intended for further interns like me, but I guess it could do the job or serve as a basis to design an easier point of entry for new collaborators?

By the way, I'll be glad to meet you in person in a couple months, as I'm coming back for a 6-month internship starting in February! 😄

[JonahRileyHuggins]

Hi @ChocolateCharlie, nice to e-meet you too! Not much has changed on the official repository in some time, but this is the exact discussion I was hoping to start!

I love what you've done in your installation guide! I did something similar, where I have an anaconda env.yml file in the project root directory and a configured .condarc to install some of the dependencies for amici and the like.

Truthfully, you probably have a better perspective than I, as I'm a wet-lab scientist turning computer scientist. The important semantics of python project building are unfamiliar at times. Your expertise would be useful.

It seems that a reproducible installation procedure on Ubuntu is more than achievable, at least for the non-OpenMPI version. Aside from the pip installable packages in the requirements file, these seem to be the only deviations:
sudo apt install libatlas-base-dev swig

conda install -c conda-forge openblas
export BLAS_LIBS=-lopenblas

Instead of using anaconda to install openblas, I wonder if the following would work:
sudo apt-get install libopenblas-dev

Now I might really be reaching beyond my abilities, but if this was the case, we could also specify the package managers needed in the requirements.txt file, like so:

# APT packages (Linux-specific)
apt install libatlas-base-dev swig # APT
apt-get install libopenblas-dev # APT

# pip packages (cross-platform)
package3  # pip
package4==4.5.6  # pip

install_dependencies.sh
Then, it would be possible to just have a single install file, called install_dependencies.sh, to install both packages, like so:

#!/bin/bash

# Install APT dependencies (Linux-specific)
while IFS= read -r line; do
    if [[ $line == *"APT"* ]]; then
        package_name="${line%%#*}"  # Extract the package name
        sudo apt-get install -y "$package_name"
    fi
done < requirements.txt

# Install pip dependencies (cross-platform)
pip install -r requirements.txt

Please share your thoughts, as well as @cerdem12!