The DesignSafe-CI Portal can be run using Docker and Docker Compose. You will need both Docker and Docker Compose pre-installed on the system you wish to run the portal on.
If you are on a Mac or a Windows machine, the recommended method is to install Docker Desktop, which will install both Docker and Docker Compose, which is required to run Docker on Mac/Windows hosts.
-
Clone the repo
$ git clone https://github.com/DesignSafe-CI/portal.git $ cd portal
-
Configure environment variables
Make a copy of designsafe.sample.env and rename it to
designsafe.env
. Configure variables as necessary. See designsafe.sample.env for details of configuration.Required variables:
DJANGO_DEBUG
: should be set toTrue
for developmentDJANGO_SECRET
: should be changed for productionTAS_*
: should be set to enable direct access todjango.contrib.admin
AGAVE_*
: should be set to enable Agave API integration (authentication, etc.)RT_*
: should be set to enable ticketing
Make copies of rabbitmq.sample.env and mysql.sample.env, then rename them to
rabbitmq.env
andmysql.env
.Make copies of mysql.sample.cnf, redis.sample.conf, and rabbitmq.sample.conf, then rename them to
mysql.cnf
,redis.conf
, andrabbitmq.conf
.Make a copy of external_resource_secrets.sample.py and rename it to
external_resource_secrets.py
. -
Build the containers and frontend package
$ make build
or
$ docker-compose -f conf/docker/docker-compose.yml build
These lines install the node packages required for DesignSafe, and build the frontend package.
$ npm ci $ npm run build
If you are working with the frontend code and want it to automatically update, use
npm run dev
rather thannpm run build
to have it build upon saving the file. -
Start local containers
$ make start
or
$ docker-compose -f ./conf/docker/docker-compose-dev.all.debug.yml up
then, in a new tab
$ docker exec -it des_django bash $ ./manage.py migrate $ ./manage.py collectstatic -i demo $ ./manage.py createsuperuser
-
Setup local access to the portal:
Add a record to your local hosts file for
127.0.0.1 designsafe.dev
sudo vim /etc/hosts
Now you can navigate to designsafe.dev in your browser.
Every file needed is in conf/nginx/certs
.
- Open mac's Keychain Access
- File > Import Items
- Navigate to
$PROJECT/conf/nginx/certificates
- Select
designsafe.dev.crt
- Search for designsafe and double click on Designsafe.dev
- Click on Trust and select "Trust Always"
- Close the window to save.
$ cd $PROJECT/conf/nginx/certificates
$ sudo mkdir /usr/local/share/ca-certificates/extra
$ sudo cp ca.pem /usr/local/share/ca-certificates/extra/designsafeCA.pem
$ sudo update-ca-certificates
$ sudo dnf install ca-certificates
or$ sudo yum install ca-certificates
$ sudo update-ca-trust enable
$ cd $PROJECT/conf/nginx/certificates
$ sudo cp ca.pem /usr/share/pki/ca-trust-source/anchors/designsafeCA.pem
or$ sudo cp ca.pem /etc/pki/ca-trust/source/anchors/designsafeCA.pem
$ sudo update-ca-trust
- Go to preferences
- Search for Authorities
- Click on "View Certificates" under "Certificates"
- On the Certificate Manager go to the "Authorities" tab
- Click on "Import..."
- Browse to
$PROJECT/conf/nginx/certificates
- Select
ca.pem
sudo apt-get install libnss3-tools
(or proper package manager)certutil -A -n "designsafeCA" -t "TCu,Cu,Tu" -i ca.pem -d ${DBDIR}
$DBDIR
differs from browser to browser for more info: Chromium: https://chromium.googlesource.com/chromium/src/+/master/docs/linux_cert_management.md Firefox: https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data?redirectlocale=en-US&redirectslug=Profiles#How_to_find_your_profile
- Generate RSA-2048 key for CA:
openssl genrsa -des3 -out ca.key 2048
(This file should already be in the repo) - Generate root CA certificate:
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.pem
(Root CA cert is valid for 365 days. Keep any form values to "Designsafe CA") - Generate RSA-2048 key for local dev site:
openssl genrsa out designsafe.dev.key 2048
(This file should already be in the repo) - Generate Cert Request (CSR):
openssql req -new -key -designsafe.dev.key -out designsafe.dev.csr
(Keep any form values to "Designsafe CA") - Make sure
designsafe.dev.ext
is correct - Generate Cert:
openssl x509 -req -in designsafe.dev.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out designsafe.dev.crt -days 365 -sha256 -extfile designsafe.dev.ext
(Cert is valid for 365 days. Keep default form values defined in .conf file) - Files created:
designsafe.dev.key
(site private key),designsafe.dev.csr
(site certificate signing request),designsafe.dev.crt
(actual site certificate),ca.key
(CA private key) andca.pem
(CA certificate).
The necessary configuration is already in conf/nginx/designsafe.dev.conf
.
Run this command to create a self signed cert using the corresponding configuration.
openssl req -config conf/nginx/designsafe.dev.conf -new -sha256 -newkey rsa:2048 -nodes -keyout conf/nginx/designsafe.dev.key -x509 -days 365 -out designsafe.dev.crt
DesignSafe custom apps should be put into designsafe/apps
. You can then enable them in
the Django settings.py
with designsafe.apps.{app_name}
.
See the DesignSafe Styles Reference for style reference and custom CSS documentation.
For simplicity the Dockerfile uses a requirements.txt
exported from Poetry. To add a new dependency:
- Run
poetry add $NEW_DEPENDENCY
. - Run
poetry export > requirements.txt --dev --without-hashes
in the repository root. - Rebuild the dev image with
docker-compose -f conf/docker/docker-compose.yml build
The easiest way to run the tests is from inside a running Docker container. While you can install all the Python/Django/npm dependencies locally (ideally within a virtualenv), this is already done in the docker container.
We assume you have the image built or checked out locally and it is called
des_django
.
Django tests should be written according to standard Django testing procedures.
You can run Django tests with the following command:
$ docker exec -it des_django pytest designsafe
Frontend tests are Jasmine tests executed using the Karma engine. Testing guidelines can be found in the AngularJS Developer Guide on Unit Testing.
To run frontend tests, ensure that all scripts and test scripts are configured in
karma-conf.js
and then run the command:
$ npm run test
Use docker-compose
to run the portal in development. The default compose file,
docker-compose.yml
runs the main django server in development
mode with a redis service for websockets support. You can optionally enable the EF sites
for testing.
$ docker-compose -f conf/docker/docker-compose.yml build
$ docker-compose -f conf/docker/docker-compose-dev.all.debug.yml up
$ npm run dev
When using this compose file, your Agave Client should be configured with a callback_url
of http://$DOCKER_HOST_IP:8000/auth/agave/callback/
.
For developing some services, e.g. Box.com integration, https support is required. To
enable an Nginx http proxy run using the docker-compose-http.yml
file. This file configures the same services as the default compose file, but it also sets
up an Nginx proxy secured by a self-signed certificate.
$ docker-compose -f docker-compose-http.yml build
$ docker-compose -f docker-compose-http.yml up
When using this compose file, your Agave Client should be configured with a callback_url
of https://$DOCKER_HOST_IP/auth/agave/callback/
.
- Delete all of the old metadata objects using this command:
metadata-list Q '{"name": "designsafe metadata"}' | while read x; do metadata-delete $x; done;
2. Run dsapi/agave/tools/bin/walker.py
to create the metadata objects for the existing files in your FS.
python portal/dsapi/agave/tools/bin/walker.py <command> <api_server> <token> <systemId> <base_folder>
base_folder
is your username, if you want to fix everything under your home dir.command
:files
: Walk through the files and print their path.meta
: Walk through the metadata objs in a filesystem-like manner and print their path.files-fix
: Check if there's a meta obj for every file, if not create the meta obj.meta-fix
: Check if there's a file for every meta obj, if not delete the meta obj.
Production deployment is managed by ansible. See https://github.com/designsafe-ci/ansible.