README.md

About

Pycroft is the current user management system of the AG DSN student network. It is based on Flask and expects a Postgres database making use of the SQLAlchemy ORM.

Cloning this directory

A basic understanding of git is advisable. The first step should be to clone this repository via git clone --recursive <url>, using what clone url shows you above this very readme.

Setup

An easy way of doing the setup is by using docker-compose.

Installing Docker and docker-compose

Follow the guides here and here. You will need at least docker engine 17.06.0+ and a docker compose 1.16.0+.

Also, note that you might have to add your user to the docker group for running docker as a non-root:

sudo usermod -aG docker $(whoami)

After adding yourself to a new group, you need to obtain a new session, by e.g. logging out and in again.

You should now be able to run docker-compose config and see the current configuration.

Docker architecture

We provide three different container environments for the project:

• dev: Development environment. The container images contain helpful tools, the containers uses persistent volumes and your local project directory of your machine is mounted inside the container.
• test: Test environment. Almost identical to the development environment. Persistent volumes are replaces by tmpfs file systems for improved performance and ephemerality.
• prod: Production environment. Contains only what is required to run Pycroft without development tools.

For each environment a docker-compose file is provided. The following diagram shows all services/containers, images and volumes at a glance:

• A base service/container for creating the base image agdsn/pycroft-base based on the Debian variant of Docker's official Python image python. The service/container is not actually needed, it's only used to build the base image. The base image contains basic system software required to run Pycroft. A pycroft user and group with UID and GID specified as build arguments is created in the image. The UID and GID of this user should match with your user on your development machine, because the development service bind mounts the project directory on your local machine in the container. The home directory of the pycroft user is created user at /opt/pycroft. A virtual environment (venv) is created at /opt/pycroft/venv and automatically activated by the image's entrypoint.
• A dev-app service/container based on agdsn/pycroft-dev derived from agdsn/pycroft-base. The development image contains additional packages for development, e.g. gcc, npm. The service uses two persistent volumes:
  • the home directory /opt/pycroft of the pycroft user, that contains among other things, the virtual environment, the pip cache, and the .bash_history.
  • the Pycroft sources on your local machine at /opt/pycroft/app.
• A test-app service/container based on the agdsn/pycroft-dev image, that runs unit and integration tests. The database tests are run against an optimized in-memory database.
• A prod-app service/container based on agdsn/pycroft-prod, which is based on agdsn/pycroft-base that contains only the basics that are required for running Pycroft without development tools, such as gcc or npm. Pycroft and its dependencies are build using an instance of the agdsn/pycroft-develop image using the multi-stage builds feature of Docker.
• A dev-db and test-db service/container based on the official postgresql image, that provides a development and test database respectively. The test database uses tmpfs for the data directory to improve performance. The dev database uses a persistent volume for the data directory.
• A dev-ldap and test-ldap service/container based on the dinkel/openldap image, that provides a development and test LDAP server respectively.
• A dev-mq and test-mq service/container based on the official rabbitq image, that provides a development and test message queue respectively.

The separate services for dev and test are mainly for isolation (you don't want tests to affect your development instance and vice versa) and also for performance (unit tests should be quick). There are no prod- services for db, ldap, and mq, because the production instances of these services are typically managed outside of Pycroft.

All services of the same type (dev and test) share the same network namespace, i.e. you can reach the database server on 127.0.0.1 from dev-app although it's running in a different container.

The services are put into different compose files for convenience:

• docker-compose.base.yml: Common definitions of services
• docker-compose.dev.yml: Development services
• docker-compose.test.yml: Test services
• docker-compose.prod.yml: Production services

The dev environment is default environment. The default compose file docker-compose.yml is a symlink to docker-compose.dev.yml.

Defining UID and GID

To set the UID and GID build arguments of the agdsn/pycroft-base image with docker-compose, use an docker-compose .env file:

UID=<your-uid>
GID=<your-gid>

An .env template is included as example.env in the project root. Copy the example to .env and set the correct values for your user, docker-compose will automatically pick up the contents of this file. The example also includes other useful environment variables, such as COMPOSE_PROJECT_NAME.

You can also use environment variables from your shell to specify the UID/GID build arguments when invoking docker-compose. The docker-compose files pass the UID and GID environment variables as build arguments to docker. Don't be fooled by your shell however by executing the following command and feeling safe, if it outputs your UID:

echo $UID

Bash and zsh automatically define this variable, but do not export it:

python3 -c 'import os; print(os.getenv("UID"))'

You have to explicitly export the variable:

export UID
# Bash does not set GID, zsh does, so you can omit the assignment with zsh:
export GID=$(id -g)

You should put these lines somewhere in your shell's startup script (e.g. .profile in your $HOME), so that it is always defined, if you want to rely on these variables instead of an .env file.

You could also the --build-arg option of docker-compose build, but this is not advised as it can easily be forgotten.

Other variables

COMPOSE_PROJECT_NAME

docker-compose uses the name of the directory, the compose file resides in, as the project name. This name is used as a prefix for all objects (containers, volumes, networks) created by docker-compose by default.

To use a different project name, use the COMPOSE_PROJECT_NAME environment variable.

TAG

The tag of the images created by docker-compose can be specified with the TAG environment variable, which defaults to latest, e.g.:

TAG=1.2.3 docker-compose -f docker-compose.prod.yml build

This will tag all generated images with the tag 1.2.3.

Starting an environment

A complete environment can be started by running

docker-compose up -d

This will start all dev environment. docker-compose will build necessary images if not already present, it will not however automatically rebuild the images if the Dockerfiles or any files used by them are modified.

If you run this command for the first time, this might take a while, as a series of packages and image are downloaded, so grab a cup of tea and relax.

All services, except base, which is only used to build the agdsn/pycroft-base image, should now be marked as UP, if you take a look at docker-compose ps. There you see which port forwardings have been set up (remember the port web has been exposed!)

Because you started them in detached mode, you will not see what they print to stdout. You can inspect the output like this:

docker-compose logs # for all services
docker-compose logs dev-app  # for one service
docker-compose logs -f --tail=50 dev-app  # Print the last 50 entries and follow the logs

The last command should tell you that the server spawned an instance at 0.0.0.0:5000 from inside the container.

But don't be too excited, pycroft will fail after the login – we have to set up the database.

To start another enviroment, run docker-compose with the-f flag to specify a different compose file, e.g.:

docker-compose -f docker-compose.test.yml up -d

This would start the test environment.

(Re-)building/Pulling images

You can (re-)build/pull a particular service/image (or all of them if no service is specified) by running:

docker-compose build --force-rm --pull [service]

PyCharm Integration

In order to integrate the setup into PyCharm, make sure that you are using the Professional edition, because the Docker integration feature is only available in the Professional edition of PyCharm. Also make sure that you have updated to a recent version, there were important bug fixes with regards to the Docker integration.

Project interpreters

The dev and test environments should be added to PyCharm as project interpreters.

Go to “Settings” → “Project: Pycroft” → “Project Interpreter” → Gear icon → “Add remote” → “Docker Compose”.

Create a new server for your local machine (use the default settings for that), if none exists yet. Select the config file docker-compose.dev.yml in the project root, select the the service: dev-app, and type in the following path for the python interpreter: /opt/pycroft/venv/bin/python.

Repeat the same thing for test environment defined in docker-compose.test.yml.

Save, and make sure the correct interpreter (dev, not test) is selected as default for the project (“Project settings” → “Project interpreter”). As a proof of concept, you can run a “Python Console” inside PyCharm.

Run Configurations

A few run configurations are already included in the project's .idea folder. If you have created the project interpreters according to the above steps, the appropriate interpreters should have been autoselected for each run configuration.

Database connections (optional)

You can access databases with PyCharm if you are so inclined. First, you need to obtain the IP address of the database container. If you didn't change the project name, the following command will yield the IP address of the database development container:

docker inspect pycroft_dev-db_1 -f '{{ .NetworkSettings.Networks.pycroft_dev.IPAddress }}'

Make sure that database container is started, show the database pane in PyCharm, and add a new data source. PyCharm may complain about missing database drivers. Install any missing driver files directly through PyCharm or your distribution's package manager (whatever you prefer). The password is password.

Setting up the Database

For this section, double check that every container is up and running via docker-compose ps, and if necessary run docker-compose up -d again.

Pycroft needs a PostgreSQL database backend. The unit tests will generate the schema and data automatically, but usually you want to run your development instance against a recent copy of our current production database.

The password for the postgres user is password.

Importing the production database into Pycroft is a three-step process:

1. A regular dump is published in our internal gitlab.

   Clone this repository to your computer.

2. Import the dump:

   psql -h 127.0.0.1 -p 55432 -U postgres -d pycroft -f ../pycroft-data/pycroft.sql

After all that, you should be able to log in into your pycroft instance with the username agdsn at localhost:5000. All users have the password password.

Congratulations!

To import a table from a CSV file, use:

psql -h 127.0.0.1 -p 55432 -U postgres -d pycroft

\copy [tablename] from 'file.csv' with delimiter ',' csv header;"

Running the test suite

For the testing setup, there exists a separate docker-compose file:

# get the stack up and running
docker-compose -f docker-compose.test.yml up -d
# run all the tests
docker-compose -f docker-compose.test.yml run --rm test-app test
# run only the frontend tests
docker-compose -f docker-compose.test.yml run --rm test-app test tests.frontend

Making changes to the database schema

Pycroft uses Alembic to manage changes to its database schema. On startup Pycroft invokes Alembic to ensure that the database schema is up-to-date. Should Alembic detect database migrations that are not yet applied to the database, it will apply them automatically.

To get familiar with Alembic it is recommended to read the official tutorial.

Creating a database migration

Migrations are python modules stored under pycroft/model/alembic/versions/.

A new migration can be created by running:

docker-compose run --rm dev-app alembic revision -m "add test table"

Alembic also has the really convenient feature to autogenerate migrations, by comparing the current status of the database against the table metadata of the application.

docker-compose run --rm dev-app alembic revision --autogenerate -m "add complex test table"

The autogeneration does not know about trigger functions, view definitons or the like. For this, you can pop up a python shell and compile the statements yourself. This way, you can just copy-and-paste them into op.execute() commands in the autogenerated schema upgrade.

import pycroft.model as m
from sqlalchemy.dialects import postgresql
print(m.ddl.CreateFunction(m.address.address_remove_orphans)
      .compile(dialect=postgresql.dialect()))
# if the statement itself has no variable like `address_remove_orphans`,
# you can try to extract it from the `DDLManager` instance:
create_stmt, drop_stmt = [(c, d) for _, c, d in m.user.manager.objects
                          if isinstance(c, m.ddl.CreateTrigger)
                          and c.trigger.name == 'TRIGGER_NAME_HERE']
print(create_stmt.compile(dialect=postgresql.dialect()))
print(drop_stmt.compile(dialect=postgresql.dialect()))

Related dependencies

Pycroft has dependencies that are not part of the Pycroft project, but are maintained by the Pycroft team. Those are:

• wtforms-widgets, for rendering forms
• bootstrap-table-py, for declaring bootstrap-tables in python

To make it easier to make changes on these dependencies, they are added as submodule in the deps folder. You need to recursively clone this repo in order to have them.

You can make changes in these sudmodules and deploy them (in your dev environment) with:

docker-compose run --rm dev-app pip install -r requirements.txt

The production build also uses the submodules. Make sure to update the commit hash of the submodule HEAD if you change something. This will be shown as unstaged change.

Additionally, new versions can be uploaded to PyPi by following these steps:

• Adjust setup.py (new version number, etc.)
• Run the distribute.sh script afterwards in order to upload the new version to PyPi

Troubleshooting

Due to laziness, prefix every bash-snippet with

alias drc=docker-compose

Webpack appears to be missing a library

Re-Install everything using npm, and re-run the webpack entrypoint.

drc run --rm dev-app shell npm install
drc run --rm dev-app webpack

Pip appears to be missing a dependency

Reinstall the pip requirements

drc run --rm dev-app pip install -r requirements.txt

I need to downgrade the schema

drc run --rm dev-app alembic downgrade $hash

Other problems (f.e. failing database initialization)

drc build