Digital Marketplace Runner ('DMRunner')

** EXPERIMENTAL / UNDER DEVELOPMENT / IN NEED OF LOVE **

A local development environment for running Digital Marketplace applications and backing services.

DMRunner also provides an interactive shell to manage processes and their logs.

Requirements

DMRunner is compatible with macOS and Ubuntu 20.04.

Running against other OSs may require some care and consideration.

• The Python version specified in .python-version, including headers if appropriate (consider using pyenv), installed with pip and virtualenv packages.
  • macOS/Homebrew users can install with brew install pyenv && pyenv shell 3.9.7
  • Add a line to your .bash_profile to source the Brewfile.env (see below)
• The Node version specified in .nvmrc (consider using Node Version Manager) and NPM 6+ installed and available in your path.
  • If you have NVM the command nvm install && nvm use will install and select the correct version of node for you
• Docker CE/Docker Desktop for Mac 18.03+ installed (if you want backing services managed for you).
  • By default, the Docker daemon starts with a max RAM allowance of only 2 GiB. This generally proves insufficient - you should consider raising it to around 4 GiB.
• If you want to automatically decrypt and inject credentials (requires SC clearance and AWS access):
  • You have a checkout of digitalmarketplace-credentials and export the DM_CREDENTIALS_REPO environment variable with the path to your local checkout. $DM_CREDENTIALS_REPO/sops-wrapper must be functional (follow instructions in README).
  • After running setup, edit the config.yml file and change the value of credentials->sops to on.

Install using Brewfile

macOS/Homebrew users can run invoke brew to install all these prerequisites.

invoke brew will also suggest you source the file Brewfile.env into your environment:

[ -s "$HOME/path/to/repo/digitalmarketplace-runner/Brewfile.env" ] && source "$HOME/path/to/repo/digitalmarketplace-runner/Brewfile.env"

This will make sure that the correct Python and Node versions are in your path, and that the correct version of Postgres is available for use with the digitalmarketplace-api repo.

Instructions

1. Ensure your environment meets the requirements above.
2. Clone this repository into an empty directory (e.g. ~/gds, ~/dm, or whatever), so that you have something like ~/gds/dmrunner.
3. Run make bootstrap
4. Run invoke setup - follow instructions.
5. Run invoke run to bring up the Digital Marketplace locally.

Using a virtual machine

If you do not use macOS or wish to use a completely isolated environment to use DM Runner you can alternatively use the provided Vagrantfile to create a virtual machine which contains DM Runner on a Linux box. To use this the only requirements are to have Vagrant and VirtualBox installed. You can then access the system using

vagrant up
vagrant ssh
cd ~/digitalmarketplace/digitalmarketplace-runner

Then follow the above instructions from step 3.

invoke commands

Options

• invoke setup - Verifies your local environment is suitable and performs basic setup.
• invoke data - Performs an abridged setup that removes your existing managed Postgres and Elasticsearch and repopulates them with fresh data.
• invoke run - Launches all repos using invoke run-app.
• invoke rebuild - Launches all repos with an initial invoke run-all to build eg frontend assets.

Commands

After the apps have done their initial boot-up cycle, you have a few commands at your disposal which are detailed below. There is tab-completion for app names and you can scroll up/down to see your command history.

H / Help

Print out a list of commands available to you.

S / Status

Print out current status of the running apps.

B / Branch / Branches

Print which branches you're running for each app.

R / Restart

Restart any failed or down apps using invoke run-app.

RB / Rebuild

Rebuild and restart any failed or down apps using invoke rebuild, which also does NPM install and frontend-build.

F / Filter

By default, incoming logs from all apps are interleaved together. You can use this to toggle on a filter to show only logs from specific apps. With no arguments, this command resets any filters, showing all logs. With arguments, the closest-matching app name for each word will be filtered in for logging purposes (eg 'f api buyer' will cause only logs for the data api and the buyer-frontend to be shown).

FE / Frontend

Runs invoke frontend-build against frontend apps. With arguments, the closest-matching app name for each word will be rebuilt (eg 'fe buyer supplier' will run a frontend-build on the buyer-frontend and the supplier-frontend. With no arguments, all frontend apps will be rebuilt. This is a one-off rebuild; for ongoing rebuilds use FW.

K / Kill

Kill apps that are running. With no arguments, all apps will be killed. With arguments, the closest-matching app name for each word will be killed (eg 'k search briefs' will take down the search-api and the briefs-frontend).

Q / Quit

Kill all running apps and quit back to your shell.

Configuration and settings

You can configure certain aspects of the runner by editing config/config.yml (after initial setup).

There are two files that determine the majority of the mutable state of the dmrunner - config/config.yml and config/settings.yml.

config.yml is intended to be a user-edited file and is derived from config/example-config.yml. Configurable options include:

• The parent directory to scan for Digital Marketplace repositories
• Whether, and where, to persistently store application logs to disk
• Highlighting in logs displayed to the terminal
• Indentation on wrapped log lines in the display
• Whether or not the runner should automatically inject required credentials/tokens for supporting services (e.g. Notify).

settings.yml should be shared state across all users of the dmrunner in most cases, and lists (for example), the URLs for each of the apps, which apps there are to run and in which order they need to be run, what frameworks need to be indexed for search, etc.

Troubleshooting

Check if your issue is listed under https://github.com/Crown-Commercial-Service/digitalmarketplace-runner/issues.

Docker backing services

On startup, if DMRunner says a backing service already exists, you can list the DMRunner Docker containers (and their ports, if running) with:

$ docker ps -a | grep dmrunner

You can remove stale DMRunner backing service containers with:

$ docker rm dmrunner_postgres_1
$ docker rm dmrunner_elasticsearch_1
$ docker rm dmrunner_nginx_1

If you're on Linux and using rootless Docker, you will need to run nginx yourself. nginx needs host networking, which is not compatible with rootless Docker.

Elasticsearch

If the elasticsearch bootstrap fails, you might need to increase the maximum memory count check.

Postgres

If you have a postgres service running, this can prevent you accessing the postgres docker container. To check for a postgres service, run sudo service postgresql status.

Code formatting

This repository uses the opinionated Python code formatter black to keep the code consistently styled. Git hooks are provided for this repository to seamlessly check and re-style your code as needed. Run ./install_hooks.sh to install the necessary hooks. Alternatively, configure your IDE of choice to run the formatter on save (note: PyCharm currently doesn't seem to support this).

Note: If you have your own global git hooks, this may not work. Global and local hooks cannot run at the same time.

Todo

• Refactoring...
• Add Nix detection to setup and, by default, run apps using Nix to avoid local requirements on node/npm/bower/etc.
• Allow use of frontend-build:watch to continually rebuild assets
• Need to install postgresql locally (api requires pg_config) even if using docker (this is a digitalmarketplace-api dependency, not a dmrunner dependency, but you're likely to come across it using this).

Credits 🏆

This project was created by ex-GDSer @samuelhwilliams as a side-project, and is now maintained by the Digital Marketplace team until he comes back 😝

Licence

Unless stated otherwise, the codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.

The documentation is © Crown copyright and available under the terms of the Open Government 3.0 licence.