From 9a08aabb080ad4fbcf9fae017c5178dd3fe12d90 Mon Sep 17 00:00:00 2001 From: Doug Bell Date: Sat, 7 Dec 2019 17:31:42 -0600 Subject: [PATCH] start docs for how to use the Docker stuff --- README.md | 130 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 96 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 566cf22..277cee8 100644 --- a/README.md +++ b/README.md @@ -1,50 +1,96 @@ # CPANTesters Deploy -This repository contains deploy scripts and configuration files for the -CPANTesters servers. +This repository contains a development environment, deploy scripts, and +configuration files for the CPANTesters servers. -## Getting Started +## Getting Started (Development Environment) -### Running the VM +This repository can be used as a development environment, and is the +recommended way to develop for CPAN Testers. -To run the VM, you need Vagrant, VirtualBox, and some Vagrant plugins. -Install Vagrant and VirtualBox according to your OS, and then to install -the Vagrant plugins: +### Prerequisites - $ vagrant plugin vbguest +To use this environment, you must have: -### Preparing the VM +1. `make` +2. `git` +3. `docker` + * On MacOS and Windows, install [Docker Desktop](https://www.docker.com/products/docker-desktop) -To prepare the VM by deploying basic packages and creating user -accounts, you need [Rex](http://rexify.org) installed. Rex is available -from CPAN: `cpan Rex`. The VM you create can take on multiple roles. To -see what roles are available and how to deploy them, run `perldoc -./Rexfile`. +### Fetch Project Code -In order to run the Rex task on your VM, make sure to pass the `-E vm` -flag to `rex`. +To fetch the project code, use `make src`. This will download the CPAN +Testers repositories into the `src/` directory. -### Deploying Applications +### Build Docker Containers -Each repository in the CPAN Testers application has its own `Rexfile` to -deploy itself into the machine prepared by this repository. All of the -repository Rexfiles also support the `-E vm` flag to deploy into the VM. +To build the initial Docker containers, use `make docker`. This will +build: -These repositories are: +1. `cpantesters/base` - The base CPAN Testers container +2. `cpantesters/schema` - Container with the CPAN Testers schema +3. `cpantesters/backend` - The backend, which runs Minion workers +4. `cpantesters/api` - The API web application +5. `cpantesters/web` - The new web application -* [CPAN::Testers::Schema](http://github.com/cpan-testers/cpantesters-schema) - Deploy the database schema -* [CPAN::Testers::Backend](http://github.com/cpan-testers/cpantesters-backend) - Deploy the backend processing tasks and - daemons -* [CPAN::Testers::API](http://github.com/cpan-testers/cpantesters-api) - Deploy the API server and message broker -* [CPAN::Testers::Web](http://github.com/cpan-testers/cpantesters-web) - Deploy the primary web application +Each build will write a log to `build-*.log` to inspect for issues. + +Docker itself is smart about not rebuilding things it does not need to, +so `make docker` will be as fast as it can be. However, if you want to +only build a certain container, you can use the following `make` +targets: + +* `make docker-base` +* `make docker-schema` +* `make docker-backend` +* `make docker-api` +* `make docker-web` + +### Run Test Cluster + +To run the test cluster, use `make start`. This uses the +`docker-compose.yml` file to run a full CPAN Testers cluster including +two database nodes, an API node, and a Web node. This also automatically +runs a database upgrade (deploying the schema if it's missing). + +Once the cluster starts, you can test the web apps with these URLs: + +* - The main web app +* - The API web app + +### Populate Test Data + +XXX + +### Hack! + +Here are some possible ways to test changes to parts of the code: + +#### Backend processing of incoming test reports + +Once a report is in the database, it must be processed. Any report in +the database can be reprocessed any number of times. + +XXX + +#### Test reporting clients + +XXX - This requires another Docker node for the legacy metabase process + +#### Web applications + +This one's pretty easy: Just make your change in the `src/` project +directory and rebuild. Your changed files will be added to the Docker +containers and ready to run! Note: Only files added to the Git repo will +be installed into the Docker containers. ## Architecture Overview This is a quick overview of the CPANTesters architecture as managed by this repository. -### Database and Metabase +### Database and Metabase (`cpantesters-schema`) The primary source of CPANTesters data is the Metabase. This is the database that the CPANTesters reporters write to. This database is @@ -54,14 +100,14 @@ The CPANTesters project reads from the Metabase and writes raw data and report summaries to a local MySQL database. This local MySQL database is managed here. -### Backend ETL +### Backend ETL (`cpantesters-backend`) The backend processes read from the Metabase and writes to the local database. These processes also send out the regular report e-mails, summarize the raw reports in to easily-queried tables, and maintain the data the web app requires. -### Web app +### Web app (`cpantesters-api`, `cpantesters-web`) The CPANTesters web app is the final piece and uses the MySQL database and the data the Backend ETL provides to make a frontend that users can @@ -75,10 +121,25 @@ CPANTesters is managed with [Rex](http://rexify.org). The `Rexfile` contains the main routines for the CPANTesters deploy processes. To see the documentation for deploying, do `perldoc ./Rexfile`. -### `backend/` +### Dockerfile + +CPAN Testers is currently being migrated to Docker containers to make it +easier to develop and deploy. During the transition, Docker is used for +development, but production deployment is done using Rex. + +The Dockerfile in this repository builds a base image with prereqs that +all the apps need. The Dockerfiles in the `src` repositories are +responsible for their own prereqs. + +### docker-compose.yml + +The docker compose configuration for a development cluster. See "Getting +Started", above. + +### `src/` -This directory contains the scripts that maintain the CPANTesters data, -reading incoming test reports and summarizing into the database tables. +The location for the CPAN Testers source repositories. Use `make src` to +populate these. ### `etc/` @@ -87,6 +148,7 @@ deployed to the servers. ### `dist/` -The application distributions. These are synced from the source by a Rex -task, and are not included in this repository. +CPAN distributions to include in the CPAN Testers base image. This +allows pre-release versions of CPAN modules to be tested in the +development environment.