Enhance the Compose development set-up

Changes are:
- Allows Docker image builds in a restricted company context (limited
  access to remote Ubuntu, Python or Node repositories) using variables,
- Centralizes variables in a .env file (not versioned),
- Adds a .env.template file as .env template with predefined variables,
- Use a distinct directory for every service dependencies,
- Use named volumes to avoid anonymous ones (PostgreSQL, Redis and
  Prometheus),
- Use a per-service environment file,
- Simplifies how development configuration files are transferred to
  Timesketch,
- Simplifies manipulation of containers using Compose CLI instead of
  the Docker one,
- Simplify and optimizes the Timesketch entrypoint,
- Updates the Bash scripts to start frontend-ng,
- Updates related documentation.

Use named volumes in Compose development

PostgreSQL, Redis and Prometheus declare volumes in their Dockerfile.
This leads to anonymous volume creations if they are not declared in
Compose.

.dockerignore

@@ -47,3 +47,6 @@
 
 # Exclude JetBrains IDE files
 /.idea/
+
+# Exclude Compose environment file
+/docker/dev/.env

.gitignore

@@ -53,3 +53,6 @@ vagrant/*.log
 
 # Exclude JetBrains IDE files
 .idea/
+
+# Exclude Compose environment file
+docker/dev/.env

docker/dev/.env.template

@@ -0,0 +1,15 @@
+GIFT_PPA_TRACK="stable"
+GIFT_PPA_URL="https://ppa.launchpadcontent.net/gift/${GIFT_PPA_TRACK}/ubuntu"
+NODE_VERSION="18.x"
+NODE_PPA_URL="https://deb.nodesource.com/node_${NODE_VERSION}"
+NODE_NPMRC=""
+PYTHON_PIP_CONF=""
+
+TIMESKETCH_BASE_IMAGE="ubuntu:22.04"
+TIMESKETCH_CONF_DIR="/etc/timesketch"
+TIMESKETCH_SECRET_KEY="L4np0jV3yAdAFdbVzWRMaBqiFMV8FKYd+Je1WKE40o8="
+TIMESKETCH_USER="dev"
+TIMESKETCH_PASSWORD="dev"
+
+POSTGRES_USER="timesketch"
+POSTGRES_PASSWORD="password"

docker/dev/README.md

@@ -5,49 +5,60 @@ Make sure to follow the docker [post-install](https://docs.docker.com/engine/ins
 
 NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM.
 
-### Start a developer version of docker containers in this directory
+### Prepare a .env file
+
+Compose requires a `.env` file with top level environment variables to be set.
+To create it, just copy the `.env.template` file as a base.
 
 ```bash
-docker compose up -d
+cp .env.template .env
 ```
 
-The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
+Note the `.env` is ignored by Git: you can safely write sensitive data in it.
 
-If you see the following message you can continue
+You can optionally edit the `.env` file.
+This is useful if you need to build images with some company restrictions (accessing
+remote Ubuntu, PyPI or Node repositories).
 
-```text
-Timesketch development server is ready!
-```
-### Find out container ID for the timesketch container
+### Start a developer version of docker containers in this directory
 
 ```bash
-CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
+docker compose up -d
 ```
 
-In the output look for CONTAINER ID for the timesketch container
-
-To write the ID to a variable, use:
-
-```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-```
+The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
 
-and test with
+If you see the following message you can continue
 
-```bash
-echo $CONTAINER_ID
+```text
+Timesketch development server is ready!
 ```
 
 ### Start a celery container shell
 
+Start the container in foreground (add `-d` to run in background):
+
 ```bash
-docker exec -it $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
+docker compose exec timesketch \
+  celery \
+  -A timesketch.lib.tasks \
+  worker \
+  --loglevel info
 ```
 
 ### Start development webserver (and metrics server)
 
+Start the container in foreground (add `-d` to run in background):
+
 ```bash
-docker exec -it $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/data/gunicorn_config.py timesketch.wsgi:application
+docker compose exec timesketch \
+  gunicorn \
+  --reload \
+  -b 0.0.0.0:5000 \
+  --log-file - \
+  --timeout 600 \
+  -c /usr/local/src/timesketch/data/gunicorn_config.py \
+  timesketch.wsgi:application
 ```
 
 You now can access your development version at http://127.0.0.1:5000/
@@ -58,18 +69,30 @@ You can also access a metrics dashboard at http://127.0.0.1:3000/
 
 ### Non-interactive
 
-Running the following as a script after `docker compose up -d` will bring up the development environment in the background for you.
+A script applies the previous commands in background for you.
 
 ```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-docker exec $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
-docker exec $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application
+docker compose up -d
+./start-frontend-ng-no-dev.sh
+```
+
+A second script starts an additional development server for the frontend
+(http://127.0.0.1:5001/).
+You need to wait a few seconds before accessing it.
+
+```bash
+docker compose up -d
+./start-frontend-ng-dev.sh
 ```
 
 ### Run tests
 
 ```bash
-docker exec -w /usr/local/src/timesketch -it $CONTAINER_ID python3 run_tests.py --coverage
+docker compose exec \
+    -w /usr/local/src/timesketch \
+    -it \
+    timesketch \
+    python3 run_tests.py --coverage
 ```
 
 That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request.