Docs

.markdownlint.yaml

@@ -1,2 +1,4 @@
 # Disable checks here
 MD013: false
+MD007:
+  indent: 4

README.md

@@ -6,195 +6,7 @@
 
 # DUNE Run Control User Interface (drunc-ui)
 
-This repo defines the web interface for various drunc tools. Including:
+This repo provides web interfaces for the Dune Run Control. See the [documentation] for
+details.
 
-- The Processes Manager
-- The Controller
-
-## Docker Setup
-
-This repository provides a Docker Compose configuration providing the requisite services
-suitable for development of the interfaces using realistic test data. Working with the
-full docker stack requires building a very larger docker base image and has heavy
-resource requirements when running. For this reason a "lite" stack is also provided that
-can be used for development of the Process Manager UI.
-
-The full and lite stack are defined using [Docker Compose profiles]. It is recommended
-to set the `COMPOSE_PROFILES` environment variable to select which stack you are working
-with. Use `COMPOSE_PROFILES=drunc` for the full stack or `COMPOSE_PROFILES=drunc-lite`
-for the lite stack.
-
-You can also use the `--profile` flag when invoking `docker compose` however the
-instructions in this file assume you are using the environment variable.
-
-[Docker Compose profiles]: https://docs.docker.com/compose/how-tos/profiles/
-
-## Process Manager UI
-
-You can use either the full or lite Docker Compose profiles when working with the
-Process Manager UI.
-
-### Getting Started with the Process Manager UI
-
-Check you have a value set for `COMPOSE_PROFILES` (see above). Start the docker stack
-with:
-
-```bash
-docker compose up
-```
-
-It can take a few moments for the services to boot but the application should then be
-available in the browser at <http://localhost:8000>.
-
-__Note that if you have the Process Manager UI already open in a browser you may see
-backtraces from unsuccessful requests generated by the UI. These should stop once drunc
-has finished starting up.__
-
-Authentication is required to work with the application so you need to create a user
-account to work with:
-
-```bash
-docker compose exec app python manage.py createsuperuser
-```
-
-and follow the prompts. You should then be able to use the details you supplied to pass
-the login screen. You can use the "boot" button on the main page to create simple
-processes to experiment with.
-
-Take the services down with `docker compose down` or by pressing Ctrl+C in the
-corresponding terminal.
-
-## Controller UI
-
-You must use the full Docker Compose profile when working with the Controller UI.
-
-### Getting Started with the Controller UI
-
-Check you have `COMPOSE_PROFILES=drunc` (see above). As with the Process Manager UI
-start the Docker Compose services and create a user account if not already done. You can
-then boot a full test session:
-
-```bash
-docker compose exec drunc /boot_test_session.sh
-```
-
-_Note that the above consumes several Gb of memory._
-
-Once booted you can interact with the root controller via:
-
-```bash
-docker compose exec drunc /entrypoint.sh drunc-controller-shell grpc://localhost:3333
-```
-
-For details of working with the controller see the [drunc wiki]. You should also be able
-to see the booted processes in the Process Manager UI.
-
-#### Note for MacOS users
-
-The above `boot_test_session.sh` will not work on M-series Macs because of limitations
-with the x86 architecture emulation used by MacOS. There is a simpler configuration that
-can be used instead. To use this, set the following environment variables before
-starting the docker containers:
-
-- `CSC_SESSION=lr-session`
-- `CSC_URL=drunc:25000`
-
-[drunc wiki]: https://github.com/DUNE-DAQ/drunc/wiki/Controller
-
-## Development
-
-Working with the full functionality of the web application requires a number of services
-to be started and to work in concert. The Docker Compose stack provides the required
-services and is suitable for development and manual testing but is not suitable for
-running QA (pre-commit) tooling or unit tests. The project directory is mounted into the
-`app` service which allows the Django development server's auto-reload mechanism to
-detect changes to local files and work as expected.
-
-It is recommended that you follow the below instructions on working with poetry to run
-the project's QA tooling and Unit Tests.
-
-### Working with Poetry
-
-This is a Python application that uses [poetry](https://python-poetry.org) for packaging
-and dependency management. It also provides [pre-commit](https://pre-commit.com/) hooks
-for various linters and formatters and automated tests using
-[pytest](https://pytest.org/) and [GitHub Actions](https://github.com/features/actions).
-Pre-commit hooks are automatically kept updated with a dedicated GitHub Action.
-
-To get started:
-
-1. [Download and install Poetry](https://python-poetry.org/docs/#installation) following the instructions for your OS.
-1. Clone this repository and make it your working directory
-1. Set up the virtual environment:
-
-   ```bash
-   poetry install
-   ```
-
-1. Activate the virtual environment (alternatively, ensure any Python-related command is preceded by `poetry run`):
-
-   ```bash
-   poetry shell
-   ```
-
-1. Install the git hooks:
-
-   ```bash
-   pre-commit install
-   ```
-
-Pre-commit should now work as expected when making commits even without the need to have
-an active poetry shell. You can also manually run pre-commit (e.g. `pre-commit run -a`)
-and the unit tests with `pytest`. Remember you'll need to prefix these with `poetry run`
-first if you don't have an active poetry shell.
-
-#### Running the web application with Poetry
-
-You can also start the web application though at a minimum this requires the drunc
-process manager to be running. Note that drunc only works on Linux so this approach will
-not work on any other platforms. See the next section on also working with
-Kafka. Assuming you have an active poetry shell for all steps:
-
-1. Start the drunc shell:
-
-   ```bash
-   drunc-unified-shell --log-level debug ./data/process-manager-no-kafka.json
-   ```
-
-1. In another terminal, run the main app:
-
-   ```bash
-   python manage.py runserver
-   ```
-
-1. As above you'll need to create a user to get past the login page:
-
-   ```bash
-   python manage.py createsuperuser
-   ```
-
-Note that if you boot any processes in the web application this will immediately die
-with an exit code of 255. This is because the drunc shell requires an ssh server on
-localhost in order to be able to run processes. In most cases this isn't very limiting.
-
-#### Running the web application with Poetry and Kafka
-
-In the event that you want to work with the full application without using Docker
-Compose you must start the required components manually. Assuming you have an active
-poetry shell for all steps.
-
-1. Start Kafka - See [Running drunc with pocket kafka].
-
-1. Start the drunc shell:
-   `drunc-unified-shell --log-level debug ./data/process-manager-pocket-kafka.json`
-
-1. Start the application server:
-   `python manage.py runserver`
-
-1. Start the Kafka consumer:
-   `python manage.py kafka_consumer --debug`
-
-From here you should be able to see broadcast messages displayed at the top of the index
-page on every refresh.
-
-[Running drunc with pocket kafka]: https://github.com/DUNE-DAQ/drunc/wiki/Running-drunc-with-pocket-kafka
+[documentation]: https://imperialcollegelondon.github.io/drunc_ui/

docs/code_overview.md

@@ -0,0 +1,109 @@
+# Code Overview
+
+## High-level Architecture
+
+The code base covers several user interfaces covering different aspects of the DUNE run
+control system. There are distinct interfaces but also shared infrastructure and
+approaches.
+
+For this reason the code is organised into the following [Django apps]:
+
+- `process_manager` - the Process Manager UI.
+- `controller` - the Controller UI.
+- `main` - common functionality and shared infrastructure between the UIs.
+
+This separation of concerns has been chosen to allow splitting the UIs into separate
+code bases at a later point if desired.
+
+## Design Goals
+
+- Compatibility with wider expertise of the DUNE project i.e. Python and avoiding
+  complex frontend code.
+- Simplicity.
+- Provide a "good enough" user experience.
+- Ease of development.
+
+## Technology Choices
+
+- [Django]
+    - A popular Python back-end framework with a batteries included approach.
+    - Standardised code layout.
+    - Abstraction over underlying database technology via Object Relational Mapping.
+    - Excellent documentation.
+    - Broad range of community packages.
+- [HTMX]
+    - Frontend library that adds dynamic behaviour to html to enable server side
+    rendering.
+    - Easy to learn.
+    - Declarative approach.
+    - Popular in the Django community.
+    - Can be used sparingly and progressively alongside standard Django templating.
+- [Hyperscript]
+    - A front-end scripting alternative to Javascript.
+    - Complements HTMX by providing small amounts of dynamic behaviour where required.
+    - Prioritises readability and conciseness.
+    - Emphasises locality of behaviour.
+    - Can be used sparingly and progressively alongside standard Django templating.
+
+## External Interactions
+
+### Dune Run Control ([drunc])
+
+The primary purpose of this code base is to provide web based user interfaces for the
+various components of drunc (e.g. the process manager or controllers). Interactions with
+drunc components use the provided [gRPC] interfaces however this is abstracted over by
+the `Driver` classes provided in the drunc codebase (see for instance the
+[ProcessManagerDriver] class). A thin wrapper over the driver classes is used to handle
+authentication (currently dummy tokens).
+
+### Kafka
+
+Kafka is used to send informational messages between different components of drunc.
+These messages are displayed in several places in the web UIs. Receiving messages from
+Kafka requires a long-running process to act as a consumer and make messages available
+within the web application.
+
+The consumer is implemented as a [Django admin command] and can be run as
+`python manage.py kafka_consumer`. This makes messages available to the web application
+by storing them in the database. This command is also responsible for pruning messages
+from the database after a configurable retention period.
+
+## Authentication
+
+In the absence of a defined authentication mechanism for use in production, the project
+currently uses the built-in [Django authentication system]. This requires manual
+creation of users via the Django command line interface and stores user data in the
+application database. **This is a placeholder that will need to be replaced**.
+
+## Permissions
+
+Permissions are used to control the actions that can be performed or information that
+can be viewed by different users. Permissions are defined on the custom user model
+defined in `main/models.py`. A fine-grained approach is taken to permissions whereby
+individual actions have their own associated permission. Once the authorisation
+model for the dune-daq systems is well defined, individual permissions can be grouped
+together into user roles via [Django groups].
+
+## Database
+
+This application requires a database to:
+
+- Store user data.
+- Manage user sessions.
+- Cache broadcast messages.
+
+An SQLite database is used in development however Django abstracts over the underlying
+database technology and supports [multiple different databases] that may be used in
+deployment.
+
+[django]: https://www.djangoproject.com/
+[django admin command]: https://docs.djangoproject.com/en/5.1/howto/custom-management-commands/
+[django apps]: https://docs.djangoproject.com/en/5.1/ref/applications/
+[django authentication system]: https://docs.djangoproject.com/en/5.1/topics/auth/default/
+[django groups]: https://docs.djangoproject.com/en/5.1/topics/auth/default/#groups
+[drunc]: https://github.com/DUNE-DAQ/drunc
+[grpc]: https://grpc.io/
+[htmx]: https://htmx.org/
+[hyperscript]: https://hyperscript.org/
+[multiple different databases]: https://docs.djangoproject.com/en/5.1/ref/databases/
+[processmanagerdriver]: https://github.com/DUNE-DAQ/drunc/blob/01ffbdfe57c3373354c2ff2d340714e3c0d90732/src/drunc/process_manager/process_manager_driver.py#L16

docs/controller.md

@@ -0,0 +1,3 @@
+# Controller UI
+
+WIP.

docs/development.md

@@ -0,0 +1,57 @@
+# Development
+
+<!-- markdownlint-disable next-line code-block-style -->
+!!! note
+
+    Make sure you've been through the instructions to run the project locally, can
+    successfully run the web application under the Docker setup, have carried out the
+    setup steps and can access the UI's you want to work with in the browser.
+
+Working with the full functionality of the web application requires a number of services
+to be started and to work in concert. The Docker Compose stack provides the required
+services and is suitable for development and manual testing but is not suitable for
+running QA (pre-commit) tooling or unit tests. The project directory is mounted into the
+`app` service which allows the Django development server's auto-reload mechanism to
+detect changes to local files and work as expected.
+
+In addition to the Docker setup, you will also need to follow the below instructions on
+working with poetry to run the project's QA tooling and unit tests.
+
+## Working with Poetry
+
+This is a Python application that uses [poetry](https://python-poetry.org) for packaging
+and dependency management. It also provides [pre-commit](https://pre-commit.com/) hooks
+for various linters and formatters and automated tests using
+[pytest](https://pytest.org/) and [GitHub Actions](https://github.com/features/actions).
+Pre-commit hooks are automatically kept updated with a dedicated GitHub Action.
+
+To get started:
+
+1. [Download and install Poetry](https://python-poetry.org/docs/#installation) following
+   the instructions for your OS.
+
+1. Clone this repository and make it your working directory
+
+1. Set up the virtual environment:
+
+    ```bash
+    poetry install
+    ```
+
+1. Activate the virtual environment (alternatively, ensure any Python-related command is
+   preceded by `poetry run`):
+
+    ```bash
+    poetry shell
+    ```
+
+1. Install the git hooks:
+
+    ```bash
+    pre-commit install
+    ```
+
+Pre-commit should now work as expected when making commits even without the need to have
+an active poetry shell. You can also manually run pre-commit (e.g. `pre-commit run -a`)
+and the unit tests with `pytest`. Remember you'll need to prefix these with `poetry run`
+first if you don't have an active poetry shell.

docs/gen_ref_nav.py

@@ -20,6 +20,8 @@
     parts = list(module_path.parts)
     if ".array_cache" in parts:
         continue
+    elif "migrations" in parts:
+        continue
     elif parts[-1] == "__init__":
         parts = parts[:-1]
         doc_path = doc_path.with_name("index.md")
@@ -34,5 +36,6 @@
 
     mkdocs_gen_files.set_edit_path(full_doc_path, Path("../") / path)
 
+
 with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
     nav_file.writelines(nav.build_literate_nav())