Docs

.markdownlint.yaml

@@ -1,2 +1,4 @@
 # Disable checks here
 MD013: false
+MD007:
+  indent: 4

README.md

@@ -6,10 +6,22 @@
 
 # DUNE Run Control User Interface (drunc-ui)
 
+This README contains a short summary on getting started with the project. For more
+details see the [documentation].
+
+[documentation]: https://imperialcollegelondon.github.io/drunc_ui/
+
 This repo defines the web interface for various drunc tools. Including:
 
-- The Processes Manager
-- The Controller
+- [The Process Manager]
+- [The Controllers]
+
+## Running the Project Locally
+
+The DUNE control system comprises multiple services working in concert (see the
+[drunc repository] for an overview). Whilst it is possible to manually run the
+individual services as required it is strongly recommended to use the Docker setup
+provided.
 
 ## Docker Setup
 
@@ -21,14 +33,11 @@ can be used for development of the Process Manager UI.
 
 The full and lite stack are defined using [Docker Compose profiles]. It is recommended
 to set the `COMPOSE_PROFILES` environment variable to select which stack you are working
-with. Use `COMPOSE_PROFILES=drunc` for the full stack or `COMPOSE_PROFILES=drunc-lite`
-for the lite stack.
+with. Use the value `drunc` for the full stack or `drunc-lite` for the lite stack.
 
 You can also use the `--profile` flag when invoking `docker compose` however the
 instructions in this file assume you are using the environment variable.
 
-[Docker Compose profiles]: https://docs.docker.com/compose/how-tos/profiles/
-
 ## Process Manager UI
 
 You can use either the full or lite Docker Compose profiles when working with the
@@ -43,8 +52,8 @@ with:
 docker compose up
 ```
 
-It can take a few moments for the services to boot but the application should then be
-available in the browser at <http://localhost:8000>.
+It can take a few moments for the services to boot but the UI should then be available
+in the browser at <http://localhost:8000/process_manager/>.
 
 __Note that if you have the Process Manager UI already open in a browser you may see
 backtraces from unsuccessful requests generated by the UI. These should stop once drunc
@@ -66,25 +75,24 @@ corresponding terminal.
 
 ## Controller UI
 
-You must use the full Docker Compose profile when working with the Controller UI.
+You must use the full Docker Compose profile when working with the Controller UI. The
+controller UI will not function correctly with the lite profile.
 
 ### Getting Started with the Controller UI
 
-Check you have `COMPOSE_PROFILES=drunc` (see above). As with the Process Manager UI
-start the Docker Compose services and create a user account if not already done. You can
-then boot a full test session:
+Check you have value of the `COMPOSE_PROFILES` variable set to `drunc` (see above). As
+with the Process Manager UI start the Docker Compose services and create a user account
+if not already done. You can then boot a full test session:
 
 ```bash
 docker compose exec drunc /boot_test_session.sh
 ```
 
 _Note that the above consumes several Gb of memory._
 
-Once booted you can interact with the root controller via:
-
-```bash
-docker compose exec drunc /entrypoint.sh drunc-controller-shell grpc://localhost:3333
-```
+The test session boots a full set of processes and controllers suitable for UI testing
+and development. Once booted you can view the status of the root controller with the UI
+at <http://localhost:8000/controller>.
 
 For details of working with the controller see the [drunc wiki]. You should also be able
 to see the booted processes in the Process Manager UI.
@@ -101,100 +109,25 @@ starting the docker containers:
 
 [drunc wiki]: https://github.com/DUNE-DAQ/drunc/wiki/Controller
 
-## Development
-
-Working with the full functionality of the web application requires a number of services
-to be started and to work in concert. The Docker Compose stack provides the required
-services and is suitable for development and manual testing but is not suitable for
-running QA (pre-commit) tooling or unit tests. The project directory is mounted into the
-`app` service which allows the Django development server's auto-reload mechanism to
-detect changes to local files and work as expected.
-
-It is recommended that you follow the below instructions on working with poetry to run
-the project's QA tooling and Unit Tests.
-
-### Working with Poetry
-
-This is a Python application that uses [poetry](https://python-poetry.org) for packaging
-and dependency management. It also provides [pre-commit](https://pre-commit.com/) hooks
-for various linters and formatters and automated tests using
-[pytest](https://pytest.org/) and [GitHub Actions](https://github.com/features/actions).
-Pre-commit hooks are automatically kept updated with a dedicated GitHub Action.
-
-To get started:
-
-1. [Download and install Poetry](https://python-poetry.org/docs/#installation) following the instructions for your OS.
-1. Clone this repository and make it your working directory
-1. Set up the virtual environment:
-
-   ```bash
-   poetry install
-   ```
-
-1. Activate the virtual environment (alternatively, ensure any Python-related command is preceded by `poetry run`):
-
-   ```bash
-   poetry shell
-   ```
-
-1. Install the git hooks:
-
-   ```bash
-   pre-commit install
-   ```
-
-Pre-commit should now work as expected when making commits even without the need to have
-an active poetry shell. You can also manually run pre-commit (e.g. `pre-commit run -a`)
-and the unit tests with `pytest`. Remember you'll need to prefix these with `poetry run`
-first if you don't have an active poetry shell.
-
-#### Running the web application with Poetry
-
-You can also start the web application though at a minimum this requires the drunc
-process manager to be running. Note that drunc only works on Linux so this approach will
-not work on any other platforms. See the next section on also working with
-Kafka. Assuming you have an active poetry shell for all steps:
-
-1. Start the drunc shell:
-
-   ```bash
-   drunc-unified-shell --log-level debug ./data/process-manager-no-kafka.json
-   ```
-
-1. In another terminal, run the main app:
-
-   ```bash
-   python manage.py runserver
-   ```
-
-1. As above you'll need to create a user to get past the login page:
-
-   ```bash
-   python manage.py createsuperuser
-   ```
-
-Note that if you boot any processes in the web application this will immediately die
-with an exit code of 255. This is because the drunc shell requires an ssh server on
-localhost in order to be able to run processes. In most cases this isn't very limiting.
-
-#### Running the web application with Poetry and Kafka
+## Not Using Docker
 
-In the event that you want to work with the full application without using Docker
-Compose you must start the required components manually. Assuming you have an active
-poetry shell for all steps.
+Brief instructions are provided below on how to manually run the various services
+required to work with the Drunc UIs. These are only meant to be indicative however and
+are not regularly tested or maintained.
 
 1. Start Kafka - See [Running drunc with pocket kafka].
 
 1. Start the drunc shell:
    `drunc-unified-shell --log-level debug ./data/process-manager-pocket-kafka.json`
 
-1. Start the application server:
-   `python manage.py runserver`
+1. Start the application server: `python manage.py runserver`
 
-1. Start the Kafka consumer:
-   `python manage.py kafka_consumer --debug`
+1. Start the Kafka consumer: `python manage.py kafka_consumer --debug`
 
-From here you should be able to see broadcast messages displayed at the top of the index
-page on every refresh.
+You may then need to perform subsequent actions such as booting test session data.
 
-[Running drunc with pocket kafka]: https://github.com/DUNE-DAQ/drunc/wiki/Running-drunc-with-pocket-kafka
+[docker compose profiles]: https://docs.docker.com/compose/how-tos/profiles/
+[drunc repository]: https://github.com/DUNE-DAQ/drunc
+[running drunc with pocket kafka]: https://github.com/DUNE-DAQ/drunc/wiki/Running-drunc-with-pocket-kafka
+[the controllers]: https://imperialcollegelondon.github.io/drunc_ui/controller
+[the process manager]: https://imperialcollegelondon.github.io/drunc_ui/process_manager

docs/code_overview.md

@@ -0,0 +1,109 @@
+# Code Overview
+
+## High-level Architecture
+
+The code base covers several user interfaces covering different aspects of the DUNE run
+control system. There are distinct interfaces but also shared infrastructure and
+approaches.
+
+For this reason the code is organised into the following [Django apps]:
+
+- `process_manager` - the Process Manager UI.
+- `controller` - the Controller UI.
+- `main` - common functionality and shared infrastructure between the UIs.
+
+This separation of concerns has been chosen to allow splitting the UIs into separate
+code bases at a later point if desired.
+
+## Design Goals
+
+- Compatibility with wider expertise of the DUNE project i.e. Python and avoiding
+  complex frontend code.
+- Simplicity.
+- Provide a "good enough" user experience.
+- Ease of development.
+
+## Technology Choices
+
+- [Django]
+    - A popular Python back-end framework with a batteries included approach.
+    - Standardised code layout.
+    - Abstraction over underlying database technology via Object Relational Mapping.
+    - Excellent documentation.
+    - Broad range of community packages.
+- [HTMX]
+    - Frontend library that adds dynamic behaviour to html to enable server side
+    rendering.
+    - Easy to learn.
+    - Declarative approach.
+    - Popular in the Django community.
+    - Can be used sparingly and progressively alongside standard Django templating.
+- [Hyperscript]
+    - A front-end scripting alternative to Javascript.
+    - Complements HTMX by providing small amounts of dynamic behaviour where required.
+    - Prioritises readability and conciseness.
+    - Emphasises locality of behaviour.
+    - Can be used sparingly and progressively alongside standard Django templating.
+
+## External Interactions
+
+### Dune Run Control ([drunc])
+
+The primary purpose of this code base is to provide web based user interfaces for the
+various components of drunc (e.g. the process manager or controllers). Interactions with
+drunc components use the provided [gRPC] interfaces however this is abstracted over by
+the `Driver` classes provided in the drunc codebase (see for instance the
+[ProcessManagerDriver] class). A thin wrapper over the driver classes is used to handle
+authentication (currently dummy tokens).
+
+### Kafka
+
+Kafka is used to send informational messages between different components of drunc.
+These messages are displayed in several places in the web UIs. Receiving messages from
+Kafka requires a long-running process to act as a consumer and make messages available
+within the web application.
+
+The consumer is implemented as a [Django admin command] and can be run as
+`python manage.py kafka_consumer`. This makes messages available to the web application
+by storing them in the database. This command is also responsible for pruning messages
+from the database after a configurable retention period.
+
+## Authentication
+
+In the absence of a defined authentication mechanism for use in production, the project
+currently uses the built-in [Django authentication system]. This requires manual
+creation of users via the Django command line interface and stores user data in the
+application database. **This is a placeholder that will need to be replaced**.
+
+## Permissions
+
+Permissions are used to control the actions that can be performed or information that
+can be viewed by different users. Permissions are defined on the custom user model
+defined in `main/models.py`. A fine-grained approach is taken to permissions whereby
+individual actions have their own associated permission. Once the authorisation
+model for the dune-daq systems is well defined, individual permissions can be grouped
+together into user roles via [Django groups].
+
+## Database
+
+This application requires a database to:
+
+- Store user data.
+- Manage user sessions.
+- Cache broadcast messages.
+
+An SQLite database is used in development however Django abstracts over the underlying
+database technology and supports [multiple different databases] that may be used in
+deployment.
+
+[django]: https://www.djangoproject.com/
+[django admin command]: https://docs.djangoproject.com/en/5.1/howto/custom-management-commands/
+[django apps]: https://docs.djangoproject.com/en/5.1/ref/applications/
+[django authentication system]: https://docs.djangoproject.com/en/5.1/topics/auth/default/
+[django groups]: https://docs.djangoproject.com/en/5.1/topics/auth/default/#groups
+[drunc]: https://github.com/DUNE-DAQ/drunc
+[grpc]: https://grpc.io/
+[htmx]: https://htmx.org/
+[hyperscript]: https://hyperscript.org/
+[multiple different databases]: https://docs.djangoproject.com/en/5.1/ref/databases/
+[processmanagerdriver]: https://github.com/DUNE-DAQ/drunc/blob/01ffbdfe57c3373354c2ff2d340714e3c0d90732/src/drunc/process_manager/process_manager_driver.py#L16

docs/controller.md

@@ -0,0 +1,3 @@
+# Controller UI
+
+WIP.

docs/development.md

@@ -0,0 +1,57 @@
+# Development
+
+<!-- markdownlint-disable next-line code-block-style -->
+!!! note
+
+    Make sure you've been through the instructions to run the project locally, can
+    successfully run the web application under the Docker setup, have carried out the
+    setup steps and can access the UI's you want to work with in the browser.
+
+Working with the full functionality of the web application requires a number of services
+to be started and to work in concert. The Docker Compose stack provides the required
+services and is suitable for development and manual testing but is not suitable for
+running QA (pre-commit) tooling or unit tests. The project directory is mounted into the
+`app` service which allows the Django development server's auto-reload mechanism to
+detect changes to local files and work as expected.
+
+In addition to the Docker setup, you will also need to follow the below instructions on
+working with poetry to run the project's QA tooling and unit tests.
+
+## Working with Poetry
+
+This is a Python application that uses [poetry](https://python-poetry.org) for packaging
+and dependency management. It also provides [pre-commit](https://pre-commit.com/) hooks
+for various linters and formatters and automated tests using
+[pytest](https://pytest.org/) and [GitHub Actions](https://github.com/features/actions).
+Pre-commit hooks are automatically kept updated with a dedicated GitHub Action.
+
+To get started:
+
+1. [Download and install Poetry](https://python-poetry.org/docs/#installation) following
+   the instructions for your OS.
+
+1. Clone this repository and make it your working directory
+
+1. Set up the virtual environment:
+
+    ```bash
+    poetry install
+    ```
+
+1. Activate the virtual environment (alternatively, ensure any Python-related command is
+   preceded by `poetry run`):
+
+    ```bash
+    poetry shell
+    ```
+
+1. Install the git hooks:
+
+    ```bash
+    pre-commit install
+    ```
+
+Pre-commit should now work as expected when making commits even without the need to have
+an active poetry shell. You can also manually run pre-commit (e.g. `pre-commit run -a`)
+and the unit tests with `pytest`. Remember you'll need to prefix these with `poetry run`
+first if you don't have an active poetry shell.

docs/gen_ref_nav.py

@@ -20,6 +20,8 @@
     parts = list(module_path.parts)
     if ".array_cache" in parts:
         continue
+    elif "migrations" in parts:
+        continue
     elif parts[-1] == "__init__":
         parts = parts[:-1]
         doc_path = doc_path.with_name("index.md")
@@ -34,5 +36,6 @@
 
     mkdocs_gen_files.set_edit_path(full_doc_path, Path("../") / path)
 
+
 with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
     nav_file.writelines(nav.build_literate_nav())

docs/index.md

@@ -1 +1,6 @@
-# Drunc Web User Interface
+# Drunc Web User Interface (drunc-ui)
+
+--8<-- "README.md:14"
+
+[the controllers]: controller.md <!-- markdownlint-disable-line MD053 -->
+[the process manager]: process_manager.md <!-- markdownlint-disable-line MD053 -->