From eae3bd2e466df7fc28f36e91e30998e20b92da50 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Thu, 5 Sep 2024 17:13:28 -0400 Subject: [PATCH] add high-level architecture diagram --- docs/developer/design/communication_flows.rst | 46 +++++-- docs/developer/design/overview.rst | 112 ++++++++++++++++++ 2 files changed, 149 insertions(+), 9 deletions(-) diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/design/communication_flows.rst index afd044f7..28f077e6 100644 --- a/docs/developer/design/communication_flows.rst +++ b/docs/developer/design/communication_flows.rst @@ -1,6 +1,10 @@ +.. _communication_flows: + ActiveMQ Communication Flows ============================ +.. contents:: :local: + Autoreducers ------------ @@ -95,7 +99,7 @@ parameters for instruments that have implemented DASMON ------ -DASMON, from Data Acquisition System (DAS) Monitor, provides process variable (PV) updates from the +DASMON, from Data Acquisition (DAQ) System Monitor, provides process variable (PV) updates from the beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database. .. mermaid:: @@ -105,13 +109,12 @@ beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directl participant Workflow DB participant Dasmon listener DASMON->>Workflow DB: PV update - Workflow DB->>Dasmon listener: PV update DASMON->>Dasmon listener: Heartbeats Dasmon listener --------------- -Streaming Message Service (SMS) +Stream Management Service (SMS) ............................... .. mermaid:: @@ -138,12 +141,35 @@ admins by email when a service has missed heartbeats (needs to be verified that participant Dasmon listener participant Workflow DB actor Subscribed users - Other services->>Dasmon listener: Heartbeat - Dasmon listener->>Workflow DB: Status update - opt There are missed heartbeats + loop Every N s + Other services->>Dasmon listener: Heartbeat + Dasmon listener->>Workflow DB: Status update + end + opt Service has 3 missed heartbeats Dasmon listener->>Subscribed users: Email end + +.. mermaid:: + + flowchart LR + SMS["SMS (per beamline)"] + PVSD["PVSD (per beamline)"] + DASMON["DASMON (per beamline)"] + STC + Autoreducers + DasmonListener + WorkflowDB[(DB)] + SMS-->|heartbeat|DasmonListener + PVSD-->|heartbeat|DasmonListener + DASMON-->|heartbeat|DasmonListener + STC-->|heartbeat|DasmonListener + Autoreducers-->|heartbeat|DasmonListener + WorkflowManager-->|heartbeat|DasmonListener + DasmonListener-->|heartbeat|DasmonListener + DasmonListener-->WorkflowDB + DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist + WebMon interaction with Live Data Server ---------------------------------------- @@ -164,13 +190,15 @@ or HTML div) to Live Data Server. participant Live Data Server WebMon->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Live Data Server: publish_plot - Note over Live Data Server: Store plot in DB + opt Publish plot + Autoreducer->>Live Data Server: publish_plot + Note over Live Data Server: Store plot in DB + end Display plot from Live Data Server ................................ -Run overview pages (monitor.sns.gov/report///) will query the Live +Run overview pages (``monitor.sns.gov/report///``) will query the Live Data Server for a plot for that instrument and run number and display it if available. .. mermaid:: diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst index f175fc1f..695984ac 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/design/overview.rst @@ -5,3 +5,115 @@ Design Overview :maxdepth: 1 communication_flows + + +High-level architecture +----------------------- + +The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are +services owned by the Data Acquisition System team that feed information to WebMon but are not part +of WebMon. The services communicate mainly through an ActiveMQ message broker. The communication +flow is described in :ref:`communication_flows`. + +.. mermaid:: + + flowchart LR + FileArchive[("`SNS/HFIR + File archive`")] + subgraph DAS + DASMON + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + PVSD + end + subgraph "Workflow Manager" + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(DB)] + end + subgraph Autoreduction + Autoreducers-->|cataloging|ONCat + LiveDataServer<-->LiveDataDB[(DB)] + LiveReduce + end + subgraph Frontend + WebMon["`WebMon + monitor.sns.gov`"] + end + SMS-->|live data stream|LiveReduce + TranslationService-->|trigger reduction|WorkflowManager + DASMON-->|status|DasmonListener + DASMON-->|PV updates|Database + WorkflowManager<-->|instructions|Autoreducers + Autoreducers-->|publish plots|LiveDataServer + Database-->WebMon + ONCat-->|run metadata|WebMon + LiveDataServer-->|plots|WebMon + LiveReduce-->|publish plots|LiveDataServer + DasmonListener-->Database + WorkflowManager-->Database + TranslationService-->|NeXus file|FileArchive + FileArchive<-->Autoreducers + style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + + +High-level architecture +----------------------- + +The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are +services owned by the Data Acquisition System team that feed information to WebMon, e.g. to trigger +autoreduction when a run is finished. The services mainly communicate through an ActiveMQ message +broker. The communication flow is described in :ref:`communication_flows`. + +.. mermaid:: + + flowchart LR + FileArchive[("`SNS/HFIR + File archive`")] + subgraph DAS + DASMON + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + PVSD + end + subgraph "Workflow Manager" + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(DB)] + end + subgraph Autoreduction + Autoreducers-->|REST API|ONCat + LiveDataServer<-->LiveDataDB[(DB)] + LiveReduce + end + subgraph Frontend + WebMon["`WebMon + monitor.sns.gov`"] + end + SMS-->LiveReduce + TranslationService-.->|ActiveMQ|WorkflowManager + DASMON-.->|ActiveMQ|DasmonListener + DASMON-->Database + WorkflowManager<-.->|ActiveMQ|Autoreducers + Autoreducers-->|REST API|LiveDataServer + Database-->WebMon + ONCat-->|REST API|WebMon + LiveDataServer-->|REST API|WebMon + LiveReduce-->|REST API|LiveDataServer + DasmonListener-->Database + WorkflowManager-->Database + TranslationService-->FileArchive + FileArchive<-->Autoreducers + style DAS fill:#D3D3D3, stroke-dasharray: 5 5