From 3ce28fbd3f4bcec072ce9ad1eb3624674226d714 Mon Sep 17 00:00:00 2001 From: Marie Backman Date: Fri, 6 Sep 2024 11:53:17 -0400 Subject: [PATCH] address comments --- docs/developer/design/communication_flows.rst | 133 ++++++++++------- docs/developer/design/overview.rst | 141 ++++++++---------- 2 files changed, 138 insertions(+), 136 deletions(-) diff --git a/docs/developer/design/communication_flows.rst b/docs/developer/design/communication_flows.rst index 28f077e6..d29b43b8 100644 --- a/docs/developer/design/communication_flows.rst +++ b/docs/developer/design/communication_flows.rst @@ -12,15 +12,38 @@ Run post-processing ................... The sequence diagram below describes the communication flow as a run gets post-processed. -The post-processing workflow is triggered when the Translation Service has finished translating and -sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the instrument, IPTS, run number and -location of the Nexus file. +The post-processing workflow is triggered when the Translation Service has finished translating the +data stream to NeXus and sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the +instrument, IPTS, run number and location of the NeXus file. -The post-processing workflow for the instrument is configurable in the database table ``Task``. +The post-processing workflow for the instrument is configurable in the database table +``report_task``. The diagram shows the three post-processing steps that are available: autoreduction, cataloging of raw data in `ONCat `_ and cataloging of reduced data in `ONCat `_. +.. mermaid:: + + sequenceDiagram + participant Translation Service + participant Workflow Manager + participant Autoreducer + participant ONCat + + Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY + Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY + Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION.STARTED + Autoreducer->>Autoreducer: run reduction + Autoreducer->>Workflow Manager: REDUCTION.COMPLETE + Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED + Autoreducer->>ONCat: pyoncat + Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE + .. .. mermaid:: @@ -28,13 +51,12 @@ raw data in `ONCat `_ and cataloging of reduced data in participant Translation Service participant Workflow Manager participant Autoreducer - participant ONCat Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY opt Cataloging Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Autoreducer->>ONCat: pyoncat + Note over Autoreducer: Ingest in ONCat Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE end opt Autoreduction @@ -46,37 +68,10 @@ raw data in `ONCat `_ and cataloging of reduced data in opt Reduced data cataloging Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Autoreducer->>ONCat: pyoncat + Note over Autoreducer: Ingest in ONCat Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE end -.. mermaid:: - - sequenceDiagram - participant Translation Service - participant Workflow Manager - participant Autoreducer - - Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY - opt Cataloging - Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY - Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE - end - opt Autoreduction - Workflow Manager->>Autoreducer: REDUCTION.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION.STARTED - Note over Autoreducer: Execute autoreduction script - Autoreducer->>Workflow Manager: REDUCTION.COMPLETE - end - opt Reduced data cataloging - Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED - Note over Autoreducer: Ingest in ONCat - Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE - end - Configuring the autoreduction ............................. @@ -92,24 +87,28 @@ parameters for instruments that have implemented actor Instrument Scientist participant WebMon participant Autoreducer + participant HFIR/SNS File archive Instrument Scientist->>WebMon: Submit form with parameter values WebMon->>Autoreducer: REDUCTION.CREATE_SCRIPT - Note over Autoreducer: Update parameter values in
instrument reduction script + Autoreducer->>HFIR/SNS File archive: Update instrument reduction script DASMON ------ -DASMON, from Data Acquisition (DAQ) System Monitor, provides process variable (PV) updates from the -beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database. +DASMON, from Data Acquisition (DAQ) System Monitor, provides instrument status and process variable +(PV) updates from the beamlines to WebMon. DASMON connects to the WebMon message broker to pass +status information, for example the current run number and count rate, to Dasmon listener. Due to +the high volume of PV updates, DASMON writes PV:s directly to the PostgreSQL database. .. mermaid:: sequenceDiagram participant DASMON - participant Workflow DB participant Dasmon listener + participant Workflow DB DASMON->>Workflow DB: PV update - DASMON->>Dasmon listener: Heartbeats + DASMON->>Dasmon listener: Instrument status + DASMON->>Workflow DB: Instrument status Dasmon listener --------------- @@ -134,20 +133,21 @@ Heartbeats from services Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting admins by email when a service has missed heartbeats (needs to be verified that this still works). -.. mermaid:: +.. + .. mermaid:: - sequenceDiagram - participant Other services - participant Dasmon listener - participant Workflow DB - actor Subscribed users - loop Every N s - Other services->>Dasmon listener: Heartbeat - Dasmon listener->>Workflow DB: Status update - end - opt Service has 3 missed heartbeats - Dasmon listener->>Subscribed users: Email - end + sequenceDiagram + participant Other services + participant Dasmon listener + participant Workflow DB + actor Subscribed users + loop Every N s + Other services->>Dasmon listener: Heartbeat + Dasmon listener->>Workflow DB: Status update + end + opt Service has 3 missed heartbeats + Dasmon listener->>Subscribed users: Email + end .. mermaid:: @@ -170,14 +170,14 @@ admins by email when a service has missed heartbeats (needs to be verified that DasmonListener-->WorkflowDB DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist -WebMon interaction with Live Data Server +Live Data Server ---------------------------------------- WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data Server database and display (fetch) plots from the database. -Publish to Live Data Server -........................... +Publish to Live Data Server from autoreduction script +..................................................... The instrument-specific autoreduction script can optionally publish plots (in either JSON format or HTML div) to Live Data Server. @@ -192,9 +192,30 @@ or HTML div) to Live Data Server. WebMon->>Autoreducer: REDUCTION.DATA_READY opt Publish plot Autoreducer->>Live Data Server: publish_plot - Note over Live Data Server: Store plot in DB end +Publish to Live Data Server from live data stream +................................................. + +Livereduce (https://github.com/mantidproject/livereduce/) allows scientists to reduce +data from an ongoing experiment, i.e. before translation to NeXus, by connecting to the live data +stream from the Stream Management Service (SMS). The instrument-specific processing +script can make the results available in WebMon by publishing plots to Live Data Server. + +.. mermaid:: + + sequenceDiagram + participant SMS + participant Livereduce + participant Live Data Server + + SMS->>Livereduce: data stream + loop Every N minutes + Livereduce->>Livereduce: run processing script + Livereduce->>Live Data Server: publish plot + end + + Display plot from Live Data Server ................................ diff --git a/docs/developer/design/overview.rst b/docs/developer/design/overview.rst index 695984ac..b5e1df00 100644 --- a/docs/developer/design/overview.rst +++ b/docs/developer/design/overview.rst @@ -10,10 +10,12 @@ Design Overview High-level architecture ----------------------- -The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are -services owned by the Data Acquisition System team that feed information to WebMon but are not part -of WebMon. The services communicate mainly through an ActiveMQ message broker. The communication -flow is described in :ref:`communication_flows`. +The diagram below describes the high-level architecture of WebMon, including some external resources +that WebMon interacts with. The gray box labeled "DAS" are services owned by the Data Acquisition +System team that feed information to WebMon. `ONCat `_ is the experiment +data catalog, which the autoreducers catalog runs to and the frontend fetches run metadata from. +The autoreducers access instrument-specific reduction scripts and experiment data files on the +HFIR/SNS file archive. The autoreducers also write reduced data files and logs to the file archive. .. mermaid:: @@ -30,90 +32,69 @@ flow is described in :ref:`communication_flows`. Management Service (SMS)`"] - PVSD end - subgraph "Workflow Manager" - WorkflowManager[Workflow Manager] - DasmonListener[Dasmon listener] - Database[(DB)] - end - subgraph Autoreduction - Autoreducers-->|cataloging|ONCat - LiveDataServer<-->LiveDataDB[(DB)] - LiveReduce - end - subgraph Frontend - WebMon["`WebMon - monitor.sns.gov`"] - end - SMS-->|live data stream|LiveReduce - TranslationService-->|trigger reduction|WorkflowManager - DASMON-->|status|DasmonListener - DASMON-->|PV updates|Database - WorkflowManager<-->|instructions|Autoreducers - Autoreducers-->|publish plots|LiveDataServer + WorkflowManager[Workflow Manager] + DasmonListener[Dasmon listener] + Database[(Workflow DB)] + Autoreducers-->ONCat + LiveDataServer-->WebMon + LiveDataServer<-->LiveDataDB[(LiveData DB)] + LiveReduce + WebMon["`WebMon + frontend`"] + SMS-->LiveReduce + TranslationService-->WorkflowManager + DASMON-->DasmonListener + DASMON-->Database + WorkflowManager-->Database + WorkflowManager<-->Autoreducers + Autoreducers-->LiveDataServer Database-->WebMon - ONCat-->|run metadata|WebMon - LiveDataServer-->|plots|WebMon - LiveReduce-->|publish plots|LiveDataServer + ONCat-->WebMon + LiveReduce-->LiveDataServer DasmonListener-->Database - WorkflowManager-->Database - TranslationService-->|NeXus file|FileArchive + TranslationService-->FileArchive FileArchive<-->Autoreducers style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + classDef webMonStyle fill:#FFFFE0 + class WorkflowManager,DasmonListener,Database,Autoreducers,LiveDataServer,LiveReduce,WebMon,LiveDataDB webMonStyle + subgraph Legend + direction LR + External["External resource"] + Internal["Internal resource"] + External ~~~ Internal + end + LiveReduce ~~~ External + style Legend fill:#FFFFFF,stroke:#000000 + class Internal webMonStyle -High-level architecture ------------------------ -The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are -services owned by the Data Acquisition System team that feed information to WebMon, e.g. to trigger -autoreduction when a run is finished. The services mainly communicate through an ActiveMQ message -broker. The communication flow is described in :ref:`communication_flows`. +Message broker +-------------- + +WebMon uses an `ActiveMQ `_ message broker for communication between +services. The message broker also serves as a load balancer by distributing post-processing jobs to +the available autoreducers in a round-robin fashion. + +Service communication flows are described in :ref:`communication_flows`. .. mermaid:: - flowchart LR - FileArchive[("`SNS/HFIR - File archive`")] - subgraph DAS - DASMON - TranslationService["`Streaming - Translation - Client - (STC)`"] - SMS["`Stream - Management - Service - (SMS)`"] - PVSD - end - subgraph "Workflow Manager" - WorkflowManager[Workflow Manager] - DasmonListener[Dasmon listener] - Database[(DB)] - end - subgraph Autoreduction - Autoreducers-->|REST API|ONCat - LiveDataServer<-->LiveDataDB[(DB)] - LiveReduce - end - subgraph Frontend - WebMon["`WebMon - monitor.sns.gov`"] - end - SMS-->LiveReduce - TranslationService-.->|ActiveMQ|WorkflowManager - DASMON-.->|ActiveMQ|DasmonListener - DASMON-->Database - WorkflowManager<-.->|ActiveMQ|Autoreducers - Autoreducers-->|REST API|LiveDataServer - Database-->WebMon - ONCat-->|REST API|WebMon - LiveDataServer-->|REST API|WebMon - LiveReduce-->|REST API|LiveDataServer - DasmonListener-->Database - WorkflowManager-->Database - TranslationService-->FileArchive - FileArchive<-->Autoreducers - style DAS fill:#D3D3D3, stroke-dasharray: 5 5 + flowchart TB + TranslationService["`Streaming + Translation + Client + (STC)`"] + SMS["`Stream + Management + Service + (SMS)`"] + Broker[ActiveMQ broker] + Broker<-->Autoreducers + Broker<-->WorkflowManager[Workflow Manager] + Broker<-->DasmonListener[Dasmon listener] + Broker<-->DASMON + Broker<-->PVSD + Broker<-->TranslationService + Broker<-->SMS