Skip to content

Commit

Permalink
address comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@backmari
backmari committed Sep 6, 2024
1 parent eae3bd2 commit 3ce28fb
Show file tree
Hide file tree
Showing 2 changed files with 138 additions and 136 deletions.
133 changes: 77 additions & 56 deletions docs/developer/design/communication_flows.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,29 +12,51 @@ Run post-processing
...................

The sequence diagram below describes the communication flow as a run gets post-processed.
The post-processing workflow is triggered when the Translation Service has finished translating and
sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the instrument, IPTS, run number and
location of the Nexus file.
The post-processing workflow is triggered when the Translation Service has finished translating the
data stream to NeXus and sends a message on the queue ``POSTPROCESS.DATA_READY`` specifying the
instrument, IPTS, run number and location of the NeXus file.

The post-processing workflow for the instrument is configurable in the database table ``Task``.
The post-processing workflow for the instrument is configurable in the database table
``report_task``.
The diagram shows the three post-processing steps that are available: autoreduction, cataloging of
raw data in `ONCat <https://oncat.ornl.gov/>`_ and cataloging of reduced data in
`ONCat <https://oncat.ornl.gov/>`_.

.. mermaid::

sequenceDiagram
participant Translation Service
participant Workflow Manager
participant Autoreducer
participant ONCat

Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY
Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY
Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED
Autoreducer->>ONCat: pyoncat
Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE
Workflow Manager->>Autoreducer: REDUCTION.DATA_READY
Autoreducer->>Workflow Manager: REDUCTION.STARTED
Autoreducer->>Autoreducer: run reduction
Autoreducer->>Workflow Manager: REDUCTION.COMPLETE
Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED
Autoreducer->>ONCat: pyoncat
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE

..
.. mermaid::
sequenceDiagram
participant Translation Service
participant Workflow Manager
participant Autoreducer
participant ONCat

Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY
opt Cataloging
Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY
Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED
Autoreducer->>ONCat: pyoncat
Note over Autoreducer: Ingest in ONCat
Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE
end
opt Autoreduction
Expand All @@ -46,37 +68,10 @@ raw data in `ONCat <https://oncat.ornl.gov/>`_ and cataloging of reduced data in
opt Reduced data cataloging
Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED
Autoreducer->>ONCat: pyoncat
Note over Autoreducer: Ingest in ONCat
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE
end

.. mermaid::

sequenceDiagram
participant Translation Service
participant Workflow Manager
participant Autoreducer

Translation Service->>Workflow Manager: POSTPROCESS.DATA_READY
opt Cataloging
Workflow Manager->>Autoreducer: CATALOG.ONCAT.DATA_READY
Autoreducer->>Workflow Manager: CATALOG.ONCAT.STARTED
Note over Autoreducer: Ingest in ONCat
Autoreducer->>Workflow Manager: CATALOG.ONCAT.COMPLETE
end
opt Autoreduction
Workflow Manager->>Autoreducer: REDUCTION.DATA_READY
Autoreducer->>Workflow Manager: REDUCTION.STARTED
Note over Autoreducer: Execute autoreduction script
Autoreducer->>Workflow Manager: REDUCTION.COMPLETE
end
opt Reduced data cataloging
Workflow Manager->>Autoreducer: REDUCTION_CATALOG.DATA_READY
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.STARTED
Note over Autoreducer: Ingest in ONCat
Autoreducer->>Workflow Manager: REDUCTION_CATALOG.COMPLETE
end


Configuring the autoreduction
.............................
Expand All @@ -92,24 +87,28 @@ parameters for instruments that have implemented
actor Instrument Scientist
participant WebMon
participant Autoreducer
participant HFIR/SNS File archive

Instrument Scientist->>WebMon: Submit form with parameter values
WebMon->>Autoreducer: REDUCTION.CREATE_SCRIPT
Note over Autoreducer: Update parameter values in<br/>instrument reduction script
Autoreducer->>HFIR/SNS File archive: Update instrument reduction script

DASMON
------
DASMON, from Data Acquisition (DAQ) System Monitor, provides process variable (PV) updates from the
beamlines to WebMon. Due to the high volume of PV updates, DASMON writes directly to the database.
DASMON, from Data Acquisition (DAQ) System Monitor, provides instrument status and process variable
(PV) updates from the beamlines to WebMon. DASMON connects to the WebMon message broker to pass
status information, for example the current run number and count rate, to Dasmon listener. Due to
the high volume of PV updates, DASMON writes PV:s directly to the PostgreSQL database.

.. mermaid::

sequenceDiagram
participant DASMON
participant Workflow DB
participant Dasmon listener
participant Workflow DB
DASMON->>Workflow DB: PV update
DASMON->>Dasmon listener: Heartbeats
DASMON->>Dasmon listener: Instrument status
DASMON->>Workflow DB: Instrument status

Dasmon listener
---------------
Expand All @@ -134,20 +133,21 @@ Heartbeats from services
Dasmon listener subscribes to heartbeats from the other services. There is a mechanism for alerting
admins by email when a service has missed heartbeats (needs to be verified that this still works).

.. mermaid::
..
.. mermaid::
sequenceDiagram
participant Other services
participant Dasmon listener
participant Workflow DB
actor Subscribed users
loop Every N s
Other services->>Dasmon listener: Heartbeat
Dasmon listener->>Workflow DB: Status update
end
opt Service has 3 missed heartbeats
Dasmon listener->>Subscribed users: Email
end
sequenceDiagram
participant Other services
participant Dasmon listener
participant Workflow DB
actor Subscribed users
loop Every N s
Other services->>Dasmon listener: Heartbeat
Dasmon listener->>Workflow DB: Status update
end
opt Service has 3 missed heartbeats
Dasmon listener->>Subscribed users: Email
end


.. mermaid::
Expand All @@ -170,14 +170,14 @@ admins by email when a service has missed heartbeats (needs to be verified that
DasmonListener-->WorkflowDB
DasmonListener-.->|if missed 3 heartbeats|InstrumentScientist

WebMon interaction with Live Data Server
Live Data Server
----------------------------------------

WebMon has two modes of interaction with Live Data Server: publish (save) plots to the Live Data
Server database and display (fetch) plots from the database.

Publish to Live Data Server
...........................
Publish to Live Data Server from autoreduction script
.....................................................

The instrument-specific autoreduction script can optionally publish plots (in either JSON format
or HTML div) to Live Data Server.
Expand All @@ -192,9 +192,30 @@ or HTML div) to Live Data Server.
WebMon->>Autoreducer: REDUCTION.DATA_READY
opt Publish plot
Autoreducer->>Live Data Server: publish_plot
Note over Live Data Server: Store plot in DB
end

Publish to Live Data Server from live data stream
.................................................

Livereduce (https://github.com/mantidproject/livereduce/) allows scientists to reduce
data from an ongoing experiment, i.e. before translation to NeXus, by connecting to the live data
stream from the Stream Management Service (SMS). The instrument-specific processing
script can make the results available in WebMon by publishing plots to Live Data Server.

.. mermaid::

sequenceDiagram
participant SMS
participant Livereduce
participant Live Data Server

SMS->>Livereduce: data stream
loop Every N minutes
Livereduce->>Livereduce: run processing script
Livereduce->>Live Data Server: publish plot
end


Display plot from Live Data Server
................................

Expand Down
141 changes: 61 additions & 80 deletions docs/developer/design/overview.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ Design Overview
High-level architecture
-----------------------

The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are
services owned by the Data Acquisition System team that feed information to WebMon but are not part
of WebMon. The services communicate mainly through an ActiveMQ message broker. The communication
flow is described in :ref:`communication_flows`.
The diagram below describes the high-level architecture of WebMon, including some external resources
that WebMon interacts with. The gray box labeled "DAS" are services owned by the Data Acquisition
System team that feed information to WebMon. `ONCat <https://oncat.ornl.gov/>`_ is the experiment
data catalog, which the autoreducers catalog runs to and the frontend fetches run metadata from.
The autoreducers access instrument-specific reduction scripts and experiment data files on the
HFIR/SNS file archive. The autoreducers also write reduced data files and logs to the file archive.

.. mermaid::

Expand All @@ -30,90 +32,69 @@ flow is described in :ref:`communication_flows`.
Management
Service
(SMS)`"]
PVSD
end
subgraph "Workflow Manager"
WorkflowManager[Workflow Manager]
DasmonListener[Dasmon listener]
Database[(DB)]
end
subgraph Autoreduction
Autoreducers-->|cataloging|ONCat
LiveDataServer<-->LiveDataDB[(DB)]
LiveReduce
end
subgraph Frontend
WebMon["`WebMon
monitor.sns.gov`"]
end
SMS-->|live data stream|LiveReduce
TranslationService-->|trigger reduction|WorkflowManager
DASMON-->|status|DasmonListener
DASMON-->|PV updates|Database
WorkflowManager<-->|instructions|Autoreducers
Autoreducers-->|publish plots|LiveDataServer
WorkflowManager[Workflow Manager]
DasmonListener[Dasmon listener]
Database[(Workflow DB)]
Autoreducers-->ONCat
LiveDataServer-->WebMon
LiveDataServer<-->LiveDataDB[(LiveData DB)]
LiveReduce
WebMon["`WebMon
frontend`"]
SMS-->LiveReduce
TranslationService-->WorkflowManager
DASMON-->DasmonListener
DASMON-->Database
WorkflowManager-->Database
WorkflowManager<-->Autoreducers
Autoreducers-->LiveDataServer
Database-->WebMon
ONCat-->|run metadata|WebMon
LiveDataServer-->|plots|WebMon
LiveReduce-->|publish plots|LiveDataServer
ONCat-->WebMon
LiveReduce-->LiveDataServer
DasmonListener-->Database
WorkflowManager-->Database
TranslationService-->|NeXus file|FileArchive
TranslationService-->FileArchive
FileArchive<-->Autoreducers
style DAS fill:#D3D3D3, stroke-dasharray: 5 5
classDef webMonStyle fill:#FFFFE0
class WorkflowManager,DasmonListener,Database,Autoreducers,LiveDataServer,LiveReduce,WebMon,LiveDataDB webMonStyle

subgraph Legend
direction LR
External["External resource"]
Internal["Internal resource"]
External ~~~ Internal
end
LiveReduce ~~~ External
style Legend fill:#FFFFFF,stroke:#000000
class Internal webMonStyle

High-level architecture
-----------------------

The diagram below describes the high-level architecture of WebMon. The gray box labeled "DAS" are
services owned by the Data Acquisition System team that feed information to WebMon, e.g. to trigger
autoreduction when a run is finished. The services mainly communicate through an ActiveMQ message
broker. The communication flow is described in :ref:`communication_flows`.
Message broker
--------------

WebMon uses an `ActiveMQ <https://activemq.apache.org/>`_ message broker for communication between
services. The message broker also serves as a load balancer by distributing post-processing jobs to
the available autoreducers in a round-robin fashion.

Service communication flows are described in :ref:`communication_flows`.

.. mermaid::

flowchart LR
FileArchive[("`SNS/HFIR
File archive`")]
subgraph DAS
DASMON
TranslationService["`Streaming
Translation
Client
(STC)`"]
SMS["`Stream
Management
Service
(SMS)`"]
PVSD
end
subgraph "Workflow Manager"
WorkflowManager[Workflow Manager]
DasmonListener[Dasmon listener]
Database[(DB)]
end
subgraph Autoreduction
Autoreducers-->|REST API|ONCat
LiveDataServer<-->LiveDataDB[(DB)]
LiveReduce
end
subgraph Frontend
WebMon["`WebMon
monitor.sns.gov`"]
end
SMS-->LiveReduce
TranslationService-.->|ActiveMQ|WorkflowManager
DASMON-.->|ActiveMQ|DasmonListener
DASMON-->Database
WorkflowManager<-.->|ActiveMQ|Autoreducers
Autoreducers-->|REST API|LiveDataServer
Database-->WebMon
ONCat-->|REST API|WebMon
LiveDataServer-->|REST API|WebMon
LiveReduce-->|REST API|LiveDataServer
DasmonListener-->Database
WorkflowManager-->Database
TranslationService-->FileArchive
FileArchive<-->Autoreducers
style DAS fill:#D3D3D3, stroke-dasharray: 5 5
flowchart TB
TranslationService["`Streaming
Translation
Client
(STC)`"]
SMS["`Stream
Management
Service
(SMS)`"]
Broker[ActiveMQ broker]
Broker<-->Autoreducers
Broker<-->WorkflowManager[Workflow Manager]
Broker<-->DasmonListener[Dasmon listener]
Broker<-->DASMON
Broker<-->PVSD
Broker<-->TranslationService
Broker<-->SMS

0 comments on commit 3ce28fb

Please sign in to comment.