Merge pull request #17 from henrypinkard/main

Documentation improvements

docs/devices.rst

@@ -0,0 +1,47 @@
+.. _devices:
+
+
+#######
+Devices
+#######
+
+Devices in ExEngine are software representations of hardware components in a microscope system. When possible, they provide a consistent way to interact with diverse equipment, abstracting away the complexities of individual hardware implementations. When not possible, devices can additionally expose specialized APIs specific to individual components.
+
+ExEngine supports multiple **backends** - individual devices or libraries of devices (e.g., Micro-Manager). The method to create devices depends on the specific backend in use.
+
+
+Here's a minimal example using the Micro-Manager backend:
+
+.. code-block:: python
+
+   from mmpycorex import create_core_instance
+   from exengine.kernel.executor import ExecutionEngine
+   from exengine.backends.micromanager.mm_device_implementations import MicroManagerSingleAxisStage
+
+   # Create the ExecutionEngine
+   executor = ExecutionEngine()
+
+   # Initialize Micro-Manager core
+   create_core_instance(config_file='MMConfig_demo.cfg')
+
+   # Access Micro-Manager device
+   z_stage = MicroManagerSingleAxisStage()
+
+   z_stage.set_position(1234.56)
+
+
+Device Hierarchies
+""""""""""""""""""
+
+Devices in ExEngine exist in hierarchies. All devices must inherit from the exengine.Device base class. Further functionality can be standardized by inheriting from one or more specific device type classes. For example, ``MicroManagerSingleAxisStage`` is a subclass of ``exengine.SingleAxisPositioner``, which is itself a subclass of ``exengine.Device``.
+
+The advantage of this hierarchical structure is that it standardizes functionality, allowing code to be written for generic device types (e.g., a single axis positioner) that will work with many different device libraries. This approach enables a common API across different libraries of devices, similar to Micro-Manager's device abstraction layer but on a meta-level - spanning multiple device ecosystems rather than just devices within a single project. However, ExEngine's device system is designed to be adaptable. While adhering to the standardized interfaces offers the most benefits, users can still leverage many advantages of the system without implementing these specialized APIs.
+
+
+TODO: thread standardization features of devices (and how to turn off)
+
+TODO: calling functions on devices directly
+
+TODO: link to guide to adding backends
+
+TODO: transition to more complex with events

docs/events.rst

@@ -0,0 +1,12 @@
+.. _events:
+
+
+#######
+Events
+#######
+
+
+Events are modular units of instructions and or computation. They can be as simple as a single command like moving the position of a hardware device, or contain multiple steps and computation. They provide building blocks to create more complex experimental workflows.
+
+
+TODO: what else should be said about events?

docs/futures.rst

@@ -0,0 +1,8 @@
+.. _futures:
+
+
+########
+Futures
+########
+
+Futures represent the outcome of asynchronous operations. They provide a way to handle long-running tasks without blocking the main execution thread. Futures allow you to submit events for execution and then either wait for their completion or continue with other tasks, checking back later for results. This enables efficient, non-blocking execution of complex workflows.

docs/index.rst

@@ -2,19 +2,13 @@
 ExEngine: An execution engine for microscope control
 #####################################################
 
-.. toggle:: Click to expand
-
-    This is the collapsible content. It is hidden by default, until the toggle button is clicked.
 
 
 
 .. toctree::
    :maxdepth: 1
    :caption: Contents:
 
-   motivation
-   overview
-   setup
+   introduction
    usage
    extending
-   examples

docs/introduction.rst

@@ -0,0 +1,74 @@
+.. _introduction:
+
+##################
+Introduction
+##################
+
+
+ExEngine is a general-purpose library for microscopy hardware control and data acquisition. It explicitly separates the specification of instructions from their execution, meaning it is not inherently specialized for any one type of microscopy and can be readily adapted to a variety of contexts. It is designed to balance between simplicity and expressiveness, enabling researchers to create sophisticated microscopy experiments without getting bogged down in low-level details.
+
+.. TODO: this first paragraph can be improved
+
+
+.. Key benefits:	
+.. 	Larger device libraries
+.. 	Existing libraries often designed for scripting/GUI. Extending to workflow manager requires custom
+.. 	Reuse of code
+.. Designed to enable convenience of development but scales to AI based workflows
+
+
+
+
+Motivation
+================
+
+
+.. raw:: html
+
+    <div style="text-align: center; max-width: 100%;">
+        <object type="image/svg+xml" data="_static/exengine_bigpicture.svg" style="width: 100%; height: auto;"></object>
+        <p style="font-style: italic; font-size: 0.9em; color: #555;"><b>The benefits of ExEngine</b></p>
+    </div>
+
+
+
+Advanced microscopes require sophisticated software to function effectively. As microscopy techniques evolve, so do the challenges of control software:
+
+1. **Scalability**: 
+   The evolution from proof-of-concept to production systems in microscopy creates a dynamic set of software requirements. In the early stages, prototype systems demand low-level hardware control and high customizability. As systems mature, the focus shifts: the hardware control layer needs to stabilize, while flexibility becomes crucial for higher-level experimental design.
+
+   This transition often results in significant software discontinuities. A common pattern illustrates this challenge: many novel microscopy systems are initially developed using LabVIEW for its rapid prototyping capabilities. However, as these systems advance towards production, teams frequently find themselves completely rewriting the control software, often transitioning to platforms like Micro-Manager. 
+
+
+2. **AI-driven Smart Microscopy**: 
+   Most current user interfaces for microscopes are designed for human operators, AI shows promise in automating complex workflows. However, we lack standardized methods to accurately capture and describe routine and complex experimental workflows, hindering their potential for automation.
+
+3. **Ecosystem Lock-in**: 
+   Current solutions often create vertically integrated control packages, tying users to specific ecosystems. These typically include:
+
+   a. A device control layer
+   b. A layer for controlling and synchronizing multiple devices
+   c. A graphical user interface
+
+   While device layers may be reusable, the acquisition engine layer is often bespoke and hard coded for specific device libraries, limiting interoperability.
+
+
+
+
+Design Philosophy
+"""""""""""""""""
+
+- **Backend-agnosticism**: While Micro-Manager is supported, ExEngine can seamlessly integrate with any (or multiple) hardware control backends, providing equal functionality and performance across different systems.
+- **Non-obtrusiveness**: Minimal assumptions about hardware devices, allowing new device support to be added with maximum flexibility and minimal effort.
+- **Versatility**: Multiple sets of independently usable features and levels of integration possibilities.
+- **Extensibility**: Hardware setup is abstracted away from execution capabilities, allowing new capabilities with entirely different functionality to be added.
+
+
+Benefits of ExEngine
+""""""""""""""""""""
+
+1. **Broad and extensible hardware support** Supports the full suite of hardware in micro-manager, but is not designed specially for them. More backends can allow more devices to be used, and used in combination
+
+2. **Parallelization and thread safety** You need parallelism with high performance software, but this often leads to hard to diagnose problems because devices that were designed with concurrency in mind get accessed from multiple places in the software, leading to problems. ExEngine can automatically handle this by rerouting calls to devices through a common pool of threads. Significantly, this happens automatically under the hood, adding little to no complexity to writing control code.
+
+3. **Built for humans and AI**: Enables complete tracking of commands sent to hardware and data received, without complex additional code. This facilitates tracking and automation of microscopes using AI.

docs/notifications.rst

@@ -0,0 +1,9 @@
+.. _notifications:
+
+
+##############
+Notifications
+##############
+
+
+Notifications provide a mechanism for asynchronous communication within the system. They allow devices, events, and other components to broadcast updates about their status or important occurrences. This feature enables reactive programming patterns, allowing your software to respond dynamically to changes in the system state or experimental conditions.

docs/usage.rst

@@ -4,64 +4,51 @@
 Usage
 ######
 
-ExEngine is built around four key abstractions: Devices, Events, Futures, and Notifications.
-
-Devices
-=======
-Devices in ExEngine are software representations of hardware components in a microscope system. When possible, they provide a consistent way to interact with diverse equipment, abstracting away the complexities of individual hardware implementations. When not possible, devices can additionally expose specialized APIs specific to individual components.
-
-ExEngine supports multiple **backends** - individual devices or libraries of devices (e.g., Micro-Manager). The method to create devices depends on the specific backend in use.
-
-
-Here's a minimal example using the Micro-Manager backend:
 
-.. code-block:: python
 
-   from mmpycorex import create_core_instance
-   from exengine.kernel.executor import ExecutionEngine
-   from exengine.backends.micromanager.mm_device_implementations import MicroManagerSingleAxisStage
 
-   # Create the ExecutionEngine
-   executor = ExecutionEngine()
+Installation and setup
+========================
 
-   # Initialize Micro-Manager core
-   create_core_instance(config_file='MMConfig_demo.cfg')
 
-   # Access Micro-Manager device
-   z_stage = MicroManagerSingleAxisStage()
+TODO
 
-   z_stage.set_position(1234.56)
 
+Key features
+=============
 
-Device Hierarchies
-""""""""""""""""""
-
-Devices in ExEngine exist in hierarchies. All devices must inherit from the exengine.Device base class. Further functionality can be standardized by inheriting from one or more specific device type classes. For example, ``MicroManagerSingleAxisStage`` is a subclass of ``exengine.SingleAxisPositioner``, which is itself a subclass of ``exengine.Device``.
-
-The advantage of this hierarchical structure is that it standardizes functionality, allowing code to be written for generic device types (e.g., a single axis positioner) that will work with many different device libraries. This approach enables a common API across different libraries of devices, similar to Micro-Manager's device abstraction layer but on a meta-level - spanning multiple device ecosystems rather than just devices within a single project. However, ExEngine's device system is designed to be adaptable. While adhering to the standardized interfaces offers the most benefits, users can still leverage many advantages of the system without implementing these specialized APIs.
+ExEngine is built around four key abstractions: Devices, Events, Futures, and Notifications.
 
 
-TODO: thread standardization features of devices (and how to turn off)
+* :ref:`Devices <devices>`: Hardware components that can be controlled, such as cameras, stages, or light sources.
+* :ref:`Events <events>`: Encapsulated units of work that represent actions to be executed, ranging from simple actions like moving a stage or capturing an image to complex ones like running an autofocus routine.
+* :ref:`Futures <futures>`: Objects that represent the result of asynchronous operations, allowing for management and retrieval of event outcomes.
+* :ref:`Notifications <notifications>`: Asynchronous messages that provide real-time updates on system status and progress.
 
-TODO: calling functions on devices directly
 
-TODO: link to guide to adding backends
+.. raw:: html
 
-TODO: transition to more complex with events
+    <div style="text-align: center; max-width: 100%;">
+        <object type="image/svg+xml" data="_static/exengine_arch.svg" style="width: 100%; height: auto;"></object>
+        <p style="font-style: italic; font-size: 0.9em; color: #555;"><b>Overview of the main components of ExEngine:</b> Events are submitted to the Execution Engine, which handles their execution on one or more internal threads. Upon execution, events can control hardware devices and produce output data. Futures allow for the management and retrieval of the results of these asynchronous operations. Notifications are produced asynchronously, enabling user applications to monitor progress and system status updates in real-time.</p>
+    </div>
 
 
-Events
-======
-Events are modular units of instructions and or computation. They can be as simple as a single command like moving the position of a hardware device, or contain multiple steps and computation. They provide building blocks to create more complex experimental workflows.
 
 
-TODO: what else should be said about events?
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
 
+   devices
+   events
+   futures
+   notifications
 
-Notifications
-=============
-Notifications provide a mechanism for asynchronous communication within the system. They allow devices, events, and other components to broadcast updates about their status or important occurrences. This feature enables reactive programming patterns, allowing your software to respond dynamically to changes in the system state or experimental conditions.
+Examples
+==========
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
 
-Futures
-=======
-Futures represent the outcome of asynchronous operations. They provide a way to handle long-running tasks without blocking the main execution thread. Futures allow you to submit events for execution and then either wait for their completion or continue with other tasks, checking back later for results. This enables efficient, non-blocking execution of complex workflows.
+   examples