Snapshot computation (#101)

* Initial commit for snapshot commputation

* Snapshot: change import of module

* Adapt SnapshotComputation further

* Adapt Snapshot test

* Snapshot: Formatting

* Prepare Snapshot class further

* First working version of snapshot computation

* Add snapshot test, improve snapshot and restructure

* Add snapshot test, improve snapshot and restructure part 2

* Improve logging, comments and minor structural improvements

* Snapshot: extend setup and github actions

* Move Snapshot into Micro Manager and update execution

* Add post-processing script option and restructure snapshot example

* Add post-processing to snapshot config file

* Make snapshot executable and rename snapshot files

* Restructure snapshot computation

* Restructure config

* Update snapshot computation still without chunking

* Remove unintentional print statement

* Update comments in tests

* Read only the necessary input on each process

* Clarify text and names

* Add HDF5 dependency

* Make h5py optional dependency

* Fix h5py optional

* Initial commit for snapshot commputation

* Snapshot: change import of module

* Adapt SnapshotComputation further

* Snapshot: Formatting

* Prepare Snapshot class further

* First working version of snapshot computation

* Add snapshot test, improve snapshot and restructure

* Add snapshot test, improve snapshot and restructure part 2

* Improve logging, comments and minor structural improvements

* Move Snapshot into Micro Manager and update execution

* Add post-processing script option and restructure snapshot example

* Make snapshot executable and rename snapshot files

* Restructure snapshot computation

* Restructure config

* Update snapshot computation still without chunking

* Remove unintentional print statement

* Update comments in tests

* Make h5py optional dependency

* Fix h5py optional

* Change snapshot case in main

* Move dt to simulation parameters in Snapshot and snapshot specific dt

* Fix init

* Fix issue of using member variable in Config class

* Fix tests

* Apply suggestions from code review

Co-authored-by: Ishaan Desai <[email protected]>

* Apply review suggestions

* Remove output dir from gitignore

* Extend CHANGELOG.md

* Update CHANGELOG.md

---------

Co-authored-by: Ishaan Desai <[email protected]>

.github/workflows/run-unit-tests.yml

@@ -22,20 +22,42 @@ jobs:
           apt-get -qq install python3-dev python3-pip git python-is-python3 pkg-config
           pip3 install --upgrade pip
 
-      - name: Install Micro Manager and uninstall pyprecice
-        working-directory: micro-manager
+      - name: Install Micro Manager and run micro_manager unit test
+        working-directory: micro-manager/
+        run: |
+          pip3 install --user .
+          pip3 uninstall -y pyprecice
+          cd tests/unit
+          python3 -m unittest test_micro_manager.py
+
+      - name: Install Micro Manager and run interpolation unit test
+        working-directory: micro-manager/
         run: |
           pip3 install --user .[sklearn]
           pip3 uninstall -y pyprecice
+          cd tests/unit
+          python3 -m unittest test_interpolation.py
 
-      - name: Run micro_manager unit test
-        working-directory: micro-manager/tests/unit
-        run: python3 -m unittest test_micro_manager.py
+      - name: Install Micro Manager and run micro simulation crash unit test
+        working-directory: micro-manager/
+        run: |
+          pip3 install --user .
+          pip3 uninstall -y pyprecice
+          cd tests/unit
+          python3 -m unittest test_micro_simulation_crash_handling.py
 
-      - name: Run interpolation unit test
-        working-directory: micro-manager/tests/unit
-        run: python3 -m unittest test_interpolation.py
+      - name: Install Micro Manager and run HDF5 read and write unit tests
+        working-directory: micro-manager/
+        run: |
+          pip3 install --user .[snapshot]
+          pip3 uninstall -y pyprecice
+          cd tests/unit
+          python3 -m unittest test_hdf5_functionality.py
 
-      - name: Run micro simulation crash unit test
-        working-directory: micro-manager/tests/unit
-        run: python3 -m unittest test_micro_simulation_crash_handling.py
+      - name: Install Micro Manager and run snapshot_computation unit tests
+        working-directory: micro-manager/
+        run: |
+          pip3 install --user .[snapshot]
+          pip3 uninstall -y pyprecice
+          cd tests/unit
+          python3 -m unittest test_snapshot_computation.py

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## latest
 
+- Add `MicroManagerSnapshot` enabling snapshot computation and storage of microdata in HDF5 format https://github.com/precice/micro-manager/pull/101
 - Make `sklearn` an optional dependency
 - Move the config variable `micro_dt` from the coupling parameters section to the simulation parameters section https://github.com/precice/micro-manager/pull/114
 - Set time step of micro simulation in the configuration, and use it in the coupling https://github.com/precice/micro-manager/pull/112

examples/clean-example.sh

@@ -4,3 +4,5 @@ rm -rfv precice-profiling/
 rm -fv *-events.json
 rm -fv cpp-dummy/micro-manager.log
 rm -fv cpp-dummy/micro_dummy.cpython-310-x86_64-linux-gnu.so
+rm -r -fv snapshot-example/output
+rm -fv snapshot-example/*.log

examples/snapshot-config.json

@@ -0,0 +1,14 @@
+{
+    "micro_file_name": "python-dummy/micro_dummy",
+    "coupling_params": {
+        "parameter_file_name": "parameter.hdf5",
+        "read_data_names": {"macro-scalar-data": "scalar", "macro-vector-data": "vector"},
+        "write_data_names": {"micro-scalar-data": "scalar", "micro-vector-data": "vector"}
+    },
+    "simulation_params": {
+        "micro_dt": 1.0
+    },
+    "snapshot_params": {
+        "post_processing_file_name": "snapshot_postprocessing"
+    }
+}

examples/snapshot_postprocessing.py

@@ -0,0 +1,24 @@
+"""
+Post-processing
+In this script a post-processing step is defined.
+A script like this can be used to post-process the simulation output before writing it to a file,
+if this is not done in the micro simulation itself.
+"""
+
+
+class Postprocessing:
+    def postprocessing(sim_output):
+        """Post-process the simulation output.
+
+        Parameters
+        ----------
+        sim_output : dict
+            Raw simulation output.
+
+        Returns
+        -------
+        sim_output : dict
+            Post-processed simulation output.
+        """
+        sim_output["micro-scalar-data"] = sim_output["micro-scalar-data"] + 20
+        return sim_output

micro_manager/__init__.py

@@ -4,19 +4,38 @@
 from .config import Config
 from .micro_manager import MicroManagerCoupling
 
+try:
+    from .snapshot.snapshot import MicroManagerSnapshot
+
+    is_snapshot_possible = True
+except ImportError:
+    is_snapshot_possible = False
+
+
+def main():
 
-def main() -> None:
     parser = argparse.ArgumentParser(description=".")
     parser.add_argument(
         "config_file", type=str, help="Path to the JSON config file of the manager."
     )
+    parser.add_argument(
+        "--snapshot", action="store_true", help="compute offline snapshot database"
+    )
 
     args = parser.parse_args()
     config_file_path = args.config_file
     if not os.path.isabs(config_file_path):
         config_file_path = os.getcwd() + "/" + config_file_path
 
-    manager = MicroManagerCoupling(config_file_path)
+    if not args.snapshot:
+        manager = MicroManagerCoupling(config_file_path)
+    else:
+        if is_snapshot_possible:
+            manager = MicroManagerSnapshot(config_file_path)
+        else:
+            raise ImportError(
+                "The Micro Manager snapshot computation requires the h5py package."
+            )
 
     manager.initialize()