add documentation (#24)

.github/workflows/docs.yml

@@ -0,0 +1,37 @@
+name: Build documentation ⚙️
+
+on:
+  push:
+    branches:
+      - master
+    paths:
+      - 'docs/**'
+  pull_request:
+    branches:
+      - master
+    paths:
+      - 'docs/**'
+  release:
+    types:
+      - released
+
+jobs:
+  main:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        include:
+          - python-version: '3.10'
+    steps:
+    - uses: actions/checkout@master
+    - uses: actions/setup-python@v5
+      name: Setup Python ${{ matrix.python-version }}
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install requirements 📦
+      run: |
+        #pip3 install -r requirements.txt
+        pip3 install -r docs/requirements.txt
+    - name: build docs 🏗️
+      run: cd docs && make html
+

.readthedocs.yaml

@@ -0,0 +1,16 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+version: 2
+
+sphinx:
+  configuration: docs/source/conf.py
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+python:
+  install:
+    - requirements: docs/requirements.txt

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/requirements.txt

@@ -0,0 +1,3 @@
+sphinx
+sphinx_rtd_theme
+readthedocs-sphinx-search

docs/source/administration.rst

@@ -0,0 +1,48 @@
+.. _administration:
+
+Administration
+==============
+
+Docker Compose
+--------------
+
+As a Docker Compose based application, ``wis2-gdc`` uses the ``docker compose`` command to manage services.  The Docker Compose setups are defined in the following configurations:
+
+- ``docker-compose.yml``
+- ``docker-compose.override.yml``
+
+To adjust service ports, edit ``docker-compose.override.yml`` accordingly.
+
+As a convenience, the ``Makefile`` in the root directory provides shortcuts to manage the Docker Compose setup (encapsulating the various ``docker compose`` commands).
+
+.. code-block:: bash
+
+   # build all images
+   make build
+
+   # build all images (no cache)
+   make force-build
+
+   # start all containers
+   make up
+
+   # reinitialize backend
+   make reinit-backend
+
+   # start all containers in dev mode
+   make dev
+
+   # view all container logs in realtime
+   make logs
+
+   # login to the wis2-gdc-management container
+   make login
+
+   # restart all containers
+   make restart
+
+   # shutdown all containers
+   make down
+
+   # remove all volumes
+   make rm

docs/source/architecture.rst

@@ -0,0 +1,55 @@
+.. _architecture:
+
+Architecture
+============
+
+``wis2-gdc`` is implemented in the spirit of the `Twelve-Factor App methodology`_.
+
+``wis2-gdc`` is a `Docker`_ and `Python`_-based platform in support of WIS2 metadata
+management, publication, and discovery.
+
+High level system context
+-------------------------
+
+The following diagram provides a high level overview of the main functions
+of ``wis2-gdc``:
+
+.. figure:: ../architecture/c4.container.png
+   :scale: 70%
+   :alt: wis2-gdc architecture
+   :align: center
+
+How ``wis2-gdc`` works
+----------------------
+
+As a containerized solution, ``wis2-gdc`` provides functionality via the following services:
+
+* **wis2-gdc-management**: ingests, validates and publishes discovery metadata published from a WIS2 Global Broker instance
+* **wis2-gdc-api**: API powered by `pygeoapi`_
+* **wis2-gdc-monitoring**: monitoring
+
+  * **metrics-collector**: metrics collector
+  * **prometheus**: metrics scraper
+  * **grafana**: analytics and visualization
+* **wis2-gdc-broker**: MQTT broker
+* **wis2-gdc-backend**: API search engine backend (default Elasticsearch)
+
+``wis2-gdc`` is primarily an event-driven system, also providing interactive functionality for managing WIS2 metadata if required.
+
+Workflows
+^^^^^^^^^
+
+``wis2-gdc`` starts up by connecting to a WIS2 Global Broker (GB), subscribing to notifications for WIS2 metadata (WMO Core Metadata Profile [WCMP2]).  On receipt of WIS2 Notification Messages (WNM) for metadata, ``wis2-gdc`` will ingest, validate and publish a WCMP2 record to its catalogue API.  In addition, WCMP2 update and deletion is supported with the appropriate WNM.
+
+The ``wis2-gdc`` monitoring capability collects and provides metrics on WCMP2 that is scraped by the WIS2 Global Monitor (GM).
+
+``wis2-gdc`` also provides its own MQTT broker that provides WCMP2 compliance and key performance indicator (KPI) reports.  The WIS2 GB subscribes to the ``wis2-gdc`` broker in order to publish these reports for data providers in support of quality assessment, scoring and corrective action.  The ``wis2-gdc`` broker also acts as an internal message bus for internal event handling.
+
+The ``wis2-gdc`` API provides an OGC API - Records endpoint that is OGC compliant.  The API provides search engine capability for WIS2 metadata, also providing OGC API - Processes functionality for WCMP2 validation and KPI quality assessment.
+
+
+
+.. _`Twelve-Factor App methodology`: https://12factor.net
+.. _`Docker`: https://www.docker.com
+.. _`Python`: https://www.python.org
+.. _`pygeoapi`: https://pygeoapi.io

docs/source/conf.py

@@ -0,0 +1,79 @@
+# =================================================================
+#
+# Authors: Tom Kralidis <[email protected]>
+#
+# Copyright (c) 2024 Tom Kralidis
+# Copyright (c) 2024 Angelos Tzotsos
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation
+# files (the "Software"), to deal in the Software without
+# restriction, including without limitation the rights to use,
+# copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following
+# conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+# =================================================================
+
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = 'wis2-gdc'
+author = 'Tom Kralidis'
+license = 'This work is licensed under a Creative Commons Attribution 4.0 International License'  # noqa
+release = '0.1.0'
+copyright = '2023-present, ' + author + ' ' + license
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ['sphinx.ext.autodoc',
+              'sphinx.ext.todo',
+              'sphinx.ext.coverage',
+              'sphinx.ext.ifconfig',
+              'sphinx.ext.viewcode',
+              'sphinx.ext.doctest',
+              'sphinx.ext.githubpages',
+              'sphinx.ext.intersphinx']
+
+today_fmt = '%Y-%m-%d'
+
+templates_path = ['_templates']
+exclude_patterns = []
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']
+
+html_favicon = '_static/favicon.ico' 
+
+#html_sidebars = { 
+#    '**': [
+#        'relations.html',  # needs 'show_related': True theme option to display
+#        'searchbox.html',
+#        'indexsidebar.html']
+#}
+
+pygments_style = 'sphinx'
+
+

docs/source/configuration.rst

@@ -0,0 +1,80 @@
+.. _configuration:
+
+Configuration
+=============
+
+``wis2-gdc`` configuration is driven by the following environment variables, which are managed in ``wis2-gdc.env``:
+
+.. csv-table:: Environment variables
+   :widths: 30 30 30
+   :header: Name,Description,Default
+
+   ``WIS2_GDC_LOGGING_LEVEL``,logging level as per the standard `Python logging levels`_,``ERROR``
+   ``WIS2_GDC_API_URL``,public URL of the GDC API,``http://localhost``
+   ``WIS2_GDC_API_URL_DOCKER``,internal Docker URL of the API,``http://wis2-gdc-api``
+   ``WIS2_GDC_BACKEND_TYPE``,API backend type,``Elasticsearch``
+   ``WIS2_GDC_BACKEND_CONNECTION``,API backend connection,``http://wis2-gdc-backend:9200/wis2-discovery-metadata``
+   ``WIS2_GDC_BROKER_URL``,URL of the GDC broker,``mqtt://wis2-gdc:wis2-gdc@wis2-gdc-broker:1883``
+   ``WIS2_GDC_CENTRE_ID``,centre identifier of the GDC,``ca-eccc-msc-global-discovery-catalogue``
+   ``WIS2_GDC_COLLECTOR_URL``,URL of metrics collector,``http://wis2-gdc-metrics-collector:8006``
+   ``WIS2_GDC_GB``,WIS2 Global Broker that the GDC connects to,``mqtts://everyone:[email protected]:8883``
+   ``WIS2_GDC_GB_TOPIC``,WIS2 topic that the GDC subscribes to,``cache/a/wis2/+/metadata/#``
+   ``WIS2_GDC_METADATA_ARCHIVE_ZIPFILE``,filepath to the metadata archive zipfile,``/data/wis2-gdc-archive.zip``
+   ``WIS2_GDC_PUBLISH_REPORTS``,whether the GDC should publish ETS and KPI reports,``true``
+   ``WIS2_GDC_REJECT_ON_FAILING_ETS``,whether the GDC should stop ingest based on on failing record,``true``
+   ``WIS2_GDC_RUN_KPI``,whether the GDC should run KPI as part of ingest,``false``
+
+Global Broker environment variables
+-----------------------------------
+
+WIS2 Global Broker environment variables are defined as comma-separated values (centre=id,url,centre-name).  ``wis2-gdc`` allows for 1..n Global Broker environment variables as required.
+
+.. note::
+
+   - the naming convention is ``WIS_GDC_GB_LINK_<LABEL>``, where ``<LABEL>`` can be named as desired to identify the GB
+   - at least one Global Broker environment variable is required
+   - the centre name may contain commas
+
+An example can be found below:
+
+.. code-block:: csv
+
+   WIS2_GDC_GB_LINK_METEOFRANCE,"fr-meteo-france-global-broker,mqtts://everyone:[email protected]:8883,Météo-France, Global Broker Service"
+
+Key settings
+------------
+
+A default installation with minimal configuration changes per below satisfies most use casess:
+
+- ``WIS2_GDC_API_URL``
+- ``WIS2_GDC_CENTRE_ID``
+- ``WIS2_GDC_GB``
+- ``WIS2_GDC_GB_LINK...``
+
+.. note::
+
+   The ``wis2-gdc`` Docker Compose file also contains additional environment variables (see ``docker-compose.yml`` to adjust accordingly).  In most cases, these values do not need adjustment.
+
+Application specific configuration
+----------------------------------
+
+Application specific configurations can be found in the following files:
+
+.. csv-table:: Applicaiton specific configuration files
+   :widths: 30 30
+   :header: Filepath,Description
+
+   ``wis2-gdc-api/docker/wis2-gdc-config.yml``,pygeoapi configuration (`documentation`_)
+   ``wis2-gdc-broker/docker/mosquitto.conf``,mosquitto main configuration
+   ``wis2-gdc-broker/docker/acl.conf``,mosquitto access control list
+   ``wis2-gdc-management/docker/pywis-pubsub.yml``,pywis-pubsub configuration
+   ``wis2-gdc-monitoring/grafana/datasource.yml``,Grafana configuration
+   ``wis2-gdc-monitoring/grafana/datasource.yml``,Grafana configuration
+   ``wis2-gdc-monitoring/prometheus/datasource.yml``,Prometheus configuration
+
+.. note::
+
+   Application specific configurations do not need adjustment in most cases.
+
+.. _`Python logging levels`: https://docs.python.org/library/logging.html#logging-levels
+.. _`documentation`: https://docs.pygeoapi.io/en/latest/configuration.html

docs/source/contributing.rst

@@ -0,0 +1,10 @@
+.. _contributing:
+
+Contributing
+============
+
+wis2-gdc is developed as a free and open source project on GitHub. Contributions to the project
+(documentation, bug fixes, enhancements, tests, etc.) are welcome and encouraged. Please consult
+the wis2-gdc `Contribution guidelines`_ for more information.
+
+.. _`Contribution guidelines`: https://github.com/wmo-im/wis2-gdc/blob/main/CONTRIBUTING.md

docs/source/index.rst

@@ -0,0 +1,35 @@
+.. _index:
+
+.. image:: /_static/wmo-logo.svg
+   :width: 10%
+   :alt: World Meteorological Organization
+
+wis2-gdc |release| documentation
+================================
+
+:Author: Tom Kralidis
+:Contact: tomkralidis at gmail.com
+:Release: |release|
+:Date: |today|
+
+Welcome to the official wis2-gdc documentation!  Here you will find complete
+reference documentation on all aspects of the project.
+
+.. toctree::
+   :maxdepth: 4
+   :caption: Table of Contents
+   :name: toc
+
+   introduction
+   architecture
+   installation
+   configuration
+   administration
+   running
+   openapi
+   transactions
+   admin-api
+   security
+   contributing
+   support
+   license