diff --git a/README.md b/README.md deleted file mode 100644 index d60ecc5..0000000 --- a/README.md +++ /dev/null @@ -1,19 +0,0 @@ -# e-soh-datastore-poc - -E-SOH datastore PoCs - -## Pre-commit - -### Setup - -1. Go to the root of the repository. -2. Install the python pre-commit package with `pip install pre-commit`. -3. Reinitialize the repository with `git init`. -4. Install the hooks defined in `.pre-commit-config.yaml` with `pre-commit install`. - -### Useful Commands - -- To update the pre-commit hooks in `.pre-commit-config.yaml`, run `pre-commit autoupdate` -- To apply the pre-commit for every file in the repository, run `pre-commit run --config './.pre-commit-config.yaml' --all-files` -- To see all options to `pre-commit run`, run `pre-commit help run` (in particular, the `--files` option can be used to apply the command to selected files only). -- To commit without the pre-commit hook, run `git commit -m "Some message" --no-verify` diff --git a/datastore/README.md b/datastore/README.md index 3b2d8f9..d60ecc5 100644 --- a/datastore/README.md +++ b/datastore/README.md @@ -1,390 +1,19 @@ -# E-SOH datastore variant 1: gRPC service written in Go +# e-soh-datastore-poc -## Overview +E-SOH datastore PoCs -This directory contains code that demonstrates how the E-SOH datastore could -be implemented as a [gRPC](https://grpc.io/) service written in -[Go](https://go.dev/). +## Pre-commit -Currently the datastore server is using a PostgreSQL server as its only -storage backend (for both metadata and observations). +### Setup -**Note:** Unless otherwise noted, all commands described below should be run from the same -directory as this README file. +1. Go to the root of the repository. +2. Install the python pre-commit package with `pip install pre-commit`. +3. Reinitialize the repository with `git init`. +4. Install the hooks defined in `.pre-commit-config.yaml` with `pre-commit install`. -The code has been tested in the following environment: +### Useful Commands -### Service - -| | | -|---|---| -| OS | [Ubuntu](https://ubuntu.com/) [22.04 Jammy](https://releases.ubuntu.com/jammy/) | -| [Docker](https://www.docker.com/) | 24.0.5 | -| [Docker Compose](https://www.docker.com/) | 2.20.2 | - -### Python client example - -| | | -|---|---| -| OS | Same as service | -| [Python](https://www.python.org/) | 3.11 | -| [grpcio-tools](https://grpc.io/docs/languages/python/quickstart/) | 1.56.2 | - -## Some examples of using docker compose to manage the service - -(**NOTE:** Any environment variables used on these examples are defined in a separate section below) - -### Check current status - -```text -docker compose ps -a -``` - -### Start service in normal mode (rolling buffer of observations within latest 24H) - -```text -docker compose down --volumes -docker compose build -docker compose up -d -``` - -### Same as above, but with a safety margin of one minute in case 'current time' isn't 100% synchronized everywhere - -```text -docker compose down --volumes -docker compose build -HITIME=-60 docker compose up -d -``` - -(note how we _subtract_ a _negative_ value to current time to get a value into the future) - -### Start service in "infinite" mode (accommodating "all" possible obs times) and run a test - -```text -docker compose down --volumes -docker compose --profile test build -DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose up -d -DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose run --rm loader -DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose run --rm integration -``` - -**NOTE:** as an alternative to specifying environment variables explicitly on the command line (which can be quite verbose), they could instead be kept in a file called `.env`: - -```text -DYNAMICTIME=false -LOTIME=1000-01-01T00:00:00Z -HITIME=9999-12-31T23:59:59Z -``` - -Using a `.env` file also makes it more practical to have all supported environment variables -explicitly defined and thus avoid warnings from `docker compose` due to undefined defaults -(defaults are defined in the Go code only). So for example, to get rid of the following warnings: - -```text -$ docker compose up -d -WARN[0000] The "CLEANUPINTERVAL" variable is not set. Defaulting to a blank string. -WARN[0000] The "PUTOBSLIMIT" variable is not set. Defaulting to a blank string. -... -``` - -, simply ensure that `CLEANUPINTERVAL` and `PUTOBSLIMIT` are both defined in `.env`. - -### Same as above, but specifying LOTIME and HITIME directly as seconds - -First ensure `.env` has the following contents: - -```text -DYNAMICTIME=false -LOTIME=-30610227208 -HITIME=253402297199 -``` - -Then run the same five docker compose commands as in the previous example (without specifying environment variables). - -------------- - -MORE DETAILS/EXAMPLES HERE! - -## Collecting profiling stats from a running datastore service - -**STEP 1:** Start the service (or ensure it already runs). - -**STEP 2:** Collect profiling stats over a certain period, for example: - -`go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=220` - -(by default, the stats will be written to `~/pprof/`; see `go tool pprof --help` to see -all options) - -**STEP 3:** Open a web page to visualize and inspect the results of a given profiling run, -for example: - -`BROWSER=firefox go tool pprof -http=:8081 ~/pprof/pprof.dsserver.samples.cpu.001.pb.gz` - -## Compiling datastore.proto and update go.sum to prevent IDEs from complaining - -Whenever `datastore.proto` changes, it should be complied locally in order for -IDEs to recognize the current types and symbols. - -```text -protoc --go_out=. --go-grpc_out=. protobuf/datastore.proto -``` - -Likewise, keeping `go.sum` up-to-date like this may also prevent certain -warnings/errors in IDEs: - -```text -go mod tidy -``` - -## Environment variables - -The following environment variables are supported: - -Variable | Mandatory | Default value | Description -:-- | :-- | :-- | :-- -`SERVERPORT` | No | `50050` | Server port number. -`PGHOST` | No | `localhost` | PostgreSQL host. -`PGPORT` | No | `5433` | PostgreSQL port number. -`PGBUSER` | No | `postgres` | PostgreSQL user name. -`PGPASSWORD` | No | `mysecretpassword` | PostgreSQL password. -`PGDBNAME` | No | `data` | PostgreSQL database name. -`DYNAMICTIME` | No | `true` | Whether the valid time range is _dynamic_ or _static_ (defined below). -`LOTIME` | No | `86400` | The _earliest_ valid time as seconds to be either [1] subtracted from the current time (if the valid time range is _dynamic_) or [2] added to UNIX epoch (1970-01-01T00:00:00Z) (if the valid time range is _static_). In the case of a _static_ valid time range, the `LOTIME` can optionally be specified as an ISO-8601 datetime of the exact form `2023-10-10T00:00:00Z`. -`HITIME` | No | `-2` | Same as `LOTIME`, but for the _latest_ valid time. Note a default leeway of 2 seconds into the future to reduce risk of missing the newest observations. -`CLEANUPINTERVAL` | No | `86400` | The minimum time duration in seconds between automatic cleanups (like removing obsolete observations from the physical store). -`PUTOBSLIMIT` | No | `100000` | Maximum number of observations allowed in a single call to `PutObservations`. - -**TODO:** Ensure that these variables are [passed properly](https://docs.docker.com/compose/environment-variables/set-environment-variables/) to the relevant `docker compose` -commands. Any secrets should be passed using a [special mechanism](https://docs.docker.com/compose/use-secrets/), etc. - -## Testing the datastore service with gRPCurl - -The datastore service can be tested with [gRPCurl](https://github.com/fullstorydev/grpcurl). Below are a few examples: - -### List all services defined in the proto file - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto list -datastore.Datastore -``` - -### Describe all services defined in the proto file - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto describe -datastore.Datastore is a service: -service Datastore { - rpc GetObservations ( .datastore.GetObsRequest ) returns ( .datastore.GetObsResponse ); - rpc PutObservations ( .datastore.PutObsRequest ) returns ( .datastore.PutObsResponse ); -} -``` - -### Describe method PutObservations - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto describe datastore.Datastore.PutObservations -datastore.Datastore.PutObservations is a method: -rpc PutObservations ( .datastore.PutObsRequest ) returns ( .datastore.PutObsResponse ); -``` - -### Describe message PutObsRequest - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto describe .datastore.PutObsRequest -datastore.PutObsRequest is a message: -message PutObsRequest { - repeated .datastore.Metadata1 observations = 1; -} -``` - -### Insert observations - -```text -$ grpcurl -d '{"observations": [{"ts_mdata": {"version": "version_dummy", "type": "type_dummy", "standard_name": "air_temperature", "unit": "celsius"}, "obs_mdata": {"id": "id_dummy", "geo_point": {"lat": 59.91, "lon": 10.75}, "pubtime": "2023-01-01T00:00:10Z", "data_id": "data_id_dummy", "obstime_instant": "2023-01-01T00:00:00Z", "value": "123.456"}}]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.PutObservations -... -``` - -### Retrieve all observations - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations -... -``` - -### Retrieve observations in a time range - -```text -$ grpcurl -d '{"interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations -... -``` - -### Retrieve observations in a polygon - -```text -$ grpcurl -d '{"inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations -... -``` - -### Retrieve observations in both a time range and a polygon - -```text -$ grpcurl -d '{"interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}, "inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations -... -``` - -### Retrieve wind speed and air temperature observations in a time range and a polygon - -```text -$ grpcurl -d '{"standard_names": ["wind_speed", "air_temperature"], "interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}, "inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations -... -``` - -### List unique occurrences of time series metadata attribute 'standard_name' - -```text -$ grpcurl -d '{"attrs": ["standard_name"]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups -{ - "groups": [ - { - "combos": [ - { - "standard_name": "air_pressure_at_sea_level" - } - ] - }, - { - "combos": [ - { - "standard_name": "air_temperature" - } - ] - }, -... -``` - -### List unique combinations of time series metadata attributes 'platform' and 'standard_name' - -```text -$ grpcurl -d '{"attrs": ["platform", "standard_name"]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups -{ - "groups": [ - { - "combos": [ - { - "platform": "06201", - "standard_name": "air_pressure_at_sea_level" - } - ] - }, - { - "combos": [ - { - "platform": "06201", - "standard_name": "air_temperature" - } - ] - }, -... -``` - -### List unique occurrences of time series metadata attribute 'standard_name', and include associated instances - -```text -$ grpcurl -d '{"attrs": ["standard_name"], "include_instances": true}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups -{ - "groups": [ - { - "combos": [ - { - "title": "Air Pressure at Sea Level 1 Min Average", - "platform": "06208", - "standard_name": "air_pressure_at_sea_level", - "unit": "hPa", - "instrument": "pp" - }, - { - "title": "Air Pressure at Sea Level 1 Min Average", - "platform": "06348", - "standard_name": "air_pressure_at_sea_level", - "unit": "hPa", - "instrument": "pp" - }, -... -``` - -### List unique combinations of time series metadata attributes 'platform' and 'standard_name', and include associated instances - -```text -$ grpcurl -d '{"attrs": ["platform", "standard_name"], "include_instances": true}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups -{ - "groups": [ - { - "combos": [ - { - "title": "Air Temperature Minimum last 12 Hours", - "platform": "06201", - "standard_name": "air_temperature", - "unit": "degrees Celsius", - "instrument": "Tn12" - }, - { - "title": "Air Temperature Minimum last 14 Hours", - "platform": "06201", - "standard_name": "air_temperature", - "unit": "degrees Celsius", - "instrument": "Tn14" - }, -... -``` - -### Get the temporal- and spatial extent of all observations currently in the storage - -```text -$ grpcurl -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetExtents -{ - "timeExtent": { - "start": "2022-12-31T00:00:00Z", - "end": "2022-12-31T23:50:00Z" - }, - "geoExtent": { - "left": -68.2758333, - "bottom": 12.13, - "right": 7.1493220605216, - "top": 55.399166666667 - } -} -``` - -## Testing the datastore service with a Python client - -### Compiling the protobuf file - -If necessary, compile the protobuf file first. The following command generates the files -`datastore_pb2.py` and `datastore_grpc.py` under `../examples/clients/python/`: - -```text -python -m grpc_tools.protoc --proto_path=protobuf datastore.proto --python_out=../examples/clients/python --grpc_python_out=../examples/clients/python -``` - -### Running the client - -The python client can be run like this: - -```text -$ python ../examples/clients/python/client.py -response from callPutObs: status: -1 -... -``` - -Testing the performance can be done with: - -```bash -python -m cProfile -o -``` - -Generate a dot graph / tree with: - -```bash -gprof2dot --colour-nodes-by-selftime -f pstats | dot -Tpng -o -``` +- To update the pre-commit hooks in `.pre-commit-config.yaml`, run `pre-commit autoupdate` +- To apply the pre-commit for every file in the repository, run `pre-commit run --config './.pre-commit-config.yaml' --all-files` +- To see all options to `pre-commit run`, run `pre-commit help run` (in particular, the `--files` option can be used to apply the command to selected files only). +- To commit without the pre-commit hook, run `git commit -m "Some message" --no-verify` diff --git a/api/Dockerfile b/datastore/api/Dockerfile similarity index 100% rename from api/Dockerfile rename to datastore/api/Dockerfile diff --git a/api/locustfile.py b/datastore/api/locustfile.py similarity index 100% rename from api/locustfile.py rename to datastore/api/locustfile.py diff --git a/api/main.py b/datastore/api/main.py similarity index 100% rename from api/main.py rename to datastore/api/main.py diff --git a/api/metadata_endpoints.py b/datastore/api/metadata_endpoints.py similarity index 100% rename from api/metadata_endpoints.py rename to datastore/api/metadata_endpoints.py diff --git a/api/requirements.in b/datastore/api/requirements.in similarity index 100% rename from api/requirements.in rename to datastore/api/requirements.in diff --git a/api/requirements.txt b/datastore/api/requirements.txt similarity index 100% rename from api/requirements.txt rename to datastore/api/requirements.txt diff --git a/ci/go/go-fmt.sh b/datastore/ci/go/go-fmt.sh similarity index 100% rename from ci/go/go-fmt.sh rename to datastore/ci/go/go-fmt.sh diff --git a/ci/go/go-vet.sh b/datastore/ci/go/go-vet.sh similarity index 100% rename from ci/go/go-vet.sh rename to datastore/ci/go/go-vet.sh diff --git a/data-loader/Dockerfile b/datastore/data-loader/Dockerfile similarity index 100% rename from data-loader/Dockerfile rename to datastore/data-loader/Dockerfile diff --git a/data-loader/client_fmi_station.py b/datastore/data-loader/client_fmi_station.py similarity index 100% rename from data-loader/client_fmi_station.py rename to datastore/data-loader/client_fmi_station.py diff --git a/data-loader/client_knmi_station.py b/datastore/data-loader/client_knmi_station.py similarity index 100% rename from data-loader/client_knmi_station.py rename to datastore/data-loader/client_knmi_station.py diff --git a/data-loader/parameters.py b/datastore/data-loader/parameters.py similarity index 100% rename from data-loader/parameters.py rename to datastore/data-loader/parameters.py diff --git a/data-loader/requirements.in b/datastore/data-loader/requirements.in similarity index 100% rename from data-loader/requirements.in rename to datastore/data-loader/requirements.in diff --git a/data-loader/requirements.txt b/datastore/data-loader/requirements.txt similarity index 100% rename from data-loader/requirements.txt rename to datastore/data-loader/requirements.txt diff --git a/database/healthcheck_postgis_uptime.sh b/datastore/database/healthcheck_postgis_uptime.sh similarity index 100% rename from database/healthcheck_postgis_uptime.sh rename to datastore/database/healthcheck_postgis_uptime.sh diff --git a/datastore-design/README.md b/datastore/datastore-design/README.md similarity index 100% rename from datastore-design/README.md rename to datastore/datastore-design/README.md diff --git a/datastore-design/frost_presos/preso1-2021-11-02.pdf b/datastore/datastore-design/frost_presos/preso1-2021-11-02.pdf similarity index 100% rename from datastore-design/frost_presos/preso1-2021-11-02.pdf rename to datastore/datastore-design/frost_presos/preso1-2021-11-02.pdf diff --git a/datastore-design/frost_presos/preso2-2022-11-16.pdf b/datastore/datastore-design/frost_presos/preso2-2022-11-16.pdf similarity index 100% rename from datastore-design/frost_presos/preso2-2022-11-16.pdf rename to datastore/datastore-design/frost_presos/preso2-2022-11-16.pdf diff --git a/datastore-design/tasks/tasks.txt b/datastore/datastore-design/tasks/tasks.txt similarity index 100% rename from datastore-design/tasks/tasks.txt rename to datastore/datastore-design/tasks/tasks.txt diff --git a/datastore/.dockerignore b/datastore/datastore/.dockerignore similarity index 100% rename from datastore/.dockerignore rename to datastore/datastore/.dockerignore diff --git a/datastore/Dockerfile b/datastore/datastore/Dockerfile similarity index 100% rename from datastore/Dockerfile rename to datastore/datastore/Dockerfile diff --git a/datastore/datastore/README.md b/datastore/datastore/README.md new file mode 100644 index 0000000..3b2d8f9 --- /dev/null +++ b/datastore/datastore/README.md @@ -0,0 +1,390 @@ +# E-SOH datastore variant 1: gRPC service written in Go + +## Overview + +This directory contains code that demonstrates how the E-SOH datastore could +be implemented as a [gRPC](https://grpc.io/) service written in +[Go](https://go.dev/). + +Currently the datastore server is using a PostgreSQL server as its only +storage backend (for both metadata and observations). + +**Note:** Unless otherwise noted, all commands described below should be run from the same +directory as this README file. + +The code has been tested in the following environment: + +### Service + +| | | +|---|---| +| OS | [Ubuntu](https://ubuntu.com/) [22.04 Jammy](https://releases.ubuntu.com/jammy/) | +| [Docker](https://www.docker.com/) | 24.0.5 | +| [Docker Compose](https://www.docker.com/) | 2.20.2 | + +### Python client example + +| | | +|---|---| +| OS | Same as service | +| [Python](https://www.python.org/) | 3.11 | +| [grpcio-tools](https://grpc.io/docs/languages/python/quickstart/) | 1.56.2 | + +## Some examples of using docker compose to manage the service + +(**NOTE:** Any environment variables used on these examples are defined in a separate section below) + +### Check current status + +```text +docker compose ps -a +``` + +### Start service in normal mode (rolling buffer of observations within latest 24H) + +```text +docker compose down --volumes +docker compose build +docker compose up -d +``` + +### Same as above, but with a safety margin of one minute in case 'current time' isn't 100% synchronized everywhere + +```text +docker compose down --volumes +docker compose build +HITIME=-60 docker compose up -d +``` + +(note how we _subtract_ a _negative_ value to current time to get a value into the future) + +### Start service in "infinite" mode (accommodating "all" possible obs times) and run a test + +```text +docker compose down --volumes +docker compose --profile test build +DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose up -d +DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose run --rm loader +DYNAMICTIME=false LOTIME=1000-01-01T00:00:00Z HITIME=9999-12-31T23:59:59Z docker compose run --rm integration +``` + +**NOTE:** as an alternative to specifying environment variables explicitly on the command line (which can be quite verbose), they could instead be kept in a file called `.env`: + +```text +DYNAMICTIME=false +LOTIME=1000-01-01T00:00:00Z +HITIME=9999-12-31T23:59:59Z +``` + +Using a `.env` file also makes it more practical to have all supported environment variables +explicitly defined and thus avoid warnings from `docker compose` due to undefined defaults +(defaults are defined in the Go code only). So for example, to get rid of the following warnings: + +```text +$ docker compose up -d +WARN[0000] The "CLEANUPINTERVAL" variable is not set. Defaulting to a blank string. +WARN[0000] The "PUTOBSLIMIT" variable is not set. Defaulting to a blank string. +... +``` + +, simply ensure that `CLEANUPINTERVAL` and `PUTOBSLIMIT` are both defined in `.env`. + +### Same as above, but specifying LOTIME and HITIME directly as seconds + +First ensure `.env` has the following contents: + +```text +DYNAMICTIME=false +LOTIME=-30610227208 +HITIME=253402297199 +``` + +Then run the same five docker compose commands as in the previous example (without specifying environment variables). + +------------- + +MORE DETAILS/EXAMPLES HERE! + +## Collecting profiling stats from a running datastore service + +**STEP 1:** Start the service (or ensure it already runs). + +**STEP 2:** Collect profiling stats over a certain period, for example: + +`go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=220` + +(by default, the stats will be written to `~/pprof/`; see `go tool pprof --help` to see +all options) + +**STEP 3:** Open a web page to visualize and inspect the results of a given profiling run, +for example: + +`BROWSER=firefox go tool pprof -http=:8081 ~/pprof/pprof.dsserver.samples.cpu.001.pb.gz` + +## Compiling datastore.proto and update go.sum to prevent IDEs from complaining + +Whenever `datastore.proto` changes, it should be complied locally in order for +IDEs to recognize the current types and symbols. + +```text +protoc --go_out=. --go-grpc_out=. protobuf/datastore.proto +``` + +Likewise, keeping `go.sum` up-to-date like this may also prevent certain +warnings/errors in IDEs: + +```text +go mod tidy +``` + +## Environment variables + +The following environment variables are supported: + +Variable | Mandatory | Default value | Description +:-- | :-- | :-- | :-- +`SERVERPORT` | No | `50050` | Server port number. +`PGHOST` | No | `localhost` | PostgreSQL host. +`PGPORT` | No | `5433` | PostgreSQL port number. +`PGBUSER` | No | `postgres` | PostgreSQL user name. +`PGPASSWORD` | No | `mysecretpassword` | PostgreSQL password. +`PGDBNAME` | No | `data` | PostgreSQL database name. +`DYNAMICTIME` | No | `true` | Whether the valid time range is _dynamic_ or _static_ (defined below). +`LOTIME` | No | `86400` | The _earliest_ valid time as seconds to be either [1] subtracted from the current time (if the valid time range is _dynamic_) or [2] added to UNIX epoch (1970-01-01T00:00:00Z) (if the valid time range is _static_). In the case of a _static_ valid time range, the `LOTIME` can optionally be specified as an ISO-8601 datetime of the exact form `2023-10-10T00:00:00Z`. +`HITIME` | No | `-2` | Same as `LOTIME`, but for the _latest_ valid time. Note a default leeway of 2 seconds into the future to reduce risk of missing the newest observations. +`CLEANUPINTERVAL` | No | `86400` | The minimum time duration in seconds between automatic cleanups (like removing obsolete observations from the physical store). +`PUTOBSLIMIT` | No | `100000` | Maximum number of observations allowed in a single call to `PutObservations`. + +**TODO:** Ensure that these variables are [passed properly](https://docs.docker.com/compose/environment-variables/set-environment-variables/) to the relevant `docker compose` +commands. Any secrets should be passed using a [special mechanism](https://docs.docker.com/compose/use-secrets/), etc. + +## Testing the datastore service with gRPCurl + +The datastore service can be tested with [gRPCurl](https://github.com/fullstorydev/grpcurl). Below are a few examples: + +### List all services defined in the proto file + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto list +datastore.Datastore +``` + +### Describe all services defined in the proto file + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto describe +datastore.Datastore is a service: +service Datastore { + rpc GetObservations ( .datastore.GetObsRequest ) returns ( .datastore.GetObsResponse ); + rpc PutObservations ( .datastore.PutObsRequest ) returns ( .datastore.PutObsResponse ); +} +``` + +### Describe method PutObservations + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto describe datastore.Datastore.PutObservations +datastore.Datastore.PutObservations is a method: +rpc PutObservations ( .datastore.PutObsRequest ) returns ( .datastore.PutObsResponse ); +``` + +### Describe message PutObsRequest + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto describe .datastore.PutObsRequest +datastore.PutObsRequest is a message: +message PutObsRequest { + repeated .datastore.Metadata1 observations = 1; +} +``` + +### Insert observations + +```text +$ grpcurl -d '{"observations": [{"ts_mdata": {"version": "version_dummy", "type": "type_dummy", "standard_name": "air_temperature", "unit": "celsius"}, "obs_mdata": {"id": "id_dummy", "geo_point": {"lat": 59.91, "lon": 10.75}, "pubtime": "2023-01-01T00:00:10Z", "data_id": "data_id_dummy", "obstime_instant": "2023-01-01T00:00:00Z", "value": "123.456"}}]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.PutObservations +... +``` + +### Retrieve all observations + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations +... +``` + +### Retrieve observations in a time range + +```text +$ grpcurl -d '{"interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations +... +``` + +### Retrieve observations in a polygon + +```text +$ grpcurl -d '{"inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations +... +``` + +### Retrieve observations in both a time range and a polygon + +```text +$ grpcurl -d '{"interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}, "inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations +... +``` + +### Retrieve wind speed and air temperature observations in a time range and a polygon + +```text +$ grpcurl -d '{"standard_names": ["wind_speed", "air_temperature"], "interval": {"start": "2023-01-01T00:00:00Z", "end": "2023-01-01T00:00:10Z"}, "inside": {"points": [{"lat": 59.90, "lon": 10.70}, {"lat": 59.90, "lon": 10.80}, {"lat": 60, "lon": 10.80}, {"lat": 60, "lon": 10.70}]}}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetObservations +... +``` + +### List unique occurrences of time series metadata attribute 'standard_name' + +```text +$ grpcurl -d '{"attrs": ["standard_name"]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups +{ + "groups": [ + { + "combos": [ + { + "standard_name": "air_pressure_at_sea_level" + } + ] + }, + { + "combos": [ + { + "standard_name": "air_temperature" + } + ] + }, +... +``` + +### List unique combinations of time series metadata attributes 'platform' and 'standard_name' + +```text +$ grpcurl -d '{"attrs": ["platform", "standard_name"]}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups +{ + "groups": [ + { + "combos": [ + { + "platform": "06201", + "standard_name": "air_pressure_at_sea_level" + } + ] + }, + { + "combos": [ + { + "platform": "06201", + "standard_name": "air_temperature" + } + ] + }, +... +``` + +### List unique occurrences of time series metadata attribute 'standard_name', and include associated instances + +```text +$ grpcurl -d '{"attrs": ["standard_name"], "include_instances": true}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups +{ + "groups": [ + { + "combos": [ + { + "title": "Air Pressure at Sea Level 1 Min Average", + "platform": "06208", + "standard_name": "air_pressure_at_sea_level", + "unit": "hPa", + "instrument": "pp" + }, + { + "title": "Air Pressure at Sea Level 1 Min Average", + "platform": "06348", + "standard_name": "air_pressure_at_sea_level", + "unit": "hPa", + "instrument": "pp" + }, +... +``` + +### List unique combinations of time series metadata attributes 'platform' and 'standard_name', and include associated instances + +```text +$ grpcurl -d '{"attrs": ["platform", "standard_name"], "include_instances": true}' -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetTSAttrGroups +{ + "groups": [ + { + "combos": [ + { + "title": "Air Temperature Minimum last 12 Hours", + "platform": "06201", + "standard_name": "air_temperature", + "unit": "degrees Celsius", + "instrument": "Tn12" + }, + { + "title": "Air Temperature Minimum last 14 Hours", + "platform": "06201", + "standard_name": "air_temperature", + "unit": "degrees Celsius", + "instrument": "Tn14" + }, +... +``` + +### Get the temporal- and spatial extent of all observations currently in the storage + +```text +$ grpcurl -plaintext -proto protobuf/datastore.proto 127.0.0.1:50050 datastore.Datastore.GetExtents +{ + "timeExtent": { + "start": "2022-12-31T00:00:00Z", + "end": "2022-12-31T23:50:00Z" + }, + "geoExtent": { + "left": -68.2758333, + "bottom": 12.13, + "right": 7.1493220605216, + "top": 55.399166666667 + } +} +``` + +## Testing the datastore service with a Python client + +### Compiling the protobuf file + +If necessary, compile the protobuf file first. The following command generates the files +`datastore_pb2.py` and `datastore_grpc.py` under `../examples/clients/python/`: + +```text +python -m grpc_tools.protoc --proto_path=protobuf datastore.proto --python_out=../examples/clients/python --grpc_python_out=../examples/clients/python +``` + +### Running the client + +The python client can be run like this: + +```text +$ python ../examples/clients/python/client.py +response from callPutObs: status: -1 +... +``` + +Testing the performance can be done with: + +```bash +python -m cProfile -o +``` + +Generate a dot graph / tree with: + +```bash +gprof2dot --colour-nodes-by-selftime -f pstats | dot -Tpng -o +``` diff --git a/datastore/common/common.go b/datastore/datastore/common/common.go similarity index 100% rename from datastore/common/common.go rename to datastore/datastore/common/common.go diff --git a/datastore/docs/puml/datastore.png b/datastore/datastore/docs/puml/datastore.png similarity index 100% rename from datastore/docs/puml/datastore.png rename to datastore/datastore/docs/puml/datastore.png diff --git a/datastore/docs/puml/datastore.puml b/datastore/datastore/docs/puml/datastore.puml similarity index 100% rename from datastore/docs/puml/datastore.puml rename to datastore/datastore/docs/puml/datastore.puml diff --git a/datastore/dsimpl/common.go b/datastore/datastore/dsimpl/common.go similarity index 100% rename from datastore/dsimpl/common.go rename to datastore/datastore/dsimpl/common.go diff --git a/datastore/dsimpl/getextents.go b/datastore/datastore/dsimpl/getextents.go similarity index 100% rename from datastore/dsimpl/getextents.go rename to datastore/datastore/dsimpl/getextents.go diff --git a/datastore/dsimpl/getobservations.go b/datastore/datastore/dsimpl/getobservations.go similarity index 100% rename from datastore/dsimpl/getobservations.go rename to datastore/datastore/dsimpl/getobservations.go diff --git a/datastore/dsimpl/gettsattrgroups.go b/datastore/datastore/dsimpl/gettsattrgroups.go similarity index 100% rename from datastore/dsimpl/gettsattrgroups.go rename to datastore/datastore/dsimpl/gettsattrgroups.go diff --git a/datastore/dsimpl/putobservations.go b/datastore/datastore/dsimpl/putobservations.go similarity index 100% rename from datastore/dsimpl/putobservations.go rename to datastore/datastore/dsimpl/putobservations.go diff --git a/datastore/go.mod b/datastore/datastore/go.mod similarity index 100% rename from datastore/go.mod rename to datastore/datastore/go.mod diff --git a/datastore/main/main.go b/datastore/datastore/main/main.go similarity index 100% rename from datastore/main/main.go rename to datastore/datastore/main/main.go diff --git a/datastore/protobuf/datastore.proto b/datastore/datastore/protobuf/datastore.proto similarity index 100% rename from datastore/protobuf/datastore.proto rename to datastore/datastore/protobuf/datastore.proto diff --git a/datastore/storagebackend/postgresql/getextents.go b/datastore/datastore/storagebackend/postgresql/getextents.go similarity index 100% rename from datastore/storagebackend/postgresql/getextents.go rename to datastore/datastore/storagebackend/postgresql/getextents.go diff --git a/datastore/storagebackend/postgresql/getobservations.go b/datastore/datastore/storagebackend/postgresql/getobservations.go similarity index 100% rename from datastore/storagebackend/postgresql/getobservations.go rename to datastore/datastore/storagebackend/postgresql/getobservations.go diff --git a/datastore/storagebackend/postgresql/gettsattrgroups.go b/datastore/datastore/storagebackend/postgresql/gettsattrgroups.go similarity index 100% rename from datastore/storagebackend/postgresql/gettsattrgroups.go rename to datastore/datastore/storagebackend/postgresql/gettsattrgroups.go diff --git a/datastore/storagebackend/postgresql/postgresql.go b/datastore/datastore/storagebackend/postgresql/postgresql.go similarity index 100% rename from datastore/storagebackend/postgresql/postgresql.go rename to datastore/datastore/storagebackend/postgresql/postgresql.go diff --git a/datastore/storagebackend/postgresql/putobservations.go b/datastore/datastore/storagebackend/postgresql/putobservations.go similarity index 100% rename from datastore/storagebackend/postgresql/putobservations.go rename to datastore/datastore/storagebackend/postgresql/putobservations.go diff --git a/datastore/storagebackend/storagebackend.go b/datastore/datastore/storagebackend/storagebackend.go similarity index 100% rename from datastore/storagebackend/storagebackend.go rename to datastore/datastore/storagebackend/storagebackend.go diff --git a/docker-compose.yml b/datastore/docker-compose.yml similarity index 100% rename from docker-compose.yml rename to datastore/docker-compose.yml diff --git a/examples/big_input_workaround/client.py b/datastore/examples/big_input_workaround/client.py similarity index 100% rename from examples/big_input_workaround/client.py rename to datastore/examples/big_input_workaround/client.py diff --git a/examples/clients/python/Dockerfile b/datastore/examples/clients/python/Dockerfile similarity index 100% rename from examples/clients/python/Dockerfile rename to datastore/examples/clients/python/Dockerfile diff --git a/examples/clients/python/client.py b/datastore/examples/clients/python/client.py similarity index 100% rename from examples/clients/python/client.py rename to datastore/examples/clients/python/client.py diff --git a/examples/clients/python/requirements.in b/datastore/examples/clients/python/requirements.in similarity index 100% rename from examples/clients/python/requirements.in rename to datastore/examples/clients/python/requirements.in diff --git a/examples/clients/python/requirements.txt b/datastore/examples/clients/python/requirements.txt similarity index 100% rename from examples/clients/python/requirements.txt rename to datastore/examples/clients/python/requirements.txt diff --git a/integration-test/Dockerfile b/datastore/integration-test/Dockerfile similarity index 100% rename from integration-test/Dockerfile rename to datastore/integration-test/Dockerfile diff --git a/integration-test/discover.py b/datastore/integration-test/discover.py similarity index 100% rename from integration-test/discover.py rename to datastore/integration-test/discover.py diff --git a/integration-test/requirements.in b/datastore/integration-test/requirements.in similarity index 100% rename from integration-test/requirements.in rename to datastore/integration-test/requirements.in diff --git a/integration-test/requirements.txt b/datastore/integration-test/requirements.txt similarity index 100% rename from integration-test/requirements.txt rename to datastore/integration-test/requirements.txt diff --git a/integration-test/response/capabilities/200/all_collections.json b/datastore/integration-test/response/capabilities/200/all_collections.json similarity index 100% rename from integration-test/response/capabilities/200/all_collections.json rename to datastore/integration-test/response/capabilities/200/all_collections.json diff --git a/integration-test/response/collection/area/200/data_within_an_area_with_two_parameters.json b/datastore/integration-test/response/collection/area/200/data_within_an_area_with_two_parameters.json similarity index 100% rename from integration-test/response/collection/area/200/data_within_an_area_with_two_parameters.json rename to datastore/integration-test/response/collection/area/200/data_within_an_area_with_two_parameters.json diff --git a/integration-test/response/collection/locations/200/locations_within_a_bbox.json b/datastore/integration-test/response/collection/locations/200/locations_within_a_bbox.json similarity index 100% rename from integration-test/response/collection/locations/200/locations_within_a_bbox.json rename to datastore/integration-test/response/collection/locations/200/locations_within_a_bbox.json diff --git a/integration-test/response/collection/locations/200/single_location_with_multiple_parameters.json b/datastore/integration-test/response/collection/locations/200/single_location_with_multiple_parameters.json similarity index 100% rename from integration-test/response/collection/locations/200/single_location_with_multiple_parameters.json rename to datastore/integration-test/response/collection/locations/200/single_location_with_multiple_parameters.json diff --git a/integration-test/response/collection/locations/404/no_data_found.json b/datastore/integration-test/response/collection/locations/404/no_data_found.json similarity index 100% rename from integration-test/response/collection/locations/404/no_data_found.json rename to datastore/integration-test/response/collection/locations/404/no_data_found.json diff --git a/integration-test/response/collection/position/200/single_coordinate_with_one_parameter.json b/datastore/integration-test/response/collection/position/200/single_coordinate_with_one_parameter.json similarity index 100% rename from integration-test/response/collection/position/200/single_coordinate_with_one_parameter.json rename to datastore/integration-test/response/collection/position/200/single_coordinate_with_one_parameter.json diff --git a/integration-test/response/metadata/200/single_collection.json b/datastore/integration-test/response/metadata/200/single_collection.json similarity index 100% rename from integration-test/response/metadata/200/single_collection.json rename to datastore/integration-test/response/metadata/200/single_collection.json diff --git a/integration-test/response/metadata/404/not_found.json b/datastore/integration-test/response/metadata/404/not_found.json similarity index 100% rename from integration-test/response/metadata/404/not_found.json rename to datastore/integration-test/response/metadata/404/not_found.json diff --git a/integration-test/test_api.py b/datastore/integration-test/test_api.py similarity index 100% rename from integration-test/test_api.py rename to datastore/integration-test/test_api.py diff --git a/integration-test/test_delete.py b/datastore/integration-test/test_delete.py similarity index 100% rename from integration-test/test_delete.py rename to datastore/integration-test/test_delete.py diff --git a/integration-test/test_knmi.py b/datastore/integration-test/test_knmi.py similarity index 100% rename from integration-test/test_knmi.py rename to datastore/integration-test/test_knmi.py diff --git a/load-test/grpc_user.py b/datastore/load-test/grpc_user.py similarity index 100% rename from load-test/grpc_user.py rename to datastore/load-test/grpc_user.py diff --git a/load-test/locustfile.py b/datastore/load-test/locustfile.py similarity index 100% rename from load-test/locustfile.py rename to datastore/load-test/locustfile.py diff --git a/load-test/requirements.in b/datastore/load-test/requirements.in similarity index 100% rename from load-test/requirements.in rename to datastore/load-test/requirements.in diff --git a/load-test/requirements.txt b/datastore/load-test/requirements.txt similarity index 100% rename from load-test/requirements.txt rename to datastore/load-test/requirements.txt diff --git a/migrate/README.md b/datastore/migrate/README.md similarity index 100% rename from migrate/README.md rename to datastore/migrate/README.md diff --git a/migrate/data/migrations/1701872471_initialise_schema.down.sql b/datastore/migrate/data/migrations/1701872471_initialise_schema.down.sql similarity index 100% rename from migrate/data/migrations/1701872471_initialise_schema.down.sql rename to datastore/migrate/data/migrations/1701872471_initialise_schema.down.sql diff --git a/migrate/data/migrations/1701872471_initialise_schema.up.sql b/datastore/migrate/data/migrations/1701872471_initialise_schema.up.sql similarity index 100% rename from migrate/data/migrations/1701872471_initialise_schema.up.sql rename to datastore/migrate/data/migrations/1701872471_initialise_schema.up.sql diff --git a/migrate/data/not_supported_yet/1702281165_geo_polygon.down.sql b/datastore/migrate/data/not_supported_yet/1702281165_geo_polygon.down.sql similarity index 100% rename from migrate/data/not_supported_yet/1702281165_geo_polygon.down.sql rename to datastore/migrate/data/not_supported_yet/1702281165_geo_polygon.down.sql diff --git a/migrate/data/not_supported_yet/1702281165_geo_polygon.up.sql b/datastore/migrate/data/not_supported_yet/1702281165_geo_polygon.up.sql similarity index 100% rename from migrate/data/not_supported_yet/1702281165_geo_polygon.up.sql rename to datastore/migrate/data/not_supported_yet/1702281165_geo_polygon.up.sql diff --git a/test-data/FMI/fmi_test_data.tar.gz b/datastore/test-data/FMI/fmi_test_data.tar.gz similarity index 100% rename from test-data/FMI/fmi_test_data.tar.gz rename to datastore/test-data/FMI/fmi_test_data.tar.gz diff --git a/test-data/KNMI/20221231.nc b/datastore/test-data/KNMI/20221231.nc similarity index 100% rename from test-data/KNMI/20221231.nc rename to datastore/test-data/KNMI/20221231.nc diff --git a/test-data/KNMI/20230101.nc b/datastore/test-data/KNMI/20230101.nc similarity index 100% rename from test-data/KNMI/20230101.nc rename to datastore/test-data/KNMI/20230101.nc diff --git a/test-data/KNMI/20230102.nc b/datastore/test-data/KNMI/20230102.nc similarity index 100% rename from test-data/KNMI/20230102.nc rename to datastore/test-data/KNMI/20230102.nc diff --git a/tstester/README.md b/datastore/tstester/README.md similarity index 100% rename from tstester/README.md rename to datastore/tstester/README.md diff --git a/tstester/common.py b/datastore/tstester/common.py similarity index 100% rename from tstester/common.py rename to datastore/tstester/common.py diff --git a/tstester/config.json b/datastore/tstester/config.json similarity index 100% rename from tstester/config.json rename to datastore/tstester/config.json diff --git a/tstester/main.py b/datastore/tstester/main.py similarity index 100% rename from tstester/main.py rename to datastore/tstester/main.py diff --git a/tstester/netcdf.py b/datastore/tstester/netcdf.py similarity index 100% rename from tstester/netcdf.py rename to datastore/tstester/netcdf.py diff --git a/tstester/netcdfsbe_tsmdatainpostgis.py b/datastore/tstester/netcdfsbe_tsmdatainpostgis.py similarity index 100% rename from tstester/netcdfsbe_tsmdatainpostgis.py rename to datastore/tstester/netcdfsbe_tsmdatainpostgis.py diff --git a/tstester/pgconnectioninfo.py b/datastore/tstester/pgconnectioninfo.py similarity index 100% rename from tstester/pgconnectioninfo.py rename to datastore/tstester/pgconnectioninfo.py diff --git a/tstester/pgopbackend.py b/datastore/tstester/pgopbackend.py similarity index 100% rename from tstester/pgopbackend.py rename to datastore/tstester/pgopbackend.py diff --git a/tstester/postgissbe.py b/datastore/tstester/postgissbe.py similarity index 100% rename from tstester/postgissbe.py rename to datastore/tstester/postgissbe.py diff --git a/tstester/storagebackend.py b/datastore/tstester/storagebackend.py similarity index 100% rename from tstester/storagebackend.py rename to datastore/tstester/storagebackend.py diff --git a/tstester/timescaledbsbe.py b/datastore/tstester/timescaledbsbe.py similarity index 100% rename from tstester/timescaledbsbe.py rename to datastore/tstester/timescaledbsbe.py diff --git a/tstester/timeseries.py b/datastore/tstester/timeseries.py similarity index 100% rename from tstester/timeseries.py rename to datastore/tstester/timeseries.py diff --git a/tstester/tstester.py b/datastore/tstester/tstester.py similarity index 100% rename from tstester/tstester.py rename to datastore/tstester/tstester.py