Common PR for new range serach for level and period #161

Teddy-1000 · 2024-08-20T07:16:06Z

This PR will serve as a common place to merge all issues realted to #151 #152 #155 #154 #158 since these issues demand changes in functionality in all 3 services.

This reverts commit 5c00783.

* Add converstion of iso duration to seconds * Fix parsing error * Change to model validator

github-actions · 2024-08-20T07:23:06Z

API Unit Test Coverage Report

File	Stmts	Miss	Cover	Missing
\_\_init\_\_.py	0	0	100%
datastore_pb2.py	65	52	20%	34–85
datastore_pb2_grpc.py	54	27	50%	15–16, 19, 65–80, 121–123, 128–130, 135–137, 142–144, 148–173, 219, 246, 273, 300
export_metrics.py	10	0	100%
grpc_getter.py	20	11	45%	15–19, 23–26, 30–32, 36–38
locustfile.py	15	15	0%	1–31
main.py	43	7	84%	45, 50, 60, 70–71, 81–82
metadata_endpoints.py	65	31	52%	45–54, 58, 85, 100–219, 223
response_classes.py	5	0	100%
utilities.py	174	51	71%	20, 38, 45, 67–70, 78–89, 94–101, 121, 125, 127, 155, 161, 179, 193–194, 198, 214–218, 222–228, 232–234, 245–249, 264, 268, 290, 295
custom_geo_json
edr_feature_collection.py	6	0	100%
formatters
\_\_init\_\_.py	11	0	100%
covjson.py	59	1	98%	91
geojson.py	21	2	90%	27, 52
openapi
custom_dimension_examples.py	4	0	100%
edr_query_parameter_descriptions.py	11	0	100%
openapi_examples.py	13	0	100%
routers
\_\_init\_\_.py	0	0	100%
edr.py	101	4	96%	348–349, 438–439
feature.py	47	19	60%	99–132, 148–153, 159–181
TOTAL	724	220	70%

API Unit Test Coverage Summary

Tests	Skipped	Failures	Errors	Time
30	0 💤	0 ❌	0 🔥	2.076s ⏱️

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

This change adds partial support for reflectable metadata fields of type int64, with the ts metadata field 'period' as the first example. This is a basis for later supporting efficient range filtering. The following two caveats apply for now: [1] the default int64 value of 0 is omitted from the output (since protoc puts 'omitempty' for all fields in the generated file); [2] it is not possible to filter at all on reflectable int64 fields. Both will be addressed in later commits.

Reflectable (i.e. handleable with Golang reflection) metadata fields are now supported with both range search (lo <= x <= hi) and regular string search (str(x) matching "12", "12*" etc.).

…el-changes

…s integers

…4-periods-and-levels-as-int

* Reject if duration not convertable to seconds and fix error occured due to change in attributes type in protobuf * remove string constraints from period

…_filter Updated proto file

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

Issue 154

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

* fix level and period for mqtt * fix warnings due to period conversions. * fix type error in send_message. * change attribute name in model.py

datastore/datastore/go.mod

datastore/datastore/storagebackend/postgresql/putobservations.go

datastore/migrate/data/migrations/1709282607_initialise_schema.up.sql

ingest/api/ingest.py

ingest/api/model.py

Update exeption to ISO8601Error.

Teddy-1000 and others added 3 commits August 20, 2024 08:48

Fix parsing error

5c00783

Revert "Fix parsing error"

6b9821a

This reverts commit 5c00783.

Add converstion of iso duration to seconds (#160)

0a17568

* Add converstion of iso duration to seconds * Fix parsing error * Change to model validator

Teddy-1000 and others added 26 commits August 26, 2024 09:37

Merge branch 'merge-backend-duration-and-level-changes' of github.com…

9e7bbc1

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

Merge main

0ad1f75

Cleaned up README

610240d

Fixed typo

66f7953

Updated go.mod

2177566

Added filter support for int64 reflectable fields

f6c0e7a

Reflectable (i.e. handleable with Golang reflection) metadata fields are now supported with both range search (lo <= x <= hi) and regular string search (str(x) matching "12", "12*" etc.).

Represented level as int64 instead of string

be08832

Added docs for integer range filter

f2d20a5

Merge branch 'main' into issues_153_155_158

029c3d9

Merge branch 'issues_153_155_158' into merge-backend-duration-and-lev…

685a82a

…el-changes

Added documentation

819586d

Add converstion of level to centi-meters

42e0b09

Update requirements

dbe90e2

Convert level in ingest

18c9508

test: update knmi data-loader

39623e7

test: update unit tests & test data

c055c4f

feat: Update endpoint return data to handle periods & levels stored a…

bb2ba22

…s integers

Fixed typo

1ea3672

Fixed typo and bug

6c5d9a6

Merge branch 'merge-backend-duration-and-level-changes' into issue-15…

9c16922

…4-periods-and-levels-as-int

refactor: format P1D -> PT24H

7b33ea9

refactor: knmi data-loader PT01H -> PT1H

6086275

test: update integration test data & integration tests

038414f

refactor: periods custom filtering

cc9812d

refactor: levels filtering to integers

116d623

fjugipe and others added 18 commits September 19, 2024 11:54

test: update client example test

c0b75f0

Updated proto file

a74f2f0

fix: allow lowercase periods as request parameters

1fb617a

refactor: clean up get_iso_8601_range function

49389e1

refactor: converting m_to_cm as a function

cb34a16

refactor: return levels as floats

e553f9e

feat: filter periods with multiple ranges

0a01434

Supported open values in int range filter

541659f

Fixed lint issues

068514f

Reject if duration not convertable to seconds (#176)

de85f44

* Reject if duration not convertable to seconds and fix error occured due to change in attributes type in protobuf * remove string constraints from period

Merge pull request #174 from EURODEO/issue_171_open_vals_in_int_range…

a7c673f

…_filter Updated proto file

Add hash of parameter_name

9a772da

Merge branch 'merge-backend-duration-and-level-changes' of github.com…

f99ee16

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

Merge pull request #172 from EURODEO/issue-154-periods-and-levels-as-int

686eed8

Issue 154

Merge branch 'merge-backend-duration-and-level-changes' of github.com…

20d4664

…:EURODEO/e-soh into merge-backend-duration-and-level-changes

fix level and period for mqtt (#182)

84430ad

* fix level and period for mqtt * fix warnings due to period conversions. * fix type error in send_message. * change attribute name in model.py

Merge remote origin/main

9cd3aa0

Fix formatting

fe63bb7

Teddy-1000 marked this pull request as ready for review September 23, 2024 08:53

Teddy-1000 requested a review from Jeffrey-Vervoort-KNMI September 23, 2024 08:55

Jeffrey-Vervoort-KNMI reviewed Sep 25, 2024

View reviewed changes

Update ingest.py

96a8e0b

fjugipe reviewed Sep 27, 2024

View reviewed changes

ingest/api/model.py Outdated Show resolved Hide resolved

shamlymajeed and others added 2 commits September 30, 2024 11:28

Update model.py

59bdfb0

Update exeption to ISO8601Error.

Removed obsolete protobuf version

730d8d3

Jeffrey-Vervoort-KNMI approved these changes Oct 1, 2024

View reviewed changes

Teddy-1000 merged commit 325e5e9 into main Oct 2, 2024
8 checks passed

Teddy-1000 deleted the merge-backend-duration-and-level-changes branch October 2, 2024 05:31

Teddy-1000 restored the merge-backend-duration-and-level-changes branch October 2, 2024 12:05

fjugipe mentioned this pull request Oct 9, 2024

[API] Value error in period return internal server error #191

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Common PR for new range serach for level and period #161

Common PR for new range serach for level and period #161

Teddy-1000 commented Aug 20, 2024

github-actions bot commented Aug 20, 2024 •

edited

Loading

Common PR for new range serach for level and period #161

Common PR for new range serach for level and period #161

Conversation

Teddy-1000 commented Aug 20, 2024

github-actions bot commented Aug 20, 2024 • edited Loading

API Unit Test Coverage Summary

github-actions bot commented Aug 20, 2024 •

edited

Loading