new normalisation #510

Dragos0000 · 2023-10-23T13:53:27Z

Do not merge this into main. This has to be tested

- add optional credentials for GitHubMappingSuitePackageDownloader - add test for credentials injection in url - add tests for semantic versioning check of mapping suite packages.

implement github download with credentials

Update conceptual_mapping_processor.py

Add repository_name param for github package downloader

Update sample application with new queries

Feature/ted4 10

resolve bug with daily notice metadata stores not days without notices in TED API

code cleaning and update to pydantic v2

optimise e2e tests

Feature/ted4 22

codecov-commenter · 2023-10-23T14:06:47Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (5aa4c19) 94.78% compared to head (80b60a7) 94.89%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #510      +/-   ##
==========================================
+ Coverage   94.78%   94.89%   +0.10%     
==========================================
  Files         116      117       +1     
  Lines        6272     6463     +191     
==========================================
+ Hits         5945     6133     +188     
- Misses        327      330       +3

Files	Coverage Δ
...tice_metadata_processor/adapters/xpath_registry.py	`99.32% <100.00%> (ø)`
...ed_sws/notice_metadata_processor/model/metadata.py	`100.00% <100.00%> (ø)`
...metadata_processor/services/metadata_normalizer.py	`98.27% <100.00%> (+0.17%)`	⬆️
...s/notice_packager/services/metadata_transformer.py	`100.00% <100.00%> (ø)`
...ed_sws/notice_packager/services/notice_packager.py	`100.00% <100.00%> (ø)`
...ta_processor/adapters/notice_metadata_extractor.py	`99.19% <99.07%> (ø)`
...a_processor/adapters/notice_metadata_normaliser.py	`97.31% <97.31%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

CaptainOfHacks and others added 30 commits July 4, 2023 16:26

implement github download with credentials

7b109a6

- add optional credentials for GitHubMappingSuitePackageDownloader - add test for credentials injection in url - add tests for semantic versioning check of mapping suite packages.

Fixed tests for notice_publisher feature

5461333

Merge pull request #485 from OP-TED/feature/TED-1401

27b4228

implement github download with credentials

Update conceptual_mapping_processor.py

aef716e

Merge pull request #486 from OP-TED/feature/TED-1401

c3a866a

Update conceptual_mapping_processor.py

wip

0ff927d

Merge pull request #487 from OP-TED/feature/TED-1401

61f8d9c

Add repository_name param for github package downloader

Update notice_processing_pipeline.py

49ac175

Update DagBatchPipelineOperator.py

2503685

Update notice_batch_processor_pipelines.py

9410465

Update notice_processor_pipelines.py

619cc4b

Update pipeline_protocols.py

7cb91d9

Create MappingSuiteTransformationPool

52bc6e7

Update documentation

188ef86

Extend notice eligibility services

7f6d107

Create notice_batch_transformer.py

6c68d78

Update docker-compose-worker.yaml

56a8152

Add AIRFLOW_NUMBER_OF_WORKERS env config

9cc2aab

Update docker-compose.yaml

bcb162c

Update docker-compose.yaml

c60ddf3

Update Airflow version to 2.7.1

e41e97c

Update rmlmapper to 6.2.1

9c3922a

Update project requirements

f0d8ebd

update sample application with new queries

905c68b

Create sample_application.xlsx

05e7895

Adjust tests and model logic to suport pydantic 2 version

4ac8b8b

Merge pull request #489 from OP-TED/feature/TDA-272

899d41f

Update sample application with new queries

Update conftest.py

bacff55

Update conftest.py

c3ff970

Update entity_deduplication.py

0c44bfa

duprijil and others added 25 commits October 4, 2023 16:32

Merge pull request #503 from OP-TED/feature/TED4-10

095801d

Feature/ted4 10

resolve bug

70be603

Update daily_notices_metadata_services.py

6ee10a5

Update test_notices_metadata_services.py

ba3cf16

Update daily_notices_metadata_update.py

3e3fcbf

Update daily_notices_metadata_update.py

dd053ed

Update daily_notices_metadata_update.py

da1977c

make start picking at dag level

b67146b

Update daily_notices_metadata_update.py

2733359

Update daily_notices_metadata_update.py

b1c5a1c

Update daily_notices_metadata_update.py

a31ae28

Merge pull request #506 from OP-TED/feature/TED4-66

e26fe8b

resolve bug with daily notice metadata stores not days without notices in TED API

code cleaning and update to pydantic v2

774c953

Merge pull request #507 from OP-TED/feature/TED4-64

39e3df4

code cleaning and update to pydantic v2

optimise e2e tests

318697e

clean tests

231af55

Merge pull request #508 from OP-TED/feature/TED4-67

8c86cd0

optimise e2e tests

add filter by mapping_suite_id for re-transform DAG

14b0319

Merge branch 'main' into feature/TED4-22

0703cb1

Merge pull request #509 from OP-TED/feature/TED4-22

5aa4c19

Feature/ted4 22

extreactor adapters

4bc2a87

define new notice extractor and normaliser architecture

c4c23b3

first draft WIP

2005615

Update test_notice_packager.py

6e5260f

test passing

80b60a7

Dragos0000 requested a review from CaptainOfHacks October 23, 2023 13:53

CaptainOfHacks force-pushed the main branch from 2701874 to 9a6a39a Compare January 16, 2024 11:13

Dragos0000 closed this May 7, 2024

Dragos0000 deleted the feature/TED4-58 branch May 7, 2024 11:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new normalisation #510

new normalisation #510

Dragos0000 commented Oct 23, 2023

codecov-commenter commented Oct 23, 2023

new normalisation #510

new normalisation #510

Conversation

Dragos0000 commented Oct 23, 2023

codecov-commenter commented Oct 23, 2023

Codecov Report