Refactor for builder changes and Individual Datasets #7

JLoveUOA · 2024-07-24T00:37:46Z

Two major changes in this PR.

Refactored based on builder changes

Encrypted graph is now expected to be written within the RO-Crate graph
Sensitive data now generates recipients entities based on public keys supplied.
Bulk encryption of output archive is now supported via the --bulk-encrypt command
Metadata holds schema url information
groups are represented as ACLS
New Data types are created in the RO-Crate
- Organizations
- facilities
- group (for both ACLS and facility manager groups)
- ACLS as Digital Document Permissions

Print Lab Extractor now Produces individual dataset Crates

Datasets now collect all sub-files and directories ONLY If 'crate children' is specified in the input worksheet
- The --duplicate-directory parameter will also cause each dataset to fully duplicate the crate source.
the --split datasets parameter will now produce an individual dataset RO-Crate for each dataset in the spreadsheet
--duplicate-directory and --split-datasets are not recommend to be used together (this duplicates your data N times where N is your number of datasets)
Datasets hold an individual copy of all relevant metatada

Added an ICD-11 lookup API for medical condition data

Added tests for all print lab data extraction and generic metadata extraction

… up from RO-Crate group

* add gpg to PR workflow * add cli tests

ChrisKSeal · 2024-09-24T01:22:45Z

pyproject.toml

+ro-crate-py = {git = "https://github.com/UoA-eResearch/ro-crate-py.git",  branch = "encrypted-metadata"}
+openpyxl = "^3.1.5"
+click = "^8.1.7"
+faker = "^27.4.0"


Do Faker, FactoryBoy and Hypothesis want to sit in general dependencies or should these be dev dependencies? I.e. do we expect our users to run the tests?

Just a thought

Moved them to a test group, I don't think I even ended up needing hypothesis and factory boy so will remove those

ChrisKSeal · 2024-09-24T01:34:04Z

src/ingestion_targets/print_lab_genomics/__init__.py

Should this be in a separate .env file? or in the conftest.py?

There is a value in the ENV file that updates this with a new pubkey this one is just hard-coded in by default.
I should definitely update the email to a dummy value.

ChrisKSeal · 2024-09-24T02:00:46Z

tests/conftest.py

+
+
+def fake_upi(faker: Faker) -> str:
+    return faker.bothify(text="???###", letters="abcdefghijklmnopqrstuvwxyz")


Is this missing a character? Should be ????### IIRC :)

ChrisKSeal · 2024-09-24T02:03:54Z

tests/examples_for_test/print_lab_test/.~lock.sampledata.xlsx#

I'm guessing this is included by mistake :)

yes removed :)

ChrisKSeal · 2024-09-24T02:07:11Z

tests/test_apis.py

+
+
+@responses.activate
+def test_mytardis_rest_agent_get(


Not sure what this test is aiming to do. The response has been mocked so asserting that the status is 200 is not useful here. It does demonstrate that a get request has been made :)

Basically all that was for, that a request happens at all. Added a few tests for bad requests to make it more complete.

ChrisKSeal · 2024-09-24T02:09:15Z

tests/test_apis.py

+
+    response = mt_rest_agent.no_auth_request("GET", url)
+    do_not_use_auth.assert_not_called()
+    assert isinstance(response.json(), dict)


These three tests are mocked so asserting them is surplus to requirements I think.

reduced just down to checking the auth is not used

JLoveUOA added 30 commits June 5, 2024 12:02

update source on dataclasses that should contain children

9651d51

update source on dataclasses that should contain children

e16a526

Updated based on builder changes to place encrypted in graph and tidy…

af3aa94

… up from RO-Crate group

resolve linting, disable old ABI parser

21e8c43

resolve linting, disable old ABI parser

eec62c1

resolve linting, disable old ABI parser

ec66742

resolve linting, disable old ABI parser

3002d19

remove interrogate and it's dodgy py dependency

a9daa90

update encrypted context entity

b611a1a

updates from refactor

e82d41e

encrypted paitent sensitive

6968ce2

removed uneeded dependencies

7684e48

made path corrections for bulk encryption output format

b3c7a7b

linting fixes

217af94

encrypt participant sensitive

3c762f2

linting for participant sensitive

4ce34be

add ICD11 API calls for disease codes

6f7cd80

add ICD-11 api for medical condition lookups

9e11797

add ICD-11 api for medical condition lookups

c0140ae

Added baked in MyTardis user from env

fb08613

Added baked in MyTardis user from env

1d2324c

update samplesheet

9574277

update samplesheet

e70418e

add test packages and example.env file

83ca24a

Update README

df1d066

Update README

71627be

un-delete line for input metadata

6e6f4dc

un-delete line for input metadata

beafabd

fix missing pubkey for bulk encryption

72bc678

update based on builder changes

1f5bc26

JLoveUOA and others added 19 commits September 10, 2024 12:09

test metadata extraction

6d42e2c

test metadata extraction (fixtures)

0bb574b

change directory to samplesheet to allow for relative paths

b205a44

add tests for API agents

3dfbce7

remove commented out section at end of API tests

b75cd5e

formatting changes for new tests

db4bf91

in-progess update to builder tests

6500e19

output path is absolute

f03c151

more options for print lab CLI

d2a7997

inital builder test fixtures

b350f40

add print lab builder tests

ba68385

add tests back into PR workflow

84a7c0d

update lock

ce1bd37

Test print lab cli (#12)

6768788

* add gpg to PR workflow * add cli tests

linting following rebase for cli tests

cf4490d

mock gpg api call

e222896

pubkeys removed for tests

f99465b

force gpg key for MyTardis user

12746a7

linting for gpg change

09ffcf8

ChrisKSeal reviewed Sep 24, 2024

View reviewed changes

ChrisKSeal approved these changes Sep 24, 2024

View reviewed changes

JLoveUOA added 3 commits September 24, 2024 15:25

fixes from PR to APIs and conftest

51dc358

remove default MT user pubkey

1927380

remove default MT user pubkey

d656564

JLoveUOA merged commit 40ca2eb into main Sep 24, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor for builder changes and Individual Datasets #7

Refactor for builder changes and Individual Datasets #7

JLoveUOA commented Jul 24, 2024 •

edited

Loading

ChrisKSeal Sep 24, 2024

JLoveUOA Sep 24, 2024

ChrisKSeal Sep 24, 2024

JLoveUOA Sep 24, 2024

ChrisKSeal Sep 24, 2024

ChrisKSeal Sep 24, 2024

JLoveUOA Sep 24, 2024

ChrisKSeal Sep 24, 2024

JLoveUOA Sep 24, 2024

ChrisKSeal Sep 24, 2024

JLoveUOA Sep 24, 2024



		def fake_upi(faker: Faker) -> str:
		return faker.bothify(text="???###", letters="abcdefghijklmnopqrstuvwxyz")

Refactor for builder changes and Individual Datasets #7

Refactor for builder changes and Individual Datasets #7

Conversation

JLoveUOA commented Jul 24, 2024 • edited Loading

Refactored based on builder changes

Print Lab Extractor now Produces individual dataset Crates

Added an ICD-11 lookup API for medical condition data

Added tests for all print lab data extraction and generic metadata extraction

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JLoveUOA commented Jul 24, 2024 •

edited

Loading