Publishing rework

[blue442]

Addressing #404 - adopted the existing infastructure to allow for the publishing of a FoundryDataset object as opposed to the previous approach of passing arguments to the Foundry.publish() function. This will allow better validation of the required components - in particular, the objects containing the datacite and metadata information.

To facilitate the validation of the datacite and metadata information, the dc_model.py and project_model.py files were created using the datamodel code generator package, which took the existing jsonschema definitions for these objects from the MDF schema repository and created a pydantic representation to be used in the validation. In an effort to ensure reproducibility for any future changes to the schemas, the resultant .py files from the datamodel code generator were left full intact. These are used as superclasses to model definitions for FoundrySchema and FoundryDatacite objects in the models.py file that add the required logic for the pydantic classes to work well with the Foundry package.

The dataset_publishing.ipynb was updated to reflect the new approach, but will require additional work once #403 is undertaken.

[what-the-diff]

PR Summary

• Method Renaming in foundry.foundry.md

  • The list method has been renamed to get_metadata_by_doi to reflect its purpose better.

• Updates to dataset_publishing.ipynb

  • The notebook's content has been expanded with explanations and examples on constructing FoundryDataset objects.
  • New information related to DataCite metadata and its application has been added.
  • The script for publishing datasets to Foundry has been revised.
  • A section which added datacite information via kwargs to the foundry.publish() method is now deprecated.
  • The notebook version has been updated from 3.9.4 to 3.10.0.

• New File: dc_model.py

  • A new data model file dc_model.py has been created. It defines a data model generated by datamodel-codegen tool.

• Changes to foundry.py

  • Two parameters parallel_https and local_cache_dir were added to the initialisation method.
  • The search method has been updated to handle variety of options for searching data.
  • The dataset_from_metadata and search_results_to_dataframe methods have been enhanced for improved functionality.
  • The publish_dataset method has been simplified and now includes metadata and datacite validation.

• Changes to foundry_dataset.py

  • An add_data method has been incorporated.
  • The FoundryDataset class now supports datacite_entry and foundry_cache parameters.
  • Some methods like get_as_dict, get_citation etc., have been streamlined by removing unused arguments.

• Updates to models.py

  • Unused imports and classes have been removed for cleaner code.
  • FoundrySchema and FoundryDatacite classes have been updated for better data handling and validation.

• Addition of foundry/project_model.py

  • New models classes have been added for project specification.

• Enhancements to Unit Tests

  • Tests in tests/test_foundry.py and tests/test_foundry_components.py have been revamped to align with new coding standards.
  • Updates to tests/test_foundry_cache.py improves testing accuracy with new source ID data and usage of the FoundrySchema class.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.63636% with 28 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (dev@8b2b61a). Click here to learn what that means.

❗ Current head 4f15455 differs from pull request most recent head 69722ba. Consider uploading reports for the commit 69722ba to get more accurate results

Files	Patch %	Lines
foundry/foundry.py	60.00%	8 Missing ⚠️
foundry/foundry_cache.py	63.63%	8 Missing ⚠️
foundry/models.py	80.00%	8 Missing ⚠️
foundry/foundry_dataset.py	80.00%	4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff           @@
##             dev     #451   +/-   ##
======================================
  Coverage       ?   74.01%           
======================================
  Files          ?       12           
  Lines          ?      943           
  Branches       ?        0           
======================================
  Hits           ?      698           
  Misses         ?      245           
  Partials       ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.