Fix bug: add support for old and new Zenodo APIs #375
Conversation
Update the Zenodo downloader to work with the new Zenodo API. Use the `filename` key for getting the filenames of all files in a repository. Update the generated download url based on heuristics: the link present in the API reference doesn't work at the moment). Update the method that populates the registry: use the new `filename` key and specify that the checksums are now md5.
Detect which API are we interacting with and generate the download url and/or populate the registry accordingly.
Test if `ZenodoRepository` correctly supports both the old and new Zenodo APIs. Use `pytest-httpserver` to run a local server that returns API response that mimics the Zenodo ones. Define a `base_api_url` class attribute for `ZenodoRepository` so we can override it in tests.
|
@santisoler thanks for the continued work on the issue by you and your colleagues. The strategy seems prudent to me: release code that tries to work regardless of whether Zenodo (a) stick with the API as it is currently operating or (b) manage to provide the backwards compatibility that they promised. Per your comment here, it seems like we can't be 100% confident that they won't (c) make further changes that are neither (a) nor (b)—but it would be impossible to predict that, anyway, so this is the best possible action for the moment. |
|
Thanks @khaeru. You're right, the best case scenario would be that Zenodo provide the details of the API they'll maintain in the long run. But considering all the issues they're trying to solve after the migration, we cannot be sure when it will happen. So I think it's better to release some fixes that would Pooch work again to download files from Zenodo, until they sort this out. Since the only information we have so far on their intentions is to keep their API backward compatible, I think that allowing our code to interact with such API is a sensible decision. I'll rerun all tests today to check that everything is working after the changes they introduced during the weekend and if all goes smoothly, tomorrow we'll have a new release of Pooch. |
Allow the Zenodo downloader class to work both with the previous Zenodo API and the new API that is running since their migration to InvenioRDM on 2023-10-13. The downloader class
ZenodoRepositorydetermines which API version is interacting with and defines the download url and populate the registry based on that. Test ifZenodoRepositorycorrectly supports both the old and new Zenodo APIs. Usepytest-httpserverto run a local server that returns API response that mimics the Zenodo ones. Define abase_api_urlclass attribute forZenodoRepositoryso we can override it in tests.Relevant issues/PRs:
Fixes #371
Related to #373
This PR was triggered by discussions on the Fatiando Meeting of 2023-10-19.