Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make signed URLs for output datasets optional #676

Merged
merged 19 commits into from
Aug 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
09b7c1b
REF: Remove unnecessary finalisation from template apps
cortadocodes Aug 14, 2024
161b847
ENH: Allow using non-signed URLs for output datasets
cortadocodes Aug 14, 2024
95b522c
REF: Stop storing `output_location` in `Analysis`
cortadocodes Aug 14, 2024
77d890e
FIX: Fix analysis finalisation when there are no output datasets
cortadocodes Aug 14, 2024
ec89235
TST: Test `Analysis.finalise` without using signed URLs
cortadocodes Aug 14, 2024
45697d1
ENH: Promote log message to info
cortadocodes Aug 14, 2024
36bd91b
REF: Remove unnecessary analysis finalisation from other template apps
cortadocodes Aug 14, 2024
04dcc06
TST: Simplify analysis test
cortadocodes Aug 15, 2024
282dec2
DOC: Add missing param strings in docstrings
cortadocodes Aug 15, 2024
a79b7ed
ENH: Handle all `requests` errors in `Dataset._get_cloud_metadata`
cortadocodes Aug 15, 2024
5a50648
ENH: Handle `requests` errors for URL-based datafiles
cortadocodes Aug 15, 2024
abbc1ec
ENH: Include `requests` exception in log messages
cortadocodes Aug 15, 2024
7d4013d
FIX: Raise for non-200 responses in URL datafile download
cortadocodes Aug 15, 2024
ebdc7a3
TST: Test working with inaccessible URL datafiles
cortadocodes Aug 15, 2024
01a2db0
TST: Test working with inaccessible URL datasets
cortadocodes Aug 15, 2024
79d85f5
ENH: Improve log message
cortadocodes Aug 15, 2024
402c8e5
FIX: Avoid trying to access buckets for URL datasets
cortadocodes Aug 15, 2024
888e052
DOC: Update app configuration docs
cortadocodes Aug 15, 2024
42940d1
CHO: Add version compatibility metadata
cortadocodes Aug 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/source/creating_apps.rst
Original file line number Diff line number Diff line change
Expand Up @@ -157,4 +157,6 @@ When the analysis has finished, it is automatically finalised. This means:

You can manually call :mod:`analysis.finalise <octue.resources.analysis.Analysis.finalise>` if you want to upload
any output datasets to a different location to the one specified in the app configuration. If you do this, the
analysis will not be finalised again - make sure you only call it when your output data is ready.
analysis will not be finalised again - make sure you only call it when your output data is ready. Note that the
``output_location`` and ``use_signed_urls_for_output_datasets`` settings in the app configuration are ignored if you
call it manually.
1 change: 1 addition & 0 deletions docs/source/creating_services.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ App configuration file (optional)
- ``configuration_manifest``
- ``children``
- ``output_location``
- ``use_signed_urls_for_output_datasets``

If an app configuration file is provided, its path must be specified in ``octue.yaml`` under the
"app_configuration_path" key.
Expand Down
176 changes: 89 additions & 87 deletions docs/source/inter_service_compatibility.rst

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions octue/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ class AppConfiguration:
:param str|dict|octue.resources.Manifest|None configuration_manifest: a manifest of datasets to configure the app
:param str|list(dict)|None children: details of the children the app requires
:param str|None output_location: the path to a cloud directory to save output datasets at
:param bool use_signed_urls_for_output_datasets: if `True`, use signed URLs instead of cloud URIs for dataset paths in the output manifest
:return None:
"""

Expand All @@ -118,12 +119,14 @@ def __init__(
configuration_manifest=None,
children=None,
output_location=None,
use_signed_urls_for_output_datasets=True,
**kwargs,
):
self.configuration_values = configuration_values
self.configuration_manifest = configuration_manifest
self.children = children
self.output_location = output_location
self.use_signed_urls_for_output_datasets = use_signed_urls_for_output_datasets

if kwargs:
logger.warning(f"The following keyword arguments were not used by {type(self).__name__}: {kwargs!r}.")
Expand Down
Loading
Loading