Access to output datasets via output manifest expires after 7 days

[cortadocodes]

Bug report

What is the current behavior?

The signed URLs generated for datasets referenced in output manifests expires after 7 days, which denies long term access to datasets. This happens even when the requester has permission to access the output datasets (because unauthenticated signed URLs are used in output manifests instead of authenticated access to gs:// addresses).

What is the expected behavior?

Signed URLs were introduced to allow parents access to output datasets generated by a child which were then stored in the child's bucket where the parent didn't have permission to access that bucket directly (e.g. parents from outside an organisation to access outputs generated by a child inside that organisation).

Proposed Solution

• For parents and children with access to the same buckets (e.g. inside the same organisation), there should be an option to use gs:// addresses in the output manifest to allow long term authenticated access to output datasets. This could be switched on via an environment variable, the application configuration, or an argument to Service.ask.
• Consider if this should be the default
• Add conversion of a signed URL to a gs:// address in the event of failure to retrieve data from it
• Raise a better error when trying to access a dataset with an expired signed URL
• Update documentation on finalise and app configuration

[cortadocodes]

@thclark I'm interested to know if you think using gs:// paths should be the default? I think it makes sense for most use cases of the sdk