Create AWS S3 Interface for TIFF Files #458
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
https://github.com/constantinpape/z5 may also be of interest to target more cloud-native data formats. |
|
Thanks @tacaswell This is just a test, not sure I am married to TIFF (as you know!). But, by using the AWS S3 async upload I can get line rate out of my office computer......... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@MarkRivers .. This is more of an exploratory "DEMO" PR, I wanted to test using the AWS S3 storage directly from AD that we talked about over e-mail.
I have verified that this works at the moment writing to a MinIO object store at SDCC here at BNL but I have no reason it would not work directly to Amazon S3 object storage.
MinIO is here:
https://min.io/
This basically gets rid of the requirement to have a file system to write images to.
This is "super hacky" and is so brute force and ugly it might even be elegant.
With some changes to the base NDFileTIFF it would only be a small shim over the files.
Comments from all welcome...
P.S. I realize that there is copied code in there. I didn't want to play too much with the NDFileTIFF at this stage. However, one could separate this out a little bit and make the NSFileTIFFS3 really simple.