Visibility of existing files/directories in /data during ingress

[DDelbarre]

When ingressing data via Azure Storage Explorer, it is possible for whovever is uploading the data to see (but not read) the other directories and files in the /data volume of the SRE.

For projects like EDoN, where there are a number of separate data providers who will ingress data into an SRE, this could be quite problematic. This is because data providers will be able to see the files and directories uploaded by the other providers. In cases where there are files like medical images (e.g. MRI scans) these typically have patient IDs in the file names, which we would not want other providers to see.

I was wondering if it was possible to restrict what data providers can see when ingressing data? I had an idea as to how this could be implemented, but there are probably other approaches too!

We currently use the following directory structure for EDoN in /data:

2023-01-01
    file_1
    file_2
2023-02-01
    file_1
    file_2

If we changed to something like the following, the data providers could be given an upload token to their specific directory. That way they could still see the data that they have uploaded previously (which could be useful for them to ensure they are not re-uploading duplicated data) but they would not be able to see anything else.

data_provider_1
    2023-01-01
        file_1
        file_2
data_provider_2
    2023-02-01
        file_1
        file_2

In the forms that are used for recording the details of the data ingress prior to it taking place, there could also be a section where the storage location could be specified to ensure that access is being given to the appropriate location.

[edwardchalstrey1]

I will have a proper think about this when I can, but just to quickly note that on Azure and it’s not obvious whether the token permission can set to a specific sub-folder like /data/data_provider_1 for instance

[JimMadge]

The SAS token permissions are listed here.

The documentation suggests using 'list' and 'write' (not 'read'). So, the uploader would then be able to see the names of existing blobs, but not their contents, and to write new blobs.

If we want to hide the names themselves we could remove the 'list' permission. You would need to be a bit careful to make sure that no one overwrites existing data. E.g. two providers upload a directory with the same name.

@DDelbarre Thanks for thinking about this. If you make this clear when making an ingress request I believe it should be possible.

@craddm @edwardchalstrey1 If either of you have a sandbox running, could you check exactly what you can see with and without the list permission? I.e. can you see all filenames, just the top-level directories or something else.

We should be confident how this works before making an recommendations. This would be a great addition to the documentation.

[JimMadge]

StackOverflow answer: "You can only get Shared Access Signature for a blob container or for a single blob, but you are NOT able to get Shared Access Signature for a blob virtual directory, since directory isn't a real concept in Azure Blob Storage."

👍 Could you also just double check that this is true?

How about this for a solution:

1. I set up a new blob container for each data uploader and provide them a token for that

2. The data uploader performs the upload

3. I copy the files over to appropriate sub-folders in the ingress container with my admin access in Azure Storage Explorer, making them available to everyone logged into the SRE at `/data`, but not the uploader

4. I delete the containers that were created for this purpose

@edwardchalstrey1 That sounds like a reasonable work around 🙌 , as long as the new containers have the same 'security configuration' as we would expect normally (e.g. restricted to specific IPs). Could you just confirm this wouldn't mean you have any more access than you normally would?

If this works, it would be good to write some guidance on supporting multiple data providers. I feel like the best thing is just to not put sensitive data in file/directory names. However, I think what you have described above @edwardchalstrey1 seems like a good way to support cases where this is difficult or a data provider feels uncomfortable about other providers seeing file/directory names.

[edwardchalstrey1]

👍 Could you also just double check that this is true?
Yes, there's no way to do it in Azure Portal

I think your suggestion to update the docs for multiple data providers and saying explicitly that in that case the best practice is to not have sensitive data in filenames, sounds like the best update to the docs

I propose to try out my suggestion for Edon's case, and even assuming this works as intended, not to document this as a best practice in the docs. But we can leave this discussion up here on GitHub so if future DSH System Manager's came across the exact specific problem they can still see what we did

[edwardchalstrey1]

Also just to check @DDelbarre are you happy with this? If so, I can generate these separate SAS urls for the data uploaders

[JimMadge]

I propose to try out my suggestion for Edon's case, and even assuming this works as intended, not to document this as a best practice in the docs. But we can leave this discussion up here on GitHub so if future DSH System Manager's came across the exact specific problem they can still see what we did

I understand not wanting to recommend this as best practice. If this works I would like to record it somewhere more authoritative/findable than in a discussion thread though.

[DDelbarre]

Hi both, thanks for looking into this. I think that the proposed solution sounds good.

If this works, it would be good to write some guidance on supporting multiple data providers. I feel like the best thing is just to not put sensitive data in file/directory names. However, I think what you have described above @edwardchalstrey1 seems like a good way to support cases where this is difficult or a data provider feels uncomfortable about other providers seeing file/directory names.

In some cases I think it might be unavoidable to have IDs in filenames for EDoN. Going back to my example of MRI scans, the layout of the directory structure and naming of the files is controlled in these datasets using industry standard specifications (e.g. BIDS) that require the ID to be in the filenames. In other projects I have worked on we have tried zipping MRI datasets for transfer but have had issues with transfers getting interrupted due to the large file sizes and needing to restart transfers multiple times.

Also some of our data providers in EDoN are large organisations that won't undertake custom data preparation so we don't have much control over the format of this data prior to ingress.