File merging broken on tip-d3 (0ab1a8e)

[evanhanders]

Just wanted to make a quick note that file merging (at least for virtual files) is broken on commit 0ab1a8e, probably before that. This generally makes sense; there seem to have been a lot of changes for how virtual files' scales and metadata are handled recently. I'll work on investigating this more soon and try to get a PR with fixes together ~this week.

[jsoishi]

The idea of the recent changes is to eliminate the need for merging virtual files. Do you have a use case for fixing this? It might not be worth the effort.

[evanhanders]

I think we're talking about two different processes when we talk about merging. Here, I'm talking about combining the data distributed across many files for a virtual file into one file. So (slices_s1/ & slices_s1.h5) => slices_s1.h5. There are two use cases for this:

1. Decreasing number of files used to store data by orders of magnitude to stay under file quotas
2. Increasing ease of moving data between computers (only having to move one file vs. a virtual file + many partials).

These are functionalities that I need in my research.

But you're right, the recent changes do make it so that all file outputs are immediately registered into virtual files, so there is no need for 'merging' from partial files, which is nice!