Guidance on when is it best to add FAIRmat contributions to offer a refactored @EBSD/export_h5.m

[mkuehbach]

Dear @ralfHielscher and team,

I have picked up working again on the EBSD-related aspects that I showed at the 2023 MTex Meeting. In the meantime, we have continued working on other topics, still would like to contribute our work to the MTex project. Specifically our HDF5 related work as code. In the meantime, MTex has evolved happily which is great to read. As I see from the repo history, the project is currently in a transition period where a v6.0 is about to become finalized but the export_h5 for EBSD has still not seen much changes, so our work could fit well. When do you think the first non-beta v6.0 of mtex is about to become released?
We would then integrate our code on a feature branch of a fork at that point from the development
branch and include the fact that now 3D grains can be handled.

Bests,
Markus Kühbach (on behalf of FAIRmat Area B)

[nyyssont]

What would this mean in practice? Would it be possible to export a .h5oina file from MTEX?

My understanding is that Matlab already saves in hdf5 format. What would be the difference here?

There was some discussion on the FAIR concept in the last workshop. Any development in this direction is welcome in my opinion.

[mkuehbach]

The work that I have implemented addresses several points, where I would be happy about guidance:

• Wrapper for HDF5 to make several low-level features of the library usable via Matlab convenience functions
• Several functionalities offering a suggestion for a semantic file format (data model + implementation) how users of MTex can export computed results not only as a *.mat file but using not just some ad hoc HDF5 format but an HDF5 file whose content is described semantically
• Examples how to export IPF, ODF and PFs using this format
• This covers the functional side of things, we have used the functionalities on that MTex feature branch to create as an example a database of several thousand EBSD datasets from more than hundred published studies worldwide using NOMAD, where preprocessing is done with MTex resulting in these datasets and the MTex-processed results explorable in Oasis.

I guess this is meaty enough content that we should discuss soonish maybe @ralfHielscher, @ppinard , @hakonanes, @benjaminbritton are also interested? Here is one of your datasets @nyyssont, MTex was used for processing about 2800 thousands of these, the remaining where processed with pyxem:

Maybe a zoom call is more efficient to get this structured?

[mkuehbach]

What would this mean in practice?

see above

Would it be possible to export a .h5oina file from MTEX?

yes and no (the hdf5 wrapper makes writing HDF5 content beyond what the high-level Matlab function h5* calls offer easier. If s.b. wishes to use this for writing an MTex to H5OINA export that is possible. Do I propose following this route though? No, because instead the MTex community should think about standardizing the content which MTex scripting enables to write to s. th. with stronger semantic expressiveness and machine-readability. This is well possible as often MTex class object member functions are called to perform specific tasks processing well-defined input with well-defined parameterization. The output of this can be stored not just ad hoc where many do it differently but in a way that people agree to do it with fewer and using semantic representations.
Exactly this is for what I have implemented a proposal and an implementation (see above).
Specifically, a semantic file format that can be used with HDF5 (but in could also be used with JSON/YAML or OMERO as container backend , but we have not implemented this). What I have proposed goes beyond the domain particularity of H5OINA as it is vendor-agnostic by design. Why is having all this beneficial. I knew that when still back in Aachen people were computing texture and showing ODFs sections with oftentimes no clear contextualization. Now with the above-mentioned and de facto already implemented solution one can even have ODF 3D sections be exported in a standardized way complementary to figures in the paper but more important interactively explorable via research data management systems like what is what we are building in NOMAD.

The benefit of having the functionality offloaded from the particular RDM system is that RDM software developer can use these as a library and have the RDM main code free from the above-mentioned highly domain-specific how-to-do-mathematically-a-specific IPF plot for a low-symmetry crystal phase (https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXem_ebsd.html#nxem-ebsd) is the respective data model for storing vendor agnostic EBSD data extending the data model that the community has already proposed with H5EBSD but going beyond that and going semantic.

My understanding is that Matlab already saves in hdf5 format. What would be the difference here?

Indeed *.mat under the hood is HDF5 (in newer versions) but HDF5 is just a container so storing content ad hoc in containers is not of much use when considering FAIR data, key is that the data are stored in a container in such a way that for each content instance piece of information there is a machine-resolvable context and contextualization, like in proposed NXem semantic file format:
https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/em-structure.html#em-structure

There was some discussion on the FAIR concept in the last workshop. Any development in this direction is welcome in my opinion.

Great.

[mkuehbach]

I saw the point "unified h5 export for EBSD from MTex" content action point here: #897
I now realize above-mentioned work adds exactly this feature as a by product.