Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata handling with mutliple input files #31

Open
DamienIrving opened this issue May 19, 2015 · 1 comment
Open

Metadata handling with mutliple input files #31

DamienIrving opened this issue May 19, 2015 · 1 comment
Labels

Comments

@DamienIrving
Copy link
Contributor

At the moment, every time a command line program is run the relevant metadata (vistrail file name with git hash, script name with git hash, and complete history of command line entries all the way back to initial generation of the data) is stored in the global attributes of the output netCDF file (the build_metadata function within utils/utils.py does this).

This works fine if there is only one input file, but for processes with more than one input file the history of only one of the files is retained. In my own work I get around this problem by looping over all the input files, extracting global history attribute, and then appending them all to the global history attribute of the output file (see the write_metadata function within general_io.py for details). I think we need a similar approach here.

As well as the global history attribute, it would also be nice to append to the global "version" attribute for each of the input files, as in CMIP5 files this specifies which version of the data was used.

@DamienIrving
Copy link
Contributor Author

We might also want to think about what to do with the metadata associated with non-netCDF outputs (e.g. file formats like .png which are not self-describing). An approach I usually use is to produce a text file of exactly the same name (except with a .met extension instead of .png) in which I dump the global history attribute information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant