You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This script currently contains a lot of logic which controls how it combines different datasets. It would be better to have this logic encoded in the output file itself. So, for example, every dataset could have a mergeable attribute that specifies how to merge (e.g. append, sum, average etc.), and the script just applies that method. It could then simply walk the group hierarchy of the files and merge as needed. Groups could have a similar attribute to specify what to do with their attributes.
Also, the script is slow on large files. This may be due to the slowness of append (or glue) in PDL. Maybe it would be better to read the datasets from all files first (or at least get their sizes) and then create a merged dataset of the full size and just fill in the entries.
The text was updated successfully, but these errors were encountered:
This script currently contains a lot of logic which controls how it combines different datasets. It would be better to have this logic encoded in the output file itself. So, for example, every dataset could have a
mergeable
attribute that specifies how to merge (e.g.append
,sum
,average
etc.), and the script just applies that method. It could then simply walk the group hierarchy of the files and merge as needed. Groups could have a similar attribute to specify what to do with their attributes.Also, the script is slow on large files. This may be due to the slowness of
append
(orglue
) in PDL. Maybe it would be better to read the datasets from all files first (or at least get their sizes) and then create a merged dataset of the full size and just fill in the entries.The text was updated successfully, but these errors were encountered: