Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate HDF5 file performance #115

Open
ManuelHu opened this issue Aug 14, 2024 · 4 comments
Open

Investigate HDF5 file performance #115

ManuelHu opened this issue Aug 14, 2024 · 4 comments
Labels
output Output Schemes

Comments

@ManuelHu
Copy link
Collaborator

Not urgent and probably not necessary, but

  • check that Geant4 HDF5 writing parameters are best for performance in our use case(s)
  • check if we can change/influence them?
@tdixon97
Copy link
Collaborator

It might also be worth checking the memory usage, I find some steps use alot of memory (maybe lh5 converting), which can cause issues...

@ManuelHu
Copy link
Collaborator Author

I will do a test. LH5 converting is actually a two-step-process (dry-run + actual convert).The dry-run runs on a (I think memory-mapped+COW, hopefully?) in-memory copy of the file, that might be expensive...

@ManuelHu
Copy link
Collaborator Author

with #138, the memory overhead should hopefully be gone

@gipert gipert added the output Output Schemes label Nov 10, 2024
@ManuelHu
Copy link
Collaborator Author

The only parameter that we could reasonably change is the deflate (?) compression level 1-9, default 1.

the toolx::hdf5 also has some options for chunking, but they are not accesible via the analysis manager interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
output Output Schemes
Projects
None yet
Development

No branches or pull requests

3 participants