Npz sniffing: do not read the whole file

[bernt-matthias]

I noticed that an upload of https://data.qiime2.org/2024.2/common/silva-138-99-nb-classifier.qza fails because of high memory load (for the file in question 18GB). One can try on usegalaxy.org.

Reason was that the npz sniffer reads the whole file (npz seems to be just zip files).

Also tried to use the mmap_mode for load (https://numpy.org/doc/stable/reference/generated/numpy.load.html) but this changed nothing.

Btw. I used memray which showed me the problem in seconds.

Could also backport further if needed (#11957)

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[mvdbeek]

Good catch, thank you!

[github-actions]

This PR was merged without a "kind/" label, please correct.

[mvdbeek]

Arg, wrong target branch, can you open against 23.2 as well ?

[bernt-matthias]

Ups. Sorry: #17674