Support for numpy<2?

[maresb]

I'm really excited about this project, but it's currently pretty difficult to install in any non-minimal environment because Numpy 2.0 is so recent. How essential is your use of 2.0 features? Is there any chance of this being relaxed?

[TomNicholas]

Hey! We're using numpy 2.0 for the new variable-length string dtype, as a good way to efficiently store arrays of paths to files.

Context here:

#33

#107 (comment)

If you really want backwards-compatibility and you want to make a contribution we can accept a PR that generalizes this, but its always going to be less efficient to use a fixed-length string dtype.

[maresb]

Thanks a lot for the explanation! Relying on a new dtype seems like fairly essential use of 2.0, and it'd probably be a significant amount of work to maintain a parallel backwards-compatible implementation.

I tracked down one of my current bottlenecks which is xagg→pytables→numpy<2, and happily that's being actively worked on 🚀!

I think my current strategy will be to build a separate container for the kerchunking step of my pipeline so that I can make use of VirtualiZarr until numpy 2 is widely supported.

Thanks so much for the quick response, it's really helpful!

[TomNicholas]

Sounds good! Let us know how you get on using VirtualiZarr.

[maresb]

With the PyTables release just now, there are no more dependency constraints blocking me from upgrading to NumPy 2 and using VirtualiZarr in my main environment. 🚀