chunk key parsing speedup

[agoodm]

Hey @ayushnag ,

I actually took a quick look your code and made a few small optimizations. This should give you a 2-3x speedup.

[ayushnag]

Thanks @agoodm! I actually just replaced the ast.literal_eval() segment with np.fromstring(chunk_tag.attrib["chunkPositionInArray"][1:-1], dtype=int, sep=',') which has also shown performance improvements. Now I am getting similar performance numbers to your method but I have only tested on files with <= 300 chunks per variable. I will keep this in mind when testing with more files/chunks

[ayushnag]

Actually there is a ~10ms improvement using this instead of the np method (just testing on one file) which might scale. However as Tom mentioned here we can have less memory pressure with numpy2.0. I will go ahead and merge this for now and we can revisit later