Dask Distributed Write Fix For Zarr

[selmanozleyen]

Hi,
This issue appears when one tries to use dask distributed then write dask/distributed#780.
I did what xarray does for this https://github.com/pydata/xarray/pull/1179/files.

Not sure why the lock is global though for now.

Ping @ivirshup @flying-sheep

[codecov]

Codecov Report

Merging #1079 (e5cd4a3) into main (5428f16) will decrease coverage by 0.02%.
The diff coverage is 84.61%.

❗ Current head e5cd4a3 differs from pull request most recent head 75dec29. Consider uploading reports for the commit 75dec29 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1079      +/-   ##
==========================================
- Coverage   82.70%   82.68%   -0.02%     
==========================================
  Files          36       36              
  Lines        5116     5128      +12     
==========================================
+ Hits         4231     4240       +9     
- Misses        885      888       +3     

Files Changed	Coverage Δ
anndata/_io/specs/methods.py	87.46% <84.61%> (-0.13%)	⬇️

... and 1 file with indirect coverage changes

[flying-sheep]

is there a way to test this?

[selmanozleyen]

@flying-sheep added tests, and it now fails for h5py :D. I thought I could just go with whatever xarray did at https://github.com/pydata/xarray/pull/1179/files. But now I realize this was from 2017 so I will need investigate a bit more closely.

[selmanozleyen]

Hey @flying-sheep @ilan-gold, it should be ready for review once this test passes. I ended up not adding h5ad support for distributed writing and just giving an error. To fix we would need to create a file manager class that tracks the locks of a file etc. Xarray has a magic ability to serialize h5py datasets but how it does is inside the file manager class: pydata/xarray#4242. There is also a module called h5py pickle for this purpose but it isn't updated since 2020 https://github.com/DaanVanVugt/h5pickle

[selmanozleyen]

Hey @flying-sheep could you perhaps merge this? I created an issue for h5py version here: #1105. But until I create a PR for that separately, I would like to create a notebook for concat_on_disk, which uses distributed writing for tests.

[ivirshup]

@flying-sheep One sec, taking a look with selman

[ivirshup]

Alright I think we can merge. But I want to follow up.

Looking through xarray, it doesn't seem like they use this kind of solution for zarr, but do for hdf5. However here, we are using this solution for zarr and it does not work for hdf5. This is a bit disconcerting and a little concerning.

I vaguely understand why this isn't working for us with hdf5, xarray has some handling for this as mentioned above.

I do not understand what the are doing as to not need this for zarr.

I will go ask xarray/ zarr people to take a look and see if they can suggest a better solution.