Support for zarr over fsspec reference ?

[steph-ben]

Thanks for this beautiful lib !

I try to use it with zarr generated with https://github.com/fsspec/kerchunk , which make usage of fsspec "Reference filesystem" . It allows indexing of existing NetCDF/GRIB files, than can be then loaded as a single dataset.

Eg. with python xarray I can load a zarr dataset with :

rpath = 's3://esip-qhub-public/noaa/hrrr/hrrr_best.json'
fs = fsspec.filesystem("reference", fo=rpath, remote_protocol='s3', remote_options={'anon':True})
ds = xr.open_dataset(fs.get_mapper(""), engine="zarr")

See this notebook for example

However it doesn't seem to be supported by this lib, I'm having the following error :

• With S3 file : CORS error
• With HTTP file : .zmetadata not found

Not much details here, but maybe you have some thoughts on this topic ?

[jhamman]

Hi @steph-ben - thanks for opening this issue. At present, zarr-js, which we use to read zarr data over http, does not have support for fsspec's reference filesystem.

Another issue issue to consider for you application is that datasets used by @carbonplan/maps need to be reprojected into multiscale web-mercator quad-trees. A typical processing workflow would be: fsspec->xarray->ndpyramid->zarr. You might want to check out ndpyramid's examples for how we've done this in the past.