input zarr extent must be global

[iferencik]

I'd like to inquire about extent of the input zarr data. I am not sure if this is the right place to inquire, but I am going to do it anyway because your repo does not feature discussions. I have a use case where I plan to publish country level high resolution GeoTiff data (30 meters). This data is a bit specific in the sense that it was produced by a model and many pixels are set to nodata (around 80%) so it looks almost like a sparse dataset.

My first step was to convert this to zarr using ndpyramid and I immediately hit a wall (size/memory). In the end i used rioxarray + rasterio + dask to reproject my dataset to EPSG:3857 and then I resampled the reprojected dataset for various zoom levels/resolutions using xarray.coarsen and created a multiscale zarr.

Of course, when I reprojected it I did not make it a global dataset but rather I sticked to it's original spatial extent (country level). I started a local python server to serve the zarr over HTTP but the dataset would not load, but i also was not able to see any errors in the console.

I compared the multilevel zarr I have created with the one I created using [ndpyramid.reproject ].(https://github.com/carbonplan/ndpyramid/blob/main/ndpyramid/regrid.py#L104) and I realized my dataset was not global. Other than that there was not difference (.zmetadata was not missing any props) . So given this context, I'd like to know:

1. does carbonplan/maps lib work only with global datasets?
2. if yes, what would be required to make it work using nonglobal datasets (country level)?
3. The example data you use is of type float. Are int types zarr supported? (i1/u1) ?
   Thanks

[katamartin]

Hi @iferencik -- thanks for starting the thread and apologies for the delay in response!

1. At a high level, yes, @carbonplan/maps expects a global dataset. Specifically, the library relies on the Zarr chunk key naming scheme to look up chunks that each exactly cover the map tiles in a standard map zoom level quadtree (this is an important piece of the ndpyramid utility). However, this does not mean that all chunks need to be defined. We added logic to handle missing chunks in 2.0.2 (we now treat as if uniformly filled with fill_value). This means that you could prepare a Zarr pyramid that adheres to a global coordinate system, but does not produce chunks over the ~80% of area that is empty. Does this make sense and seem reasonable for your needs?
2. Because the global extent allows us to share the map tile scheme with the standard set used by Mapbox and others, I think we're unlikely to revisit this anytime soon.
3. Yes, int types are supported. You can find the full list of supported dtypes here.

Thanks again for the questions. This has been a helpful nudge to spend a little time fleshing out some of these requirements in the README.

[keltonhalbert]

A question along similar lines regarding non global data - it appears that using various forms of fill_value, _fillValue, etc in both zarr and the js library don't mask out tiles containing only the fill value. It's almost like the shader doesn't recognize the fill value passed.

I have some example tiled zarr data here: https://mrms.satsquatch.com/tilesdata/hrrr/

The main issue is that the fill_value is not being respected and data are being plotted.