Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document input data format specification #49

Open
jhamman opened this issue Jan 26, 2022 · 2 comments
Open

Document input data format specification #49

jhamman opened this issue Jan 26, 2022 · 2 comments

Comments

@jhamman
Copy link

jhamman commented Jan 26, 2022

We should write down the multiscale-schema we depend on here. There are various conversations happening about the use of the multiscales spec for mapping applications (see zarr-developers/zarr-specs#125, zarr-developers/zarr-specs#124, carbonplan/ndpyramid#18, and #14).

@jhamman
Copy link
Author

jhamman commented Jan 28, 2022

@katamartin / @freeman-lab - here's a short sample .zattrs from a recent dataset we've been using:

{
    "history": "Created: Wed Dec 22 04:14:40 2021",
    "hostname": "prefect-job-9f003c8e-85nkn",
    "institution": "CarbonPlan",
    "multiscales": [
        {
            "datasets": [
                {
                    "path": "0",
                    "pixels_per_tile": 128
                },
                {
                    "path": "1",
                    "pixels_per_tile": 128
                }
            ],
            "metadata": {
                "args": [],
                "kwargs": {
                    "levels": 2,
                    "method": "bilinear",
                    "regridder_kws": {
                        "periodic": true
                    },
                    "weights_template": null
                },
                "method": "pyramid_regrid",
                "version": "0.0.post60"
            },
            "type": "reduce"
        }
    ],
    "source": "/srv/conda/envs/notebook/bin/prefect",
    "username": "jovyan",
    "version": "0.4.0"
}

Can you confirm that the maps-relevant spec can be entirely confined to the multiscales[0].datasets key?

@katamartin
Copy link
Member

@jhamman yeah I think that's right!

This is all we look at from the metadata (starting in soon-to-be-released v2 of maps):

maps/src/utils.js

Lines 251 to 278 in a3cab7c

export const getPyramidMetadata = (metadata) => {
const multiscales = metadata.metadata['.zattrs'].multiscales
if (!multiscales) {
throw new Error(
'Missing `multiscales` value in .zattrs. Please check your pyramid generation code.'
)
}
const datasets = multiscales[0].datasets
if (!datasets || datasets.length === 0) {
throw new Error(
'No datasets provided in `multiscales` metadata. Please check your pyramid generation code.'
)
}
const levels = datasets.map((dataset) => Number(dataset.path))
const maxZoom = Math.max(...levels)
const tileSize = datasets[0].pixels_per_tile
if (!tileSize) {
throw new Error(
'Missing required `pixels_per_tile` value in `multiscales` metadata. Please check your pyramid generation code.'
)
}
return { levels, maxZoom, tileSize }
}

@katamartin katamartin mentioned this issue May 25, 2022
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants