Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: Object of type Shuffle is not JSON serializable #223

Open
kthyng opened this issue Aug 16, 2023 · 3 comments · May be fixed by #225
Open

TypeError: Object of type Shuffle is not JSON serializable #223

kthyng opened this issue Aug 16, 2023 · 3 comments · May be fixed by #225

Comments

@kthyng
Copy link

kthyng commented Aug 16, 2023

Hi! I have some model output I am trying to serve with xpublish (and xpublish-host and kerchunk). This model output has compression, and when I try to access it through the server I get the following error, though I am able to just open the model output fine (pointing to the kerchunked file):

TypeError: Object of type Shuffle is not JSON serializable

Indeed, a "Shuffle" object is in the metadata, for example here is what it shows in the xpublish code for an example variable:

'Uwind_eastward/.zarray': {'compressor': {'id': 'zlib', 'level': 3}, 'filters': [Shuffle(elementsize=4)], 'chunks': [1, 362, 794], 'dtype': '<f4', 'fill_value': 'NaN', 'order': 'C', 'shape': [1, 362, 794], 'zarr_format': 2}

I'm not sure why the compressor items remained a dictionary but the filter item was converted to an object, which is what seems to cause the issue. In the metadata coming out of kerchunk the filter item looked like

"filters":[{"elementsize":8,"id":"shuffle"}]

(I'm not sure if that change in "elementsize" is important as I go back and forth between two cases I'm testing, but I can track it down if it seems relevant.)

Does anyone follow this? I am not sure how to set up an actual working example for this workflow yet.

@kthyng
Copy link
Author

kthyng commented Aug 16, 2023

For reference, I decompressed the file with

ncks -L 0 nwgoa_1999-02-02.nc nwgoa_1999-02-02_decompressed.nc

and I do not hit any error anymore. Is there an issue with serving compressed model output?

@kthyng
Copy link
Author

kthyng commented Aug 16, 2023

Update: I removed the filters encoding in my plugin for serving the model output and that seems to have fixed the issue. I've been trying to resist reset all of the encoding with ds.reset_encoding() so I did:

    for varname in ds.variables:
        del ds[varname].encoding["filters"]

I'll close this for now as I seem to have sidestepped the issue but will open it if needed. Thanks. 🦆

@kthyng kthyng closed this as completed Aug 16, 2023
@kthyng kthyng reopened this Aug 16, 2023
@kthyng
Copy link
Author

kthyng commented Aug 16, 2023

Unfortunately of course that didn't work. Still not sure how to handle having filters necessary in the model output metadata.

@kthyng kthyng linked a pull request Aug 17, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant