Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data type and compression in the output of cutmaps #42

Open
casadoj opened this issue Nov 7, 2023 · 0 comments
Open

Data type and compression in the output of cutmaps #42

casadoj opened this issue Nov 7, 2023 · 0 comments

Comments

@casadoj
Copy link
Collaborator

casadoj commented Nov 7, 2023

The output NetCDF files from cutmaps do not inherit the data type and compression of the original NetCDF files. The results is that the output file size might be excessively large compared with the original file.

For instance, I've used the cutmaps on the GloFAS4 reservoir fill maps (rsfil.nc):

cutmaps -f ./path_in/ -c "-114.125 -103.475 41.925 50.875" -o ./path_out/

The original files are the daily global maps for every year in the GloFAS long run; their size is approximately 300 Mb. The resulting maps represent a small portion of the Mississippi river, however their size is over 100 Mb. I've done a dummy test with Xarray and the resulting maps are 330 kb, instead.

# load dataset
da = xr.open_dataset(path_in / f'rsfil_2019.nc')
# cut the data
da = da.sel(lon=slice(-114.125, -103.475), lat=slice(50.875, 41.925))
# export NetCDF
encoding = {'rsfil': {'dtype': 'float64', 'zlib': True, 'complevel': 4}}
da.to_netcdf(path / f'giuseppe/test/test_{year}.nc', encoding=encoding)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant