Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Earth Data Collection Requirements #23

Open
tinaok opened this issue May 4, 2024 · 7 comments
Open

Earth Data Collection Requirements #23

tinaok opened this issue May 4, 2024 · 7 comments

Comments

@tinaok
Copy link
Collaborator

tinaok commented May 4, 2024 •
edited
Loading

Objective

The goal of this issue is to outline and track the collection of essential earth data products necessary for our project. This table will serve as a reference to ensure we have all the necessary data ready and accessible for project phases requiring them.

Data Requirements

We have two main data categories for our project:

  • Past Data: Required for quick-track construction from biologging data sets.
  • Future Data: Needed to estimate future habitat environmental stress.

Please find below the table detailing each data product needed for the project. Each row represents a specific data type, with details about its purpose, source, deadline, and preferred format.

Past Data:

Sea Temperature, Sea surface hight, hourly and Daily

CMEMS:*

CMEMS Data Product Name spatial extent and resolution Temporal extent Source(DOI) Format Availability DEDL
Atlantic: Iberia-Biscay-IrelandAtlantic: North IBI_ANALYSISFORECAST_PHY_005_001 Lat 26° to 56°Lon -19° to 5°, 0.083 deg 1 Dec 2020 to now https://doi.org/10.48670/moi-00027 NetCDF & ZARR (if zarr , chunked spatially) - -
Atlantic: IBI_MULTIYEAR_PHY_005_002 Lat 26° to 56°Lon -19° to 5°, 0.028° × 0.028° 1 Dec 2020 to now https://doi.org/10.48670/moi-00027 NetCDF & ZARR (if zarr , chunked spatially) - -
Atlantic: European North West Shelf- Ocean Physics Reanalysis: NWSHELF_ANALYSISFORECAST_PHY_004_013 Lat 46° to 61.28°Lon -16° to 9.98°,0.014° × 0.03° 1 Sep 2021 to now https://doi.org/10.48670/moi-00054, decommissioned from CMEMS, see https://www.metoffice.gov.uk/services/data/met-office-marine-data-service NetCDF & ZARR (if zarr , chunked spatially) - -
NWSHELF_ANALYSISFORECAST_PHY_004_009 Lat 46° to 61.28°Lon -16° to 9.98°,0.014° × 0.03° NetCDF & ZARR (if zarr , chunked spatially) - -
GLOBAL_ANALYSISFORECAST_PHY_001_024 Global 2020 to now. All the version of data 7km resolution NetCDF & ZARR (if zarr , chunked spatially) - -
GLOBAL_MULTIYEAR_PHY_001_030 Global two product. one untill 2021, another from 2021, separated in 2 zarr file NetCDF & ZARR (if zarr , chunked spatially) - -

| ... | ... | ... | ... | ... | ... |

Future Data:

(https://easy.gems.dkrz.de/DYAMOND/NextGEMS/cycle3.html#nextgems-cycle-3 for the model description?)

Data Category Data Product Description Source(DOI) Format Availability DEDL
FESOM (70 levels) Temperature, SSH, Salinity,... ?reference paper? Grib by polytope transform to zarr or kechunk of Grib ? ?
NEMO (75 levels) Temperature, SSH, Salinity,... ?reference paper? Grib by polytope transform to zarr or kechunk of Grib ? ?
ICON (? levels) Temperature, SSH, Salinity,... ?reference paper? Grib by polytope transform to zarr or kechunk of Grib ? ?
RIOMAR (? levels) Temperature, SSH, Salinity,... ?reference paper? NetCDF, stored on HPC center. transform to zarr or kechunk of Grib ? ?

Data Management and Tracking

Each data product can have a sub-issue to track its availability:

  • Is the data available on DEDL (Data Environment for Data Libraries)?
  • If not, the request to DEDL is made?
  • While waiting for availability on DEDL, should we make a copy to our DESP (Data Environment for Special Projects) space?
  • Status updates for copying data, e.g., "Copied in Zarr format?"
  • If in GRIB/NetCDF, is Kerchunk preparation completed?
  • Is the Intake catalog prepared for accessing the data?

Request for Contribution

I encourage team members to contribute by identifying potential data sources or by assisting in the data collection process. Please update this table or comment on this issue as you make progress or if you encounter any challenges.

Additional Notes

  • Ensure all collected data complies with legal and privacy requirements.
  • Regularly update the status of data preparation and accessibility.

Thank you for your contributions and let's make sure all necessary data is collected promptly and efficiently!
@annefou
Copy link
Collaborator

annefou commented May 5, 2024

For the ClimateDT, I managed to get one month (June as requested by Tina) for ifs-nemo (I also tried icon but I can't get much; not sure why):

160366332 avg_hc300m_ifs-nemo_20210601-20210615.grib
160590884 avg_hc300m_ifs-nemo_20210616-20210630.grib
175705501 avg_hc300m_ifs-nemo_20220601-20220615.grib
171527445 avg_hc300m_ifs-nemo_20220616-20220630.grib
160499497 avg_hc300m_ifs-nemo_20230601-20230615.grib
165171522 avg_hc300m_ifs-nemo_20230616-20230630.grib
160620926 avg_hc300m_ifs-nemo_20240601-20240615.grib
160756733 avg_hc300m_ifs-nemo_20240616-20240630.grib
154134651 avg_hc700m_ifs-nemo_20210601-20210615.grib
154250888 avg_hc700m_ifs-nemo_20210616-20210630.grib
153740121 avg_hc700m_ifs-nemo_20220601-20220615.grib
153932408 avg_hc700m_ifs-nemo_20220616-20220630.grib
154003557 avg_hc700m_ifs-nemo_20230601-20230615.grib
154146803 avg_hc700m_ifs-nemo_20230616-20230630.grib
153969132 avg_hc700m_ifs-nemo_20240601-20240615.grib
154056305 avg_hc700m_ifs-nemo_20240616-20240630.grib
6495121699 avg_so_ifs-nemo_20210601-20210615.grib
6494203690 avg_so_ifs-nemo_20210616-20210630.grib
6497145446 avg_so_ifs-nemo_20220601-20220615.grib
6498916071 avg_so_ifs-nemo_20220616-20220630.grib
6477147301 avg_so_ifs-nemo_20230601-20230615.grib
6478819417 avg_so_ifs-nemo_20230616-20230630.grib
6492560276 avg_so_ifs-nemo_20240601-20240615.grib
6493759416 avg_so_ifs-nemo_20240616-20240630.grib
124430145 avg_sos_ifs-nemo_20210601-20210615.grib
124504649 avg_sos_ifs-nemo_20210616-20210630.grib
125003963 avg_sos_ifs-nemo_20220601-20220615.grib
124958912 avg_sos_ifs-nemo_20220616-20220630.grib
124773729 avg_sos_ifs-nemo_20230601-20230615.grib
124706330 avg_sos_ifs-nemo_20230616-20230630.grib
125208677 avg_sos_ifs-nemo_20240601-20240615.grib
125210521 avg_sos_ifs-nemo_20240616-20240630.grib
9959434512 avg_thetao_icon_20210601-20210615.grib
9956652167 avg_thetao_icon_20210616-20210630.grib
9961834898 avg_thetao_icon_20220601-20220615.grib
9234391819 avg_thetao_ifs-nemo_20210601-20210615.grib
9231699251 avg_thetao_ifs-nemo_20210616-20210630.grib
9242519770 avg_thetao_ifs-nemo_20220601-20220615.grib
9242566457 avg_thetao_ifs-nemo_20220616-20220630.grib
9234889485 avg_thetao_ifs-nemo_20230601-20230615.grib
9235449623 avg_thetao_ifs-nemo_20230616-20230630.grib
9257534960 avg_thetao_ifs-nemo_20240601-20240615.grib
9256437221 avg_thetao_ifs-nemo_20240616-20240630.grib
152915639 avg_tos_ifs-nemo_20210601-20210615.grib
152917464 avg_tos_ifs-nemo_20210616-20210630.grib
153419426 avg_tos_ifs-nemo_20220601-20220615.grib
153259330 avg_tos_ifs-nemo_20220616-20220630.grib
152716339 avg_tos_ifs-nemo_20230601-20230615.grib
152801249 avg_tos_ifs-nemo_20230616-20230630.grib
153594860 avg_tos_ifs-nemo_20240601-20240615.grib
153345960 avg_tos_ifs-nemo_20240616-20240630.grib
11080412455 avg_uoe_ifs-nemo_20210601-20210615.grib
10889924371 avg_uoe_ifs-nemo_20210616-20210630.grib
10929602005 avg_uoe_ifs-nemo_20220601-20220615.grib
10865601865 avg_uoe_ifs-nemo_20220616-20220630.grib
10977099930 avg_uoe_ifs-nemo_20230601-20230615.grib
10950679268 avg_uoe_ifs-nemo_20230616-20230630.grib
10965176810 avg_uoe_ifs-nemo_20240601-20240615.grib
10915231921 avg_uoe_ifs-nemo_20240616-20240630.grib
11126417501 avg_von_ifs-nemo_20210601-20210615.grib
11025381071 avg_von_ifs-nemo_20210616-20210630.grib
11069287606 avg_von_ifs-nemo_20220601-20220615.grib
10921799480 avg_von_ifs-nemo_20220616-20220630.grib
11132575892 avg_von_ifs-nemo_20230601-20230615.grib
11025466889 avg_von_ifs-nemo_20230616-20230630.grib
11230409480 avg_von_ifs-nemo_20240601-20240615.grib
11123705070 avg_von_ifs-nemo_20240616-20240630.grib
9169116191 avg_wo_ifs-nemo_20210601-20210615.grib
9245560393 avg_wo_ifs-nemo_20210616-20210630.grib
9314574859 avg_wo_ifs-nemo_20220601-20220615.grib
9276597759 avg_wo_ifs-nemo_20220616-20220630.grib
9332851551 avg_wo_ifs-nemo_20230601-20230615.grib
9391269058 avg_wo_ifs-nemo_20230616-20230630.grib
9290668141 avg_wo_ifs-nemo_20240601-20240615.grib
9478879455 avg_wo_ifs-nemo_20240616-20240630.grib
135325122 avg_zos_icon_20240616-20240630.grib
159751128 avg_zos_ifs-nemo_20210601-20210615.grib
159893845 avg_zos_ifs-nemo_20210616-20210630.grib
159575287 avg_zos_ifs-nemo_20220601-20220615.grib
159600267 avg_zos_ifs-nemo_20220616-20220630.grib
159870669 avg_zos_ifs-nemo_20230601-20230615.grib
160013276 avg_zos_ifs-nemo_20230616-20230630.grib
159815137 avg_zos_ifs-nemo_20240601-20240615.grib
159794729 avg_zos_ifs-nemo_20240616-20240630.grib
  6963826 lsm.grib
215529769 slhf_icon_20210601-20210615.grib
211385292 slhf_icon_20210616-20210630.grib
218094425 slhf_icon_20220601-20220615.grib
215816467 slhf_icon_20220616-20220630.grib
222054855 slhf_icon_20230601-20230615.grib
222463963 slhf_icon_20230616-20230630.grib
215814211 slhf_icon_20240601-20240615.grib
220679898 slhf_icon_20240616-20240630.grib
413306346 slhf_ifs-nemo_20210601-20210615.grib
412586498 slhf_ifs-nemo_20210616-20210630.grib
406136551 slhf_ifs-nemo_20220601-20220615.grib
402732836 slhf_ifs-nemo_20220616-20220630.grib
413505983 slhf_ifs-nemo_20230601-20230615.grib
410592215 slhf_ifs-nemo_20230616-20230630.grib
411291551 slhf_ifs-nemo_20240601-20240615.grib
406442753 slhf_ifs-nemo_20240616-20240630.grib
199254606 sshf_icon_20210601-20210615.grib
186454593 sshf_icon_20220601-20220615.grib
196349123 sshf_icon_20220616-20220630.grib
199066550 sshf_icon_20230601-20230615.grib
199342620 sshf_icon_20230616-20230630.grib
200491041 sshf_icon_20240601-20240615.grib
200357028 sshf_icon_20240616-20240630.grib
395691070 sshf_ifs-nemo_20210601-20210615.grib
399767163 sshf_ifs-nemo_20210616-20210630.grib
395065383 sshf_ifs-nemo_20220601-20220615.grib
394989618 sshf_ifs-nemo_20220616-20220630.grib
396834844 sshf_ifs-nemo_20230601-20230615.grib
396999891 sshf_ifs-nemo_20230616-20230630.grib
399883620 sshf_ifs-nemo_20240601-20240615.grib
398481366 sshf_ifs-nemo_20240616-20240630.grib

@tinaok would you need other months, etc?

@tinaok
Copy link
Collaborator Author

tinaok commented May 5, 2024

Thanks @annefou, I would like to have them in kerchunk if possible.

@annefou
Copy link
Collaborator

annefou commented May 5, 2024

Thanks @annefou, I would like to have them in kerchunk if possible.

Yes, I will create Kerchunk catalogs for the ClimateDT data too but as we only have access to the data until May 17th, I want to make sure we get all we need.

@tinaok
Copy link
Collaborator Author

tinaok commented May 5, 2024

as we only have access to the data until May 17th, I want to make sure we get all we need.

OK, then having thetao and zos for all available data (from the oldest (2010?) to all available future) can be great.

@annefou
Copy link
Collaborator

annefou commented May 7, 2024

OK, then having thetao and zos for all available data (from the oldest (2010?) to all available future) can be great.

there is no data before 2020 available from polytope. (and at the moment, we can only get ifs-demo).

@tinaok
Copy link
Collaborator Author

tinaok commented Jul 25, 2024 •
edited
Loading

I listed product names from Copernicus Marine services that for our interest here. I'll add it in the description of issue. Not all has 3d temperature available for full time.

 name={'GLOBAL_ANALYSISFORECAST_PHY_001_024':
      {'thetao':
      {'H':'cmems_mod_glo_phy-thetao_anfc_0.083deg_PT6H-i_202406',
      'D':'cmems_mod_glo_phy-thetao_anfc_0.083deg_P1D-m_202406' },
      'zos':
       {'H':'cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406' ,
      'D':'cmems_mod_glo_phy_anfc_0.083deg_P1D-m_202406'},
       'deptho': 'cmems_mod_glo_phy_anfc_0.083deg_static_202211--ext--bathy'},
    'GLOBAL_MULTIYEAR_PHY_001_030':
      {'thetao':
      {'NEW':'cmems_mod_glo_phy_myint_0.083deg_P1D-m_202311',
      'OLD':'cmems_mod_glo_phy_my_0.083deg_P1D-m_202311' },
      'zos':
       {'NEW':'cmems_mod_glo_phy_myint_0.083deg_P1D-m_202311' ,
      'OLD':'cmems_mod_glo_phy_anfc_0.083deg_P1D-m_202406'},
       'deptho': 'cmems_mod_glo_phy_anfc_0.083deg_static_202211--ext--bathy'},
          'IBI_MULTIYEAR_PHY_005_002':
      {'thetao':
      {'H':'cmems_mod_ibi_phy_anfc_0.027deg-3D_PT1H-m_202211',
      'D':'cmems_mod_ibi_phy_anfc_0.027deg-3D_P1D-m_202211' },
      'zos':
       {'H':'cmems_mod_ibi_phy_anfc_0.027deg-2D_PT1H-m_202211' ,
      'D':'cmems_mod_ibi_phy_anfc_0.027deg-3D_P1D-m_202211'},
       'deptho': 'cmems_mod_ibi_phy_anfc_0.027deg-3D_static_202211--ext--bathy'},
                'IBI_ANALYSISFORECAST_PHY_005_001':
      {'thetao':
      {'H':'',
      'D':'cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m_202012' },
      'zos':
       {'H':'cmems_mod_ibi_phy_my_0.083deg-2D_PT1H-m_202012' ,
      'D':'cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m_202012'},
       'deptho': 'cmems_mod_ibi_phy_my_0.083deg-3D_static_202012--ext--bathy'},
                'NWSHELF_ANALYSISFORECAST_PHY_004_013':
      {'thetao':
      {'H':'cmems_mod_nws_phy_anfc_0.027deg-3D_PT1H-m_202309',
      'D':'cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_202309' },
      'zos':
       {'H':'cmems_mod_nws_phy_anfc_0.027deg-2D_PT1H-m_202309' ,
      'D':'cmems_mod_nws_phy_anfc_0.027deg-3D_P1D-m_202309'},
       'deptho': 'cmems_mod_nws_phy_anfc_0.027deg-3D_static_202309--ext--bathy'},
          'NWSHELF_MULTIYEAR_PHY_004_009':
      {'thetao':
      {'H':'',
      'D':'cmems_mod_nws_phy-t_my_7km-3D_P1D-m_202012' },
      'zos':
       {'H':'' ,
      'D':'cmems_mod_nws_phy-ssh_my_7km-2D_P1D-m_202012'},
       'deptho': 'cmems_mod_nws_phy-bottomt_my_7km-2D_P1D-m_202012'},
         }

@tinaok
Copy link
Collaborator Author

tinaok commented Jul 25, 2024

We need a way to query time-range(of tag) and bbox( of simulation) to all available 'thetao' and 'zos' to query the available dataset.
unfortunately the catalogue from cmems gives back the time-range for 'all' the catalogue itself and

We need some sort of translation (refined search) of it to, to all available data on DestinE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@annefou @tinaok