Pangu-DART interface to support DA with ensemble ML models

[Nuo-Chen]

Description:

Adding Pangu-DART interface to support DA with ensemble ML models.

Fixes issue

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Documentation changes needed?

• My change requires a change to the documentation.
  • I have updated the documentation accordingly.

Tests

Please describe any tests you ran to verify your changes.

Checklist for merging

• Updated changelog entry
• Documentation updated
• Update conf.py

Checklist for release

• Merge into main
• Create release from the main branch with appropriate tag
• Delete feature-branch

Testing Datasets

• Dataset needed for testing available upon request
• Dataset download instructions included
• No dataset needed

Testing reanalysis dataset at /glade/work/chennuo/code/pangu/dart_pangu/era5_data/
Testing observation data at /glade/work/chennuo/code/cice_old/DART/observations/obs_converters/NCEP/prep_bufr/data/20240120/prepout

[hkershaw-brown]

Thanks for this Nuo!

[hkershaw-brown]

note on AI models, no terrain surface - problems for altimeter obs.

[criedel40]

Yeah I had to make the code read in terrain surface data for this reason. -Chris

…

On Fri, Nov 8, 2024 at 7:58 AM Helen Kershaw ***@***.***> wrote: note on AI models, no terrain surface - problems for altimeter obs. — Reply to this email directly, view it on GitHub <#737 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVLWAOUXE66YBMJD77J53Z7S7SDAVCNFSM6AAAAABOZG3FVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHAZTIMZTHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hkershaw-brown]

Chris come back to dares! We were talking about this at the standup, reading in terrain from another source.
We've got two pangu projects on the go, and I think we're going to hit a few things (e.g. terrain) that apply to AI atmosphere models in general.

[criedel40]

So I think it will be dependent on the grid you are using right? Our training dataset is ERA5 so it makes perfect sense that I use the terrain data from ERA5. I guess it depends on the data source used for Pangu really. However I imagine it depends more on the grid structure and resolution. I have to write data from binary numpy files (yuck I know) to netcdf to make it clean for DART so I just add terrain data to the first member's netcdf file. The AI group at Windborne loves numpy binary files. I actually have some real reservations on how DA will actually work with AI models due to their makeup. The other thing I absolutely love (really hate) is our AI model is defined on pressure levels, which goes down to 1000hPa. In some areas, heights at 1000hPa are below the surface level hahaha. Just nuts that it actually works and produces forecasts that are that much different, however, we are just at cold-start DA at the moment. Tell Jeff to get some money! -Chris

…

On Fri, Nov 8, 2024 at 8:25 AM Helen Kershaw ***@***.***> wrote: Chris come back to dares! We were talking about this at the standup, reading in terrain from another source. We've got two pangu projects on the go, and I think we're going to hit a few things (e.g. terrain) that apply to AI atmosphere models in general. — Reply to this email directly, view it on GitHub <#737 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVLWGN5MEK3UZ3NQJV7ATZ7TCWBAVCNFSM6AAAAABOZG3FVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHA4TKMRWGE> . You are receiving this because you commented.Message ID: ***@***.***>

[Nuo-Chen]

I think convert_pgout_to_nc.py reads in terrain.npy, though the path was linked to my work directory. terrain.npy and landmask.npy are located in model/pangu, both are created them with WRF-WPS.

[hkershaw-brown]

awesome thanks Nuo!