Wildcard on Single netCDF based on OpenDAP

[haze1986]

Hi I am trying to share GFS wind forecast, i.e. daily it will source for
https://nomads.ncep.noaa.gov/dods/gfs_0p25_1hr/gfs20220913/gfs_0p25_1hr_00z
with the correct date.

Is there a way to specify date commands so the auto update can work?
https://nomads.ncep.noaa.gov/dods/gfs_0p25_1hr/gfs${date +"%Y%m%d"}/gfs_0p25_1hr_00z/gfs_0p25_1hr_00z

Or is there a better way to have the latest forecast on LAS?

[noaaroland]

The short answer is no, this is not possible.

There are many tools (like ERDDAP, Hyrax and TDS) which can dynamically read a collection of data files and serve them as an aggregation. Then configure LAS to see the aggregation and set up the automatic updates in LAS to check for new data periodically.

Unfortunately since NCEP is using a tool that is decades old (GDS) and has not aggregated these data, your only hope is to download the data (maybe via ftp) and set up your own server to serve the data to LAS.

[haze1986]

Understood, this is clear.

One way I could download the files daily and store them on my server, rename them as a generic "3 day forecast". I set up the autoupdate to run daily some time after the generation of the "3 day forecast" for the day. However, when I check the data of the plot in the interface it still seems to be stuck at the day which I added them into LAS. Is this a known issue or do I have to create a script delete the dataset and add the latest daily? I have no issues with the date if I delete the old data and manually add them back.

Would just like to confirm if the autoupdate does not obtain the date info from the netcdf file? The date limits will always be set from the initial time when the data was added.

[noaaroland]

The auto update feature in LAS is designed specifically to examine the data source, extract the time and update the UI. If it's not working something is probably not configured correctly. Check the logs to see if the update is firing as you expect, Use https://crontab.guru/ to check your cron expression. I admit, I've never run it where it was reading the same file over and over again, but I don't see any reason why it wouldn't work.

[haze1986]

Hi Roland, understood. I did verify that the cron scripts fired correctly. Actually an easy way to check is to put a file that is totally out of the range of the previous file. i.e. current file 21 - 25 sep, replaced with a file with the same filename consisting of data from 13 - 17 sep. The dates will still remain as 21 - 25. I'm still figuring out why but it might have something to do with the jnl files in content/las/scripts not being updated correctly as they are still showing the previous date.

[noaaroland]

It's possible that we are talking past each other. I'm sure your process to update your file is working fine. But, unless and until you enter the admin interface on your LAS and direct LAS to use its internal facilities to automatically update the data set, the end dates in the UI will never change regardless of what's in the file. Have you visited the pictured page for this data set and set up and update?

[haze1986]

Yes I'm referring to this auto update page exactly.

1. I set it to update every half an hour.
2. Replace the netcdf file to the data consisting of 13 - 17 sep
3. you will notice the dates are still 21 - 25

I could provide the files to help with troubleshoot if needed.

[noaaroland]

Are you reading the file directly in LAS (by entering in the absolute path to the file in LAS) or are you looking at the file via a server like THREDDS. If it's the latter the the issue might be that the dates are being cached by TDS.

[haze1986]

Yes the file is residing at the $HOME/data folder of the server where I installed las. I keyed in the full path. This is how I added it

Can I confirm the hash and history will not cause any issues when it gets updated?

[noaaroland]

Great. Seems like you have everything set up correctly. The hash should be the same if the file "url" is the same. History value won't affect the automatic update. The time axis is the only thing being examined.

Try resetting the trigger for the update. Use the automatic update page and send a blank cron spec. Verify in the log you see a:
Stopped updates for data set id=
message.

Then set it again and verify in the log you see a message:
Setting up cron-based updates for data set id= XXX for cron spec * * * * *
matching your data set.

Then when the update fires, look at the log output for updating the data set. There should be plenty of messages, especially if you logback.xml is set to debug for the las packages.

Post all that output, and I'll see if I can figure out what's going wrong.

[haze1986]

I have not been able to remove a cron after setting it for the first time in the interface. When I send a blank cron and hit save it just appear back and when I check back the cron spec is still there. The outputs of the catalina.out is as below, as shown it resets back to the old cron spec:

08:38:47.321 [http-nio-8080-exec-9] INFO p.sdig.las.UpdateDatasetJobService - Setting up cron-based updates for data set id=206 for cron spec 0 0,30 * * * ? 08:38:54.204 [http-nio-8080-exec-8] DEBUG pmel.sdig.las.DatasetController - Starting response for dataset 206 08:38:57.268 [http-nio-8080-exec-10] INFO p.sdig.las.UpdateDatasetJobService - Setting up cron-based updates for data set id=206 for cron spec 0 0,30 * * * ?

From the checks of the whole catalina.out, I was not able to receive the below message until I deleted the dataset.
Stopped updates for data set id=

EDIT:
09:30:00.090 [quartzScheduler_Worker-9] ERROR p.sdig.las.UpdateDatasetJobService - Exception during update of data set 206 with message No signature of method: pmel.sdig.las.Variable.getTimeAxis() is applicable for argument types: () values: []
Obtained the error message regarding the time axis.

[noaaroland]

Thanks for this information. From this output we know that you can ingest the initial file into LAS (assuming you tested it after you did the initial ingest), but when the update runs it fails. So either there is a bug in LAS, or there is something different about the initial file and the once being updated. (The failure to remove the cron spec is probably another bug which I'll tackle separately).

I'd like set up the same scenario locally and watch the update in the debugger. I looking around for GFS output, but it be easier if you could give me the specifics of the files you're using (or links to similar or whatever). Thanks!

[haze1986]

Sure, appreciate your help in looking into this.

I was also suspecting it could be the dataset but I have tried 2 different datasets, one is direct from GFS and another an output from WavewatchIII. For ease, I will share my script to download the GFS dataset:

curl "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t${CC}z.pgrb2.0p25.f${FFF}&lev_10_m_above_ground=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.${YYYYMMDD}%2F${CC}%2Fatmos" -o gfs.t${CC}z.f${FFF}.grib2

then convert with wgrib2 to netcdf
wgrib2 gfs.t00z.f000-f384.grib2 -netcdf gfs.t00z.f000-f384.nc

[noaaroland]

Anyway you can give me a link or an ftp of two of the converted netCDF files? Otherwise, I have to install/build wgrib2 or set up a TDS for the grib.

[haze1986]

Yes in fact any netcdf file I checked have the same error with getTimeAxis() error when refreshed

You can try these 2
25/09
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20220925/00/atmos/gfs.t00z.atmf000.nc
17/09
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20220917/00/atmos/gfs.t00z.atmf000.nc

[noaaroland]

Thanks. It's a bug, obviously. I'm working on a fix.

[noaaroland]

The strategy which haze1986 adopted was to use a single file with the latest forecast configured into LAS. Then an external process would replace that file (using the same file name) with the latest forecast and then configured LAS to automatically update the time axis in LAS using the auto update feature. This should have worked, but there was a bug which is now fixed and this method of replacing the file and automatically updating should now work.

I believe this is fixed with the 9.7.6 release. Thank you for opening the discussion and helping me track down the bug.

Edited: for missing word and fuller explanation of the issue and resolution.

[haze1986]

Thanks Roland, will update the version and test it