Synergy with pyglider

[dhruvbalwada]

Just came across this package called pyglider (https://github.com/jklymak/pyglider/) by @jklymak . Would be curious to know if synergy between pyglider and glidertools will be useful?

[richardsc]

First, I am not a python expert (long-time R user, actually) but I have been learning pyglider to put a L0-L2 processing chain in place for our SeaExplorer glider fleet (actually, we discussed this a bit separately in a Slack workspace I set up to connect the global SeaExplorer community). I'm also new with glidertools, so this could be a good chance for someone who is "starting from scratch" to work at making these packages connect in a useful way.

TL;DR: I think it's a good idea and I'm happy to help where I can!

[jklymak]

Pyglider and glidertools are a little disjoint in that pyglider translates the raw glider data and makes netcdf files, whereas glidertools operates on netcdf files for qa / qc. I don't think their netcdf conventions are compatible, but we have been waiting for the OG 2.0 standard to stabilize before bothering to translate.

We also have a very informal processing chain to do lag corrections and despiking that are just ipython notebooks.

[jklymak]

Note I've moved pyglider out of my personal repository: https://github.com/c-proof/pyglider

Looking at this again, I think the question is what sort of netcdf file is GliderTools willing to ingest? c-proof/pyglider makes ones that are almost compatible with the IOOS gdac, and we are planning to have proper compliance by the end of summer. OG 2.0 will also have a standard, loosely based on the EGO standard.

It seems from my quick look that GliderTools is using yet a different netcdf format, as provided by the sea glider base station?

It would be great if a critical mass of folks decided on a single format and then GliderTools was able to ingest and return that format. My assumption is that would be the OG2.0 format?

Maybe folks here are more plugged into what OG2.0 is doing.

I know @kerfoot is also involved in helping to make the standard, and maybe he knows where we should be looking ahead for what the standard will be in the nearish future.

[soerenthomsen]

Thanks @jklymak for your detailed response. Indeed we are envision to harmonise with OceanGliders data formats. You can see already some discussion here #49 and #50 and are exchanging with @vturpin about this.

Any additional information would be helpful to understand where we are heading to @kerfoot

[jklymak]

That'd be great, then GliderTools could just claim to take the blessed format, and the lower level libraries could provide that format. I don't see anything in GliderTools that would be wildly different between glider manufacturers, though I could be missing things.

One thing I think all the new formats have in common is that each data variable has a parallel QC variable.

[kerfoot]

From what I've seen and read in the GliderTools doco, the compatible data structures are numpy arrays, pandas data frames and xarray datasets. It doesn't matter where they come from as long as it's one of these and they have an appropriate index such as time, so I don't think we need to get hung up on a "blessed" format. So the trick is to either import the NetCDF into an xarray dataset (trivial) or write a parser for any glider type that doesn't report data in a time-series-like NetCDF file. I work with slocum gliders, so this has been the sticking point for me. Slocum data are basically ascii tables that contain poorly named variables with no attributes (units, standard names, etc.). I wrote a python package (https://github.com/kerfoot/gdm) that parses these ascii table and, with a few yaml configuration files, returns either fully attributed xarray dataset or pandas data frame with more familiar variable names (i.e.: sci_water_temp renamed to temperature, etc.). I think this will do the trick for the slocum users. I've been developing the wiki for that package if you'd like to take a look: https://github.com/kerfoot/gdm/wiki. The OG1.0 (or whatever the first version number is) is a time-series based NetCDF, so there may be some minor issues with certain variables, but nothing to tricky IMO. Happy to help and contribute on this going forward.

[soerenthomsen]

For interested people here is the discussion of the pyglider package we are talking about: c-proof/pyglider#16

[soerenthomsen]

No also the OG1.0 format is on GitHub and people are welcome to feedback and interact there:
https://github.com/OceanGlidersCommunity/OG-format-user-manual