QCing and Visualizing glider data

[dhruvbalwada]

I have been working on a visualization dashboard for gliders. It lives here at the moment: https://github.com/dhruvbalwada/glider-panel-demo and you should be able to see it in action by using the first binder button.

@marcelduplessis and @bastienqueste had raised the point about "Diagnostic / Visualisation tools to assess data quality". I am wondering if they can expand on that more on that over here. Is this dashboard style approach something that they were envisioning for part of it?

At this moment I am not thinking of incorporating the dashboard into GT because it does not seem like it belongs here (plus lot of extra dependencies), but open to discussion whether some more minimal cases should be included as examples to provide users a jump off platform to go explore things.

Any comments from others about approaching glider visualization is also helpful.

[soerenthomsen]

Super nice @dhruvbalwada !

In general I think it is nice to have quick visualisation tools to check data quality before going into detailed quality control procedures.

After the data is quality controlled and has the common OceanGliders Format (as planned) it would be amazing to be able to use your tools.

[marcelduplessis]

Hi @dhruvbalwada. These tools are fantastic! Really nice to visualise the data like this. For one, this tool can potentially be very useful for glider piloting, to make real-time decisions based on the in-situ mesoscale field. I do agree, this would probably be separate to GT as I'm not sure how this could be brought in.

I'll take a look at the detail of the code more, but I particularly like this anomaly function and the ability to switch between time/distance to get a feel for the scale of features the glider is covering.

[jbusecke]

These are dope! I am not generally opposed to having both QC and postprocessed Visualization tools as part of the package (we can handle optional dependencies, e.g. folks can install them if they want to use this specific tool). But, I think these would make for some phenomenal tutorials first and foremost. Giving users an idea how to set a dashboard like this up and how to customize it, might be even more valuable than having a baked in solution!

[dhruvbalwada]

Thanks for liking the dashboard, it does look cool :)
However, I want to reiterate that the purpose of that repo is not to build a stand alone platform or a plug and play tool, but instead to provide examples on how to use holoviz to make relatively complex interactive visualizations. A sort of what is possible example. I doubt that most people will want to look at the same things I do, and so will have very different needs. The repo gives a good jumping point to go and explore holoviz for earth data.

I don't know what QCing gliders exactly entails, and which part of it was going to live in GT. So would be nice if people with more upstream experience than me in glider processing can chime in.

[bastienqueste]

Very nice work! Starring and following that!

When I wrote the earlier comment, I had a holoviz applet kind of thing in mind, with a few recipes of simple figures which give good insight into glider data for beginners. Nothing groundbreaking, figures that are probably obvious to anyone that has worked with glider data for a litte bit but for example:

1. a T/S plot colored by dive direction to assess temp and salinity lags, gridded up profiles - down profiles or histograms of up and down. Most glider beginners don't realise you can leverage the repeated up/down profiling to assess data quality. The differences are obvious with dirty data.
   2 ) For flight model regressions, an N**2 vs w(H2O) scatter a la Frajka-Williams to assess quality of flight model, a plot of estimated w(H2O), there are so many for this...
2. For oxygen: up v down for lags; plots of surface saturation vs wind speed; plots of d(AOU)/dt for different T&S bins
3. For chlorophyll, near surface time/depth sections with a 24hr high pass filter to assess quenching presence/absence (spoiler: it's always there!)

There are figures that should always have a clear pattern on good data, and that really obviously do not when working with bad data. Such figures, accompanied by a quick tooltip can be super helpful.

[castelao]

It looks really cool. Nice job @dhruvbalwada et. al.

I tend to split my projects by objectives instead of big monolithic do all. As long as there is a clear common standard, all tools would talk with each other, and someone could always install all. The main advantage is to make it easier for other packages to absorb parts of your work as a dependency.

I agree with @soerenthomsen , and your work could turn into a powerful tool for delayed mode QC. I agree with you that different people could be interested in seeing different things, but there are some basic procedures DM QC that everyone should check, and just that would justify a nice package by itself.

[soerenthomsen]

Yes. As I was told by some folks (i.e. who are new with gliders) that not for everyone the difference between Real Time QC (RTQC) and Delayed Mode QC (DMQC) is clear. Here I started #97 a discussion thread to quickly explaining the differences and discuss.