Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Ocean Gliders format output option to seaxexplorer processing #212

Closed
wants to merge 2 commits into from
Closed

add Ocean Gliders format output option to seaxexplorer processing #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8d2bc96
add og1 output option to seaxexplorer processing
callumrollo Dec 12, 2024
778a6b9
add example yml and script
callumrollo Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 20 additions & 9 deletions pyglider/seaexplorer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@

def raw_to_timeseries(indir, outdir, deploymentyaml, kind='raw',
profile_filt_time=100, profile_min_time=300,
maxgap=10, interpolate=False, fnamesuffix=''):
maxgap=10, interpolate=False, fnamesuffix='', og_format=False):
"""
A little different than above, for the 4-file version of the data set.
"""
Expand All @@ -320,7 +320,12 @@
_log.info(f'Opening combined payload file {indir}/{id}-{kind}pld.parquet')
sensor = pl.read_parquet(f'{indir}/{id}-{kind}pld.parquet')
sensor = _remove_fill_values(sensor)

if og_format:
# temporarily translate from og names to pyglider names
for pyglider_var, og_var in utils.pyglider_og_var_dict.items():
if og_var in ncvar.keys():
ncvar[pyglider_var] = ncvar.pop(og_var)

Check warning on line 327 in pyglider/seaexplorer.py

View check run for this annotation

Codecov / codecov/patch

pyglider/seaexplorer.py#L325-L327 

Added lines #L325 - L327 were not covered by tests

# build a new data set based on info in `deploymentyaml.`
# We will use ctd as the interpolant
ds = xr.Dataset()
Expand Down Expand Up @@ -470,8 +475,6 @@
ds = ds.assign_coords(longitude=ds.longitude)
ds = ds.assign_coords(latitude=ds.latitude)
ds = ds.assign_coords(depth=ds.depth)
# ds = ds._get_distance_over_ground(ds)

ds = utils.fill_metadata(ds, deployment['metadata'], device_data)

start = ds['time'].values[0]
Expand All @@ -485,18 +488,26 @@
except:
pass
id0 = ds.attrs['deployment_name']
outname = outdir + id0 + fnamesuffix + '.nc'
_log.info('writing %s', outname)
if 'units' in ds.time.attrs.keys():
ds.time.attrs.pop('units')
if 'calendar' in ds.time.attrs.keys():
ds.time.attrs.pop('calendar')
if 'ad2cp_time' in list(ds):
if 'units' in ds.ad2cp_time.attrs.keys():
ds.ad2cp_time.attrs.pop('units')
ds.to_netcdf(outname, 'w',
encoding={'time': {'units': 'seconds since 1970-01-01T00:00:00Z',
'dtype': 'float64'}})
if og_format:
ds = utils.add_og1_metadata(ds, deployment)
outname = f"{outdir}/{ds.attrs['id']}.nc"
_log.info('writing %s', outname)
ds.to_netcdf(outname, 'w',

Check warning on line 502 in pyglider/seaexplorer.py

View check run for this annotation

Codecov / codecov/patch

pyglider/seaexplorer.py#L499-L502 

Added lines #L499 - L502 were not covered by tests
encoding={'TIME': {'units': 'seconds since 1970-01-01T00:00:00Z',
'dtype': 'float64'}})
else:
outname = outdir + id0 + fnamesuffix + '.nc'
_log.info('writing %s', outname)
ds.to_netcdf(outname, 'w',
encoding={'time': {'units': 'seconds since 1970-01-01T00:00:00Z',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, 'time'/'TIME' could be replaced by a time-encoder yaml entry. eg calls to ds.time could be replaced by ds[time_var_name] or similar?

Not sure about 'latitude' etc, but why do they need to be translated back and forth? For distance over ground? Maybe just use inside that function?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can localise lon/lat to just change within those functions, I'll make a new PR with ds[time_var_name] and see what it looks like

'dtype': 'float64'}})
return outname


Expand Down
81 changes: 81 additions & 0 deletions pyglider/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -633,6 +633,87 @@
return index


pyglider_og_var_dict = {'time': 'TIME',
'longitude': 'LONGITUDE',
'latitude': 'LATITUDE',
'depth': 'DEPTH',
'pressure': 'PRES'}


def add_og1_metadata(ds, deployment_yaml):
# Translate back to og names from pyglider names
for pyglider_var, og_var in pyglider_og_var_dict.items():
if pyglider_var in ds.keys():
ds = ds.rename({pyglider_var: og_var})

Check warning on line 647 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L645-L647 

Added lines #L645 - L647 were not covered by tests

# add some global attributes
attrs = ds.attrs
attrs["start_date"] = attrs["time_coverage_start"]
ds.attrs = attrs

Check warning on line 652 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L650-L652 

Added lines #L650 - L652 were not covered by tests

# Add empty variables for OG1
for variable, variable_dict in deployment_yaml['oceangliders_empty_variables'].items():
attrs = {name: str(val) for name, val in variable_dict.items() if name != 'value'}
ds[variable] = xr.DataArray(variable_dict['value'], attrs=attrs)

Check warning on line 657 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L655-L657 

Added lines #L655 - L657 were not covered by tests


# Add sensors
for device, device_dict in deployment_yaml['glider_devices'].items():
attrs = {name: str(val) for name, val in device_dict.items() if name != 'value'}
sensor_name = f"SENSOR_{device}_{device_dict['sensor_serial_number']}"
ds[sensor_name] = xr.DataArray(attrs=attrs)

Check warning on line 664 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L661-L664 

Added lines #L661 - L664 were not covered by tests

# add GPS variables
for vname in ["LATITUDE", "LONGITUDE", "TIME"]:
ds[f"{vname}_GPS"] = ds[vname].copy()
nan_val = np.nan
if vname == 'TIME':
nan_val = np.datetime64("NaT")
ds[f"{vname}_GPS"].values[ds["dead_reckoning"].values != 0] = nan_val
ds[f"{vname}_GPS"].attrs["long_name"] = f"{vname.lower()} of each GPS location"
ds["LATITUDE_GPS"].attrs["URI"] = (

Check warning on line 674 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L667-L674 

Added lines #L667 - L674 were not covered by tests
"https://vocab.nerc.ac.uk/collection/OG1/current/LAT_GPS/"
)
ds["LONGITUDE_GPS"].attrs["URI"] = (

Check warning on line 677 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L677 

Added line #L677 was not covered by tests
"https://vocab.nerc.ac.uk/collection/OG1/current/LON_GPS/"
)

ds["TRAJECTORY"] = xr.DataArray(

Check warning on line 681 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L681 

Added line #L681 was not covered by tests
ds.attrs["id"],
attrs={"cf_role": "trajectory_id", "long_name": "trajectory name"},
)
ds["PLATFORM_MODEL"] = xr.DataArray(

Check warning on line 685 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L685 

Added line #L685 was not covered by tests
ds.attrs["glider_model"],
attrs={
"long_name": "model of the glider",
"platform_model_vocabulary": "None",
},
)
ds["PLATFORM_SERIAL_NUMBER"] = xr.DataArray(

Check warning on line 692 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L692 

Added line #L692 was not covered by tests
f"sea{ds.attrs['glider_serial'].zfill(3)}",
attrs={"long_name": "glider serial number"},
)
ds["DEPLOYMENT_TIME"] = np.nanmin(ds.TIME.values)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The yamls have a "conversion" field for each variable that tries to fine the method in utils.py. So this could look like

netcdf_variables:
    DEPLOYMENT_TIME:
         "long_name": "date of deployment"
         ...
         "conversion": "nanmin_time"

with the accompanying change in utils.py. We may want to think of a way to make that more general (np.nan_min, "DEPLOYMENT_TIME) if yaml accepts tuples.

ds["DEPLOYMENT_TIME"].attrs = {

Check warning on line 697 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L696-L697 

Added lines #L696 - L697 were not covered by tests
"long_name": "date of deployment",
"standard_name": "time",
"units": "seconds since 1970-01-01T00:00:00Z",
"calendar": "gregorian",
}
ds["DEPLOYMENT_LATITUDE"] = ds.LATITUDE.values[0]
ds["DEPLOYMENT_LATITUDE"].attrs = {"long_name": "latitude of deployment"}
ds["DEPLOYMENT_LONGITUDE"] = ds.LONGITUDE.values[0]
ds["DEPLOYMENT_LONGITUDE"].attrs = {"long_name": "longitude of deployment"}
for var_name in ds.keys():
if "time" in var_name.lower() and var_name is not "TIME":
if 'units' in ds[var_name].attrs.keys():
ds[var_name].attrs.pop('units')
if 'calendar' in ds[var_name].attrs.keys():
ds[var_name].attrs.pop('calendar')
ds = ds.rename_dims({'TIME': 'N_MEASUREMENTS'})
return ds

Check warning on line 714 in pyglider/utils.py

View check run for this annotation

Codecov / codecov/patch

pyglider/utils.py#L703-L714 

Added lines #L703 - L714 were not covered by tests


def _parse_gliderxml_pos(fname):
"""
DEPRECATED: use slocum.parse_gliderState instead
Expand Down
Loading
Loading