Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update csv2bufr exercise, add daycli #115

Merged
merged 6 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions documentation/docs/assets/tables/averaging-method-table.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Integer,Description
0,Average of minimum and maximum temperatures
1,Average of 8 tri-hourly observations
2,Average of 24 hourly observations
3,Weighted average of three observations
4,"Weighted average of three observations, as well as the minimum and maximum values"
5,AWS complete integration from minute data
6,Average of 4 six-hourly observations
24 changes: 12 additions & 12 deletions documentation/docs/assets/tables/aws-minimal.csv
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
"Column","Units","Data type","Description"
"wsi_series","","Integer","WIGOS identifier series"
"wsi_issuer","","Integer","WIGOS issuer of identifier"
"wsi_issue_number","","Integer","WIGOS issue number"
"wsi_local","","Character","WIGOS local identifier"
"wmo_block_number","","Integer","WMO block number"
"wmo_station_number","","Integer","WMO station number"
"station_type","","Integer","Type of observing station, encoding using code table 0 02 001 (set to 0, automatic)"
"year","","Integer","Year (UTC), the time of observation (based on the actual time the barometer is read)"
"month","","Integer","Month (UTC), the time of observation (based on the actual time the barometer is read)"
"day","","Integer","Day (UTC), the time of observation (based on the actual time the barometer is read)"
"hour","","Integer","Hour (UTC), the time of observation (based on the actual time the barometer is read)"
"minute","","Integer","Minute (UTC), the time of observation (based on the actual time the barometer is read)"
"wsi_series"," " ,"Integer","WIGOS identifier series"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add a space, we want an empty string if no units, not a space.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise the page will display "nan", with an empty space the table looks better.

"wsi_issuer"," " ,"Integer","WIGOS issuer of identifier"
"wsi_issue_number"," " ,"Integer","WIGOS issue number"
"wsi_local"," " ,"Character","WIGOS local identifier"
"wmo_block_number"," " ,"Integer","WMO block number"
"wmo_station_number"," " ,"Integer","WMO station number"
"station_type"," " ,"Integer","Type of observing station, encoding using code table 0 02 001 (set to 0, automatic)"
"year"," " ,"Integer","Year (UTC), the time of observation (based on the actual time the barometer is read)"
"month"," " ,"Integer","Month (UTC), the time of observation (based on the actual time the barometer is read)"
"day"," " ,"Integer","Day (UTC), the time of observation (based on the actual time the barometer is read)"
"hour"," " ,"Integer","Hour (UTC), the time of observation (based on the actual time the barometer is read)"
"minute"," " ,"Integer","Minute (UTC), the time of observation (based on the actual time the barometer is read)"
"latitude","degrees","Decimal","Latitude of the station (to 5 decimal places)"
"longitude","degrees","Decimal","Longitude of the station (to 5 decimal places)"
"station_height_above_msl","meters","Decimal","Height of the station ground above mean sea level (to 1 decimal place)"
Expand Down
53 changes: 53 additions & 0 deletions documentation/docs/assets/tables/daycli-table.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
Column,Units,Data Type,Description
wsi_series, ,Integer,"WIGOS Identifier series, set to 0 for stations"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above re space vs length zero string.

wsi_issuer, ,Integer,"WIGOS Identifier issue, ISO 3 digit (number) country code or 20000 series"
wsi_issue_number, ,Integer,WIGOS Identifier issue number
wsi_local, ,Character,"WIGOS Identifier local identifier, alphanumeric, max 16 characters"
wmo_block_number, ,Integer,WMO block number for station 0 - 99
wmo_station_number, ,Integer,WMO station number 0 - 999
latitude,Degrees,Decimal,Latitude of the station (to 5 decimal places)
longitude,Degrees,Decimal,Longitude of the station (to 5 decimal places)
station_height_above_msl,Meters,Decimal,Height of the station ground above mean sea level (to 1 decimal place)
averaging_method, ,Integer,Method used to calculate daily average temperature
temperature_siting_classification, ,Integer,"Combined sensor siting and measurement quality classification (temperature). See references for siting classification"
precipitation_siting_classification, ,Integer,"Combined sensor siting and measurement quality classification (precipitation). See references for siting classification"
year, ,Integer,Year (UTC) of nominal reporting day
month, ,Integer,Month (UTC) of nominal reporting day
day, ,Integer,Day (UTC) of month for nominal reporting day
precipitation_day_offset, ,Integer,"Start of reporting period for precipitation, offset in days relative to nominal reporting day (-1 or 0)"
precipitation_hour, ,Integer,Beginning hour (UTC) over which the precipitation is measured
precipitation_minute, ,Integer,Beginning minute (UTC) over which the precipitation is measured
precipitation_second, ,Integer,Beginning second (UTC) over which the precipitation is measured
precipitation,kg m-2,Decimal,"Total accumulated precipitation over indicated 24 hour period (kg m-3, equivalent to mm)"
precipitation_flag, ,Integer,Quality control flag for total accumulated precipitation
fresh_snow_day_offset, ,Integer,"Start of reporting period for fresh snow depth, offset in days relative to nominal reporting day (-1 or 0)"
fresh_snow_hour, ,Integer,Beginning hour (UTC) over which the fresh snow depth is measured
fresh_snow_minute, ,Integer,Beginning minute (UTC) over which the fresh snow depth is measured
fresh_snow_second, ,Integer,Beginning second (UTC) over which the fresh snow depth is measured
fresh_snow_depth,Meters,Decimal,"Fresh snow depth over indicated 24 hour period (m, 2 d.p.)"
fresh_snow_depth_flag, ,Integer,Quality control flag for the fresh snow depth
total_snow_day_offset, ,Integer,"Start of reporting period for total snow depth, offset in days relative to nominal reporting day (-1 or 0)"
total_snow_hour, ,Integer,Beginning hour (UTC) over which the total snow depth is measured
total_snow_minute, ,Integer,Beginning minute (UTC) over which the total snow depth is measured
total_snow_second, ,Integer,Beginning second (UTC) over which the total snow depth is measured
total_snow_depth,Meters,Decimal,"Total snow depth over indicated 24 hour period (m, 2 d.p.)"
total_snow_depth_flag, ,Integer,Quality control flag for the total snow depth
thermometer_height,Meters,Decimal,Height of thermometer above local ground level (m)
maximum_temperature_day_offset, ,Integer,"Start of reporting period for maximum temperature, offset in days relative to nominal reporting day (-1 or 0)"
maximum_temperature_hour, ,Integer,Beginning hour (UTC) over which the maximum temmperature is measured
maximum_temperature_minute, ,Integer,Beginning minute (UTC) over which the maximum temperature is measured
maximum_temperature_second, ,Integer,Beginning second (UTC) over which the maximum temperature is measured
maximum_temperature,Kelvin,Decimal,"Maximum temperature over indicated 24 hour period (K, 2 d.p.)"
maximum_temperature_flag, ,Integer,Quality control flag for the maximum temperature
minimum_temperature_day_offset, ,Integer,"Start of reporting period for minimum temperature, offset in days relative to nominal reporting day (-1 or 0)"
minimum_temperature_hour, ,Integer,Beginning hour (UTC) over which the minimum temmperature is measured
minimum_temperature_minute, ,Integer,Beginning minute (UTC) over which the minimum temperature is measured
minimum_temperature_second, ,Integer,Beginning second (UTC) over which the minimum temperature is measured
minimum_temperature,Kelvin,Decimal,"Minimum temperature over indicated 24 hour period (K, 2 d.p.)"
minimum_temperature_flag, ,Integer,Quality control flag for the minimum temperature
average_temperature_day_offset, ,Integer,"Start of reporting period for average temperature, offset in days relative to nominal reporting day (-1 or 0)"
average_temperature_hour, ,Integer,Beginning hour (UTC) over which the average temmperature is measured
average_temperature_minute, ,Integer,Beginning minute (UTC) over which the average temperature is measured
average_temperature_second, ,Integer,Beginning second (UTC) over which the average temperature is measured
average_temperature,Kelvin,Decimal,"Average temperature over indicated 24 hour period (K, 2 d.p.)"
average_temperature_flag, ,Integer,Quality control flag for the average temperature
9 changes: 9 additions & 0 deletions documentation/docs/assets/tables/quality_flag.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Integer,Description
0,Data checked and declared good
1,Data checked and declared suspect
2,Data checked and declared aggregated
3,Data checked and declared out of instrument range
4,Data checked and declared aggregated and out of instrument range
5,Parameter is not measured at the station
6,Daily value not provided
7,Data unchecked
17 changes: 17 additions & 0 deletions documentation/docs/csv2bufr-templates/aws-template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: AWS Template
---

# csv2bufr template for Automated Weather Stations reporting hourly GBON data

The **AWS Template** uses a standardized CSV format to ingest data from Automatic Weather Stations in support of GBON reporting requirements. This mapping template converts CSV data to BUFR sequence 301150, 307096.

The format is intended for use with automatic weather stations reporting a minimum number of parameters, including pressure, air temperature and humidity, wind speed and direction and precipitation on an hourly basis.

## CSV columns and description

{{ read_csv("docs/assets/tables/aws-minimal.csv") }}

## Example

Example CSV file that conforms to the AWS template: [aws-example.csv](/sample-data/aws-example.csv).
43 changes: 43 additions & 0 deletions documentation/docs/csv2bufr-templates/daycli-template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: DAYCLI Template
---

# csv2bufr template for daily climate data (DAYCLI)

The **DAYCLI** template provides a standardized CSV format for converting daily climate data to BUFR sequence 307075.

The format is intended for use with Climate Data Management Systems to publish data on WIS2, in support of reporting requirements for daily climate observations.

This templates maps daily observations of:

- Minimum, maximum and average temperature over 24 hours period
- Total accumulated precipitation over 24 hours period
- Total snow depth at time of observation
- Depth of fresh snow over 24 hours period

This template requires additional metadata with respect to the simplified AWS-template: method of calculating average temperature; sensor and station heights; exposure and measurement quality classification.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a caveat that the BUFR sequence is planned to be updated.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK added

!!! Note "About the DAYCLI template"
Please note that the DAYCLI BUFR sequence will be updated during 2025 to include additional information and revised QC flags. The DAYCLI template included the wis2box will be updated to reflect these changes. WMO will communicate when the wis2box-software is updated to include the new DAYCLI template, to allow users to update their systems accordingly.

## CSV columns and description

{{ read_csv("docs/assets/tables/daycli-table.csv") }}

## Averaging method

{{ read_csv("docs/assets/tables/averaging-method-table.csv") }}

## Quality flag

{{ read_csv("docs/assets/tables/quality_flag.csv") }}

## References for siting classification

[Reference for "temperature_siting_classification"](https://library.wmo.int/idviewer/35625/839).

[Reference for "precipitation_siting_classification"](https://library.wmo.int/idviewer/35625/840).

## Example

Example CSV file that conforms to the DAYCLI template: [daycli-example.csv](/sample-data/daycli-example.csv).
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ cp ~/exercise-materials/gts-headers-exercises/gts_headers_mapping.csv ~/wis2box-
Then restart the wis2box-management container to apply the changes:

```bash
docker restart wis2box-managment
docker restart wis2box-management
```

## Exercise 2: Ingesting data with GTS headers
Expand Down
Loading
Loading