Daily amsr2 prototype fields #140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Still TODO is adding the code to combine the F17 and AMSR2 fields and publish
everything in the way we want it to be.
As of this commit, output looks like this:
```
/share/apps/G02202_V5/25km/combined/
└── intermediate
└── north
├── checksums
│ ├── daily
│ │ └── 2022
│ └── monthly
├── daily
│ └── 2022
├── initial_daily
├── monthly
└── temporal_interp
```
trey-stafford
changed the base branch from
main
to
revert-143-revert-142-update-pm_icecon
trey-stafford
force-pushed
the
combine-amsr2-prototype-fields
branch
from
7b9ab90 to
858e7ec
Compare
These are group-specific. All other attrs are inherited from the root group
trey-stafford
changed the base branch from
revert-143-revert-142-update-pm_icecon
to
ecdr-at-25km
* Remove `valid_range` from corodinate variables * Add `coverage_content_type: coordinate` attr to coordinate variables * Add `coordinates: "time y x"` attr to data variables. This lets Panoply find the root group coordinate variables for plotting.
resolves mypy error. More cleanup to come.
trey-stafford
force-pushed
the
combine-amsr2-prototype-fields
branch
from
af998d4 to
6cf10f2
Compare
sc0tts
reviewed
Sep 17, 2024
sc0tts
reviewed
Sep 17, 2024
sc0tts
reviewed
Sep 17, 2024
sc0tts
reviewed
Sep 17, 2024
sc0tts
reviewed
Sep 17, 2024
| hemisphere=hemisphere, | ||
| is_nrt=False, | ||
| ) | ||
| cdr_var_fieldnames = [ |
There was a problem hiding this comment.
Oh man, this list looks so NICE compared to prior versions!
sc0tts
reviewed
Sep 17, 2024
sc0tts
reviewed
Sep 17, 2024
| config = PlatformConfig( | ||
| platforms=[ | ||
| Platform( | ||
| id="ame", |
There was a problem hiding this comment.
I don't think we're allowing AMSR-E, are we?
There was a problem hiding this comment.
Not yet (maybe not ever?) but the code exists to support it. This unit test just uses ame as an example.
sc0tts
approved these changes
Sep 17, 2024
There was a problem hiding this comment.
I added some single-comments to specific locations in the code where I think we should add "TODO" statements to remind us to replace hard-coded values with configuration values.
Also, there is mention of AMSR-E at at least one place in the code. Is that OBE now?
But these are minor and should not hold up the merging of this valuable PR!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial code to support creating daily files with an
amsr2_prototypesubgroup for AMSR2 fields.There are still a fair number of TODOs here, but they can wait for if we have more time after adding support for daily aggregate files, monthly files, validation, and NRT.
For usage, see
make_25km_ecdr.sh.Creating the AMSR2 files relies on setting an environment variable at import-time, which means that the code cannot run as a single process. First, daily data is produced using the default platform ranges. Then, a new process is spawned (see
cli/daily.py) with thePLATFORM_START_DATES_CONFIG_FILEPATHset to the prototype platform start dates config (containing just AMSR2), which produces the amsr2 fields. Finally, the files get staged for publication by combining the F17 outputs w/ AMSR2 outputs in theprototype_amsr2group (publish_daily.py).Longer term, a complete refactor of the code to support this scheme from foundations would be ideal. We could have a single config file with
cdr_platform_start_datesandprototype_platform_start_dateskeys that the program would know how to parse and act accordingly without needing to spawn separate processes to access different platform configs. We don't have the time for that now, and the present approach is a byproduct of development that originally did not have this in mind (we were going to produce separate 12.5km data files for AMSR2 instead of adding it as a "prototype" group to F17 output files - just a single platform start date config).