Update metadata with frame_id, version and temporal_baseline_days

[cmarshak]

Updates CMR metadata to include:

frame_number: <frame_id which is -1 if frames are not used>,
version: <dataset_version; for this PR set to 3.0.0>,
temporal_baseline_days: <difference in reference and secondary center time in days>

Also, resolves #69 as we only use topsApp to process VV SLC (as noted in the ticket).

[dbekaert]

@cmarshak Do you use the interferogram extents for the start/stop time or the derive it based of the SLC input lists.
I recall the former from talking to you.
For the dateline corssing IFGs could it occur that reference and secondary are on opposite sides of the dateline making it 5/7 days? If it are the IFGs start/stop time then this will likely not occur as its already cropped to same utc start/stop time.

[cmarshak]

@dbekaert

Do you use the interferogram extents for the start/stop time or the derive it based of the SLC input lists.
I recall the former from talking to you.

I was not as precise as I could have been start/stop time. Literally just updated so it is based on the earliest start time from all SLCs in reference pass using the metadata collected from ASF stored during workflow - should be identical to SLCs.

For the dateline crossing IFGs, could it occur that reference and secondary are on opposite sides of the dateline making it 5/7 days? If it are the IFGs start/stop time then this will likely not occur as its already cropped to same utc start/stop time.

I think everything is in UTC so it's not an issue. That said, we can't process datelines because of Issue #78 (i.e. this comment with the relevant ISCE2 ticket).

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[cmarshak]

Providing some context for @dbekaert important comments about obtaining an accurate temporal baseline in days. We expect this temporal baseline in days to be a multiple of 6 based on S1's repeat pass frequency.

Suppose the temporal baseline (in days) is for example 5.999999. If delta is a datetime.timedelta object, then delta.days will be 5 (not 6). Therefore, we want to do the rounding using total_seconds(). This resolves David's concerns.

Here is an example:

import datetime
d0 = datetime.datetime(2021, 1, 1, 0, 0, 0)
d1 = datetime.datetime(2021, 1, 6, 23, 59, 59)
(d1 - d0).days

Will be 5.

To be clear, this concern can occur anywhere and anytime, not just at pairs in which acquisitions are close to midnight in UTC.

[cmarshak]

@dbekaert - I updated the dataset version for the coming processing campaign to v3.0.0 (it should be backward compatible with v2.0.*. Just let me know your thoughts.

[cmarshak]

@asjohnston-asf - ready for review and to coordinate on ingest.

[cmarshak]

@asjohnston-asf and @jhkennedy - should we also add reference_date so that we can uniquely request a GUNW from CMR based on:

• reference_date (can this be a string or should it be a datetime? - not sure what data types are permitted here exactly)
• temporal_baseline_days and
• frame_id

Let me know your thoughts. We already have start_time but this is would require specifying a range. Might be easier to simply to regex in the GUNW name which has the date in it. Not sure what the most straightforward approach would be. Would like to get this moving. Thanks!

[asjohnston-asf]

@asjohnston-asf and @jhkennedy - should we also add reference_date so that we can uniquely request a GUNW from CMR based on:

• reference_date (can this be a string or should it be a datetime? - not sure what data types are permitted here exactly)

I'd rather not duplicate the temporal information as an additional attribute when it's already available via the standard temporal fields in CMR. CMR is designed around searches by collection, temporal, and spatial searches first, and attribute searches second.

We set the temporal values in CMR based on what you send in the start_date and end_date fields in the metadata file. That's currently the start/end dates of the first reference scene. As you mention, you can do that search by providing the reference scene date as both the start/end of your temporal filter:

https://cmr.earthdata.nasa.gov/search/granules?provider=ASF&collection_concept_id=C1595422627-ASF&temporal=2023-03-20T18:02:51.000Z,2023-03-20T18:02:51.000Z

finds all scenes with reference scene S1A_IW_SLC__1SDV_20230320T180251_20230320T180309_047731_05BBDB_DCA0 (with and aquisition start of 2023-03-20T18:02:51.000Z)

[cmarshak]

@asjohnston-asf and @jhkennedy - let's coordinate on next steps.

[cmarshak]

We are also adding this to the metadata.

[cmarshak]

@asjohnston-asf

[asjohnston-asf]

@cmarshak Can you provide us an example command to run to get an output product with the latest and greatest meta.json file with the new attributes? @cirrusasf is running isce2_topsapp --reference-scenes S1B_IW_SLC__1SDV_20171117T145926_20171117T145953_008323_00EBAB_AFB8 --secondary S1A_IW_SLC__1SDV_20171111T150004_20171111T150032_019219_0208AF_EE89 from this branch and still getting a v2.0.6 product

[cmarshak]

I am not sure...

The DATASET_VERSION only being changed on this branch currently.

I am checking the version with this test here: https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/frame-metadata/tests/test_packaging.py#L1-L16

I just ran the following from this branch too:

(topsapp_env) --- bekaert-team/DockerizedTopsApp ‹frame-metadata› » ipython
Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:42:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from isce2_topsapp.packaging import DATASET_VERSION, get_gunw_id

In [2]: DATASET_VERSION
Out[2]: '3.0.0'

Is there a chance you are on dev branch?

[cmarshak]

Or am I misunderstanding the question entirely?

[cirrusasf]

Hi Charlie, My fault. I was on the dev branch. I switched to the frame-metadata branch now. Thank you for help. Jiang

…

On Thu, Apr 13, 2023 at 3:13 PM Charlie Marshak ***@***.***> wrote: Or am I misunderstanding the question entirely? — Reply to this email directly, view it on GitHub <#102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO3CPWGMBQY5UDQ26Y7FJU3XBCCANANCNFSM6AAAAAAU24OOPQ> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[cmarshak]

@cirrusasf @asjohnston-asf - I am going to merge this branch and run a single job using the operational user (should be published to UAT venue). We can hopefully discuss in the coming weeks how to finalize the metadata changes in person. Super appreciate your help on this.