diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b00f57..d736233 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,8 +26,8 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Changed * The CLI now *requires* `frame_id` (use `frame_id = -1` for old API and what is now considered a "non"-standard product) -* Water mask now uses `tile-mate` to download and merge ESA World cover tiles on -* All water masks applied to processing/packaging use ESA 10 meter world cover: ionosphere processing, browse imagery, and global attributes associate with mean coherence +* Water mask now uses `tile-mate>=0.0.8` to download and merge water mask tiles (Pekel Occurence data >= 95 is the default) +* All water masks applied to processing/packaging use Pekel Occurence (>= 95 percent occurence): ionosphere processing, browse imagery, and global attributes associate with mean coherence * Some function names associated to writing global attributes in the netcdf file were renamed to be more descriptive e.g. `record_stats` became `record_stats_as_global_attrs` ## [0.3.0] diff --git a/README.md b/README.md index bf35575..e96f67d 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,21 @@ We note all the input datasets are publicly available using a NASA Earthdata acc 3. Create a new environment and install requirements using `conda env update --file environment.yml` (or use [`mamba`](https://github.com/mamba-org/mamba) to speed install up) 4. Install the package from cloned repo using `python -m pip install -e .` +### For Mac M1 Silicon Users + +`ISCE2` requires Intel `x86_64` complied, conda-forge packages. Please follow the directions [here](https://conda-forge.org/docs/user/tipsandtricks.html#installing-apple-intel-packages-on-apple-silicon) i.e. + +``` +CONDA_SUBDIR=osx-64 conda create -n topsapp_env python +conda activate topsapp_env +conda config --env --set subdir osx-64 +``` +Then check +``` +python -c "import platform;print(platform.machine())" # Should print "x86_64" +echo "CONDA_SUBDIR: $CONDA_SUBDIR" # Should print "CONDA_SUBDIR: osx-64" +``` + ## Additional setup 1. Ensure that your `~/.netrc` file has: @@ -158,7 +173,6 @@ or as a json: ## Expedient Docker Test for GUNW Generation Create a new directory (for all the intermediate files) and navigate to it. - ``` docker run -ti -v $PWD:/home/ops/topsapp_data topsapp_img \ --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E \ @@ -209,8 +223,8 @@ EARTHDATA_PASSWORD=... The main entrypoint is in the `__main__.py` file [here](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/__main__.py). The whole standard `++slc` workflow that generates the `ARIA-S1-GUNW` product is summarized in this file. The workflow takes 1.5 - 3 hours to complete. -For developing a new feature, we need to modify a very small parts of this long running workflow e.g. the packaging of ISCE2 outputs into a netcdf file, staging/preparation of auxiliary data, etc. -Thus, for development it is not necessary to rerun the workflow each time to test a new feature, but utilize the intermediate ISCE data files and fix relevant parts of the code. +Likely, when developing a small new feature, we need to modify only of this long running workflow e.g. the packaging of ISCE2 outputs into a netcdf file, staging/preparation of auxiliary data, etc. +Thus, for development, it is recommended to not rerun the workflow each time, but utilize the intermediate ISCE data files and the metadata stored as json by this plugin. We have some sample [notebooks](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsapp-Debugging-NBs) to load the relevant metadata to make this "jumping" into the code slightly easier. We note that `ISCE2` generates *a lot* of interemdiate files. For our workflow, this can be between 100-150 GBs of disk required. @@ -225,6 +239,7 @@ However, the plugin takes about 1.5 to 3 hours (depending on the number of corre Therefore, these integration are *not* sufficient to permit a new release (see more instructions below). Until we have a complete end-to-end test of the workflow (via Hyp3), any new feature cannot be integrated into official production (i.e. the `main` branch). As a first step, it is imperative to share the output of a new feature (i.e. the GUNW file and the command to generate it). +Here is a notebook that demonstrates how to compare GUNWs that will be very helpful: https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsapp-Debugging-NBs/blob/dev/2_Compare_GUNWs.ipynb ## Hyp3 and Cloud Submission @@ -261,9 +276,10 @@ Here are more detailed [notes](https://github.com/ACCESS-Cloud-Based-InSAR/CICD- ### Additional (manual) checks -Even though there are some integration tests and unit tests in our test suites, this CPU intensive workflow cannot be tested end-to-end using github actions (there is simply not enough memory and CPU on a github runner). +Even though there are some integration tests and unit tests in our test suites, this CPU intensive workflow cannot be tested end-to-end using github actions (there is simply not enough memory and CPU for these workflows). Therefore, even if all the tests pass, there is still a nontrivial chance a GUNW is not successfully generated (e.g. the `topo` step of topsapp fizzles out because `numpy` API was not successfully tracked in the latest ISCE2 release). We request a sample GUNW be shared in a PR. +Ideally, a comparison of the GUNW created with a new branch and an existing one (as done in this [notebook](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsapp-Debugging-NBs/blob/dev/2_Compare_GUNWs.ipynb)) is ideal. Even if we go through careful accounting, once a PR is merged into `dev`, we will use `hyp3` to further inspect the new plugin e.g. through this [notebook](https://github.com/ACCESS-Cloud-Based-InSAR/s1_frame_enumerator/blob/dev/notebooks/Submitting_to_Hyp3.ipynb) using a few sites. It is important to use `INSAR_ISCE_TEST` job to ensure the features from the `dev` branch are used. Only after these **manual** checks, will we continue with a release of the plugin. diff --git a/environment.yml b/environment.yml index d2431c0..3d96de9 100644 --- a/environment.yml +++ b/environment.yml @@ -44,4 +44,4 @@ dependencies: - tqdm - dem_stitcher>=2.5.5 - aiohttp # only needed for manifest and swath download - - tile_mate>=0.0.7 + - tile_mate>=0.0.8 diff --git a/isce2_topsapp/__main__.py b/isce2_topsapp/__main__.py index 94551ac..ca42ed4 100644 --- a/isce2_topsapp/__main__.py +++ b/isce2_topsapp/__main__.py @@ -66,7 +66,7 @@ def localize_data( # For ionospheric correction computation if water_mask_flag: out_water_mask = download_water_mask( - out_slc["extent"], water_mask_name="esa_world_cover_2021_10m") + out_slc["extent"], water_mask_name="pekel_water_occurrence_2021") out_aux_cal = download_aux_cal() diff --git a/isce2_topsapp/localize_mask.py b/isce2_topsapp/localize_mask.py index 0c99a4f..3147a07 100644 --- a/isce2_topsapp/localize_mask.py +++ b/isce2_topsapp/localize_mask.py @@ -10,7 +10,7 @@ def download_water_mask( - extent: list, water_mask_name: str = "esa_world_cover_2021_10m", buffer: float = 0.1 + extent: list, water_mask_name: str = "pekel_water_occurrence_2021", buffer: float = 0.1 ) -> dict: output_dir = Path(".").absolute() @@ -39,6 +39,22 @@ def download_water_mask( mask_filename = fix_image_xml(mask_filename) + elif water_mask_name == 'pekel_water_occurrence_2021': + X, p = get_raster_from_tiles(extent_buffered, tile_shortname='pekel_water_occ_2021') + mask = (X >= 95).astype(np.uint8) + mask[mask.astype(bool)] = 255 + mask_filename = 'water_mask_derived_from_pekel_water_occurrence_2021_with_at_least_95_perc_water.geo' + + # Remove esa nodata, change gdal driver to ISCE, and generate VRT as in localize DEM + p_isce = p.copy() + p_isce['nodata'] = None + p_isce['driver'] = 'ISCE' + + with rasterio.open(mask_filename, 'w', **p_isce) as ds: + ds.write(mask) + + mask_filename = fix_image_xml(mask_filename) + elif water_mask_name == "SWBD": # Download SRTM-SWDB water mask # Water mask dataset extent diff --git a/isce2_topsapp/water_mask.py b/isce2_topsapp/water_mask.py index 2cec0bb..6ea8564 100644 --- a/isce2_topsapp/water_mask.py +++ b/isce2_topsapp/water_mask.py @@ -21,8 +21,8 @@ def get_water_mask_raster_for_browse_image(profile: dict) -> np.ndarray: """ extent = array_bounds(profile["height"], profile["width"], profile["transform"]) - X_esa, p_esa = get_raster_from_tiles(extent, tile_shortname="esa_world_cover_2021") - X_esa_r, _ = reproject_arr_to_match_profile(X_esa, p_esa, profile, resampling='nearest') - mask = (X_esa_r == 80).astype(bool) + X_occ, p_occ = get_raster_from_tiles(extent, tile_shortname="pekel_water_occ_2021") + X_occ_r, _ = reproject_arr_to_match_profile(X_occ, p_occ, profile, resampling='bilinear') + mask = (X_occ_r >= 95).astype(bool) mask = mask[0, ...] return mask diff --git a/tests/test_packaging.py b/tests/test_packaging.py index 4a5c2af..845d2e2 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -66,7 +66,7 @@ def test_mean_of_geocoded_isce_outputs(): "mean_filtered_coherence_without_water_mask": 0.4395995, "mean_filtered_coherence_with_water_mask": 0.4649538, "mean_unfiltered_coherence_without_water_mask": 0.33125195, - "mean_unfiltered_coherence_with_water_mask": 0.33531728, + "mean_unfiltered_coherence_with_water_mask": 0.3347433, "mean_incidence_angle": 38.845047, "mean_azimuth_angle": -169.84756469726562, }