Unit test for IONEX functionalities #80

vbrancat · 2023-01-04T21:00:58Z

This PR addresses issue #61 and adds the unit test for testing the IONEX reader and interpolation methods.

At the moment, the PR includes the IONEX golden dataset inside the repository. Eventually, the unit test can include functionality to download the required data for the test.

Not sure how to set up the test to run on the CI system

force different entry point for unit tests

ionex testing with less repeat code and ci-capable

scottstanie

I think I found some fixes for it. Note that I've got pytests installing as a command in the docker run part, but we might consider having a tests/requirements.txt file if/when we have more requirements needed for testing

.circleci/config.yml

tests/conftest.py

.circleci/config.yml

scottstanie · 2023-01-10T16:32:54Z

A separate note-
The way the Dockerfile is currently set up isn't great for testing our own code changes.
The 4th command (line 11) is COPY . /home/compass_user/OPERA/COMPASS
Then after that, all of the installing and setup happens. So if you change any file, it invalidates the cache and makes you run everything after that. Since code changes are more ocmmon than environment/requirement changes the COPY should go as late as possible @seongsujeong

vbrancat · 2023-01-13T05:49:04Z

A separate note- The way the Dockerfile is currently set up isn't great for testing our own code changes. The 4th command (line 11) is COPY . /home/compass_user/OPERA/COMPASS Then after that, all of the installing and setup happens. So if you change any file, it invalidates the cache and makes you run everything after that. Since code changes are more ocmmon than environment/requirement changes the COPY should go as late as possible @seongsujeong

@seongsujeong any opinion on this?

works for scipy < 1.10

…nex_unit_test

LiangJYu · 2023-01-20T21:18:47Z

The unit test only works for scipy < 1.10. Do we want use scipy > 1.10 and debug/refactor compass.utils.iono? Or force scipy the environment scipy <= 1.9?

LiangJYu · 2023-01-20T23:12:19Z

The unit test only works for scipy < 1.10. Do we want use scipy > 1.10 and debug/refactor compass.utils.iono? Or force scipy the environment scipy <= 1.9?

Disregard above. Fixed with e0f2b02

vbrancat · 2023-01-24T22:33:02Z

@scottstanie @LiangJYu ready for you to review whenever you have time. All your comments have been addressed.

src/compass/utils/iono.py

docker - moved copy of source after miniconda install

…nex_unit_test

LiangJYu · 2023-01-26T02:20:00Z

A separate note- The way the Dockerfile is currently set up isn't great for testing our own code changes. The 4th command (line 11) is COPY . /home/compass_user/OPERA/COMPASS Then after that, all of the installing and setup happens. So if you change any file, it invalidates the cache and makes you run everything after that. Since code changes are more ocmmon than environment/requirement changes the COPY should go as late as possible @seongsujeong

fixed?

LiangJYu · 2023-01-26T02:28:31Z

@seongsujeong @scottstanie Can you guys please take a look? I think the pressing issues in the PR have been addressed. Thanks.

seongsujeong · 2023-01-26T02:33:25Z

A separate note- The way the Dockerfile is currently set up isn't great for testing our own code changes. The 4th command (line 11) is COPY . /home/compass_user/OPERA/COMPASS Then after that, all of the installing and setup happens. So if you change any file, it invalidates the cache and makes you run everything after that. Since code changes are more ocmmon than environment/requirement changes the COPY should go as late as possible @seongsujeong

I agree that can be an efficient way of building Docker file, but I think I'd prefer to invalidate the cache when we make any changes to the file we are copying into. That would make docker build slower, but we can make sure any changes in the code are properly applied into the layers of the Docker images. Note that we are using the same Dockerfile not only for test but also release.

LiangJYu · 2023-01-26T02:36:15Z

I'm not sure if anything can be done to address remaining codacy issues. If anyone has fixes, please share!

seongsujeong · 2023-01-26T02:42:03Z

I'm not sure if anything can be done to address remaining codacy issues. If anyone has fixes, please share!

I see codacy issue on the line 46 in Dockerfile.
Can you try just removing the echo commands that prints out the messages into stdout?

The line is just to show the progress, but those messages are hard to read anyhow because the Docker build process gives out tons of lines of messages.

seongsujeong

LGTM. Just left a comment for Docker image, which I think I will address when preparing for the upcoming delivery.

Please make sure that Scott's suggestions on tests/conftest.py are addressed. (e.g. https://github.com/opera-adt/COMPASS/pull/80/files#r1065984932 )

docker/Dockerfile

* Unit test for IONEX functionalities * Add ionex unit test data * adopt pytest fixtures * Allocate variables in conftest * rewrite Docker command * Test with pytest in circleci docker - moved copy of source after miniconda install * fix tec URL

@LiangJYu

* Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * Write `NaN` into HDF5 instead of `None` for metadata (#82) * Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * write NaN instead of None Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> * Apply corrections in geocodeSlc module (#78) * Add Boolean flag to acitivate/disactivate LUT corrections * Feed LUT corrections to geocode SLC * Save LUTs in CSLC product when computed, skip otherwise * Remove trailing space * Add cumulative LUT function * Modify CSLC validation script (#77) * Throw error when percentage of pixels is above threshold * Address comments on threshold and relative difference: * Specify threshold in the print message. Suppress numpy warnings * Use masked array; modify failed pixel percentage computation * Use masked array to compute NaNs * Correct division by reference pixels * More verbose way to compute absolute difference * compare raster per pixel use masked arrays assign thresholds to variables report on both real and imag % fails Co-authored-by: Liang Yu <liangjyu@gmail.com> * make common burst filtering optional in stack processor (#60) * make common burst filtering optional in stack processor * also add gix for bad function name * Update dependencies (#88) * Add pandas dependency * add lmxl * Include default values in runconfig written to HDF5 metadata (#87) * add default options to metadata runconfig str * Update src/compass/utils/runconfig.py Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com> * Unit test for IONEX functionalities (#80) * Unit test for IONEX functionalities * Add ionex unit test data * adopt pytest fixtures * Allocate variables in conftest * rewrite Docker command * Test with pytest in circleci docker - moved copy of source after miniconda install * fix tec URL * ionospheric correction - working version * `ionex_file` added into `dynamic_ancillary_file_group` in runconfig * Interface revision for iono / tropo corrections; test code removed * `ionex_path` added * shortcut to ionex path in cfg * reverting the changes in parameters for calling `az_fm_rate_mismatch_mitigation` * ionospheric correction writeout to CSLC * removed unnecessary lines * fixing codacy issue * PEP8 issue * updating runconfig template for CI * revision on runconfig scheme and defaults * PR comments addressed * docstring revised * revision on the description in corrections * addressing comments in the 2nd round review * PEP8 --------- Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Scott Staniewicz <scott.j.staniewicz@jpl.nasa.gov>

@LiangJYu

* Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * Write `NaN` into HDF5 instead of `None` for metadata (opera-adt#82) * Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * write NaN instead of None Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> * Apply corrections in geocodeSlc module (opera-adt#78) * Add Boolean flag to acitivate/disactivate LUT corrections * Feed LUT corrections to geocode SLC * Save LUTs in CSLC product when computed, skip otherwise * Remove trailing space * Add cumulative LUT function * Modify CSLC validation script (opera-adt#77) * Throw error when percentage of pixels is above threshold * Address comments on threshold and relative difference: * Specify threshold in the print message. Suppress numpy warnings * Use masked array; modify failed pixel percentage computation * Use masked array to compute NaNs * Correct division by reference pixels * More verbose way to compute absolute difference * compare raster per pixel use masked arrays assign thresholds to variables report on both real and imag % fails Co-authored-by: Liang Yu <liangjyu@gmail.com> * make common burst filtering optional in stack processor (opera-adt#60) * make common burst filtering optional in stack processor * also add gix for bad function name * Update dependencies (opera-adt#88) * Add pandas dependency * add lmxl * Include default values in runconfig written to HDF5 metadata (opera-adt#87) * add default options to metadata runconfig str * Update src/compass/utils/runconfig.py Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com> * Unit test for IONEX functionalities (opera-adt#80) * Unit test for IONEX functionalities * Add ionex unit test data * adopt pytest fixtures * Allocate variables in conftest * rewrite Docker command * Test with pytest in circleci docker - moved copy of source after miniconda install * fix tec URL * ionospheric correction - working version * `ionex_file` added into `dynamic_ancillary_file_group` in runconfig * Interface revision for iono / tropo corrections; test code removed * `ionex_path` added * shortcut to ionex path in cfg * reverting the changes in parameters for calling `az_fm_rate_mismatch_mitigation` * ionospheric correction writeout to CSLC * removed unnecessary lines * fixing codacy issue * PEP8 issue * updating runconfig template for CI * revision on runconfig scheme and defaults * PR comments addressed * docstring revised * revision on the description in corrections * addressing comments in the 2nd round review * PEP8 --------- Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Scott Staniewicz <scott.j.staniewicz@jpl.nasa.gov>

vbrancat added 2 commits January 4, 2023 12:56

Unit test for IONEX functionalities

308ee99

Add ionex unit test data

81c89e4

vbrancat requested review from LiangJYu, scottstanie and seongsujeong January 4, 2023 21:01

LiangJYu and others added 3 commits January 4, 2023 17:34

adopt pytest fixtures

99249d2

force different entry point for unit tests

Merge pull request #4 from LiangJYu/ionex_unit_test

953dc88

ionex testing with less repeat code and ci-capable

Add docstring remove unused variables

01fcffd

scottstanie requested changes Jan 10, 2023

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

tests/conftest.py Outdated Show resolved Hide resolved

tests/conftest.py Show resolved Hide resolved

tests/conftest.py Show resolved Hide resolved

.circleci/config.yml Outdated Show resolved Hide resolved

vbrancat and others added 5 commits January 12, 2023 21:49

Allocate variables in conftest

c53032f

rewrite Docker command

11925a0

replace journal logging

0bd648e

updated docker + circleci

e033b2e

works for scipy < 1.10

Merge branch 'ionex_unit_test' of github.com:vbrancat/COMPASS into io…

6fdbf4a

…nex_unit_test

make contiguous

e0f2b02

vbrancat added 2 commits January 23, 2023 17:36

Merge remote-tracking branch 'upstream/main' into ionex_unit_test

7101112

Test with pytest

0b69895

scottstanie reviewed Jan 25, 2023

View reviewed changes

src/compass/utils/iono.py Outdated Show resolved Hide resolved

LiangJYu added 3 commits January 25, 2023 18:14

simplify circleci by a bit

c167a46

docker - moved copy of source after miniconda install

Merge branch 'ionex_unit_test' of github.com:LiangJYu/COMPASS into io…

01a1869

…nex_unit_test

fix tec URL

f42af4d

address codacy issues

ca1add9

remove echos for codacy

dfca0b6

scottstanie self-requested a review January 26, 2023 14:01

scottstanie approved these changes Jan 26, 2023

View reviewed changes

seongsujeong approved these changes Feb 2, 2023

View reviewed changes

docker/Dockerfile Show resolved Hide resolved

LiangJYu merged commit 8d04477 into opera-adt:main Feb 2, 2023

vbrancat deleted the ionex_unit_test branch February 9, 2023 16:01

vbrancat mentioned this pull request Feb 13, 2023

Unit test for IONEX reader #61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unit test for IONEX functionalities #80

Unit test for IONEX functionalities #80

vbrancat commented Jan 4, 2023

scottstanie left a comment •

edited

Loading

scottstanie commented Jan 10, 2023

vbrancat commented Jan 13, 2023

LiangJYu commented Jan 20, 2023

LiangJYu commented Jan 20, 2023

vbrancat commented Jan 24, 2023

LiangJYu commented Jan 26, 2023

LiangJYu commented Jan 26, 2023

seongsujeong commented Jan 26, 2023 •

edited

Loading

LiangJYu commented Jan 26, 2023

seongsujeong commented Jan 26, 2023

seongsujeong left a comment

Unit test for IONEX functionalities #80

Unit test for IONEX functionalities #80

Conversation

vbrancat commented Jan 4, 2023

scottstanie left a comment • edited Loading

Choose a reason for hiding this comment

scottstanie commented Jan 10, 2023

vbrancat commented Jan 13, 2023

LiangJYu commented Jan 20, 2023

LiangJYu commented Jan 20, 2023

vbrancat commented Jan 24, 2023

LiangJYu commented Jan 26, 2023

LiangJYu commented Jan 26, 2023

seongsujeong commented Jan 26, 2023 • edited Loading

LiangJYu commented Jan 26, 2023

seongsujeong commented Jan 26, 2023

seongsujeong left a comment

Choose a reason for hiding this comment

scottstanie left a comment •

edited

Loading

seongsujeong commented Jan 26, 2023 •

edited

Loading