-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modify CSLC validation script #77
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the revision on the validation script @vbrancat. Here I leave some comments as below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing my comments above. My output data have passed the validation with the new script (yay).
My last question is if it is okay to hard-code the thresholds?
And I think it would be helpful if the script displays some information about the comparison (thresholds, ratio of pixels above threshold etc.) when the validation fails. But I don't want to hold this PR too long for that feature, so I would suggest to write an issue and make that happen in the follow-up PRs.
Thank you for your review @seongsujeong. Regarding of having the threshold exposed as an option, I am a bit conflicted about it. The idea behind the script is that we come up with something that makes the quality of the generated product to be acceptable. Having the threshold to be tunable would allow the users to make that choice (and eventually tune it to make the validation script pass). What would be the advantage of exposing it? |
Sorry that my question was not so straightforward. I agree that the thresholds / criteria are fixed in the code. What I meant was that we declare some constants in the code (e.g. |
@vbrancat @hfattahi For more information about the difference, here I share the plot below. The plot shows the ratio of pixels (y axis) whose relative difference is larger than the value in the x axis. Left and right panel are for real and imaginary parts respectively. The output from Docker on Aurora is the reference, and the one from Docker on NISAR server is the secondary. The blue solid horizontal line is the relative tolerance, and the red solid vertical line is the threshold for the ratio of the pixels which did not pass the test. Based on the plot, less than 0.0002% of the pixels have relative difference larger than 1.0e-5. More information about the test data are as below: |
src/compass/utils/validate_cslc.py
Outdated
tot_pixels_ref = np.count_nonzero(~np.isnan(np.abs(slc_ref))) | ||
tot_pixels_sec = np.count_nonzero(~np.isnan(np.abs(slc_ref))) | ||
|
||
# Check that total number of pixel is the same | ||
assert tot_pixels_ref == tot_pixels_sec |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think comparing number of Nan's is a sufficiently robust check. The following toy example of 2 4x4 rasters (red Nan and green valid) would pass, when I think it ought to fail.
I think following per-pixel test would better validate that geolocation is consistent the same between the rasters:
ref_nan = isnan(slc_ref)
sec_nan = isnan(slc_sec)
np.testing.assert_array_equal(ref_nan, sec_nan)
I left out np.abs
because if either real or imaginary component is NaN, np.nan
evaluates to True
. e.g.:
In [4]: np.isnan([1, np.nan + np.nan * 1j, np.nan + 1j, 1 + np.nan * 1j])
Out[4]: array([False, True, True, True])
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uhm, I am a bit confused. My lines of code are comparing pixels that are not NaNs and leaving aside those pixels which are NaNs. With your code, we would compare only NaNs. The idea of the test is that the number of valid pixels (not NaNs) in the reference and secondary RSLCs are the same. Your code would do the opposite, doesn't it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By using np.count_nonzero
, only the number of non-NaN values is compared and no consideration is given for where the values are located in the array. Both toy rasters have 2 non-zero pixels at different locations and it incorrectly passes the current assert.
np.testing.assert_array_equal
does an element-by-element comparison of the NaN mask and it would correctly fail with the toy rasters since the non-NaNs are at different places.
src/compass/utils/validate_cslc.py
Outdated
np.seterr(invalid='ignore') | ||
diff_real = np.abs((slc_ref.real - slc_sec.real) / slc_sec.real) | ||
diff_imag = np.abs((slc_ref.imag - slc_sec.imag) / slc_sec.imag) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
np.seterr(invalid='ignore') | |
diff_real = np.abs((slc_ref.real - slc_sec.real) / slc_sec.real) | |
diff_imag = np.abs((slc_ref.imag - slc_sec.imag) / slc_sec.imag) |
Let's move this after NaN pixel comparison. There's no need to perform this computation if the pixel-wise NaN test fails.
src/compass/utils/validate_cslc.py
Outdated
# Compute the number of pixels in real part above threshold | ||
failed_pixels_real = np.count_nonzero(diff_real > 1.0e-5) | ||
failed_pixels_imag = np.count_nonzero(diff_imag > 1.0e-5) | ||
|
||
# Compute percentage of pixels in real and imaginary part above threshold | ||
percentage_real = failed_pixels_real / tot_pixels_ref | ||
percentage_imag = failed_pixels_imag / tot_pixels_ref | ||
|
||
# Check that percentage of pixels above threshold is lower than 0.1 % | ||
print('Check that the percentage of pixels in the difference between reference' | ||
'and secondary products real parts above the threshold 1.0e-5 is below 0.1 %') | ||
assert percentage_real < 0.001 | ||
|
||
print('Check that the percentage of pixels in the difference between reference' | ||
'and secondary products imaginary parts above the threshold 1.0e-5 is below 0.1 % %') | ||
assert percentage_imag < 0.001 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Compute the number of pixels in real part above threshold | |
failed_pixels_real = np.count_nonzero(diff_real > 1.0e-5) | |
failed_pixels_imag = np.count_nonzero(diff_imag > 1.0e-5) | |
# Compute percentage of pixels in real and imaginary part above threshold | |
percentage_real = failed_pixels_real / tot_pixels_ref | |
percentage_imag = failed_pixels_imag / tot_pixels_ref | |
# Check that percentage of pixels above threshold is lower than 0.1 % | |
print('Check that the percentage of pixels in the difference between reference' | |
'and secondary products real parts above the threshold 1.0e-5 is below 0.1 %') | |
assert percentage_real < 0.001 | |
print('Check that the percentage of pixels in the difference between reference' | |
'and secondary products imaginary parts above the threshold 1.0e-5 is below 0.1 % %') | |
assert percentage_imag < 0.001 | |
percent_fail_threshold = 0.001 | |
for dtype_op, dtype_str in zip([np.real, np.imag], | |
['real', 'imag']): | |
diff = (dtype_op(ma_slc_ref) - dtype_op(ma_slc_secsec)) / dype_op(ma_slc_sec) | |
percent_fail = diff / tot_pixels_ref | |
# Check that percentage of pixels above threshold is lower than 0.1 % | |
print(f'Check that the percentage of pixels in the difference between reference' | |
'and secondary products {dtype} parts above the threshold {max_diff_threshold} is below {percent_fail_threshold*100} %') | |
assert percent_fail < percent_fail_threshold |
Percentage failed computation moved into loop to avoid unnecessary repeat computation.
Another approach would be run both real and imag tests and saving the error string should either test fail. This is more thorough since both real and imag pixels are checked. Code as follows:
percent_fail_threshold = 0.001
any_percent_fail = False
err_str = []
for dtype_op, dtype_str in zip([np.real, np.imag],
['real', 'imag']):
diff = (dtype_op(ma_slc_ref) - dtype_op(ma_slc_sec)) / dype_op(ma_slc_sec)
percent_fail = diff / tot_pixels_ref
# Check that percentage of pixels above threshold is lower than 0.1 %
err_str.append(f'Check that the percentage of pixels in the difference between reference'
'and secondary products {dtype} parts above the threshold {max_diff_threshold} is below {percent_fail_threshold*100} %')
any_percent_fail = percent_fail < percent_fail_threshold
assert any_percent_fail, '\n'.join(err_str)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the last chunk of code, I am not sure what is the intent with 2 items:
percent_fail
is a matrix but we do want a ballpark number describing the percentage of pixels failing to meet the condition, what is the intent here in your code?- In the last line with the
assert
condition,any_percent_fail
is a matrix again and theassert
condition will fail. Is this intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current code will only report an error in the imaginary if the real test pass. i.e.:
compute real_percentage
assert real_percentage < percentage_threshold
# check imaginary only if previous assert passes
compute imaginary_percentage
assert imaginary_percentage < percentage_threshold
The 2nd chunk of code (when properly coded) will test for both real and imaginary. i.e.:
for real_imag in [real, imag]:
if read_imag_percentage > percentage_threshold:
save that real or imag failed
no_fail = False
assert no_fail, error_message
Apologies for the errors in my code suggestions. The following should address the failing you listed above:
# declare thresholds
pixel_diff_threshold = 1e-5
percent_fail_threshold = 0.001
# container to capture all possible dtype failures
# tests passes if empty i.e. no failures were captured
dtype_fails = []
# iterate to compute and check %age diff in real and imaginary
for dtype_op, dtype_str in zip([np.real, np.imag],
['real', 'imag']):
# compute diff per pixel
diff = (dtype_op(ma_slc_ref) - dtype_op(ma_slc_sec)) / dype_op(ma_slc_sec)
# count number that exceed threshold
n_diff_gt_threshold = np.count_non_zero(diff > pixel_diff_threshold)
# compute % that fail
percent_fail = n_diff_gt_threshold / tot_pixels_ref
# Check that percentage of pixels above percent_fail_threshold
if percent_fail > percent_fail_threshold
dtype_fails.append(dtype_str)
# check no fails occurred by check of
err_str = f'Percentage of pixels in the difference between reference and '
'secondary products parts above the threshold {pixel_diff_threshold} '
'is above {percent_fail_threshold*100}% for: '
assert len(dtype_fails) == 0, err_str + ','.join(dtype_fails)
@LiangJYu would it be fine for you to keep the current implementation? It seems to be clearer albeit more verbose |
use masked arrays assign thresholds to variables report on both real and imag % fails
Fix bugs in suggestions made in PR opera-adt#77
@seongsujeong and @LiangJYu this PR should be good to go. It would be great if you can have another look. Merci |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Nice work!
* Throw error when percentage of pixels is above threshold * Address comments on threshold and relative difference: * Specify threshold in the print message. Suppress numpy warnings * Use masked array; modify failed pixel percentage computation * Use masked array to compute NaNs * Correct division by reference pixels * More verbose way to compute absolute difference * compare raster per pixel use masked arrays assign thresholds to variables report on both real and imag % fails Co-authored-by: Liang Yu <liangjyu@gmail.com>
* Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * Write `NaN` into HDF5 instead of `None` for metadata (#82) * Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * write NaN instead of None Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> * Apply corrections in geocodeSlc module (#78) * Add Boolean flag to acitivate/disactivate LUT corrections * Feed LUT corrections to geocode SLC * Save LUTs in CSLC product when computed, skip otherwise * Remove trailing space * Add cumulative LUT function * Modify CSLC validation script (#77) * Throw error when percentage of pixels is above threshold * Address comments on threshold and relative difference: * Specify threshold in the print message. Suppress numpy warnings * Use masked array; modify failed pixel percentage computation * Use masked array to compute NaNs * Correct division by reference pixels * More verbose way to compute absolute difference * compare raster per pixel use masked arrays assign thresholds to variables report on both real and imag % fails Co-authored-by: Liang Yu <liangjyu@gmail.com> * make common burst filtering optional in stack processor (#60) * make common burst filtering optional in stack processor * also add gix for bad function name * Update dependencies (#88) * Add pandas dependency * add lmxl * Include default values in runconfig written to HDF5 metadata (#87) * add default options to metadata runconfig str * Update src/compass/utils/runconfig.py Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com> * Unit test for IONEX functionalities (#80) * Unit test for IONEX functionalities * Add ionex unit test data * adopt pytest fixtures * Allocate variables in conftest * rewrite Docker command * Test with pytest in circleci docker - moved copy of source after miniconda install * fix tec URL * ionospheric correction - working version * `ionex_file` added into `dynamic_ancillary_file_group` in runconfig * Interface revision for iono / tropo corrections; test code removed * `ionex_path` added * shortcut to ionex path in cfg * reverting the changes in parameters for calling `az_fm_rate_mismatch_mitigation` * ionospheric correction writeout to CSLC * removed unnecessary lines * fixing codacy issue * PEP8 issue * updating runconfig template for CI * revision on runconfig scheme and defaults * PR comments addressed * docstring revised * revision on the description in corrections * addressing comments in the 2nd round review * PEP8 --------- Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Scott Staniewicz <scott.j.staniewicz@jpl.nasa.gov>
* Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * Write `NaN` into HDF5 instead of `None` for metadata (opera-adt#82) * Mod on Dockerfile for cslc_s1 point release and few improvement * Docker image size optimized * Removing the commands in the header comment * Initial commit after branch-out * minor fix; metadata writeout * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/elevation_antenna_pattern.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Comments on PR addressed; docstring for `apply_eap_correction` fixed * comment clarification * comment clarification * addressing codacy issues * Applying EAP correction (or skip) before `range_split_spectrum` * addressing codacy issue * metadata field name changed * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Update src/compass/s1_geocode_slc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * converting burst ID to `str` * using EPSG from geogrid, not from input DEM * Revert "Merge branch 'eap_correction_into_s1_geocode_slc'" This reverts commit d2d5971, reversing changes made to b5e9341. * adding azimuth FM rate mismatch into lut.py * docstring fixed; variable name changed * docstring for file added; raise error when DEM is not valid * Update src/compass/utils/lut.py Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * updates on local * returning az_lut as `isce3.core.LUT2d()` code based on the suggestion from @LiangJYu * docstring revised * removing redundant renaming of the variables * Dockerfile for beta release * Specfile updated * specifile updated * tag on beta.Dockerfile updated * specifile updated * typo fix * beta.Dockerfile updated * updates on `s1_geocode_slc.py` * but fix * updates on beta.Dockerfile * Dockerfile renamed * untested writing az/rg correction LUT to hdf5 moved EAP to corrections group * restructed HDF5 LUT data not working yet * fix doppler return type clean up debugs * fix paths * fix correction group organization * camel to snake fix correction spacing * restore s1_burst_metadata reorg doppler corrections * PEP8 and syntax fixes * fix correction grouping * specfile updated * Specfile updated * s1-reader version updated * Added `0.2` into `version.Tag` * remove radar grid usage from geo grid orbit item names all snake correction reorg * fix CSLC raster validation * fix typo Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * docstring revised * ref -> reference w/r/t _epoch add spacing datasets add metadata path comparision in validation * fix burst border _ID to _id added more metadata to enable RadarGridProduct reconstruction * Separate bistatic and azimuth FM rate * change function return unpacking * add all parameters needed to reconstruct burst object * wrap value assignment with try+except move helper to helpers up top for visibility * burst DB now static correct h5 write exception type * fix burst.center type * fix writing empty burst polynomials * fix misformed poly1d items * rename geocoding interpolator item * move runconfig * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * fix product name * add az fm rate mismatch into `h5_helpers.py` * reverting s1_geocode_slc.py * reverting `lut.py` * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * Update src/compass/utils/h5_helpers.py Co-authored-by: Seongsu Jeong <sjeong.kr@gmail.com> * spelling error * fix descriptions * Version number changed * add `validate_cslc.py` to `setup.cfg` * Flexibility on Docker's entrypoint to run the validation script * PEP8 * add entrypoint to `validate_cslc.py` * fix on `setup.cfg` * Update src/compass/utils/validate_cslc.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * Change version for s1-reader * Removing default value for `dem_path` * Update src/compass/utils/lut.py Co-authored-by: Liang Yu <liangjyu@gmail.com> * release date updated for `0.1.3` * Change base Docker image into Oracle Linux * typo fix * write NaN instead of None Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> * Apply corrections in geocodeSlc module (opera-adt#78) * Add Boolean flag to acitivate/disactivate LUT corrections * Feed LUT corrections to geocode SLC * Save LUTs in CSLC product when computed, skip otherwise * Remove trailing space * Add cumulative LUT function * Modify CSLC validation script (opera-adt#77) * Throw error when percentage of pixels is above threshold * Address comments on threshold and relative difference: * Specify threshold in the print message. Suppress numpy warnings * Use masked array; modify failed pixel percentage computation * Use masked array to compute NaNs * Correct division by reference pixels * More verbose way to compute absolute difference * compare raster per pixel use masked arrays assign thresholds to variables report on both real and imag % fails Co-authored-by: Liang Yu <liangjyu@gmail.com> * make common burst filtering optional in stack processor (opera-adt#60) * make common burst filtering optional in stack processor * also add gix for bad function name * Update dependencies (opera-adt#88) * Add pandas dependency * add lmxl * Include default values in runconfig written to HDF5 metadata (opera-adt#87) * add default options to metadata runconfig str * Update src/compass/utils/runconfig.py Co-authored-by: Scott Staniewicz <scott.stanie@gmail.com> * Unit test for IONEX functionalities (opera-adt#80) * Unit test for IONEX functionalities * Add ionex unit test data * adopt pytest fixtures * Allocate variables in conftest * rewrite Docker command * Test with pytest in circleci docker - moved copy of source after miniconda install * fix tec URL * ionospheric correction - working version * `ionex_file` added into `dynamic_ancillary_file_group` in runconfig * Interface revision for iono / tropo corrections; test code removed * `ionex_path` added * shortcut to ionex path in cfg * reverting the changes in parameters for calling `az_fm_rate_mismatch_mitigation` * ionospheric correction writeout to CSLC * removed unnecessary lines * fixing codacy issue * PEP8 issue * updating runconfig template for CI * revision on runconfig scheme and defaults * PR comments addressed * docstring revised * revision on the description in corrections * addressing comments in the 2nd round review * PEP8 --------- Co-authored-by: Seongsu Jeong <seongsu.jeong@jpl.nasa.gov> Co-authored-by: vbrancat <virginia.brancato@jpl.nasa.gov> Co-authored-by: Liang Yu <liangjyu@gmail.com> Co-authored-by: Heresh Fattahi <hersh.fattahi@gmail.com> Co-authored-by: Zhang Yunjun <yunjunzgeo@gmail.com> Co-authored-by: Scott Staniewicz <scott.j.staniewicz@jpl.nasa.gov>
The CSLC validation script performs a pixel-wise comparison of the difference between the real and imaginary parts of a reference and secondary products and fails when even one pixel is above a fixed threshold (1.0e-5).
This PR keeps the pixel-wise comparison but makes the validation script fail if the percentage of pixels for which the difference between the real and imaginary parts is above 0.1 %.