EHN: Tdiff Correction for Elevation Data

[carleyjmartin]

Scope

This pull request includes a new utility to return 'amended elevation' values for different tdiff values.
You can return the entire dmap data structure to use with plotting, or just the new elevation values by using the overwrite keyword. In amending the data structure with new values, rather than the plot, you can use the amended elevation in any plot type, not just RTPs.

issue: no issue

Approval

Number of approvals: 2, including a code review please

Test

matplotlib version: 3.6.2
Note testers: please indicate what version of matplotlib you are using

import pydarn
import matplotlib.pyplot as plt

# Read in original data
file = "/Users/carley/Documents/data/20160120.1602.00.cly.fitacf"
SDarn_read = pydarn.SuperDARNRead(file)
fitacf_data = SDarn_read.read_fitacf()

# Summary plot with original data
a = pydarn.RTP.plot_summary(fitacf_data, beam_num=15, groundscatter=True,
                            range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()
# Use new recalc function to get new elevation in dmap data dictionary
new_data = pydarn.recalculate_elevation(fitacf_data, tdiff=0.013, overwrite=True)

# Summary plot with new data
a = pydarn.RTP.plot_summary(new_data,beam_num=15, groundscatter=True,
                            range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()

# Check the values of the first elements in the dictionaries
print(fitacf_data[0]['elv'])
print(new_data[0]['elv'])

# Example of difference in a fan plot format
# Original Data:
pydarn.Fan.plot_fan(fitacf_data, parameter='elv', scan_index=60, groundscatter=True)
plt.show()
# New data:
pydarn.Fan.plot_fan(new_data, parameter='elv',scan_index=60, groundscatter=True)
plt.show()

Calling the function with overwrite=False (or left off) will return the elevation data only in a dictionary, this output will not work if you put it straight into a plot:

new_elevation = pydarn.recalculate_elevation(fitacf_data, tdiff=0.013, overwrite=False)

Please test for accuracy, when looking at the same plots made from the DaViT library, it appears to be similar, but I don't know for sure

*Reminder, please check your code is:

• copyrighted (if applicable)
• Appropriate disclaimer added
• Modification line filled in (if applicable)*

[egthomas]

@carleyjmartin thank you for adding this - it will be a very useful capability for pydarn.

It appears the IDL Davit code was based on the original elevation angle algorithm, which could only handle interferometer offsets in a single direction. Since then a more generalized algorithm has been published which can support interferometer offets in all 3 dimensions (Shepherd [2017]), eg

https://github.com/SuperDARN/rst/blob/main/codebase/superdarn/src.lib/tk/elevation.1.0/src/elevation_v2.c

[bharatreddy]

@carleyjmartin I updated the code so that we now use the more generalized version developed by Simon. Seems to work on my machine. Can you please test it out?

@egthomas Thanks for pointing this out Evan. Can you please help us test this using CV data? I tried to use it for BKS and it seems to be working on a qualitative sense. But it would be great if this can be tested out on a radar where the tdiff parameter is tuned.

[carleyjmartin]

Code looks great, just ran flake8 on it to tidy up the styling. I've tested with my website back end and it works great, the plot outputs when tuning the tdiff look reasonable, but it would be nice to have a scientist to have a test to make sure the results are correct.

EDIT: Also merged with current develop below to pull any updates we did on the release for the rest of the branch.

[bharatreddy]

Thanks @carleyjmartin. I feel a good way to validate the output is by reproducing some of the results presented in previous studies. I can work on it.

[bharatreddy]

I tried to reproduce Figure 6 from the Ponomarenko et al., 2015 paper as a method of validation (see plot). I think there is agreements in a qualitative sense.

[Shirling-VT]

I have reviewed the codes and compared it with the RST C code and another Python version Shibaji wrote. They look good to me. I have an issue accessing the fitacf data that Bharat tested from my lab computer and thus cannot provide any help on testing it now.

[shibaji7]

@bharatreddy Code looks good to me. One quick comment, the 'recalculate_elevation' method takes data(list of dict) and tdiff as inputs. But I recall while working with Pasha, some of SAS radar observation were from back lobe and he requested to input new 'interferometer_offset' position (other than values taken from hwd.dat files) to reevaluate the angles. If you think (might not be in this release, maybe next) can you please include 'interferometer_offset' as an input (tuple) just like tdiff so that it can be used for other back lobe returns [just a suggestion, feel free to ignore].

[carleyjmartin]

A keyword interferometer_offset has been added so that you can experiment with your own inteferometer offset. If you don't set one then it will be read from the hdw file.
It can be used as below:

new_data = pydarn.recalculate_elevation(fitacf_data, tdiff=0.013, overwrite=True, interferometer_offset=[0.0,20.0,1.0])

:)

[hiyadutuje]

I am using the "20221220.2200.00.rkn.a.fitacf" data.