EHN: Boxcar Filter

[carleyjmartin]

Scope

This PR extends previous PR #204
Includes a module filters.py which can be extended if we want to in the future.
filters.py includes the code provided by @shibaji7 in the previous PR but I've made a new branch as the rest of the code was still at v2.1 so this was easier than pulling the old branch to be up to date.

For @shibaji7, I had to add some additional code in to deal with ground scatter flags and repopulating the data dictionary to be able to use the outputted data in pyDARN methods afterwards. Please let me know if the changes at the bottom for groundscatter are okay. If you have a copyright line let me know and I can add it in at the top with mine.

Also included here is an addition to the documentation on how to use the boxcar filter.

issue: will close #171

Approval

Number of approvals: 2 (testing and code review where possible please)

Test

matplotlib version: 3.6.3
Note testers: please indicate what version of matplotlib you are using

The boxcar filter can take 5-10 minutes to run on a 2 hours file, please bear that in mind, it is not a bug, it's just how it is.
Reminder to also downgrade numpy to 1.23.0 as newer numpy breaks pydarnio, this is in a patch release waiting now. If you get a hardware file issue see solutions here: #292

import bz2
import datetime as dt
import matplotlib.pyplot as plt
import warnings

import pydarn

with bz2.open('/Users/carley/Documents/data/20150308.1400.03.cly.fitacf.bz2') as fp:
    fitacf_stream = fp.read()
fitacf_data = pydarn.SuperDARNRead(fitacf_stream, True).read_fitacf()

# Before filtering:
pydarn.RTP.plot_summary(fitacf_data, beam_num=7,
                        range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()

pydarn.Fan.plot_fan(fitacf_data, scan_index=10, coastline=True)
plt.show()

# Evoke filter on data
bx = pydarn.Boxcar(
    thresh=0.7,
    w=None
)
filtered_data = bx.run_filter(fitacf_data)

# After filtering
pydarn.RTP.plot_summary(filtered_data, beam_num=7,
                        range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()

pydarn.Fan.plot_fan(filtered_data, scan_index=10, coastline=True)
plt.show()

The code produces the following plots to compare:

Summary unfiltered:

Summary filtered:

Fan unfiltered:

Fan filtered:

[carleyjmartin]

Discussion from meeting: include a warning here for the user with reference to relevant literature, and add to documentation how the data is being modified. Let me know any references and I can add in a warning when using and add a page in the documentation on what is happening when using the filter with some warnings (like don't use on twofsound data like me 👀 )

[KhanalKrishna]

Matplotlib version: 3.5.3
numpy version: 1.23.0

Hi, I am getting the following error. I tried by dropping plot_coastline option as I am having trouble with cartopy.

Script:
import bz2
import datetime as dt
import matplotlib.pyplot as plt
import warnings

import pydarn

with bz2.open('20150331.1801.00.rkn.fitacf.bz2') as fp:
fitacf_stream = fp.read()
fitacf_data = pydarn.SuperDARNRead(fitacf_stream, True).read_fitacf()

Before filtering:

pydarn.RTP.plot_summary(fitacf_data, beam_num=7,
range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()

pydarn.Fan.plot_fan(fitacf_data, scan_index=10)
plt.show()

Evoke filter on data

bx = pydarn.Boxcar(
thresh=0.7,
w=None
)
filtered_data = bx.run_filter(fitacf_data)

After filtering

pydarn.RTP.plot_summary(filtered_data, beam_num=7,
range_estimation=pydarn.RangeEstimation.RANGE_GATE)
plt.show()

pydarn.Fan.plot_fan(filtered_data, scan_index=10)
plt.show()

error message:

[Shirling-VT]

I confirm the same error with the summary plot for the filtered data:

pydarn.RTP.plot_summary(filtered_data, beam_num=7,
range_estimation=pydarn.RangeEstimation.RANGE_GATE)
/Users/xueling/pydarn_test/pydarn/pydarn/plotting/rtp.py: 934: UserWarning: WARNING: matplotlib Default dpi may cause distortion in range gates and time period. The figure size can be adjusted with the option figsize and dpi can be adjusted when saving the file.
Traceback (most recent call last):
File "", line 1, in
File "/Users/xueling/pydarn_test/pydarn/pydarn/plotting/rtp.py", line 1136, in plot_summary
cls.plot_range_time(dmap_data, beam_num=beam_num,
File "/Users/xueling/pydarn_test/pydarn/pydarn/plotting/rtp.py", line 342, in plot_range_time
dmap_record[parameter][j]
IndexError: index 1 is out of bounds for axis 0 with size 1

Although the fan plot for the filtered data seems to work fine for me:

[Shirling-VT]

I also checked individual RTP plot and confirmed that only plotting the elevation angle has the same issue, plotting other parameters works fine.

After filtering

a = pydarn.RTP.plot_summary(filtered_data, beam_num=7,
range_estimation=pydarn.RangeEstimation.RANGE_GATE)

a = pydarn.RTP.plot_range_time(filtered_data, beam_num=7,parameter='elv',
range_estimation=pydarn.RangeEstimation.RANGE_GATE)

a = pydarn.RTP.plot_range_time(filtered_data, beam_num=7,parameter='p_l',
range_estimation=pydarn.RangeEstimation.RANGE_GATE)

a = pydarn.RTP.plot_range_time(filtered_data, beam_num=7,parameter='w_l',
range_estimation=pydarn.RangeEstimation.RANGE_GATE)

plt.show()

a = pydarn.RTP.plot_range_time(filtered_data, beam_num=7,parameter='elv',
range_estimation=pydarn.RangeEstimation.RANGE_GATE)
Traceback (most recent call last):
File "", line 1, in
File "/Users/xueling/pydarn_test/pydarn/pydarn/plotting/rtp.py", line 342, in plot_range_time
dmap_record[parameter][j]
IndexError: index 1 is out of bounds for axis 0 with size 1

[carleyjmartin]

Thanks for testing both!
So, elevation is not included in the run_filter algorithm, so it's not getting modified with new values, so we're seeing an issue with mis-matching data lengths as slist is modified. You can see in my example at the top that it's not smooth, just so happens that the mismatching issue doesn't pop up in the data that I originally tested with, but the elevation data is likely wrong anyway in that example.
@shibaji7 should we be filtering the elevation data too? Or should we just modify the length of the array instead?

EDIT: after looking further at the arrays, it looks like slist isn't just reduced in size but some new gates are filled in when filtering (is this intended?) so it does't make sense to modify the elevation length, as then we're missing data, so for now I have also added elevation to the filtering - testing now will push soon. Let me know what you think!

[shibaji7]

@carleyjmartin Yes, I think elev angle data should be filtered. But I have not gone through the code it. I might change a few lines in this branch.

[shibaji7]

@carleyjmartin The code looks good to me. I approve of the modification under ehn/boxcar-filter.

[carleyjmartin]

Excellent! I think we should just hang on merging this into develop until the Iceland radar changes are put into develop and then made into a patch release, if you're happy for this to be in the next major/minor release.

[hiyadutuje]

Using different data.