Make FoV routine

[aburrell]

A preliminary version of the routine that automatically determines the backscatter return direction. This partially addresses #135.

Test results using a fitacf 3.0 file:

make_fov -cn a ~/Programs/Data/SuperDARN/Downloads/19960916.0400.00.han.fitacf3 > han.tst_fov

This creates an ascii file with 13727 lines and the following columns:

#STID DATE TIME BMNUM BMAZM CPID INTT_SC INTT_US NAVE FRANG RSEP RXRISE FREQ NOISE ATTEN CHANNEL NRANG RG GFLG FOVFLG FOV_PAST GRPFLG GRPNUM GRPID P_0 P_0_ERR V V_ERR W_L W_L_ERR P_L P_L_ERR PHI0 PHI0_ERR ELV ELV_LOW ELV_HIGH VH VH_ERR VH_METHOD REGION HOP DIST MED_P_0 MED_P_0_ERR MED_V MED_V_ERR MED_W_L MED_W_L_ERR MED_P_L MED_P_L_ERR MED_PHI0 MED_PHI0_ERR OPP_ELV OPP_ELV_LOW OPP_ELV_HIGH OPP_VH OPP_VH_ERR OPP_VH_METHOD OPP_REGION OPP_HOP OPP_DIST

Re-creating one of the figures from my paper leads to similar results. This scan is from a different time that was created during various spot checks.

Before moving this from draft status we need to:

• merge Find frequency bands #444
• Move general utility functions into appropriate libraries
• Create a fitacf-esq non-ascii output file option
• Finalize command-line options

[pasha-ponomarenko]

Yep, I believe that what you are supposed to do with XMLs.

[aburrell]

So, opening any of the files in a web browser doesn't do anything but display the code. I guess the good news is that my xml behaves the same as the rest of the xml. None of them open.

Given that it seems none of us have the knowledge to resolve this issue, here's what you need to run the code:

Create ascii output: make_fov -ascii
Use your own tdiff: make_fov -update-tdiff -tdiff <value_in_microseconds>
Specify your own frequency band: make_fov -tfmin -tfmax
Specify channel a: make_fov -cn a

Because you need to look at the range and azimuthal variations, you need to separate frequency bands for the analysis. For best results, use the TDIFF suited to the choses radar channel and frequency!

To convert the dmap binary to ascii, use fitmultbsidtoascii

[pasha-ponomarenko]

Yep, the XML business looks like a separate issue.
:-(
Thanks for the clarification. Do you have a description of the extra fields in a more robust format like DOC or PDF? Or simple ASCII?

[ecbland]

To view the documentation in a web browser, I think you first need to run make.doc from the root directory of RST.

cd $RSTPATH
make.doc
cd doc/html
google-chrome index.html &   # or another web browser of your choice

You can then navigate through the html documentation in the web browser.
The documentation for the make_fov binary is here: $RSTPATH/doc/html/superdarn/src.bin/tk/tool/make_fov/index.html

[aburrell]

No, I put all the documentation in xml since that was the RST standard. The most important thing is FOVFLG, which is 1 for the front FoV, -1 for the rear FoV, and 0 if a FoV could not be cleanly identified. GFLG is also updated so that it is -1 if it was formerly groundscatter and doesn't look like groundscatter.

[ecbland]

@aburrell Now I am confused. Here's a screenshot from following the instructions I gave in my previous comment. Is this what @pasha-ponomarenko was looking for?

[aburrell]

Is this what @pasha-ponomarenko was looking for?

Yes, @ecbland that is it!

[pasha-ponomarenko]

@ecbland, thanks a lot, I can see this page after I run make.doc
;-)
I was able to process a FIATCF file last night, and I wanted to figure out what is *.fov data format so that I would be able to visualise it one way or another. From dmapdump output I saw that it is similar to FTITACF but with extra fields.

[pasha-ponomarenko]

@aburrell, I might miss something, but there seem to be fewer fields in a binary output of make_fov than in its ASCII one, e.g., MED_* and GRPFLG, GRPNUM, GRPID are all missing.

[aburrell]

Yes, those fields are for future routines down the backscatter ID stream. Because the binaries are the default output for RST, I wanted them to be streamlined and only have the appropriate information. Since the ASCII output is primarily diagnostic, I added less logic steps to remove currently unused parameters. From: pasha-ponomarenko ***@***.***> Reply-To: SuperDARN/rst ***@***.***> Date: Wednesday, February 9, 2022 at 17:53 To: SuperDARN/rst ***@***.***> Cc: "Burrell, Angeline" ***@***.***>, Mention ***@***.***> Subject: Re: [SuperDARN/rst] Make FoV routine (#449) @aburrell<https://github.com/aburrell>, I might miss something, but there seem to be fewer fields in a binary output of make_fov than in its ASCII one, e.g., MED_* and GRPFLG, GRPNUM, GRPID are all missing. — Reply to this email directly, view it on GitHub<#449 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABVYF7QPA6ZDYTJI7C4M3DDU2LV6DANCNFSM5BWQM2NQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pasha-ponomarenko]

OK, here are some preliminaries.
I used 20150223.2001.00.sas.fitacf (v3, twofrequency mode) which contains both ionospheric and ground scatter. I also selected SAS data because the ground scatter here is normally dominated by signals received through the grating lobe. Here are plots for the 10 MHz band of the twofrequency mode. Importantly, for this interval there was no detectable tdiff offset for this frequency band. The data show both ionospheric and ground scatter, and ground scatter coming from rather high elevation angles which is indicative of the "back" lobe scatter.

I processed this data for frequencies between 10 and 11 MHZ using following command line options:
make_fov -vb -tfmin 10000 -tfmax 11000 20150223.2001.00.sas.fitacf > 20150223.2001.00.10.sas.fov
Then I converted DMAP *.fov file into NetCDF with following steps:

• Analysed DMAP structure and produced supplementary files:
  dmaptocdl 20150223.2001.00.10.sas.fov fov.cdl fov.cdlmap
• Adjusted the field lengths in the "mask" file fov.cdl to 75.
• Generated an empty NetCDF file
  ncgen fov.cdl -o fov_10.netcdf
• Filled the empty file with data from DMAP file:
  dmaptoncdf 20150223.2001.00.10.sas.fov fov.cdlmap fov_10.netcdf

After that used IDL to read and plot NetCDF FOV data:

Velocity

Elevation

FOV

GSCT

SCT

[pasha-ponomarenko]

My understanding is that the majority of the 1.5 ionospheric scatter above rage gate 40 is removed due to one of the selection criteria (exceeding the maximum virtual height?).
The FOV determination seems to be a bit shaky for the majority of the ground scatter before 21:30 UT. I guess it might be related to an apparent partial overlap between the front- and "back"-lobe ground scatter. Indeed, the closer edge of the GS band between gates 18 and 25 shows very high elevation angle values in excess of 35 deg followed by a sharp boundary with elevation angle values dropping to 20-25 deg at gates ~25-32.

@aburrell, please correct me if I am wrong here.
BTW, I am not sure what SCT flag means.

[aburrell]

Ok, I played around with things and the main reason some things aren't working well is the propagation grouping. I applied the HAN defaults (basically the same, but -fhmax 900) and got much more of the ionospheric scatter back. Fine tuning the propagation virtual height ranges would further improve the results. I added a note to the docs to make sure users know how important this is, and know where the defaults come from.

Looking at the azimuthal variations, it makes sense that much of the groundscatter returns from the front FoV, since the variations are consistent across the beams.

[aburrell]

BTW, I am not sure what SCT flag means.

It's the sct parameter from the fitacf structure, I believe it just flags whether the scatter is valid or not.

[pasha-ponomarenko]

It's the sct parameter from the fitacf structure, I believe it just flags whether the scatter is valid or not.

I don think there is such a parameter in the fitacf structure
https://radar-software-toolkit-rst.readthedocs.io/en/latest/references/general/fitacf/,
that is why I asked
:-P
It does not look like a quality flag...

[pasha-ponomarenko]

@aburrell , What language did you make your plots in? Python? IDL?
Is there a way to integrate some plotting into this package (or utilise some existing RST plotting routines) in order to make tester's/user's life a bit easier?
:-{

[aburrell]

I don think there is such a parameter in the fitacf structure

There is! Just grep "sct" in $RSTPATH/include/superdarn/*.h and you'll see which structures it's in. You can also examine the code to see how it's used. It may not be included in the user-description of the output, but it's a code structure parameter.

What language did you make your plots in

Python. I don't know how to plot in C, so I won't be adding this to RST. They're just RTI plots like the ones you made, but further separated by hop or propagation path. The plots you made were fine, they confirmed the code is working as it should be!