Subbeamnod routine cannot access plnum

[jfrothingham]

Unable to retrieve a subbeamnod scan because of apparent problems accessing 'plnum'

I've been following the SubBeamNod example (https://dysh.readthedocs.io/en/latest/examples/subbeamnod.html) with a different dataset.

I am using dysh-0.2.0b with Python3.11 on a computer that I think is running RedHat 8. (Thales data reduction machine)
Data is located at /home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas in the Green Bank filesystem.

Here is a screenshot of the error I'm seeing. I get this error regardless of whether I specify kwargs or not, and regardless of which scan I use.

However, sdfits.summary() shows that this scan does have values for PLNUM and FDNUM.

[jfrothingham]

Yes, I think you can test against it. It's a dataset we were considering using as a training example

[tchamberlin]

I can reproduce this:

sdfits = GBTFITSLoad("/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas")
sbn = sdfits.subbeamnod(scan=18, fdnum=0, plnum=1)

Output:

File <snip>dysh/fits/gbtfitsload.py:815, in GBTFITSLoad.subbeamnod(self, scan, bintable, **kwargs)
    813 df = df[df["IFNUM"].isin([ifnum])]
    814 df = df[df["FDNUM"].isin([fdnum])]
--> 815 df = df[df["PLNUM"].isin([plnum])]
    816 df_on = df[df["CAL"] == "T"]
    817 df_off = df[df["CAL"] == "F"]

UnboundLocalError: cannot access local variable 'plnum' where it is not associated with a value

Indeed, plnum is not defined, since rx is not "Rcvr26_40". But upon fixing that, I get

File <snip>/dysh/spectra/scan.py:228, in TPScan.__init__(self, gbtfits, scan, sigstate, calstate, scanrows, calrows, bintable, calibrate)
    225 # print("BINTABLE = ", bintable)
    226 # @TODO deal with data that crosses bintables
    227 if bintable is None:
--> 228     self._bintable_index = self._sdfits._find_bintable_and_row(self._scanrows[0])[0]
    229 else:
    230     self._bintable_index = bintable

IndexError: index 0 is out of bounds for axis 0 with size 0

[tchamberlin]

After fixing the plnum bug (see 153-subbeamnod-routine-cannot-access-plnum)

It appears that this causes an error:

sdfits = GBTFITSLoad("/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas")
sbn = sdfits.subbeamnod(scan=18, fdnum=0, plnum=1)

But this does not:

sdfits = GBTFITSLoad("/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas/AGBT19B_096_08.raw.vegas.A.fits")
sbn = sdfits.subbeamnod(scan=18, fdnum=0, plnum=1)

The difference being that the former (the path in the original issue) is the path to the SDFITS folder, while the latter is the specific path of a specific bank (as per the example).

@mpound Should both of these paths work? i.e. is giving the SDFITS folder path a supported operation?

[jfrothingham]

I should have attached screenshots of my full code, sorry. I ran dysh from within the folder "/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas" and did specify the path of the bank:

filename='AGBT19B_096_08.raw.vegas.A.fits'
sdfits = GBTFITSLoad(filename)
sbn = sdfits.subbeamnod(scan=18, fdnum=0)

This code produced the error I reported here.

[tchamberlin]

Ah okay, then I think I fixed the bug. You can install the updated dysh code via:

pip install git+https://github.com/GreenBankObservatory/dysh.git@153-subbeamnod-routine-cannot-access-plnum

[jfrothingham]

I'm still encountering the issue, but I think that's due to installation problems on my end rather than anything to do with this bug.

[astrofle]

After fixing the plnum bug (see 153-subbeamnod-routine-cannot-access-plnum)

It appears that this causes an error:

sdfits = GBTFITSLoad("/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas")
sbn = sdfits.subbeamnod(scan=18, fdnum=0, plnum=1)

But this does not:

sdfits = GBTFITSLoad("/home/scratch/jfrothin/testdata/AGBT19B_096_08.raw.vegas/AGBT19B_096_08.raw.vegas.A.fits")
sbn = sdfits.subbeamnod(scan=18, fdnum=0, plnum=1)

The difference being that the former (the path in the original issue) is the path to the SDFITS folder, while the latter is the specific path of a specific bank (as per the example).

@mpound Should both of these paths work? i.e. is giving the SDFITS folder path a supported operation?

@tchamberlin yes. dysh should be able to load multiple fits files in a directory. This feature was introduced as part of 0.2.0. (Thanks for looking into this issue.)

[astrofle]

@jfrothingham I had to uninstall dysh and then reinstall using the command Thomas provided for this to work. Here's what I did (from the Python environment with dysh):

pip uninstall dysh
pip install git+https://github.com/GreenBankObservatory/dysh.git@153-subbeamnod-routine-cannot-access-plnum

Hope this helps.

[tchamberlin]

Ah right! You can do it in one command via the -I/--ignore-installed flag:

pip install -I git+https://github.com/GreenBankObservatory/dysh.git@153-subbeamnod-routine-cannot-access-plnum

[jfrothingham]

With the uninstall/reinstall, I can now successfully process subbeamnod data with dysh. Thanks all!

[mpound]

Late to this party, but I do think there is still a potential bug in subbeamnod having to do with the receiver assumption Rcvr26_40, which I got from the GBTIDL implementation. I will work with Pedro to improve my understanding of which Rxs have or historically had polarization capability.

[astrofle]

@mpound currently, only Argus (the 3 mm 16 feed receiver being used for the GBT EDGE survey) has single polarization capabilities. For observations with every other spectral line receiver the data should have at least two polarizations. The if else clause in dysh.fits.GBTFITSLoad.subbeamnod is there because the Ka receiver (Rcvr26_40) has had its feeds mislabelled for some time now. So, if you want to calibrate the data properly for that particular receiver you have to swap the polarization and feed values. For the rest of the receivers the method should use the same polarization for the selected feeds. I hope that clears a bit the confusion.

Apologies for not explaining that earlier.

PS. There is an open item to fix the mislabelling of the Ka receiver feeds. That is also why there is a commented part of the else clause referring to the date of the observations. For observations taken after the Ka beam mislabelling issue is fixed the polarizations should not be swapped.