Extraction aperture centering for NIRSpec FS data needs a fix

[stscijgbot-jp]

Issue JP-3244 was created on JIRA by James Muzerolle:

The extract_1d step currently uses an automated centering routine to place the extraction aperture, based on the source coordinates. For reasons that aren't completely clear but likely related to the accuracy of the pointing and/or target coordinate keywords, the centering rarely if ever works correctly for FS data. If these keywords can't be fixed, then a new algorithm is needed.

Because FS observations are always done using target acquisition, the position of the source in the slit should always be known to very high accuracy (~5-10 mas given the typical TA uncertainties). So, it would make sense for the extraction aperture to be centered at the known dither position, independent of the actual source sky coordinates.

There are several potential options for implementing this in stage 2:

The pipeline could base the trace location on the dither keywords XOFFSET & YOFFSET, by shifting from the center of the s2d spectrum by that amount. This assumes that the center can be determined consistently.

Use dither-specific reference files to specify the extraction aperture locations. This would require separate files for each dither point in each dither pattern, along with new CRDS selection criteria, which would be cumbersome.

For stage 3, if the current algorithm can't be made to work, the only other option would be to use reference files. However, this is complicated by the fact that the spatial extent of the combined 2D spectrum depends on the number of dithers, so as with stage 2, multiple reference files would be required. 

[stscijgbot-jp]

Comment by David Law on JIRA:

James Muzerolle Keep in mind that https://jira.stsci.edu/browse/JP-3011 implemented a fix for IFU mode observations, which is now available in the 1.11.0 pipeline.  In brief, set the optional flag ifu_autocen=True in calls to Extract1d and it will auto center up on your source.  This is done by median collapsing the cube across wavelengths, and running a standard photometric source finder to identify the brightest object in the scene.  I mostly verified it for MRS, but did test that it worked in a couple examples for NIRSpec IFU as well.  If you want to use it all you should have to do is change the NIRSpec parameter reference files to turn on that flag.

I know the MIRI LRS folks are thinking about how to do something similar for slit mode observations.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Ah, thanks David Law ! That had slipped by me. I'll take "IFU" off of this ticket, then.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Added NIRSpec reps Cheryl Pavlovsky and Stephan Birkmann to the watchlist – we received an update from Howard Bushouse who described some progress on this by Tyler Pauly

Could we ask Tyler Pauly to describe here in this JP ticket the algorithm(s) he's implementing, and perhaps some of the results he's finding, if he'd like the other spectroscopy folks on this ticket (incl Greg Sloan) to take a look at it or give input?

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Hi Anton,

I've implemented a simple trace-finding algorithm in the PR linked to this ticket. It relies on the rectification of the trace in prior processing, such that the dispersion direction described in the header is limited to one axis of the detector array.

The algorithm sums the 2D spectrum in the dispersion direction, creating a spatial profile of the source. The profile is fit with a function (at the moment, a gaussian), and returns the location of the fitted peak and the width. These values are used to set the extraction limits in the cross-dispersion direction. The width may be considered "fixed" for point-like sources through a given optical path and so could be provided via parameter reference file if this gets to that stage, but for now is a fitted parameter. There is currently no consideration for a source with "spatio-spectral" dependence, i.e. position of the trace moves with wavelength such that the fitted peak would depend on wavelength. It also does not attempt to extract a "tapered column" to allow for the growing PSF size with wavelength - it just modifies the size and location of the extraction "box".

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

hi Tyler Pauly thanks for the update! (and I'll link the PR here explicitly, for those who might miss it up in the top right of this ticket:  https://github.com/spacetelescope/jwst/pull/7763])

The algorithm sounds like a good initial approach; a couple of caveats I can imagine (in addition to your "spatio-spectral" possible structural change of the source with wavelength, and the wavelength-dependent PSF change) are possible cases where there might be another object in the slit, close to the target of interest, which could perhaps confuse the functional fit, and also the dependence on how accurately the spectral trace has been rectified, which would depend on the current state of the reference files.

It would be interesting to have additional input/ discussion from the other spectral folks on this ticket ( Cheryl Pavlovsky  Stephan Birkmann  Greg Sloan David Law  James Muzerolle), also to see some results posted here from running this on test datasets to evaluate its performance in a few different scenarios, and thanks again,

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

As James points out in the original ticket, the position of the source in the slit is already known with high accuracy, even if the absolute pointing location is inaccurate.

In the slit data model, the source position is stored as {}source_ypos{}.  This is in the slit frame, but the associated WCS can transform it to the detector frame.  For example, at a central wavelength {}middle_wl{}, for a slit model {}slit{},

slit2det = slit.meta.wcs.get_transform('slit_frame', 'detector')
loc_x, loc_y = slit2det(slit.source_xpos, slit.source_ypos, middle_wl)

The aperture should be placed at {}loc_y{}.

In the pipeline (jwst/extract_1d/extract.py), this method can be used in place of the locn_from_wcs method that uses target RA/Dec to find the aperture location.  In testing, the recorded ystart/ystop from this method looks appropriate to the data, and the x1d products are much improved.  I checked with a nodded fixed slit point source, spec2 and spec3, and a MOS point source, spec2 and spec3.  The aperture location looked perfect in all cases.

Here's the test data I used:

• Fixed slit:
  ** spec3: jw01125-o001_s00001_nirspec_f290lp-g395h-s200a1-subs200a1_s2d.fits
  ** spec2: jw01125001001_0310g_0000[1-9]_nrs[1-2]_s2d.fits
• MOS:
  ** spec3:
  *** jw01345-o061_s07662_nirspec_f290lp-g395m_s2d.fits
  *** jw01345-o061_s02919_nirspec_f290lp-g395m_s2d.fits
  ** spec2: jw01345061001_07101_00003_nrs1_s2d.fits
  *** extension 29, slit 37, source alias 7662
  *** extension 183, slit 59, source alias 2919

Auto centering from spectral tracing is also likely to be useful in some cases, but the assumed source location may be a better choice for a default algorithm.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Melanie Clarke for the win! I didn't know that transform was available in the s2d data model.

How does this work for level 3, though? The individual FS dithers can have different source_ypos.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

The first dataset I tried returned nans for the transform ('jw01514003001_03102_00002_nrs1_cal.fits') - I'll look into why. I'm also curious how the level 3 reductions worked so well. I'll get the code into a separate PR.

Edit: need to use the s2d, clearly...

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I haven't tracked it down in the code, but it looks like the pipeline propagates the source position appropriately for the level 3 products.  They looked right to me for my test cases, anyway.

I'm writing up a quick notebook now and will attach it here for demo and testing purposes.

Edit: notebook attached

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Regarding the level 3 reductions and why they worked...

The output WCS for both level 2 and level 3 s2d products is propagated in the build_nirspec_output_wcs function in {}jwst/resample/resample_spec.py{}.  This includes propagation for the slit frame transform, as a "virtual" slit, extended to include all the input data.  So, the original slit frame for each slit extends from -0.5 to 0.5; the resampled slit frame may have significantly different extent.  Sources from all the input data are located at the same place in the virtual slit, even though they were at different locations in the input slits.  The source position ({}source_xpos/ypos{}) are taken from the reference data model (usually the first input file), also used as the base for the output WCS.  So, to the extent that the resampling works to combine all targets at the same location in the output image, the source location in the new slit frame should also be properly identifying that location.

The output WCS for MIRI is propagated differently, so it looks like the solution for NIRSpec will not extend to their case.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

discussed at JWST CalWG Meeting 2023-09-05

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #7796

[stscijgbot-jp]

Comment by Glenn Wahlgren on JIRA:

I have tested the Extract1D fix for the FS mode sets with a point source (PID 1125) in stages 2 and 3. The attached images document that the fix is working as intended in identifying the location for the spectrum. The extraction aperture used in pipeline version 1.12.0 matches the re-centering workaround notebook extraction aperture used in version 1.11.4. This issue is resolved. 

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

One more note from testing - the fix is working as intended for standard FS modes, but we've found that it doesn't work for BOTS data, since it is not resampled before extraction.  We're discussing recommendations for this mode and will follow up with a separate ticket.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

This ticket was closed without any testing from MIRI LRS. For completeness, I tested these changes with MIRI LRS slit data and they give a nice improvement to the extracted products at L2b. A nice additional benefit is that the extraction now works on cal as well as s2d files. The L3 extraction is unaffected by the change. 

cc Greg Sloan Katherine Murray