Implement option to use likelihood-based jump and ramp fitting methods in stcal

[stscijgbot-jp]

Issue JP-3576 was created on JIRA by David Law:

On 3/13/24 there was a meeting to discuss the likelihood-based approach to jump and ramp fitting described by Timothy Brandt in a pair of recent papers.  Abbreviated notes from this meeting are in the comments below.

This ticket is to implement this method, coded currently at https://github.com/t-brandt/fitramp, in the stcal environment with hooks so that it can be called as an option from the jwst code.  Once this is done there will need to be an extensive suite of tests performed to flush out edge cases and compare performance across the different available methods.

[stscijgbot-jp]

Comment by David Law on JIRA:

Extremely broad meeting summary from 3/13/24

Attendees: David Law Michael Regan Timothy Brandt Tyler Pauly Howard Bushouse Kenneth MacDonald Jane Morrison Karl Gordon Nadia Dencheva Eddie Schlafly 

• Ken is largely done with a C extension implementation of the existing approach that is nearly ready to merge.  This shows typical speed gains of a factor ~20x, ranging from 2x to 130x in the cases tested.   This also resolves some outstanding bugs in the python implementation.  This will be extremely useful for our testing.
• Tim summarizes the likelihood-based approach.  Current pipeline uses an approximation of the Fixsen method, which is itself an approximation to using the covariance matrix of samples.  Inverting the covariance matrix is expensive, but doing it with sequential differences makes the matrix tridiagonal so that it can be inverted easily.
• This happens in order-N time, which is fast even in pure python, and can check every possible jump location as well with order-N runtime.  This also provides a chi-2 based evaluation of the fit quality for each ramp, which can evaluate how accurate the input read noise values were (although additional noise sources could also affect this).
• Based on testing on different depths of NIRCam data, this approach may be more reliable about finding jumps that the existing algorithm.  In the poisson-dominated regime it should perform roughly comparable to existing routine, but should do substantially better in the read-noise dominated regime.
• Code is also very compact (600 lines or so) and runs roughly 10x faster than the existing python implementation, and may be possible to parallelize in a similar manner.
• The new code does not account for showers/snowball flagging, so those would need to be added separately.
• Read noise for the NIR detectors has temporal covariance, likely due to 1/f noise, which may have implications for the efforts to model and remove 1/f noise also currently ongoing.
• We would like to see this implemented as an option for JWST (which Roman would be particularly interested in using too).  This group and the INS teams would then need to subject it to extensive testing of edge cases.

[stscijgbot-jp]

Comment by David Law on JIRA:

See https://jira.stsci.edu/browse/JP-3577 for collection of test cases.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

David Law - the first version for this algorithm is close to being ready for testing, but there are still some questions about exactly how to incorporate it in the pipeline (JP-3658).  Time is a little short for Build 11.0, so we're leaning towards deferring the merge until after this build closes to make sure there's plenty of time to make those decisions and test them.  Is that okay for INS or would you like us to make the first version available in this build, with the caveat that the interface may need to change later?

[stscijgbot-jp]

Comment by David Law on JIRA:

Melanie Clarke Just catching up on everything from the last week now; I'd also be inclined to defer merging until after the build closes and handle these details in 11.1 instead.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Okay, sounds good.  Thanks!

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Initial version implemented in spacetelescope/stcal#278

[stscijgbot-jp]

Comment by David Law on JIRA:

Confirmed that the new step is working; further testing will take place in dedicated ticket https://jira.stsci.edu/browse/JP-3765