specviz does not support x1dints files

[Jdaviz-Triage-Bot]

Reporter: Kyle Conroy

We should either implement support or at least in the meantime detect that we do not support the file format and provide a useful error message.  See #1486

---

DISCLAIMER: This issue was autocreated by the Jdaviz Issue Creation Bot on behalf of the reporter. If any information is incorrect, please contact Duy Nguyen

[pllim]

I encountered that last night and I wasn't sure if it was the software or data. It only has ASDF extension and no FITS extension, which is weird. And the "JWST x1d" loader cannot recognize it over at specutils.

When I inspect the ASDF tree, it is unclear where the flux is stored.

[pllim]

Why is MAST distributing x1dint file anyway? Where is the final x1d?

[duytnguyendtn]

According to @camipacifici, the x1dints are level 2 products, and only Mosviz is supposed to be able to view level 2 products. Which tool was the user trying to observe in?

[havok2063]

We do not have yet have full x1dints support in MAST. As in those products in MAST do not show up as load-able in Jdaviz. But I believe Specviz can load them as a SpectrumList. I think those files identify as "JWST x1d multi" format. I have successfully loaded these files.

[havok2063]

Also the file the user was loading in #1486, jw02734-o002_t002_niriss_clear-gr700xd-substrip256_x1dints.fits is a stage 3 product filename. See https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/file_naming.html#stage-3-file-names. And they are in the list of stage 3 data products. See https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/product_types.html#stage-3-data-products.

[pllim]

@havok2063 , can you please tell me in plain English what is the way forward here for Jdaviz? I don't use the pipeline and the jwst doc is hard to read.

[havok2063]

There are x1dints files that a user can download that should be supported by Jdaviz. These minimally behave as specutils.SpectrumList objects, identified as "JWST x1d multi" format, and can be loaded into Specviz as a list of spectra. I don't have my computer right now due to an OS upgrade to verify this and provide example code, but I can provide that when I get it back. This doesn't necessarily mean that Specviz is the ideal or preferred viztool for loading x1dints files. That may be Mosviz, but I'm not sure about that.

[havok2063]

Here is some example code that loads a level 3 x1dints file into Specviz. It loads spectra from all the EXTRACT1D extensions found in the file and I believe creates a combined spectrum. I think this got de facto supported after Ivo worked on a similar ticket to get multi-extension x1d files supported.

file = 'jw00721-o012_t004_nircam_f444w-grismr-subgrism256_x1dints.fits'
s = SpecViz()
s.app
s.load_spectrum(file)

and

import specutils
specutils.io.identify_spectrum_format(file, specutils.SpectrumList)
"JWST x1d multi"

[pllim]

So... you are saying it is supposed to work out of the box already, then why is the ERO file crashing?

[havok2063]

Looking at the original ticket, they are trying to load the spectrum with viz.load_spectrum(jwst_wasp, data_label='JWST Spectrum', format='JWST x1d'). That may be the problem. These spectra do not identify as "JWST x1d". Either the user needs to remove the incorrect format keyword, or specify format='JWST x1d multi'. But without seeing the full traceback of the error message, it's hard to say if this is the issue. It could also be an issue with the latest pipeline data not being correctly handled by Jdaviz anymore.

[pllim]

Thanks, @havok2063 . I think that clarified things enough that I was able to track it to bad header in the data released. See #1486 (comment)

Nothing much we can do here.

[havok2063]

Sounds good! If it's a pipeline output error, then perhaps the relevant people on jdat-pipeline should be notified so a ticket can get filed for a bug fix.