JWST custom reduction pipelines

Reference | Setup | MIRI Pipeline | NIRSpec Pipeline | Support | Examples

Custom JWST data reduction and visualisation code used to call and extend the standard JWST reduction pipeline to process, reduce and analyse NIRSpec and MIRI data for solar system observations.

Reference

King et al., (2023). Custom JWST NIRSpec/IFU and MIRI/MRS Data Reduction Pipelines for Solar System Targets. Research Notes of the AAS, 7(10), 223, https://doi.org/10.3847/2515-5172/ad045f

BibTeX entry

@article{king_2023_pipelines,
  doi       = {10.3847/2515-5172/ad045f},
  url       = {https://dx.doi.org/10.3847/2515-5172/ad045f},
  year      = {2023},
  month     = {oct},
  publisher = {The American Astronomical Society},
  volume    = {7},
  number    = {10},
  pages     = {223},
  author    = {Oliver R. T. King and Leigh N. Fletcher and Jake Harkett and Michael T. Roman and Henrik Melin},
  title     = {Custom JWST NIRSpec/IFU and MIRI/MRS Data Reduction Pipelines for Solar System Targets},
  journal   = {Research Notes of the AAS},
  abstract  = {We present custom JWST data reduction pipelines for JWST NIRSpec/IFU and MIRI/MRS observations of solar system objects. The pipelines simplify the process of reducing the JWST observations, and include custom steps to significantly improve the data quality. Our custom processing routines include a “desaturation” routine to reduce the effect of saturation while still maintaining high signal-to-noise ratio, and custom flat field correction code to remove the significant artifacts found in MIRI/MRS observations. The pipelines also automatically generate a series of quick look plots and animations to simplify exploration of a dataset. These custom JWST pipelines can be downloaded from https://github.com/JWSTGiantPlanets/pipelines.}
}

Each release is also archived on Zenodo at https://doi.org/10.5281/zenodo.7891559.

Setup

Requirements:

• Python 3.10 or above
• ~10GB of disk space (for SPICE kernels and the CRDS cache files downloaded when running the JWST data reduction)

To run the pipelines yourself, first create a local clone of this repository with:

git clone https://github.com/JWSTGiantPlanets/pipelines.git

Then install the required Python modules with:

cd pipelines
pip install -r requirements.txt

If it is not already installed, you will also need to install FFmpeg to enable the creation of animations from the data cubes. If you are using a conda environment, you can install FFmpeg with:

conda install -c conda-forge ffmpeg

The standard JWST reduction pipeline requires the CRDS_PATH and CRDS_SERVER_URL environment variables to be set to instruct the pipeline where to find and save the various reference files used to reduce the JWST data. These can be set in your .bashrc file with:

export CRDS_PATH="path/to/crds_cache"
export CRDS_SERVER_URL="https://jwst-crds.stsci.edu"

The navigation and visualisation pipeline steps use SPICE kernels to calculate the observation geometry. The location that the pipeline looks for the kernels can be set with the PLANETMAPPER_KERNEL_PATH environment variable. These can be set in your .bashrc file with:

export PLANETMAPPER_KERNEL_PATH="path/to/spice_kernels"

If you don't already have the SPICE kernels downloaded, you will need to download the appropriate set of kernels for your targets and for JWST from the NAIF archive. For information on downloading and saving the SPICE kernels, see the SPICE kernel documentation for the PlanetMapper module. For example, to download the required kernels for a JWST observation of jupiter, you can run the following in Python:

from planetmapper.kernel_downloader import download_urls
# Ensure you have set the PLANETMAPPER_KERNEL_PATH environment variable before running 
# these commands (otherwise the kernels will be downloaded to the wrong location)

# Standard kernels
download_urls('https://naif.jpl.nasa.gov/pub/naif/generic_kernels/lsk/')
download_urls('https://naif.jpl.nasa.gov/pub/naif/generic_kernels/pck/')

# JWST kernels
download_urls('https://naif.jpl.nasa.gov/pub/naif/JWST/kernels/spk/')

# Locations of planetary system barycentres:
download_urls('https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/planets/de430.bsp')

# Locations of Jupiter (& its major satellites)
# THE URL WILL BE DIFFERENT FOR OTHER TARGETS...
download_urls('https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/satellites/jup365.bsp')

# Note, the exact URLs in this example may not work if new kernel versions are published

MIRI MRS pipeline

The miri_pipeline.py script is used to reduce and process MIRI MRS observations of solar system targets. The full custom pipeline includes:

1. The standard JWST reduction pipeline is used to reduce stage0 uncalibrated data into stage3 calibrated data cubes. Reduced cubes are produced for each individual dither, and with dithers combined. Background subtraction and residual fringe correction can also be applied to the data in this step.
2. Each reduced cube is navigated, and backplanes are created to provide useful coordinates (e.g. latitude, longitude, emission angle, RA, Dec etc.) for each pixel.
3. Darkening caused by the telescope's PSF is approximately corrected by convolving a forward model of the observation with a Gaussian PSF.
4. Saturated parts of the cubes are identified and desaturated where possible using data reduced using fewer groups.
5. Flat field effects (e.g. striping and swirling patterns) in the data can be corrected using synthetic flat fields generated from dithered observations. To use these, see the flat field section below.
6. Extreme outlier pixels are identified and flagged and removed from the data. Note that for small objects or point source observations, this despike step may be overaggressive and remove real data.
7. Quick look plots and animations are generated for each cube.

This pipeline is described in Fletcher et al. (2023).

Running the MIRI pipeline

First download the stage0 data from the MAST archive:

1. Search for the observations you are interested in, then add them to the download basket
2. Open the download basket
3. Uncheck the Minimum Recommend Products box on the left
4. Check the SCIENCE box in the 'Product Category' section and the UNCAL box in the 'Group' section
5. Select the files you want to download in 'Files' section in the middle of the screen
6. Download the data to a local directory e.g. /path/to/your/miri/data/stage0

If you want to apply the flat field corrections to your data, you will need to download (or construct) a set of synthetic flat fields. See the flat field section below for more details about getting the flats. If you don't want to flat field correct your data, you can run the pipeline with the --skip_steps flat argument.

Once you have downloaded the stage0 data and downloaded any required flat fields, you can run the full pipeline with:

python3 miri_pipeline.py /path/to/your/miri/data

or if you are working from a different directory, you can use:

python3 /path/to/pipelines/miri_pipeline.py /path/to/your/miri/data

The pipeline can be customised by calling with various arguments, for example:

python3 miri_pipeline.py /path/to/your/miri/data --parallel --desaturate --skip-steps despike --cube-build-weighting emsm

For more detailed documentation, including the various customisation options, see the instructions at the top of the miri_pipeline.py file or run python3 miri_pipeline.py --help.

MIRI flat fields

Synthetic flat fields derived from the observations of Saturn in November 2022 can be downloaded from the supplementary material for Fletcher et al. (2023). Flats in the pipelines/flat_field directory will be automatically used by the pipeline if they are present, or you can manually specify the path of the flat fields with the flat_data_path argument when running the MIRI pipeline.

The Saturn observations have poor SNR at some wavelengths (particularly in channel 1B) which reduces the quality of the flats at some wavelengths. This means that data corrected with the Saturn flats will have increased noise at the wavelengths where there was poor SNR in the original Saturn observations.

Our testing suggests that the flat field appears to be slightly different for different observations, potentially due to non-linear variations in the flat with brightness or time variability of the flat. If the Saturn flats described above do not adequately remove striping from your data, you may wish to generate synthetic flats from your dataset instead.

You can create flat fields yourself using the construct_flat_field.py script to create a set fo flat fields directly from your dataset. The script takes a set of stage3 or stage3_desaturated dithered observations and uses these to construct a flat field which minimises variation in brightness of each location on the target between dithers. This was designed to work for extended source observations which fill the MIRI field of view and have at least 4 dithers. Observations of objects which do not fill the FOV or have fewer dithers are unlikely to produce reliable flat fields. See generate_saturn_flats.py for an example of how to use construct_flat_field.py to generate flat fields from a set of observations.

NIRSpec pipeline

The nirspec_pipeline.py script is used to reduce and process NIRSpec IFU observations of solar system targets. The full custom pipeline includes:

1. The standard JWST reduction pipeline is used to reduce stage0 uncalibrated data into stage3 calibrated data cubes. Reduced cubes are produced for each individual dither, and with dithers combined. Background subtraction can also be applied to the data in this step.
2. Each reduced cube is navigated, and backplanes are created to provide useful coordinates (e.g. latitude, longitude, emission angle, RA, Dec etc.) for each pixel.
3. Saturated parts of the cubes are identified and desaturated where possible using data reduced using fewer groups.
4. Extreme outlier pixels are identified and flagged and removed from the data. Note that for small objects or point source observations, this despike step may be overaggressive and remove real data.
5. Quick look plots and animations are generated for each cube.

Running the NIRSpec pipeline

Running the NIRSpec pipeline is virtually identical to running the MIRI pipeline as described above. Simply download the stage0 data from the MAST archive as described above, then you can run the full pipeline with:

python3 nirspec_pipeline.py /path/to/your/nirspec/data

For more detailed documentation, including the various customisation options, see the instructions at the top of the nirspec_pipeline.py file or run python3 nirspec_pipeline.py --help.

Point source targets

These pipelines were originally designed to reduce extended-source solar system targets, but are also able to reduce data for point-source targets. It is worth noting, however, that the steps run after the standard pipeline (navigate, defringe_1d, despike, psf etc.) all work with spectral cubes (*_s3d.fits), so may not be as useful if you are using extracted 1D spectra (*_x1d.fits) instead.

Support

If you have any issues running the code in this repository, please open an issue or contact ortk2@leicester.ac.uk.

Common issues

SPICE kernel errors

If you are getting errors from SpiceyPy, it is likely that your SPICE kernels are not being found correctly. See the PlanetMapper SPICE kernel documentation for more details and instructions on how to fix this.

FFmpeg errors

If you are getting errors such as FileNotFoundError: [Errno 2] No such file or directory: 'ffmpeg' when generating animations, it is likely that FFmpeg (the software used by Matplotlib to create animations) is not installed on your system. If you are using a conda environment, you can install FFmpeg with:

conda install -c conda-forge ffmpeg

Examples

Figure from King et al. (2023) showing example cube slices of JWST/MIRI observations of Saturn (Fletcher et al. 2023) at different stages of our custom data reduction process. The first column shows the output of the standard JWST pipeline, which contains saturation (missing data around Saturn's North Pole in panel i), partial saturation (dark pixels on Saturn in i), significant flat fields effects (a and g), and limb darkening caused by the instrument PSF (m). The second column shows the data after the application of the desaturation step, the third column shows the data after flat field correction, and the fourth column shows the "final" version of the data after PSF correction. The application of the custom data reduction steps significantly improves the data quality, such as revealing banded structure of Saturn's atmosphere (panels d and h) that is impossible to see in the standard pipeline output.

Example quick look plot generated by jwst_summary_plots.py showing a Saturn cube from Fletcher et al. (2023). Plots like this are generated for every cube when the pipeline is run with the default settings.

quick_look_animation_example.mp4

Example quick look animation generated by jwst_summary_animation.py showing a set of dithered Saturn cubes from Fletcher et al. (2023). Animations like this are automatically generated for every set of dithered cubes when the pipeline is run with the default settings.