Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude source-specific WFSS files from observation page #1651

Merged

Conversation

bhilbert4
Copy link
Collaborator

WFSS source specific files have started showing up on JWQL's observation level pages. I'm not sure how this happened. The filenames come in from the MAST query. Maybe the metadata examined by MAST has changed? This was leading to a lot of extra thumbnail placeholders being added to WFSS programs on the observation level pages. I don't think we want to keep track of these files, even though they are an output of the spec2 pipeline, and JWQL generally supports stage 2 outputs. In WFSS there is one file for each extracted source. By default the pipeline now keeps only the top 100 brightest sources per exposure. I think adding 100*n_exposures files to the page could quickly become overwhelming. Maybe if we come up with a better way to organize these, we can add them back in. They are 2D images of extracted spectra.

This is not tested yet.

@bhilbert4 bhilbert4 self-assigned this Oct 16, 2024
@bhilbert4
Copy link
Collaborator Author

I think the (now commented out) lines that created a dictionary when the filename_parser doesn't recognize the filename doesn't make any sense to keep. It assumes a filename structure, except that structure is recognized by the filename_parser. So if the parser fails, this assumed dictionary probably won't contain correct information either.

@bhilbert4
Copy link
Collaborator Author

@mfixstsci I think this is ready for review. The update at line 2200 solves the problem with the WFSS files that were getting through. Local testing looks good. There are a few MIRI proposals (e.g. 1219) where the observation level page is failing, but that looks like it's due to missing filter names in my local rootfileinfo db. We should certainly confirm this once on the test server.

With the fix at line 2200, we technically could leave the lines in place that create the dummy dictionary in cases where the filename parser fails. But this strikes me as asking for more trouble down the line. If the filename parser fails, there won't be rootfileinfo entries for the files, leading to more problems. Do you agree?

@bhilbert4 bhilbert4 changed the title [WIP]: Exclude source-specific WFSS files from observation page Exclude source-specific WFSS files from observation page Oct 17, 2024
Copy link
Collaborator

@mfixstsci mfixstsci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me @bhilbert4 I am going to do some testing and will let you know what I think.

jwql/website/apps/jwql/data_containers.py Show resolved Hide resolved
@mfixstsci mfixstsci merged commit 2d655d8 into spacetelescope:develop Oct 22, 2024
11 checks passed
@bhilbert4 bhilbert4 deleted the filter-wfss-source-specific-files branch October 29, 2024 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants