Clarify basic documentation

[bhilbert4]

While very basic documentation has been put together (see #44), there were a lot of confusing details. Below is a list of questions I have sent to Armin. The documentation should be clarified with the answers to these.

Reftypes – what is the point of this list? The reflabels list determines what is actually run. So why do we also need a list of reference file types?

Default_reflabels – what are the defaults for? Is there a way to say “Make me a readnoise file.” and have mkrefs proceed using a default module? I can see this being useful down the road when we have officially selected algorithms and people don’t want to explicitly add that module to the reflabels list. Is that the intention?

Dark pattern and flat pattern – These are strings that mkrefs looks for in the filenames, in order to identify them as flats and darks? If so, this won’t work on in-flight jwst data.

Basename pattern – Is this here so that users don’t need to provide a search string in the command line call?

Mjdobs header keyword entry is blank in my mkrefs example. Is this used? Is this a backup if date-obs and time-obs are not provided?

Required fits keys: Is this something the user is going to modify? If not maybe this is something that should go somewhere less user-visible.

Optional fits keys: same question. Also, what are these used for at the moment?

runIDNdigits – This is also something the average user probably won’t want to think about, and can probably be moved somewhere less visible. Also, if this is defined as the number of leading zeros, then when you get to run 10, will mkrefs turn that into 00010? And run 100 will be 000100? Or is it the total number of digits in the run ID, so that run 10 is 010, and run 100 is 100?

cmdID is shown as part of the output reffile basename, but isn’t defined anywhere. I’m not sure what to do about this because we don’t want to clutter the cfg file with lots of definitions and explanations, but it should be defined somewhere. Maybe the entire “reference file output basename” line should be removed and definitions left to the documentation?

Skip_runID_ssbdir. If True then what is skipped? Looking for exisiting SSB output files? Creation of SSB output files?

Refiles_for_ssb – If a filepattern is entered, do you expect the full name of the file? Or is there a glob search involved and the user can enter wildcards? In that case, what do you do if more than one matching file is found?

Validation – This means the output reference file is validated by using it in a pipeline run? What is the difference between setting CRDS or entering a filename?

I think we need to separate the entries for the plugged-in packages a little better. Maybe all we need is a couple of commented out lines ( #--------------), but it would be nice to make it clear that this is where you are entering values for your various packages.

I’ve been calling mkrefs using the command:

mkrefs.py -vvv --cfgfile main_config_file.cfg example_readnoise_module /path/to/dark/files/DARK_uncal.fits /path/to/flatfield/files/NRCN815A-LIN-53650[5678]*uncal.fits

Do I need to include “example_readnoise_module”? Is this just overriding whatever is in the reflabels list in the config file?

It might be helpful to make the file paths into keywords. (e.g. dark_files=/path/to/darks/*uncal.fits flat_files=/path/to/flats/*uncal.fits) That way people won’t have to remember what order to add them in. Or is mkrefs just lumping everything together and figuring out which are darks and which are flats? In that case we need to make that clear in the documentation. And at the moment there is support only for darks and flats right? Nothing for external observations?

[bhilbert4]

A couple answers from Armin:
The list of reflabels in the call to mkrefs.py is what is actually run. The list of reflabels in the config file is simply a list of allowed reflabels. This should now be generated automatically by the code, and reflabels should be removed from the config file.

The files found from the (multuple) provided file patterns are combined into a single list, and jwst_reffiles determines what kind each one is based on header keywords.

External observations are not yet supported.

[arminrest]

On 1/28/19 3:52 PM, Bryan Hilbert wrote: A couple answers from Armin: The list of reflabels in the call to mkrefs.py is what is actually run. The list of reflabels in the config file is simply a list of allowed reflabels. This should now be generated automatically by the code,

This is not yet done, but soon!

…

-A

[bhilbert4]

I thought this is the part that I wrote last week. Should be working, right? See self.allowed_reflabels in mkrefs.py

[arminrest]

yes, you are right! What I had in mind was that only the reflables are loaded that are also in the config file, and that is not implemented yet...

…

On 1/28/19 3:55 PM, Bryan Hilbert wrote: I thought this is the part that I wrote last week. Should be working, right? See self.allowed_reflabels in mkrefs.py — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#45 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWd30wt-2b6pMuQExbCU0F5jYRxq5Kifks5vH2PegaJpZM4aWtap>.