VAC notebook

[gschwend]

Write an e2e notebook to serve as the user interface to create VACs on the LIneA OPen OnDemand platform. The bulk code should be hidden in ancillary Python modules. The notebook must be as clean as possible and mostly used to get parameters from the user.

Preparation steps outside the notebook - 1st time

Beta Give feedback

1. create conda env
2. create jupyter kernel

Preparation steps outside the notebook - before every new process

Beta Give feedback

1. prepare pz-compute process dir inside user's scratch area (for provenance tracking) (it should include a copy of the VAC notebook inside the process dir)

Steps inside the notebook

Beta Give feedback

1. make footprint map from the combination of survey conditions maps available (optional, default=False)
2. create temporary skinny tables in lustre T0 scratch area as the result of region selection + object selection + apply photometric corrections
3. make input data QA plots
4. run pipeline TSM from pzserver lib to create train/test sets
5. run RAIL inform on training set (optional, if False, reuse existing .pkl file)
6. run RAIL estimate on test set
7. run RAIL evaluate on test results
8. run pipeline pz-compute on the skinny tables
9. make output data QA plots
10. create wazp input files = FITS files with skinny table columns + pz PDF percentiles partitioned by healpix pixels of fixed resolution, maglim file, pz metrics file
11. delete temporary skinny tables
12. export notebook to HTML
13. save a copy of notebook (.ipynb without outputs and .HTML with outputs) inside the process dir with process_id and timestamp prefixes
14. update process_info.yaml with notebook filenames
15. if production mode, move process dir to wazp project shared directory on lustre T1