VLITE Database Pipeline (VDP)

Scripts for automating source extraction and database storage.

Version 2.0

A pipeline to adaptively extract and catalog point sources is being developed to enhance the scientific value and accessibility of data products generated by the VLA Low-band Ionosphere and Transient Experiment (VLITE) on the Karl G. Jansky Very Large Array (VLA). In contrast to other radio sky surveys, the commensal observing mode of VLITE results in varying depths, sensitivities, and spatial resolutions across the sky based on the configuration of the VLA, location on the sky, and time on source specified by the primary observer for their independent science objectives. Therefore, previously developed tools and methods for generating source catalogs and survey statistics are not always appropriate for VLITE's diverse and growing set of data. A catalog of radio sources detected in every VLITE image is created from source fit parameters measured using the Python Blob Detector and Source Finder software (PyBDSF; Mohan & Rafferty 2015). Detected VLITE sources are positionally associated with each other in a resolution-dependent manner, and are cross-matched to other radio sky surveys and catalogs to aid in the detection of transient sources and enable creation of radio flux spectra for sources across the entire northern sky.

Files

This repository contains the following files under the vdp/ directory:

• break_handler.py: class to trap keyboard interrupts
• database/createdb.py: functions to create the database tables
• database/dbclasses.py: module defining classes used in vdp and their functions
• database/dbio.py: functions which handle database I/O for vdp
• database/pybdsfcat.py: function to read list of sources output by PyBDSF
• docs/: documentation files
• errors.py: module containing class definition for ConfigError exception
• example_config.yaml: example YAML configuration file used to set run parameters for vdp
• maintenance.py: module containing code to cluster & analyze the spatially indexed database tables
• matching/matchfuncs.py: module containing functions for positional cross-matching
• matching/radioxmatch.py: driver script for performing association and catalog cross-matching
• notebooks/VDPGuide.ipynb: vdp demo in an IPython notebook
• notebooks/VDPUsageExamples: examples demonstrating different ways to run vdp
• radiocatalogs/catalogio.py: module for reading other radio sky survey source lists and catalogs
• radiocatalogs/radcatdb.py: module for creation of separate database schema and insertion of other radio catalogs
• sourcefinding/beam_tools.py: functions to primary beam correct VLITE flux measurements
• sourcefinding/runbdsf.py: script which runs PyBDSF source finding
• tests/functest_single.yaml: configuration file used in tests/FunctionalTests.ipynb
• tests/functest_two.yaml: configuration file used in tests/FunctionalTests.ipynb
• tests/FunctionalTests.ipynb: IPython notebook to perform interactive functional tests
• tests/test_cmbranches.py: tests for correct logic & data flow through all catalog matching stage options
• tests/test_sabranches.py: tests for correct logic & data flow through all source association stage options
• test_sfbranches.py: tests for correct logic & data flow through all source finding stage options
• timeout.py: Python decorator to timeout long running functions
• vdp.py: module that drives the whole pipeline

Running the Code

$ python vdp.py example_config.yaml

To see all optional command line arguments, type:

$ python vdp.py -h

or

$ python vdp.py --help

Dependencies

Python packages:

• astropy
• numpy
• pandas
• psycopg2
• pyephem

Other software:

• PostgreSQL
• PyBDSF
• PyYAML
• Q3C