Skip to content

v3.1.1 FBH5 (HDF5), Multiantenna, and NFPC Support [updated]

Latest
Latest
Compare
Choose a tag to compare
@texadactyl texadactyl released this 16 Dec 19:19
· 35 commits to master since this release
v3.0.0
5accb91
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

FBH5 Support

Filterbank HDF5 Format (FBH5) is one of the two science product file formats that are found in the various pipeline stages. The other is the SIGPROC Filterbank format (.fil). A simple picture of the file evolution in the various operations pipelines:

Guppi file (.raw) ---rawspec---> SIGPROC Filterbank format (.fil) ---fil2h5---> FBH5 (.h5)

With this release of rawspec, the existing pipelines can be simplified as follows:

Guppi file (.raw) ---rawspec---> FBH5 (.h5)

This will have the following advantages:

  • Reduced consumption of processing resources in file conversion.
  • Reduced consumption of disk storage.
  • The addition of a new file header field, nfpc, representing the number of fine channels per coarse channel. This is discussed in a subsequent section.

The FBH5 output option is enabled by the use of --fbh5 or -j on the rawspec command line.

New Output Header Field (FBH5 only)

In current production, the SIGPROC Filterbank file header has no information about the relationship between coarse and fine channels. In this release, a new FBH5 header field, nfpc, is set to the number of fine channels per coarse channel.

In addition to documenting this relationship, it has a practical use. For turbo_seti, the presence of nfpc in the FBH5 header can trigger an automatic calculation of the number of coarse channels, enhancing coarse channel search accuracy.

The following is the turbo_seti methodology for deriving the number of coarse channels:

  1. If the user specifies the number of coarse channels, use that value.
  2. Else, if the FBH5 header contains the nfpc field, compute the number of coarse channels = the number of fine channels divided by nfpc.
  3. Else, use blimpy as a last resort to calculate the number of coarse channels.

Multiantenna Output

In the case where multiple antennas have contributed data to a guppi raw file, there are 2 options for processing:

Rawspec_Option Feature
--splitant or -S Split the output data by antenna. This is the action taken if no multiantenna option is specified.
--ics=W0[,W1,...]
or -i=W0[,W1,...]		 Incoherent sum over all antennas for the output data, as specified by antenna weight for each antenna (Wi). If only W0 is specified, that value is applied to all antenna.

Package Requirements

This release of rawspec requires an HDF5 run-time development package for building the rawspec shared object file and executable. For Ubuntu, the following package must be installed by a systems administrator using the apt utility: libhdf5-dev. This package contains the development files (.h and .so), used for compilation and link edit in the rawspec make process.

It is recommended that one additional package be installed if not already present: hdf5-tools. These are useful FBH5 utility programs.

Installation Instructions

Base Installation

Rawsspec is installed by a systems administrator using the following steps:

Bitshuffle Plugin Installation

For this release, the FBH5 feature requires a Bitshuffle plugin used for compression (libh5bshuf.so). This plugin must be copied by a systems administrator to a specific directory that is designated for this purpose by the installed HDF5 run-time (see the section, Package Requirements). If not already known, the path of the HDF5 run-time plugin directory can be displayed as follows: rawspec -h. The following is sample console output of this information:

The plugin directory is /usr/local/hdf5/lib/plugin.
Please copy the bitshuffle plugin to that directory if you haven't already done so.

If the compression plugin is not yet available, the Python package hdf5plugin contains a reasonably up-to-date version that can be copied to the HDF5 run-time plugin directory or pointed to through the use of the HDF5_PLUGIN_PATH environment variable.

Similar to any other Python package, the hdf5plugin package can be installed as follows:

python3 -m pip install -U --user hdf5plugin

Then, the desired plugin can be found in the following directory: $HOME/.local/lib/python<VERSION>/site-packages/hdf5plugin/plugins/ where <VERSION> refers to the Python version (3.7, 3.8, 3.9, etc.).

Bitshuffle Plugin Installation - Julia Language Compatibility

It was found in testing that the above Bitshuffle Plugin Installation section does not work for Julia language programs. The cause is still unknown but it seems tp be related to HDF5.jl sensitivity to how third-party filters are produced.

Therefore, for compatibility with the Julia language, it would be better to download the bitshuffle plugin github source, build, and install it in a commonly-accessible directory. Then, use the same set of plugins for both rawspec and Julia programs.

The bitshuffle source github project can be found at https://github.com/kiyo-masui/bitshuffle

Assets 2
Loading