Data log file format description

HDF5 Data-Log structure

The data-logs produced by the SpectraLogger tool.

Broadly, the logged spectra data is saved in two tables in the produced HDF5 data-file:

• One table contains the actual spectrum data, with a small amount of information pertaining to the current running acquisition.
• The other table contains longer-duration metadata about the acquisition in general, as well as things like recalibration messages, exhaustive acquisition mode description, and other miscellany.

###The first table is fairly simple in structure:

Unix Timestamp	Frequency of center of first Bin	Bin Width	Averaged samples	Actual data (~16K/22K columns depending on BB60a/BB60c)

• All inter-table correlation is done via unix timestamps.
• The frequency is the frequency of the lower-most bin of the acquisition, and is the center of the bin. The frequency is in Hertz.
• Bin width. Also in Hertz.
• Averaged sampled - This is more complex. This variable encodes the effective integration time of the row, in units of n * single-sample-duration. Effectively, it is the number of frontend samples averaged to form the row.
  Under normal operation, where the dwell time at the specified frequency is a integer multiple of the number of samples averaged, this number should simply be the number of samples averaged. However, if the dwell time is not an integer multiple, this column will contain data to reflect that.
  For example, if the frequency dwell time were 650 samples, and the row averaging were 100 samples, the first 6 rows at each frequency would be the average of 100 samples. However, the last row would only accumulate 50 samples before the acquisition frequency was changed, so the last row would only contain 50 samples, which would be reflected in the averaged samples column.
• Actual data: This is the actual data from the SignalHound. It is a power of 2 number of frequencies, currently 2^14 using the current acquisition settings.

###Metadata table:

This table is somewhat more complex, primarily because it doesn't contain HDF-5 data proper. Rather, it is a variable length string, composed of a pickled python dictionary. This was done because there are several different messages that can be stored in this table. It contains general running status for the current acquisition.

Cross-correspondence between the two tables must be done by comparing timestamps. Both tables contain the timestamp of each row, and there is no otherwise 1:1 correspondence on a row basis, or any other mechanism.

Currently, everytime the front-end frequency is changed, a full snapshot of all the running acquisition settings is dumped to the metadata table, as well as periodic polling of the hardware temperature, and a message indicating the frontend was returned whenever it needs it. The signalhound hardware requires retuning whenever the signalhound temperature drifts > 2°C from the internal temperature when it was started. The acquisition system takes care of this, but because recalibration could potentially generate a disconnect in the data, every recalibration event is marked with a timestamped message in the metadata table.