Records sometimes missing fields (develop branch)

[RemingtonRohel]

Some records in data files are missing fields. For RAWACF files, it seems to be consistently the following fields:

The following fields in record 1694629335109 of file 20230913.1800.02.rkn.0.rawacf.hdf5.site are missing: {
'num_slices', 
'scan_start_marker', 
'sqn_timestamps', 
'slice_comment', 
'num_sequences', 
'slice_id', 
'scheduling_mode', 
'xcfs', 
'rx_sample_rate', 
'pulses', 
'tx_pulse_len', 
'tau_spacing', 
'noise_at_freq', 
'range_sep', 
'station', 
'tx_antenna_phases', 
'samples_data_type', 
'slice_interfacing'}

I suspect it is a missing conditional in data_write.py at Line 595, where there are a bunch of if ... elif statements, but no final else to catch if data is None.

[RemingtonRohel]

Investigating the logs on PGR from when this issue most recently occurred. The data_write log reports a failure at 01:30:48, with the issue being noise_at_freq = [] and Borealis trying to index noise_at_freq[0]. However, earlier in the log, there is a gap between 01:30:38 and 01:30:48 where no sequences are parsed by data_write. noise_at_freq is written as [0.0] * aveperiod_meta.num_sequences, so if num_sequences is zero this returns an empty list. Radar control also has the same gap, then when it continues the radar is behind (scanbound is set) so two averaging periods are skipped and have zero sequences. The metadata is still sent to data_write, and hence leads to the error. The Brian log covering this gap is included below, and reports a 9.45 second delay waiting for the driver to be ready. It could be some blip in the network communication with the Ettus devices. There is no other information in any of the logs or the system logs to indicate why the 10 second gap occurred.

‌

{
  "dsp_time": 0.020487183704972267,
  "event": "dsp done with data",
  "logger": "brian",
  "level": "info",
  "timestamp": "2024-03-19T01:30:39.010764Z",
  "module": "brian",
  "func_name": "start_new",
  "process": 3499,
  "unix_timestamp": 1710811839.0108805
}
{
  "driver_time": 9.450346434488893,
  "event": "driver ready",
  "logger": "brian",
  "level": "info",
  "timestamp": "2024-03-19T01:30:48.461120Z",
  "module": "brian",
  "func_name": "start_new",
  "process": 3499,
  "unix_timestamp": 1710811848.4612558
}