-
Notifications
You must be signed in to change notification settings - Fork 18
Reformatting Data Files
Original Author: R. J. Barnes (JHU/APL)
For many years the primary archival data file types for SuperDARN have been the fit and dat files. The dat files store the raw ACF data derived from the radar observations, and the fit files store the fitted parameters generated by the FitACF algorithm. The formats for these file types were defined many years ago and require special software to read and write the binary format.
Although these formats have served SuperDARN well over the years, the limitations that they impose are starting to become a burden.
The next generation of radar software will use a new self-describing format called dmap for storing both the raw ACF data and the fitted parameters. To distinguish the new format data from the old the suffixes rawacf and fitacf have been adopted.
Obviously there is a lot of legacy data in the old format and also a lot of legacy software that will only work with the old format. To solve this problem, tools have been developed for transforming the old data format into the new and vice versa.
To convert a dat file to a rawacf file, use the utility dattorawacf:
dattorawacf 20021219kap.dat > 20021219.kap.rawacf
The utility reads the dat file specified on the command line and writes the rawacf file to standard output (redirected to a file using >).
The only command line flag is the -vb verbose option.
To convert a rawacf file to a dat file, use the utility rawacftodat:
rawacftodat 20021219.kap.rawacf > 20021219.kap.dat
The utility reads the rawacf file specified on the command line and writes the dat file to standard output (redirected to a file using >).
The -t command line flag sets a limit on the signal to noise ratio stored in the file. Ranges with a lag-zero power to noise ratio lower than this threshold are not stored in the "dat" file. The other command line flag, -vb, sets the verbose mode.
To convert a fit file to a fitacf file, use the utility fittofitacf:
fittofitacf 20021219.kap.fit > 20021219.kap.fitacf
The utility reads the fit file specified on the command line and writes the fitacf file to standard output (redirected to a file using >).
The only command line flag is the -vb verbose option.
To convert a fitacf file to a fit file, use the utility fitacftofit:
fitacftofit 20021219.kap.fitacf > 20021219.kap.fit
The utility reads the fitacf file specified on the command line and writes the fit file to standard output (redirected to a file using >).
The only command line flag is the -vb verbose option.
The rawacf and fitacf files can be viewed using the utility dmapdump:
dmapdump 20021219.kap.fitacf
By default dmapdump only prints out the values of scalar variables, to print out all data values you must specify the -d option.
One of the obvious issues when converting from one file format to another is to make sure that the transformation does not effect the values of the data in the file. The best way of checking this is to compare the differences between an existing file and a copy created by converting it first to the new format and then converting it back to the old. Unfortunately, due to the way fit and dat files are written it is impossible to do a byte by byte comparison. Instead, the two utilities cmpraw and cmpfit, can be used:
cmpraw 20021219.kap.dat.old 20021219.kap.dat.new
cmpfit 20021219.kap.fit.old 20021219.kap.fit.new
The utilities compare the fit and dat files specified on the command line and report any differences.