Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolve warnings from FITS verify tool #63

Closed
2 of 3 tasks
jmbhughes opened this issue Dec 6, 2023 · 3 comments · Fixed by #62
Closed
2 of 3 tasks

Resolve warnings from FITS verify tool #63

jmbhughes opened this issue Dec 6, 2023 · 3 comments · Fixed by #62
Assignees
@lowderchris

Comments

@jmbhughes
Copy link
Member

jmbhughes commented Dec 6, 2023 •
edited by lowderchris
Loading

We ran an example file through https://fits.gsfc.nasa.gov/fits_verify.html and it flagged a few warnings.

Tasks:

  • remove duplicated MJDRef
  • resolve "Warning: The HDU 3 and 2 have identical type/name/version"
  • add test to run astropy's automated verify

Output:

File name:  PUNCH_L3_PTM_20231205185656.fits
                                           Run Number 35706
 
              fitsverify 4.20 (CFITSIO V3.470)              
              --------------------------------              
 

3 Header-Data Units in this file.
 
=================== HDU 1: Primary Array ===================
 
   1 | SIMPLE  =                    T / conforms to FITS standard
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    0 / number of array dimensions
   4 | EXTEND  =                    T
   5 | CHECKSUM= 'A6dFD4Z9A4bEA4Z9'   / HDU checksum updated 2023-12-05T11:56:59
   6 | DATASUM = '0       '           / data unit checksum updated 2023-12-05T11:56:59
   7 | END     
 
 7 header keywords
 
 Null data array; NAXIS = 0 
 
=================== HDU 2: BINARY Table ====================
 
*** Warning: Keyword #170, COMP_RAT has a null value.
*** Warning: Keyword #192, RSUN_ARC has a null value.
*** Warning: Keyword #193, RSUN_REF has a null value.
*** Warning: Keyword #194, SOLAR_EP has a null value.
*** Warning: Keyword #195, CAR_ROT has a null value.
*** Warning: Keyword MJDREF is duplicated in card #66 and card #106.
 
   1 | XTENSION= 'BINTABLE'           / binary table extension
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    2 / number of array dimensions
   4 | NAXIS1  =                   32 / width of table in bytes
   5 | NAXIS2  =                 8192 / number of rows in table
   6 | PCOUNT  =             28595565 / number of group parameters
   7 | GCOUNT  =                    1 / number of groups
   8 | TFIELDS =                    4 / number of fields in each row
   9 | TTYPE1  = 'COMPRESSED_DATA'    / label for field 1
  10 | TFORM1  = '1PB(6745)'          / data format of field: variable length array
  11 | TTYPE2  = 'GZIP_COMPRESSED_DATA' / label for field 2
  12 | TFORM2  = '1PB(13520)'         / data format of field: variable length array
  13 | TTYPE3  = 'ZSCALE  '           / label for field 3
  14 | TFORM3  = '1D      '           / data format of field: 8-byte DOUBLE
  15 | TTYPE4  = 'ZZERO   '           / label for field 4
  16 | TFORM4  = '1D      '           / data format of field: 8-byte DOUBLE
  17 | ZIMAGE  =                    T / extension contains compressed image
  18 | ZSIMPLE = 'T       '           / Conforms to FITS Standard
  19 | ZBITPIX =                  -32 / Number of bits per pixel
  20 | ZNAXIS  =                    3 / Number of axes
  21 | ZNAXIS1 =                 4096 / Length of the first axis
  22 | ZNAXIS2 =                 4096 / Length of the second axis
  23 | ZNAXIS3 =                    2 / Length of the third axis
  24 | ZTILE1  =                 4096 / size of tiles to be compressed
  25 | ZTILE2  =                    1 / size of tiles to be compressed
  26 | ZTILE3  =                    1 / size of tiles to be compressed
  27 | ZCMPTYPE= 'RICE_1  '           / compression algorithm
  28 | ZNAME1  = 'BLOCKSIZE'          / compression block size
  29 | ZVAL1   =                   32 / pixels per block
  30 | ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)
  31 | ZVAL2   =                    4 / bytes per pixel (1, 2, 4, or 8)
  32 | ZNAME3  = 'NOISEBIT'           / floating point quantization level
  33 | ZVAL3   =                 16.0 / floating point quantization level
  34 | ZQUANTIZ= 'NO_DITHER'          / No dithering during quantization
  35 | EXTNAME = 'COMPRESSED_IMAGE'   / name of this binary table extension
  36 | COMMENT ----- FITS Required ----------------------------------------------------
  37 | LONGSTRN= 'OGIP 1.0'           / The OGIP long string convention may be used
  38 | COMMENT ----- Documentation, Contact, and Collection Metadata ------------------
  39 | DOI     = 'https://doi.org/TBD' / Data reference DOI
  40 | PROJECT = 'PUNCH   '
  41 | TITLE   = 'PUNCH Level-3 Polarized Mosaic'
  42 | KEYVOCAB= 'Unified Astronomy Thesaurus Keywords'
  43 | KEYWORDS= 'Solar Corona (1483), Solar K Corona (2042), Solar F Corona (1991), &'
  44 | CONTINUE  'Solar Coronal Streamers (1486), Solar Coronal Plumes (2039), Solar &'
  45 | CONTINUE  'Wind (1534), Fast Solar Wind (1872), Slow Solar Wind (1873), Solar &'
  46 | CONTINUE  'Coronal Mass Ejection (310), Heliosphere (711), Polarimetry (1278)'
  47 | LICENSE = 'Creative Commons Attribution 4.0 International | CC BY 4.0'
  48 | DESCRPTN= 'PUNCH Level-3 data, Composite mosaic in output coordinates'
  49 | DOC_URL = 'https://punch.spaceops.swri.org'
  50 | COMMENT ----- File Type and Provenance -----------------------------------------
  51 | FILENAME= 'PUNCH_L3_PTM_20231205185656.fits' / Name of file
  52 | LEVEL   = '3       '           / Product Level
  53 | OBSTYPE = 'Polarized mosaic'   / Plain text observation
  54 | TYPECODE= 'PT      '           / Observation product type code
  55 | OBSCODE = 'M       '           / Observatory spacecraft code
  56 | PIPEVRSN= '' / PUNCHPipe software version number
  57 | FILE_RAW= '' / Raw telemetry filename
  58 | ORIGIN  = 'SwRI    '           / Institution responsible for creating the file
  59 | COMMENT ----- Temporal Information ---------------------------------------------
  60 | TIMESYS = 'UTC     '           / Principal time system
  61 | DATE-BEG= '2023-12-05T18:56:56.454062' / UTC time observation
  62 | DATE-OBS= '2023-12-05T18:56:56.454042' / UTC reference time
  63 | DATE-AVG= '2023-12-05T18:58:56.454067' / UTC reference time
  64 | DATE-END= '2023-12-05T19:00:56.454074' / UTC time of observation end
  65 | DATE    = '2023-12-05T19:00:56.454078' / UTC file generation date and time
  66 | MJDREF  =                  0.0 / [d] MJD of fiducial time
  67 | COMMENT ----- Instrument and Spacecraft State ----------------------------------
  68 | WAVELNTH=                  530 / [nm] average peak response
  69 | WAVEUNIT= 'nanometer'          / Unit of observation measurement
  70 | OBS-MODE= 'Polar_BpB'          / Image Mode (Unpolarized, Polar_MZP, or Polar_Bp
  71 | OBSLAYR1= 'Polar_B '           / Image Mode for first datacube layer
  72 | OBSLAYR2= 'Polar_pB'           / Image Mode for second datacube layer
  73 | INSTRUME= 'WFI+NFI Mosaic'     / Instrument name
  74 | TELESCOP= 'PUNCH 1-2-3-4'      / Satellite name
  75 | OBSRVTRY= 'PUNCH   '           / Observatory name
  76 | OBJECT  = 'Heliosphere white light' / Object observed
  77 | COMMENT ----- World Coordinate System ------------------------------------------
  78 | WCSAXES =                    3 / Number of coordinate axes
  79 | CRPIX1  =                  0.0 / Pixel coordinate of reference point
  80 | CRPIX2  =               2048.5 / Pixel coordinate of reference point
  81 | CRPIX3  =               2048.5 / Pixel coordinate of reference point
  82 | PC1_1   =                  1.0 / Coordinate transformation matrix element
  83 | PC1_2   =                  0.0 / Coordinate transformation matrix element
  84 | PC1_3   =                  0.0 / Coordinate transformation matrix element
  85 | PC2_1   =                  0.0 / Coordinate transformation matrix element
  86 | PC2_2   =                  1.0 / Coordinate transformation matrix element
  87 | PC2_3   =                  0.0 / Coordinate transformation matrix element
  88 | PC3_1   =                  0.0 / Coordinate transformation matrix element
  89 | PC3_2   =                  0.0 / Coordinate transformation matrix element
  90 | PC3_3   =                  1.0 / Coordinate transformation matrix element
  91 | CDELT1  =                  1.0 / [deg] Coordinate increment at reference point
  92 | CDELT2  =             6.25E-06 / [deg] Coordinate increment at reference point
  93 | CDELT3  =             6.25E-06 / [deg] Coordinate increment at reference point
  94 | CUNIT1  = 'deg     '           / Units of coordinate increment and value
  95 | CUNIT2  = 'deg     '           / Units of coordinate increment and value
  96 | CUNIT3  = 'deg     '           / Units of coordinate increment and value
  97 | CTYPE1  = 'STOKES  '           / Coordinate type
  98 | CTYPE2  = 'HPLN-ARC'           / Coordinate type codezenithal/azimuthal equidist
  99 | CTYPE3  = 'HPLT-ARC'           / Coordinate type codezenithal/azimuthal equidist
 100 | CRVAL1  =                  0.0 / [deg] Coordinate value at reference point
 101 | CRVAL2  =                  0.0 / [deg] Coordinate value at reference point
 102 | CRVAL3  =                  0.0 / [deg] Coordinate value at reference point
 103 | LONPOLE =                180.0 / [deg] Native longitude of celestial pole
 104 | LATPOLE =                  0.0 / [deg] Native latitude of celestial pole
 105 | CNAME1  = 'STOKES  '           / Axis name for labelling purposes
 106 | MJDREF  =                  0.0 / [d] MJD of fiducial time
 107 | WCSAXESA=                    3 / Number of coordinate axes
 108 | CRPIX1A =                  0.0 / Pixel coordinate of reference point
 109 | CRPIX2A =               2048.5 / Pixel coordinate of reference point
 110 | CRPIX3A =               2048.5 / Pixel coordinate of reference point
 111 | PC1_1A  =                  1.0 / Coordinate transformation matrix element
 112 | PC1_2A  =                  0.0 / Coordinate transformation matrix element
 113 | PC1_3A  =                  0.0 / Coordinate transformation matrix element
 114 | PC2_1A  =                  0.0 / Coordinate transformation matrix element
 115 | PC2_2A  =   0.9689375369683749 / Coordinate transformation matrix element
 116 | PC2_3A  = -0.24730557909933853 / Coordinate transformation matrix element
 117 | PC3_1A  =                  0.0 / Coordinate transformation matrix element
 118 | PC3_2A  =  0.24730557909933853 / Coordinate transformation matrix element
 119 | PC3_3A  =   0.9689375369683749 / Coordinate transformation matrix element
 120 | CDELT1A =                  1.0 / [deg] Coordinate increment at reference point
 121 | CDELT2A =             6.25E-06 / [deg] Coordinate increment at reference point
 122 | CDELT3A =             6.25E-06 / [deg] Coordinate increment at reference point
 123 | CUNIT1A = 'deg     '           / Units of coordinate increment and value
 124 | CUNIT2A = 'deg     '           / Units of coordinate increment and value
 125 | CUNIT3A = 'deg     '           / Units of coordinate increment and value
 126 | CTYPE1A = 'STOKES  '           / Coordinate type
 127 | CTYPE2A = 'RA-ARC  '           / Coordinate type codezenithal/azimuthal equidist
 128 | CTYPE3A = 'DEC-ARC '           / Coordinate type codezenithal/azimuthal equidist
 129 | CRVAL1A =                  0.0 / [deg] Coordinate value at reference point
 130 | CRVAL2A =    7.282772648229448 / [deg] Coordinate value at reference point
 131 | CRVAL3A =                  0.0 / [deg] Coordinate value at reference point
 132 | LONPOLEA=                180.0 / [deg] Native longitude of celestial pole
 133 | LATPOLEA=                  0.0 / [deg] Native latitude of celestial pole
 134 | CNAME1A = 'STOKES  '           / Axis name for labelling purposes
 135 | MJDREFA =                  0.0 / [d] MJD of fiducial time
 136 | COMMENT ----- Camera and Readout State -----------------------------------------
 137 | DETECTOR= 'CCD 2Kx2K'          / CCD detector 2048x2048 pixels
 138 | EXPTIME =                  3.0 / [s] commanded exposure time
 139 | OBT_BEG =          317813545.8 / MET when first pixel of first image received
 140 | OBT_END =          317813586.4 / MET when first pixel of last image is received
 141 | NSUMEXP =                    3 / total number images stacked on onboard
 142 | NSUMBAD =                    0 / number of images with invalid pixel counts
 143 | READTIME=               2.4576 / [s] time to readout raw image from camera
 144 | TELAPSE =              48.1296 / [s] time between begin and end of exposure
 145 | REGION  =                    1 / region
 146 | READOUT0=                    3 / readout region if region=0
 147 | CAMERA  = 'FMCFMD  '           / detector name (TBD)
 148 | PXBEG1  =                    1 / first read-out detector row
 149 | PXEND1  =                 2048 / last read-out detector row
 150 | PXBEG2  =                    1 / first read-out detector column
 151 | PXEND2  =                 2048 / last read-out detector column
 152 | NBIN1   =                    1 / number of pixels binned along axis1
 153 | NBIN2   =                    1 / number of pixels binned along axis2
 154 | NBIN    =                    1 / number of pixels binned
 155 | DSTART1 =                    1 / first column of image array on data array
 156 | DSTOP1  =                 2048 / last column of image array on data array
 157 | DSTART2 =                    1 / first row of image area on data array
 158 | DSTOP2  =                 2048 / last row of image area on data array
 159 | IMGCTR  =                 8370 / image counter from IDPU
 160 | LEDSTATE= 'Off     '           / state of LED (TBD)
 161 | LEDDAC  =                    0 / last commanded setting of LED
 162 | OFFSET  =                   40 / commanded offset value used in camera
 163 | GAINCMD =                 12.0 / commanded gain value
 164 | XFBYTES =              1374028 / number of bytes sent
 165 | COMMENT ----- Onboard Image Processing -----------------------------------------
 166 | RAWBITS =                   16 / [bits] raw image bits per pixel
 167 | COMPBITS=                   10 / [bits] compressed image bits per pixel
 168 | SQRTCODE= '' / square-root coding table name
 169 | COMPRESS= '' / image compression type
 170 | COMP_RAT=  / compressed/uncompressed data ratio
 171 | COMMENT ----- Image Statistics and Properties ----------------------------------
 172 | BUNIT   = 'Mean Solar Brightness' / units of observation
 173 | BSUN_DEF=           20090000.0 / [W/m2/sr] Mean Solar Brightness
 174 | DATAZER =             15384941 / number of pixels=0
 175 | DATASAT =                    0 / number of saturated pixels
 176 | DSATVAL =              65535.0 / saturation value in calibrated units
 177 | DATAAVG = 2.66907591139897E-05 / average non-zero pixel value
 178 | DATAMDN = 2.65701203261414E-07 / median non-zero pixel value
 179 | DATASIG = 0.001541828620247542 / standard deviation of non-zero pixels
 180 | DATAP01 = 5.13851261985109E-08 / 1% of non-zero pixels are LE this value
 181 | DATAP10 = 8.47963264050122E-08 / 10% of non-zero pixels are LE this value
 182 | DATAP25 = 1.28057614290355E-07 / 25% of non-zero pixels are LE this value
 183 | DATAP50 = 2.65701203261414E-07 / 50% of non-zero pixels are LE this value
 184 | DATAP75 = 8.07493108823109E-07 / 75% of non-zero pixels are LE this value
 185 | DATAP90 = 2.89900822281197E-06 / 90% of non-zero pixels are LE this value
 186 | DATAP95 = 8.21391859062714E-06 / 95% of non-zero pixels are LE this value
 187 | DATAP98 = 3.27936075336765E-05 / 98% of non-zero pixels are LE this value
 188 | DATAP99 = 9.25617583561692E-05 / 99% of non-zero pixels are LE this value
 189 | DATAMIN = -1.4374144235995E-12 / minimum valid physical value
 190 | DATAMAX =   0.6713011860847473 / maximum valid physical value
 191 | COMMENT ----- Solar Reference Data ---------------------------------------------
 192 | RSUN_ARC=  / [arcsec] photospheric solar radius
 193 | RSUN_REF=  / [m] assumed physical solar radius
 194 | SOLAR_EP=  / [deg] S/C ecliptic North to solar North angle
 195 | CAR_ROT =  / Carrington Rotation number
 196 | COMMENT ----- Fixity -----------------------------------------------------------
 197 | CHECKSUM= 'kMAAnJ97kJAAkJ75'   / HDU checksum updated 2023-12-05T11:57:01
 198 | DATASUM = '340358026'          / data unit checksum updated 2023-12-05T11:57:01
 199 | ZHECKSUM= 'Y6Naa4NXY4Naa4NU'   / HDU checksum updated 2023-12-05T11:57:01
 200 | ZDATASUM= '1844929411'         / data unit checksum updated 2023-12-05T11:57:01
 201 | COMMENT ----- History ----------------------------------------------------------
 202 | HISTORY Records of processing from pipeline
 203 | END     
 
 203 header keywords
 
 COMPRESSED_IMAGE  (4 columns x 8192 rows)
 
 Col# Name (Units)       Format
   1 COMPRESSED_DATA      1PB(6745) 
   2 GZIP_COMPRESSED_DATA 1PB(13520)
   3 ZSCALE               1D        
   4 ZZERO                1D        
 
=================== HDU 3: BINARY Table ====================
 
*** Warning: The HDU 3 and 2 have identical type/name/version
 
   1 | XTENSION= 'BINTABLE'           / binary table extension
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    2 / number of array dimensions
   4 | NAXIS1  =                    8 / width of table in bytes
   5 | NAXIS2  =                 8192 / number of rows in table
   6 | PCOUNT  =               449185 / number of group parameters
   7 | GCOUNT  =                    1 / number of groups
   8 | TFIELDS =                    1 / number of fields in each row
   9 | TTYPE1  = 'COMPRESSED_DATA'    / label for field 1
  10 | TFORM1  = '1PB(170)'           / data format of field: variable length array
  11 | ZIMAGE  =                    T / extension contains compressed image
  12 | ZTENSION= 'IMAGE   '           / Image extension
  13 | ZBITPIX =                    8 / data type of original image
  14 | ZNAXIS  =                    3 / dimension of original image
  15 | ZNAXIS1 =                 4096 / length of original image axis
  16 | ZNAXIS2 =                 4096 / length of original image axis
  17 | ZNAXIS3 =                    2 / length of original image axis
  18 | ZPCOUNT =                    0 / number of parameters
  19 | ZGCOUNT =                    1 / number of groups
  20 | ZTILE1  =                 4096 / size of tiles to be compressed
  21 | ZTILE2  =                    1 / size of tiles to be compressed
  22 | ZTILE3  =                    1 / size of tiles to be compressed
  23 | ZCMPTYPE= 'RICE_1  '           / compression algorithm
  24 | ZNAME1  = 'BLOCKSIZE'          / compression block size
  25 | ZVAL1   =                   32 / pixels per block
  26 | ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)
  27 | ZVAL2   =                    1 / bytes per pixel (1, 2, 4, or 8)
  28 | EXTNAME = 'COMPRESSED_IMAGE'   / name of this binary table extension
  29 | WCSAXES =                    3
  30 | CRPIX1  =                  0.0
  31 | CRPIX2  =               2048.5
  32 | CRPIX3  =               2048.5
  33 | CDELT1  =                  1.0
  34 | CDELT2  =             6.25E-06
  35 | CDELT3  =             6.25E-06
  36 | CUNIT1  = 'deg     '
  37 | CUNIT2  = 'deg     '
  38 | CUNIT3  = 'deg     '
  39 | CTYPE1  = 'STOKES  '
  40 | CTYPE2  = 'HPLN-ARC'
  41 | CTYPE3  = 'HPLT-ARC'
  42 | CRVAL1  =                  0.0
  43 | CRVAL2  =                  0.0
  44 | CRVAL3  =                  0.0
  45 | LONPOLE =                180.0
  46 | LATPOLE =                  0.0
  47 | CNAME1  = 'STOKES  '
  48 | TIMESYS = 'UTC     '
  49 | MJDREF  =                  0.0
  50 | DATE-OBS= '2023-12-05T18:56:56.454042'
  51 | MJD-OBS =      60283.789542292
  52 | DATE-BEG= '2023-12-05T18:56:56.454062'
  53 | MJD-BEG =      60283.789542292
  54 | DATE-AVG= '2023-12-05T18:58:56.454067'
  55 | MJD-AVG =      60283.790931181
  56 | DATE-END= '2023-12-05T19:00:56.454074'
  57 | MJD-END =       60283.79232007
  58 | TELAPSE =              48.1296
  59 | WCSAXESA=                    3
  60 | CRPIX1A =                  0.0
  61 | CRPIX2A =               2048.5
  62 | CRPIX3A =               2048.5
  63 | CDELT1A =                  1.0
  64 | CDELT2A =             6.25E-06
  65 | CDELT3A =             6.25E-06
  66 | CUNIT1A = 'deg     '
  67 | CUNIT2A = 'deg     '
  68 | CUNIT3A = 'deg     '
  69 | CTYPE1A = 'STOKES  '
  70 | CTYPE2A = 'RA-ARC  '
  71 | CTYPE3A = 'DEC-ARC '
  72 | CRVAL1A =                  0.0
  73 | CRVAL2A =    7.282772648229448
  74 | CRVAL3A =                  0.0
  75 | LONPOLEA=                180.0
  76 | LATPOLEA=                  0.0
  77 | CNAME1A = 'STOKES  '
  78 | TIMESYSA= 'UTC     '
  79 | MJDREFA =                  0.0
  80 | HIERARCH DATE-OBSA = '2023-12-05T18:56:56.454042'
  81 | MJD-OBSA=      60283.789542292
  82 | HIERARCH DATE-BEGA = '2023-12-05T18:56:56.454062'
  83 | MJD-BEGA=      60283.789542292
  84 | HIERARCH DATE-AVGA = '2023-12-05T18:58:56.454067'
  85 | MJD-AVGA=      60283.790931181
  86 | HIERARCH DATE-ENDA = '2023-12-05T19:00:56.454074'
  87 | MJD-ENDA=       60283.79232007
  88 | TELAPSEA=              48.1296
  89 | PC2_2A  =   0.9689375369683749
  90 | PC2_3A  = -0.24730557909933853
  91 | PC3_2A  =  0.24730557909933853
  92 | PC3_3A  =   0.9689375369683749
  93 | ZHECKSUM= 'oAaGq8WFoAaFo7WF'   / HDU checksum updated 2023-12-05T11:57:01
  94 | ZDATASUM= '3453025111'         / data unit checksum updated 2023-12-05T11:57:01
  95 | CHECKSUM= 'B4YbE4YZB4YaB4YY'   / HDU checksum updated 2023-12-05T11:57:02
  96 | DATASUM = '3755330424'         / data unit checksum updated 2023-12-05T11:57:02
  97 | END     
 
 97 header keywords
 
 COMPRESSED_IMAGE  (1 columns x 8192 rows)
 
 Col# Name (Units)       Format
   1 COMPRESSED_DATA      1PB(170)  
 
++++++++++++++++++++++ Error Summary  ++++++++++++++++++++++
 
 HDU#  Name (version)       Type             Warnings  Errors
 1                          Primary Array    0         0     
 2     COMPRESSED_IMAGE     Binary Table     6         0     
 3     COMPRESSED_IMAGE     Binary Table     1         0     
 
**** Verification found 7 warning(s) and 0 error(s). ****
@jmbhughes
Copy link
Member Author

jmbhughes commented Dec 6, 2023 •
edited
Loading

Should we change EXTNAME to something else? See Section 4.4.2.6 in FITS 4 standard.

Make an official test to check this verification each time we update.

@lowderchris
Copy link
Contributor

A note - hdus and hdu lists contain a .verify() function that should warn / fix some non-compliance with FITS standards. It seems in this case to not be warning about anything, maybe requiring validation outside of the pipeline.

@jmbhughes jmbhughes linked a pull request Dec 7, 2023 that will close this issue
Improve wcs metadata handling #62
Merged
3 tasks
@lowderchris
Copy link
Contributor

As of the latest code in the linked pull request above, validation on a sample output file raises no warnings or errors:

File name:  test_header_validation.fits
                                           Run Number 35723
 
              fitsverify 4.20 (CFITSIO V3.470)              
              --------------------------------              
 

3 Header-Data Units in this file.
 
=================== HDU 1: Primary Array ===================
 
   1 | SIMPLE  =                    T / conforms to FITS standard
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    0 / number of array dimensions
   4 | EXTEND  =                    T
   5 | CHECKSUM= 'B7eHB5Z9B5bGB5Z9'   / HDU checksum updated 2023-12-07T17:15:33
   6 | DATASUM = '0       '           / data unit checksum updated 2023-12-07T17:15:33
   7 | END     
 
 7 header keywords
 
 Null data array; NAXIS = 0 
 
=================== HDU 2: BINARY Table ====================
 
   1 | XTENSION= 'BINTABLE'           / binary table extension
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    2 / number of array dimensions
   4 | NAXIS1  =                   32 / width of table in bytes
   5 | NAXIS2  =                 8192 / number of rows in table
   6 | PCOUNT  =               450560 / number of group parameters
   7 | GCOUNT  =                    1 / number of groups
   8 | TFIELDS =                    4 / number of fields in each row
   9 | TTYPE1  = 'COMPRESSED_DATA'    / label for field 1
  10 | TFORM1  = '1PB(0)  '           / data format of field: variable length array
  11 | TTYPE2  = 'GZIP_COMPRESSED_DATA' / label for field 2
  12 | TFORM2  = '1PB(55) '           / data format of field: variable length array
  13 | TTYPE3  = 'ZSCALE  '           / label for field 3
  14 | TFORM3  = '1D      '           / data format of field: 8-byte DOUBLE
  15 | TTYPE4  = 'ZZERO   '           / label for field 4
  16 | TFORM4  = '1D      '           / data format of field: 8-byte DOUBLE
  17 | ZIMAGE  =                    T / extension contains compressed image
  18 | ZSIMPLE = 'T       '           / Conforms to FITS Standard
  19 | ZBITPIX =                  -32 / Number of bits per pixel
  20 | ZNAXIS  =                    3 / Number of axes
  21 | ZNAXIS1 =                 4096 / Length of the first axis
  22 | ZNAXIS2 =                 4096 / Length of the second axis
  23 | ZNAXIS3 =                    2 / Length of the third axis
  24 | ZTILE1  =                 4096 / size of tiles to be compressed
  25 | ZTILE2  =                    1 / size of tiles to be compressed
  26 | ZTILE3  =                    1 / size of tiles to be compressed
  27 | ZCMPTYPE= 'RICE_1  '           / compression algorithm
  28 | ZNAME1  = 'BLOCKSIZE'          / compression block size
  29 | ZVAL1   =                   32 / pixels per block
  30 | ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)
  31 | ZVAL2   =                    4 / bytes per pixel (1, 2, 4, or 8)
  32 | ZNAME3  = 'NOISEBIT'           / floating point quantization level
  33 | ZVAL3   =                 16.0 / floating point quantization level
  34 | ZQUANTIZ= 'NO_DITHER'          / No dithering during quantization
  35 | COMMENT ----- FITS Required ----------------------------------------------------
  36 | EXTNAME = 'PRIMARY DATA ARRAY' / Name of this binary table extension
  37 | LONGSTRN= 'OGIP 1.0'           / The OGIP long string convention may be used
  38 | COMMENT ----- Documentation, Contact, and Collection Metadata ------------------
  39 | DOI     = 'https://doi.org/TBD' / Data reference DOI
  40 | PROJECT = 'PUNCH   '
  41 | TITLE   = 'PUNCH Level-3 Polarized Mosaic'
  42 | KEYVOCAB= 'Unified Astronomy Thesaurus Keywords'
  43 | KEYWORDS= 'Solar Corona (1483), Solar K Corona (2042), Solar F Corona (1991), &'
  44 | CONTINUE  'Solar Coronal Streamers (1486), Solar Coronal Plumes (2039), Solar &'
  45 | CONTINUE  'Wind (1534), Fast Solar Wind (1872), Slow Solar Wind (1873), Solar &'
  46 | CONTINUE  'Coronal Mass Ejection (310), Heliosphere (711), Polarimetry (1278)'
  47 | LICENSE = 'Creative Commons Attribution 4.0 International | CC BY 4.0'
  48 | DESCRPTN= 'PUNCH Level-3 data, Composite mosaic in output coordinates'
  49 | DOC_URL = 'https://punch.spaceops.swri.org'
  50 | COMMENT ----- File Type and Provenance -----------------------------------------
  51 | FILENAME= '' / Name of file
  52 | LEVEL   = '3       '           / Product Level
  53 | OBSTYPE = 'Polarized mosaic'   / Plain text observation
  54 | TYPECODE= 'PT      '           / Observation product type code
  55 | OBSCODE = 'M       '           / Observatory spacecraft code
  56 | PIPEVRSN= '' / PUNCHPipe software version number
  57 | FILE_RAW= '' / Raw telemetry filename
  58 | ORIGIN  = 'SwRI    '           / Institution responsible for creating the file
  59 | COMMENT ----- Temporal Information ---------------------------------------------
  60 | TIMESYS = 'UTC     '           / Principal time system
  61 | DATE-BEG= '2023-12-08T00:15:31.565775' / UTC time observation
  62 | DATE-OBS= '2023-12-08T00:15:31.565765' / UTC reference time
  63 | DATE-AVG= '2023-12-08T00:17:31.565778' / UTC reference time
  64 | DATE-END= '2023-12-08T00:19:31.565782' / UTC time of observation end
  65 | DATE    = '2023-12-08T01:15:31.565785' / UTC file generation date and time
  66 | COMMENT ----- Instrument and Spacecraft State ----------------------------------
  67 | WAVELNTH=                  530 / [nm] average peak response
  68 | WAVEUNIT= 'nanometer'          / Unit of observation measurement
  69 | OBS-MODE= 'Polar_BpB'          / Image Mode (Unpolarized, Polar_MZP, Polar_BpB)
  70 | OBSLAYR1= 'Polar_B '           / Image Mode for first datacube layer
  71 | OBSLAYR2= 'Polar_pB'           / Image Mode for second datacube layer
  72 | INSTRUME= 'WFI+NFI Mosaic'     / Instrument name
  73 | TELESCOP= 'PUNCH 1-2-3-4'      / Satellite name
  74 | OBSRVTRY= 'PUNCH   '           / Observatory name
  75 | OBJECT  = 'Heliosphere white light' / Object observed
  76 | COMMENT ----- World Coordinate System ------------------------------------------
  77 | WCSAXES =                    3 / Number of coordinate axes
  78 | CRPIX1  =                  0.0 / Pixel coordinate of reference point
  79 | CRPIX2  =               2048.5 / Pixel coordinate of reference point
  80 | CRPIX3  =               2048.5 / Pixel coordinate of reference point
  81 | PC1_1   =                  1.0 / Coordinate transformation matrix element
  82 | PC1_2   =                  0.0 / Coordinate transformation matrix element
  83 | PC1_3   =                  0.0 / Coordinate transformation matrix element
  84 | PC2_1   =                  0.0 / Coordinate transformation matrix element
  85 | PC2_2   =                  1.0 / Coordinate transformation matrix element
  86 | PC2_3   =                  0.0 / Coordinate transformation matrix element
  87 | PC3_1   =                  0.0 / Coordinate transformation matrix element
  88 | PC3_2   =                  0.0 / Coordinate transformation matrix element
  89 | PC3_3   =                  1.0 / Coordinate transformation matrix element
  90 | CDELT1  =                  1.0 / [deg] Coordinate increment at reference point
  91 | CDELT2  =             6.25E-06 / [deg] Coordinate increment at reference point
  92 | CDELT3  =             6.25E-06 / [deg] Coordinate increment at reference point
  93 | CUNIT1  = 'deg     '           / Units of coordinate increment and value
  94 | CUNIT2  = 'deg     '           / Units of coordinate increment and value
  95 | CUNIT3  = 'deg     '           / Units of coordinate increment and value
  96 | CTYPE1  = 'STOKES  '           / Coordinate type
  97 | CTYPE2  = 'HPLN-ARC'           / Coordinate type codezenithal/azimuthal equidist
  98 | CTYPE3  = 'HPLT-ARC'           / Coordinate type codezenithal/azimuthal equidist
  99 | CRVAL1  =                  0.0 / [deg] Coordinate value at reference point
 100 | CRVAL2  =                  0.0 / [deg] Coordinate value at reference point
 101 | CRVAL3  =                  0.0 / [deg] Coordinate value at reference point
 102 | LONPOLE =                180.0 / [deg] Native longitude of celestial pole
 103 | LATPOLE =                  0.0 / [deg] Native latitude of celestial pole
 104 | CNAME1  = 'STOKES  '           / Axis name for labelling purposes
 105 | MJDREF  =                  0.0 / [d] MJD of fiducial time
 106 | WCSAXESA=                    3 / Number of coordinate axes
 107 | CRPIX1A =                  0.0 / Pixel coordinate of reference point
 108 | CRPIX2A =               2048.5 / Pixel coordinate of reference point
 109 | CRPIX3A =               2048.5 / Pixel coordinate of reference point
 110 | PC1_1A  =                  1.0 / Coordinate transformation matrix element
 111 | PC1_2A  =                  0.0 / Coordinate transformation matrix element
 112 | PC1_3A  =                  0.0 / Coordinate transformation matrix element
 113 | PC2_1A  =                  0.0 / Coordinate transformation matrix element
 114 | PC2_2A  =   0.9727905771259087 / Coordinate transformation matrix element
 115 | PC2_3A  = -0.23168619521896736 / Coordinate transformation matrix element
 116 | PC3_1A  =                  0.0 / Coordinate transformation matrix element
 117 | PC3_2A  =  0.23168619521896736 / Coordinate transformation matrix element
 118 | PC3_3A  =   0.9727905771259087 / Coordinate transformation matrix element
 119 | CDELT1A =                  1.0 / [deg] Coordinate increment at reference point
 120 | CDELT2A =             6.25E-06 / [deg] Coordinate increment at reference point
 121 | CDELT3A =             6.25E-06 / [deg] Coordinate increment at reference point
 122 | CUNIT1A = 'deg     '           / Units of coordinate increment and value
 123 | CUNIT2A = 'deg     '           / Units of coordinate increment and value
 124 | CUNIT3A = 'deg     '           / Units of coordinate increment and value
 125 | CTYPE1A = 'STOKES  '           / Coordinate type
 126 | CTYPE2A = 'RA-ARC  '           / Coordinate type codezenithal/azimuthal equidist
 127 | CTYPE3A = 'DEC-ARC '           / Coordinate type codezenithal/azimuthal equidist
 128 | CRVAL1A =                  0.0 / [deg] Coordinate value at reference point
 129 | CRVAL2A =   167.46361044314384 / [deg] Coordinate value at reference point
 130 | CRVAL3A =    7.217469536789277 / [deg] Coordinate value at reference point
 131 | LONPOLEA=                180.0 / [deg] Native longitude of celestial pole
 132 | LATPOLEA=                  0.0 / [deg] Native latitude of celestial pole
 133 | CNAME1A = 'STOKES  '           / Axis name for labelling purposes
 134 | MJDREFA =                  0.0 / [d] MJD of fiducial time
 135 | COMMENT ----- Camera and Readout State -----------------------------------------
 136 | DETECTOR= 'CCD 2Kx2K'          / CCD detector 2048x2048 pixels
 137 | EXPTIME =                  3.0 / [s] commanded exposure time
 138 | OBT_BEG =          317813545.8 / MET when first pixel of first image received
 139 | OBT_END =          317813586.4 / MET when first pixel of last image is received
 140 | NSUMEXP =                    3 / total number images stacked on onboard
 141 | NSUMBAD =                    0 / number of images with invalid pixel counts
 142 | READTIME=               2.4576 / [s] time to readout raw image from camera
 143 | TELAPSE =              48.1296 / [s] time between begin and end of exposure
 144 | REGION  =                    1 / region
 145 | READOUT0=                    3 / readout region if region=0
 146 | CAMERA  = 'FMCFMD  '           / detector name (TBD)
 147 | PXBEG1  =                    1 / first read-out detector row
 148 | PXEND1  =                 2048 / last read-out detector row
 149 | PXBEG2  =                    1 / first read-out detector column
 150 | PXEND2  =                 2048 / last read-out detector column
 151 | NBIN1   =                    1 / number of pixels binned along axis1
 152 | NBIN2   =                    1 / number of pixels binned along axis2
 153 | NBIN    =                    1 / number of pixels binned
 154 | DSTART1 =                    1 / first column of image array on data array
 155 | DSTOP1  =                 2048 / last column of image array on data array
 156 | DSTART2 =                    1 / first row of image area on data array
 157 | DSTOP2  =                 2048 / last row of image area on data array
 158 | IMGCTR  =                 8370 / image counter from IDPU
 159 | LEDSTATE= 'Off     '           / state of LED (TBD)
 160 | LEDDAC  =                    0 / last commanded setting of LED
 161 | OFFSET  =                   40 / commanded offset value used in camera
 162 | GAINCMD =                 12.0 / commanded gain value
 163 | XFBYTES =              1374028 / number of bytes sent
 164 | COMMENT ----- Onboard Image Processing -----------------------------------------
 165 | RAWBITS =                   16 / [bits] raw image bits per pixel
 166 | COMPBITS=                   10 / [bits] compressed image bits per pixel
 167 | SQRTCODE= '' / square-root coding table name
 168 | COMPRESS= '' / image compression type
 169 | COMP_RAT=                 -1.0 / compressed/uncompressed data ratio
 170 | COMMENT ----- Image Statistics and Properties ----------------------------------
 171 | BUNIT   = 'Mean Solar Brightness' / Units of observation
 172 | BSUN_DEF=           20090000.0 / [W/m2/sr] Mean Solar Brightness
 173 | DATAZER =                    0 / number of pixels=0
 174 | DATASAT =                    0 / number of saturated pixels
 175 | DSATVAL =              65535.0 / saturation value in calibrated units
 176 | DATAAVG =                  1.0 / average non-zero pixel value
 177 | DATAMDN =                  1.0 / median non-zero pixel value
 178 | DATASIG =                  0.0 / standard deviation of non-zero pixels
 179 | DATAP01 =                  1.0 / 1% of non-zero pixels are LE this value
 180 | DATAP10 =                  1.0 / 10% of non-zero pixels are LE this value
 181 | DATAP25 =                  1.0 / 25% of non-zero pixels are LE this value
 182 | DATAP50 =                  1.0 / 50% of non-zero pixels are LE this value
 183 | DATAP75 =                  1.0 / 75% of non-zero pixels are LE this value
 184 | DATAP90 =                  1.0 / 90% of non-zero pixels are LE this value
 185 | DATAP95 =                  1.0 / 95% of non-zero pixels are LE this value
 186 | DATAP98 =                  1.0 / 98% of non-zero pixels are LE this value
 187 | DATAP99 =                  1.0 / 99% of non-zero pixels are LE this value
 188 | DATAMIN =                  1.0 / minimum valid physical value
 189 | DATAMAX =                  1.0 / maximum valid physical value
 190 | COMMENT ----- Solar Reference Data ---------------------------------------------
 191 | RSUN_ARC=    973.6616658871211 / [arcsec] photospheric solar radius
 192 | RSUN_REF=          695700000.0 / [m] assumed physical solar radius
 193 | SOLAR_EP=    13.39636545570722 / [deg] S/C ecliptic North to solar North angle
 194 | CAR_ROT =   2278.4904405242123 / Carrington Rotation number
 195 | COMMENT ----- Fixity -----------------------------------------------------------
 196 | CHECKSUM= '1ref4ocZ1ocf1ocZ'   / HDU checksum updated 2023-12-07T17:15:34
 197 | DATASUM = '178261149'          / data unit checksum updated 2023-12-07T17:15:34
 198 | ZHECKSUM= 'loIMmmGKlmGKlmGK'   / HDU checksum updated 2023-12-07T17:15:34
 199 | ZDATASUM= '8323072 '           / data unit checksum updated 2023-12-07T17:15:34
 200 | COMMENT ----- History ----------------------------------------------------------
 201 | HISTORY Records of processing from pipeline
 202 | END     
 
 202 header keywords
 
 PRIMARY DATA ARRAY  (4 columns x 8192 rows)
 
 Col# Name (Units)       Format
   1 COMPRESSED_DATA      1PB(0)    
   2 GZIP_COMPRESSED_DATA 1PB(55)   
   3 ZSCALE               1D        
   4 ZZERO                1D        
 
=================== HDU 3: BINARY Table ====================
 
   1 | XTENSION= 'BINTABLE'           / binary table extension
   2 | BITPIX  =                    8 / array data type
   3 | NAXIS   =                    2 / number of array dimensions
   4 | NAXIS1  =                    8 / width of table in bytes
   5 | NAXIS2  =                 8192 / number of rows in table
   6 | PCOUNT  =               401408 / number of group parameters
   7 | GCOUNT  =                    1 / number of groups
   8 | TFIELDS =                    1 / number of fields in each row
   9 | BSCALE  =                    1
  10 | BZERO   =                 -128
  11 | TTYPE1  = 'COMPRESSED_DATA'    / label for field 1
  12 | TFORM1  = '1PB(49) '           / data format of field: variable length array
  13 | ZIMAGE  =                    T / extension contains compressed image
  14 | ZTENSION= 'IMAGE   '           / Image extension
  15 | ZBITPIX =                    8 / data type of original image
  16 | ZNAXIS  =                    3 / dimension of original image
  17 | ZNAXIS1 =                 4096 / length of original image axis
  18 | ZNAXIS2 =                 4096 / length of original image axis
  19 | ZNAXIS3 =                    2 / length of original image axis
  20 | ZPCOUNT =                    0 / number of parameters
  21 | ZGCOUNT =                    1 / number of groups
  22 | ZTILE1  =                 4096 / size of tiles to be compressed
  23 | ZTILE2  =                    1 / size of tiles to be compressed
  24 | ZTILE3  =                    1 / size of tiles to be compressed
  25 | ZCMPTYPE= 'RICE_1  '           / compression algorithm
  26 | ZNAME1  = 'BLOCKSIZE'          / compression block size
  27 | ZVAL1   =                   32 / pixels per block
  28 | ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)
  29 | ZVAL2   =                    1 / bytes per pixel (1, 2, 4, or 8)
  30 | EXTNAME = 'UNCERTAINTY ARRAY'  / extension name
  31 | WCSAXES =                    3
  32 | CRPIX1  =                  0.0
  33 | CRPIX2  =               2048.5
  34 | CRPIX3  =               2048.5
  35 | CDELT1  =                  1.0
  36 | CDELT2  =             6.25E-06
  37 | CDELT3  =             6.25E-06
  38 | CUNIT1  = 'deg     '
  39 | CUNIT2  = 'deg     '
  40 | CUNIT3  = 'deg     '
  41 | CTYPE1  = 'STOKES  '
  42 | CTYPE2  = 'HPLN-ARC'
  43 | CTYPE3  = 'HPLT-ARC'
  44 | CRVAL1  =                  0.0
  45 | CRVAL2  =                  0.0
  46 | CRVAL3  =                  0.0
  47 | LONPOLE =                180.0
  48 | LATPOLE =                  0.0
  49 | CNAME1  = 'STOKES  '
  50 | MJDREF  =                  0.0
  51 | MJD-BEG =      60286.010782011
  52 | MJD-AVG =        60286.0121709
  53 | MJD-END =      60286.013559789
  54 | WCSAXESA=                    3
  55 | CRPIX1A =                  0.0
  56 | CRPIX2A =               2048.5
  57 | CRPIX3A =               2048.5
  58 | CDELT1A =                  1.0
  59 | CDELT2A =             6.25E-06
  60 | CDELT3A =             6.25E-06
  61 | CUNIT1A = 'deg     '
  62 | CUNIT2A = 'deg     '
  63 | CUNIT3A = 'deg     '
  64 | CTYPE1A = 'STOKES  '
  65 | CTYPE2A = 'RA-ARC  '
  66 | CTYPE3A = 'DEC-ARC '
  67 | CRVAL1A =                  0.0
  68 | CRVAL2A =   167.46361044314384
  69 | CRVAL3A =    7.217469536789277
  70 | LONPOLEA=                180.0
  71 | LATPOLEA=                  0.0
  72 | CNAME1A = 'STOKES  '
  73 | MJDREFA =                  0.0
  74 | MJD-BEGA=      60286.010782011
  75 | MJD-AVGA=        60286.0121709
  76 | MJD-ENDA=      60286.013559789
  77 | PC2_2A  =   0.9727905771259087
  78 | PC2_3A  = -0.23168619521896736
  79 | PC3_2A  =  0.23168619521896736
  80 | PC3_3A  =   0.9727905771259087
  81 | RSUN_ARC=    973.6616658871211
  82 | SOLAR_EP=    13.39636545570722
  83 | CAR_ROT =   2278.4904405242123
  84 | ZHECKSUM= '5aMdAUMc8ZMcAZMc'   / HDU checksum updated 2023-12-07T17:15:34
  85 | ZDATASUM= '3233857728'         / data unit checksum updated 2023-12-07T17:15:34
  86 | CHECKSUM= 'ApEACmB6AmBAAmB5'   / HDU checksum updated 2023-12-07T17:15:34
  87 | DATASUM = '1846483980'         / data unit checksum updated 2023-12-07T17:15:34
  88 | END     
 
 88 header keywords
 
 UNCERTAINTY ARRAY  (1 columns x 8192 rows)
 
 Col# Name (Units)       Format
   1 COMPRESSED_DATA      1PB(49)   
 
++++++++++++++++++++++ Error Summary  ++++++++++++++++++++++
 
 HDU#  Name (version)       Type             Warnings  Errors
 1                          Primary Array    0         0     
 2     PRIMARY DATA ARRAY   Binary Table     0         0     
 3     UNCERTAINTY ARRAY    Binary Table     0         0     
 
**** Verification found 0 warning(s) and 0 error(s). ****

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lowderchris lowderchris

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve wcs metadata handling punch-mission/punchbowl
2 participants
@lowderchris @jmbhughes