DSS2Red FITS file is corrupt

[cdeil]

@tboch - I was trying to add a test for HiPS Allsky.fits at
https://github.com/hipspy/hips/blob/master/hips/tiles/tests/test_allsky.py#L10
and got this error trying to read datasets/samples/DSS2Red/Norder3/Allsky.fits :

>>> from io import BytesIO
>>> from pathlib import Path
>>> from astropy.io import fits
>>> raw_data = Path('datasets/samples/DSS2Red/Norder3/Allsky.fits').read_bytes()
>>> fits.open(BytesIO(raw_data))
>>> WARNING: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters [astropy.io.fits.py3compat]
WARNING: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters [astropy.io.fits.py3compat]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/hdulist.py", line 158, in fitsopen
    lazy_load_hdus, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/hdulist.py", line 411, in fromfile
    lazy_load_hdus=lazy_load_hdus, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/hdulist.py", line 1021, in _readfrom
    read_one = hdulist._read_next_hdu()
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/hdulist.py", line 1096, in _read_next_hdu
    hdu = _BaseHDU.readfrom(fileobj, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/base.py", line 332, in readfrom
    **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/hdu/base.py", line 397, in _readfrom_internal
    header = Header.fromfile(data, endcard=not ignore_missing_end)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/header.py", line 449, in fromfile
    padding)[1]
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/fits/header.py", line 518, in _from_blocks
    raise IOError('Header missing END card.')
OSError: Header missing END card.

ftverify also says that this is not a valid FITS file:

$ ftverify datasets/samples/DSS2Red/Norder3/Allsky.fits 
 
               ftverify 4.17 (CFITSIO V3.370)               
               ------------------------------               
 
HEASARC conventions are being checked.
 
File: datasets/samples/DSS2Red/Norder3/Allsky.fits

1 Header-Data Units in this file.
 
=================== HDU 1: Primary Array ===================
 
*** Error:   Keyword #6, BLANK: value = -32768.0 is not an integer.
*** Error:   checking data fill: tried to move past end of file
 
 7 header keywords
 
 16-bit integer pixels,  2 axes (1728 x 1856), 
 
++++++++++++++++++++++ Error Summary  ++++++++++++++++++++++
 
 HDU#  Name (version)       Type             Warnings  Errors
 1                          Primary Array    0         2     
 
**** Verification found 0 warning(s) and 2 error(s). ****

I think ./datasets/samples/IRAC4/Norder3/Allsky.fits might be OK, I'll try to continue with that now.

@tboch - is this something CDS can fix in the coming months, i.e. re-generate FITS files to be valid FITS files?

Or do we need to implement a workaround in our FITS reader? (when I use fits.open passing directly the filename it seems to work, i.e. it must also be somehow possible to extract the data numpy array from the BytesIO object (but I don't know how).

[tboch]

@cdeil actually, it seems that some of the Allsky.fits files are actually gzipped!
I'm going to investigate further.

[cdeil]

@cdeil actually, it seems that some of the Allsky.fits files are actually gzipped!
I'm going to investigate further.

@tboch - Any news / conclusions?

If indeed the issue is just that the files are gzipped and there won't be a change in Allsky.fits maps at CDS, we could just do a try-except and try to unzip, i.e. make it work in the hips Python package now.

[tboch]

We will update these tiles, as to conform with the HiPS standard. I don't have a proper timescale for this yet, but my suggestion would be not to support the gzipped files in our hips package, unless you have an urgent need to access one of these HiPS.

[cdeil]

We will update these tiles, as to conform with the HiPS standard. I don't have a proper timescale for this yet, but my suggestion would be not to support the gzipped files in our hips package, unless you have an urgent need to access one of these HiPS.

If the change happens in the coming months, I think that's fine. I think (true?) currently we never load allsky images for drawing in the hips package, so users won't encounter a problem.
My motivation for adding the Allsky class was just because I wanted to generate my own HiPS, which I can as-is.

[cdeil]

@tboch - Status?

I'd suggest we close issues here on the https://github.com/hipspy/hips-extra/issues/ and disable that issue tracker and only use https://github.com/hipspy/hips/issues/ . Two issue trackers is too confusing.