Error about missing key when reading CR4 file

[mwest007]

I created a Jupyter notebook using:

# we can load data in two wayas, 

# load in a template using the load_template
meta = NormalizedMetadata.load_template("CR4", "1")

# make the date time now
meta['DATE-OBS'] = str(datetime.now())

# set the wcs
wcs = WCS(naxis=2)

# create a PUNCHdata object using meta, wcs, data
obj = PUNCHData(data, wcs, meta)

# write a fits file using thr object
file_path = obj.filename_base + ".fits"

obj.write(file_path, overwrite=True, skip_wcs_conversion=True)

--- When opening the generated file:

reloaded = PUNCHData.from_fits(file_path)

---- it generates this error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[44], line 2
      1 # try loading with PUNCHdata object and astropy
----> 2 reloaded = PUNCHData.from_fits(file_path)
      3 reloaded_using_fits =  fits.open(file_path)
      4 reloaded_using_fits_hdu = reloaded_using_fits[1]

File [~/work/software/python/punchbowl/punchbowl/data.py:918](http://localhost:8888/~/work/software/python/punchbowl/punchbowl/data.py#line=917), in PUNCHData.from_fits(cls, path, key)
    916 header["CHECKSUM"] = ""
    917 header["DATASUM"] = ""
--> 918 meta = NormalizedMetadata.from_fits_header(header)
    919 wcs = WCS(header, hdul, key=key)
    920 unit = u.ct

File [~/work/software/python/punchbowl/punchbowl/data.py:468](http://localhost:8888/~/work/software/python/punchbowl/punchbowl/data.py#line=467), in NormalizedMetadata.from_fits_header(cls, h)
    466     if k not in ("COMMENT", "HISTORY"):
    467         if k not in m:
--> 468             raise RuntimeError(
    469                 f"Unexpected key of {k} found in header for Level" f"{level} {type_code + obs_code} type meta."
    470             )
    471         m[k] = v
    472 m.history = History.from_fits_header(h)

RuntimeError: Unexpected key of  found in header for Level1 CR4 type meta.

[jmbhughes]

I'm unable to recreate this error. Let's make a test to see if we can check for things like this automatically.

[jmbhughes]

@mwest007 and @lowderchris since you both got this error, can you put the output of pip list here so I can check versions?

[lowderchris]

Absolutely! Find that below for the virtual environment I was running the notebook from the sdf in:

(.venv) clowder@lowdermac sdf % pip list
Package                   Version
------------------------- -------------------
aioftp                    0.22.3
aiohttp                   3.9.3
aiosignal                 1.3.1
aiosqlite                 0.20.0
alembic                   1.13.1
annotated-types           0.7.0
anyio                     3.7.1
appnope                   0.1.4
apprise                   1.8.0
asciitree                 0.3.3
asdf                      3.1.0
asdf-astropy              0.6.0
asdf_coordinates_schemas  0.3.0
asdf_standard             1.1.1
asdf_transform_schemas    0.5.0
asdf_unit_schemas         0.2.0
asdf_wcs_schemas          0.4.0
asgi-lifespan             2.1.0
asteval                   0.9.32
astropy                   6.0.0
astropy_healpix           1.0.3
astropy-iers-data         0.2024.3.11.18.33.3
asttokens                 2.4.1
asyncpg                   0.29.0
attrs                     23.2.0
beautifulsoup4            4.12.3
bs4                       0.0.2
cachetools                5.3.3
ccsdspy                   1.2.1
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
cloudpickle               3.0.0
colorama                  0.4.6
comm                      0.2.2
contourpy                 1.2.0
coolname                  2.2.0
croniter                  2.0.5
cryptography              42.0.7
cycler                    0.12.1
Cython                    3.0.10
dask                      2024.5.1
dateparser                1.2.0
debugpy                   1.8.1
decorator                 5.1.1
dill                      0.3.8
dnspython                 2.6.1
docker                    6.1.3
drms                      0.7.1
email_validator           2.1.1
executing                 2.0.1
fasteners                 0.19
fonttools                 4.49.0
frozenlist                1.4.1
fsspec                    2024.5.0
future                    1.0.0
google-auth               2.29.0
graphviz                  0.20.3
greenlet                  3.0.3
griffe                    0.45.1
gwcs                      0.21.0
h11                       0.14.0
h2                        4.1.0
h5py                      3.11.0
hpack                     4.0.0
httpcore                  1.0.5
httpx                     0.27.0
humanize                  4.9.0
hyperframe                6.0.1
idna                      3.6
imageio                   2.34.1
importlib_metadata        7.0.2
importlib_resources       6.1.3
ipykernel                 6.29.3
ipython                   8.22.2
isodate                   0.6.1
itsdangerous              2.2.0
jedi                      0.19.1
Jinja2                    3.1.4
jinja2-humanize-extension 0.4.0
jmespath                  1.0.1
jsonpatch                 1.33
jsonpointer               2.4
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
jupyter_client            8.6.1
jupyter_core              5.7.2
kiwisolver                1.4.5
kubernetes                29.0.0
lazy_loader               0.4
lmfit                     1.3.1
locket                    1.0.0
lxml                      5.2.2
Mako                      1.3.5
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.8.3
matplotlib-inline         0.1.6
mdurl                     0.1.2
multidict                 6.0.5
ndcube                    2.2.0
nest-asyncio              1.6.0
networkx                  3.3
numcodecs                 0.12.1
numpy                     1.26.4
oauthlib                  3.2.2
orjson                    3.10.3
packaging                 24.0
pandas                    2.2.2
parfive                   2.0.2
parso                     0.8.3
partd                     1.4.2
pathspec                  0.12.1
pendulum                  3.0.0
pexpect                   4.9.0
pillow                    10.2.0
pip                       24.0
platformdirs              4.2.0
prefect                   2.19.1
prompt-toolkit            3.0.43
psutil                    5.9.8
ptyprocess                0.7.0
punchbowl                 0.0.1
pure-eval                 0.2.2
pyasn1                    0.6.0
pyasn1_modules            0.4.0
pycparser                 2.22
pydantic                  2.7.1
pydantic_core             2.18.2
pyerfa                    2.0.1.1
Pygments                  2.17.2
pylibjpeg                 2.0.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-multipart          0.0.9
python-slugify            8.0.4
pytz                      2024.1
PyYAML                    6.0.1
pyzmq                     25.1.2
readchar                  4.1.0
referencing               0.35.1
regex                     2024.5.15
regularizepsf             0.3.2
reproject                 0.13.1
requests                  2.32.1
requests-file             2.1.0
requests-oauthlib         2.0.0
requests-toolbelt         1.0.0
rfc3339-validator         0.1.4
rich                      13.7.1
rpds-py                   0.18.1
rsa                       4.9
ruamel.yaml               0.18.6
ruamel.yaml.clib          0.2.8
scikit-image              0.23.2
scipy                     1.12.0
semantic-version          2.10.0
sep                       1.2.1
setuptools                70.0.0
shellingham               1.5.4
six                       1.16.0
sniffio                   1.3.1
solpolpy                  0.1.2
soupsieve                 2.5
SQLAlchemy                2.0.30
stack-data                0.6.3
sunkit-image              0.5.1
sunpy                     5.1.1
text-unidecode            1.3
tifffile                  2024.5.10
time-machine              2.14.1
toml                      0.10.2
toolz                     0.12.1
tornado                   6.4
tqdm                      4.66.2
traitlets                 5.14.2
typer                     0.12.3
typing_extensions         4.11.0
tzdata                    2024.1
tzlocal                   5.2
ujson                     5.10.0
uncertainties             3.1.7
urllib3                   2.2.1
uvicorn                   0.28.1
wcwidth                   0.2.13
websocket-client          1.8.0
websockets                12.0
yarl                      1.9.4
zarr                      2.18.1
zeep                      4.2.1
zipp                      3.18.1

[lowderchris]

This test I cobbled together, directly inside the punchbowl current main branch, seemed to run alright:

def test_generate_write_read():
    meta = NormalizedMetadata.load_template("CR4", "1")
    meta['DATE-OBS'] = str(datetime.now())
    data = np.random.random((10,10))
    wcs = WCS(naxis=2)

    write_obj = PUNCHData(data, wcs, meta)

    filepath = 'test_generate_write_read.fits'

    write_obj.write(filepath, overwrite=True, skip_wcs_conversion=True)

    read_obj = PUNCHData.from_fits(filepath)

    assert isinstance(write_obj, PUNCHData)
    assert isinstance(read_obj, PUNCHData)

[mwest007]

Matt's
Package Version Editable project location

---

aioftp 0.22.3
aiohttp 3.9.5
aiosignal 1.3.1
aiosqlite 0.20.0
alembic 1.13.1
annotated-types 0.6.0
anyio 3.7.1
appnope 0.1.4
apprise 1.7.6
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asciitree 0.3.3
asdf 3.2.0
asdf-astropy 0.6.1
asdf_coordinates_schemas 0.3.0
asdf_standard 1.1.1
asdf_transform_schemas 0.5.0
asdf_wcs_schemas 0.4.0
asgi-lifespan 2.1.0
asteval 0.9.32
astropy 6.0.1
astropy_healpix 1.0.3
astropy-iers-data 0.2024.4.22.0.29.50
asttokens 2.4.1
async-lru 2.0.4
async-timeout 4.0.3
asyncpg 0.29.0
attrs 23.2.0
Babel 2.14.0
beautifulsoup4 4.12.3
bleach 6.1.0
cachetools 5.3.3
ccsdspy 1.2.1
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
colorama 0.4.6
comm 0.2.2
contourpy 1.2.1
coolname 2.2.0
croniter 2.0.5
cryptography 42.0.5
cycler 0.12.1
Cython 3.0.10
dask 2024.4.2
dateparser 1.2.0
debugpy 1.8.1
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.8
dnspython 2.6.1
docker 6.1.3
drms 0.7.1
email_validator 2.1.1
executing 2.0.1
fasteners 0.19
fastjsonschema 2.19.1
fonttools 4.51.0
fqdn 1.5.1
frozenlist 1.4.1
fsspec 2024.3.1
future 1.0.0
google-auth 2.29.0
graphviz 0.20.3
greenlet 3.0.3
griffe 0.44.0
gwcs 0.21.0
h11 0.14.0
h2 4.1.0
h5py 3.11.0
hpack 4.0.0
httpcore 1.0.5
httpx 0.27.0
humanize 4.9.0
hyperframe 6.0.1
idna 3.7
imageio 2.34.1
importlib_metadata 7.1.0
importlib_resources 6.1.3
ipykernel 6.29.4
ipython 8.24.0
ipywidgets 8.1.2
isodate 0.6.1
isoduration 20.11.0
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.3
jinja2-humanize-extension 0.4.0
jmespath 1.0.1
json5 0.9.25
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter 1.0.0
jupyter_client 8.6.1
jupyter-console 6.6.3
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.5
jupyter_server 2.14.0
jupyter_server_terminals 0.5.3
jupyterlab 4.1.7
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.1
jupyterlab_widgets 3.0.10
kiwisolver 1.4.5
kubernetes 29.0.0
lazy_loader 0.4
lmfit 1.3.1
locket 1.0.0
lxml 5.2.2
Mako 1.3.3
Markdown 3.6
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.8.4
matplotlib-inline 0.1.7
mdurl 0.1.2
mistune 3.0.2
multidict 6.0.5
nbclient 0.10.0
nbconvert 7.16.3
nbformat 5.10.4
ndcube 2.2.0
nest-asyncio 1.6.0
networkx 3.3
notebook 7.1.3
notebook_shim 0.2.4
numcodecs 0.12.1
numpy 1.26.4
oauthlib 3.2.2
orjson 3.10.1
overrides 7.7.0
packaging 24.0
pandas 2.2.2
pandocfilters 1.5.1
parfive 2.1.0
parso 0.8.4
partd 1.4.1
pathspec 0.12.1
pendulum 2.1.2
pexpect 4.9.0
pillow 10.3.0
pip 24.0
platformdirs 4.2.1
prefect 2.18.1
prometheus_client 0.20.0
prompt-toolkit 3.0.43
psutil 5.9.8
ptyprocess 0.7.0
punchbowl 0.0.1 /Users/mwest/work/software/python/punchbowl
pure-eval 0.2.2
pyasn1 0.6.0
pyasn1_modules 0.4.0
pycparser 2.22
pydantic 2.7.1
pydantic_core 2.18.2
pyerfa 2.0.1.4
Pygments 2.17.2
pyparsing 3.1.2
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
python-multipart 0.0.9
python-slugify 8.0.4
pytz 2024.1
pytzdata 2020.1
PyYAML 6.0.1
pyzmq 26.0.2
qtconsole 5.5.1
QtPy 2.4.1
readchar 4.0.6
referencing 0.35.0
regex 2024.4.16
regularizepsf 0.3.2
reproject 0.13.1
requests 2.31.0
requests-file 2.0.0
requests-oauthlib 2.0.0
requests-toolbelt 1.0.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.7.1
rpds-py 0.18.0
rsa 4.9
ruamel.yaml 0.18.6
ruamel.yaml.clib 0.2.8
scikit-image 0.23.2
scipy 1.13.0
semantic-version 2.10.0
Send2Trash 1.8.3
sep 1.2.1
setuptools 67.6.1
shellingham 1.5.4
six 1.16.0
sniffio 1.3.1
soupsieve 2.5
SQLAlchemy 2.0.29
stack-data 0.6.3
sunpy 5.1.2
terminado 0.18.1
text-unidecode 1.3
tifffile 2024.4.24
tinycss2 1.3.0
toml 0.10.2
toolz 0.12.1
tornado 6.4
tqdm 4.66.2
traitlets 5.14.3
typer 0.12.3
types-python-dateutil 2.9.0.20240316
typing_extensions 4.11.0
tzdata 2024.1
tzlocal 5.2
ujson 5.9.0
uncertainties 3.1.7
uri-template 1.3.0
urllib3 2.2.1
uvicorn 0.28.1
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.8.0
websockets 12.0
widgetsnbextension 4.0.10
yarl 1.9.4
zarr 2.17.2
zeep 4.2.1
zipp 3.18.1

[jmbhughes]

Seems to have something to do with float versus int for STEREO data but it's unclear why

[jmbhughes]

The problem was that BSCALE and BZERO get converted to empty header keys when being converted, so I just ignore that case now. It won't come up for PUNCH anyway really.