Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CoDICE decompression algorithms for science data #278

Merged

Conversation

bourque
Copy link
Collaborator

@bourque bourque commented Nov 14, 2023

Change Summary

Overview

This PR implements the six possible CoDICE decompression algorithms for science data, with supporting unit tests. The algorithms are:

  1. No compression
  2. Lossy compression A (lookup table)
  3. Lossy compression B (lookup table)
  4. Lossless compression (LZMA)
  5. Lossy A + Lossless
  6. Lossy B + Lossless

This PR closes #249 , however there will likely need to be some tweaks down the line to get this working with the greater CoDICE L0-L1A decom algorithm

New Dependencies

None

New Files

  • imap_processing/codice/l0/decompress_codice.py
    • Contains methods that implement the decompression algorithms
  • imap_processing/codice/tests/test_decompress_codice.py
    • Parameterized unit tests for decompress_codice.py

Deleted Files

None

Updated Files

  • imap_processing/codice/l0/__init__.py
    • This is where I decided to store the Lossy compression lookup tables

Testing

test_decompress_codice.py contains a unit test for each of the 6 algorithms using an input value of 234 (I picked this randomly), as well as an attempt to use an unsupported algorithm, which should result in a decompressed value of None.

@bourque bourque added Ins: CoDICE Related to the CoDICE instrument Level: L1 Level 1 processing labels Nov 14, 2023
@bourque bourque requested a review from a team November 14, 2023 19:51
@bourque bourque self-assigned this Nov 14, 2023
@bourque bourque requested review from sdhoyt, greglucas, tech3371, bryan-harter, laspsandoval, GFMoraga and maxinelasp and removed request for a team November 14, 2023 19:51
@bourque bourque force-pushed the codice-decompression branch from 571521a to 1682eae Compare November 14, 2023 19:57
Copy link
Contributor

@tech3371 tech3371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

imap_processing/codice/l0/decompress_codice.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decompress_codice.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@greglucas greglucas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Especially the comments and tests.

imap_processing/codice/l0/__init__.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/__init__.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decompress_codice.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decompress_codice.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decompress_codice.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/__init__.py Outdated Show resolved Hide resolved
imap_processing/codice/tests/test_decompress_codice.py Outdated Show resolved Hide resolved
imap_processing/codice/tests/test_decompress_codice.py Outdated Show resolved Hide resolved
@bourque bourque force-pushed the codice-decompression branch from 1682eae to acb9487 Compare November 15, 2023 20:46
@bourque bourque force-pushed the codice-decompression branch from e6d6407 to cbfea18 Compare November 15, 2023 21:04
Copy link
Contributor

@tech3371 tech3371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great!

Copy link
Contributor

@sdhoyt sdhoyt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code was weirdly pleasant to look at lol everything looks great!

decompressed_value : int
The 24- or 32-bit decompressed value
"""
if algorithm == CoDICECompression.NO_COMPRESSION:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could pass a function into this method instead of this if/else table. So, you could pass in _apply_lossy_a as algorithm and then this whole table could just be algorithm(decompress). Not sure if that makes sense for the rest of your code, but it could be an interesting option to reduce the number of functions and get rid of that enum.

https://stackoverflow.com/questions/1349332/python-passing-a-function-into-another-function

Copy link
Contributor

@maxinelasp maxinelasp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left one suggestion that probably isn't a good match for this piece of code, but might help with later development based on this style

Copy link
Collaborator

@greglucas greglucas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit on the xfail/raises test. But looks good to me!

@bourque bourque force-pushed the codice-decompression branch from cbfea18 to e8921ab Compare November 20, 2023 18:09
Copy link
Contributor

@GFMoraga GFMoraga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful work. Everything looks great. I thought your last commit was a good catch. I think its ready to merge, and at a good spot whenever updates need to happen. (hopefully not anytime soon or too monstrous of changes)

@bourque bourque merged commit 0675991 into IMAP-Science-Operations-Center:dev Nov 20, 2023
14 checks passed
@bourque bourque deleted the codice-decompression branch November 20, 2023 21:37
laspsandoval pushed a commit to laspsandoval/imap_processing that referenced this pull request Apr 2, 2024
…dice-decompression

CoDICE decompression algorithms for science data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ins: CoDICE Related to the CoDICE instrument Level: L1 Level 1 processing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create module for decompressing CoDICE science data
6 participants