Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Housekeeping data testing and validation for CoDICE l0 #176

Merged
merged 25 commits into from
Oct 25, 2023
Merged

Housekeeping data testing and validation for CoDICE l0 #176

merged 25 commits into from
Oct 25, 2023

Conversation

GFMoraga
Copy link
Contributor

@GFMoraga GFMoraga commented Sep 28, 2023

Change Summary

This is the first pass at l0 decom for codice- specifically NHK, which is the raw housekeeping data. This data has been validated. Testing is done in this PR and it validates against a csv we recieved from the instrument team at swRi.

Overview

Updated the decom code in test_decom.py
Added testing code that validates SHCOARSE

@GFMoraga GFMoraga added Ins: CoDICE Related to the CoDICE instrument Level: L0 Level 0 processing labels Sep 28, 2023
@GFMoraga GFMoraga self-assigned this Sep 28, 2023
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/housekeeping_data.bin Outdated Show resolved Hide resolved
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
imap_processing/codice/l0/decom_nhk.py Outdated Show resolved Hide resolved
@bourque bourque changed the title First try at pkt NHK(housekeeping) decom for l0 First try at pkt NHK(housekeeping) decom for CoDICE l0 Sep 29, 2023
@bourque
Copy link
Collaborator

bourque commented Sep 29, 2023

I added an initial pass at some unit tests (basically took what @tech3371 has for SWE). We will have to think about what to add to validate CoDICE housekeeping data specifically, but this adds in the basic testing structure for now.

@bourque bourque changed the title First try at pkt NHK(housekeeping) decom for CoDICE l0 NHK(housekeeping) testing and validation for CoDICE l0 [WIP] Oct 3, 2023
@bourque bourque marked this pull request as draft October 3, 2023 16:27
@bourque bourque changed the title NHK(housekeeping) testing and validation for CoDICE l0 [WIP] Housekeeping data testing and validation for CoDICE l0 [WIP] Oct 3, 2023
@bourque bourque added Level: L1 Level 1 processing and removed Level: L0 Level 0 processing labels Oct 10, 2023
@bourque bourque added this to the Decom and L1A Complete milestone Oct 10, 2023
@bourque bourque mentioned this pull request Oct 10, 2023
10 tasks
@GFMoraga GFMoraga marked this pull request as draft October 23, 2023 16:17
@GFMoraga GFMoraga marked this pull request as ready for review October 25, 2023 17:53
@GFMoraga GFMoraga requested review from bourque and removed request for sdhoyt, joeymukherjee, bryan-harter, laspsandoval and maxinelasp October 25, 2023 21:04
@GFMoraga
Copy link
Contributor Author

I had to change a few things that where outside the general change summary. I had to update the conf.py due to an external issue. Names where changed and data was swapped.

Copy link
Collaborator

@bourque bourque left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work here @GFMoraga!

@GFMoraga GFMoraga merged commit 0c6fcea into IMAP-Science-Operations-Center:dev Oct 25, 2023
12 checks passed
@GFMoraga GFMoraga deleted the l0_decom branch October 25, 2023 21:42
laspsandoval pushed a commit to laspsandoval/imap_processing that referenced this pull request Nov 15, 2023
…Operations-Center#176)

Validated housekeeping data with tests plus minor updates

---------

Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>
maxinelasp pushed a commit to maxinelasp/imap_processing that referenced this pull request Nov 16, 2023
…Operations-Center#176)

Validated housekeeping data with tests plus minor updates

---------

Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>
maxinelasp added a commit that referenced this pull request Jan 8, 2024
* update telemetry_generator to be more generic

* Updates to telemetry generator

* updating docs

* updating to remove local-specific testing changes

* first pass at glows xtce

* update way of checking if description exists or not. update generator template.

* updated generator to use correct IntegerParameterType

* First pass at GLOWS decom

* updating telemetry generator

* Updating glows decom to properly process data into dictionary

* Finished histogram decom

* Updating decom to include direct events

* Finalizing code, adding smaller test data file and combined XML, updating documentation and tests

* Updating test file path

* Fixing test runs

* updating docs

* Updating data classes to use attributes, and changing names

* Removing keys from data class and getting attribute names from the packet

* Move INT to SINT in telemetry_generator

Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com>

* Addressing code review comments

* Fixing test failure

* Review updates

* fixing typo

* Updating IntEnum to Enum to fix automatic documentation

* Move L0 data classes to a dataclass type with specific attributes

* Initial GLOWS L1A data classes

* Adding required attributes to L0 data class

* Adding L1A histogram data class

* fixing test failures

* Finishing ccsds dataclass

* Minor updates

* Finishing GLOWS L1A histograms

* Updating to include direct events data

* Updating glows l1a

* Adding useful GLOWS code from GLOWS team and Marek Strumik

* Adding direct event information

Co-authored-by: GLOWS team <maro@cbk.waw.pl>

* Updates for direct event creation

* Updates to L1A direct events

* Updated release workflow to reflect removal of "main" branch and use of version branches

* Reduced the filesize of git-workflow.png

* Glows L0 changes to data class to accomodate L1A needs (#264)

* Glows L0 changes to data class to accomodate L1A needs

* DOC: Add intersphinx to documentation for external references

* DOC: Add a clean option to the documentation Makefile build command

* DOC: Update glows types in dataclasses

* DOC: Change glows currentmodule references

* DOC: Add ccsds header to glows documentation

* DOC: Add inherited member description for glows

This adds the inherited attributes from the dataclass directly
into the subclasses. There may be a better way to do this, so
perhaps some setting/configuration parameter needs to be updated?
But this at least fixes the warnings for now.

* FIX: Run black on glows test

* PR updates

---------

Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com>

* Housekeeping data testing and validation for CoDICE l0 (#176)

Validated housekeeping data with tests plus minor updates

---------

Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>

* L1 cdf creation (#179)

* Extracting and adding new data to the IDEX L1

* Adding some of the values to the XML, rather than shifting bits in the code
(work in progress, need to do a few more variables)

* Finishing up having space packet parser decom

* Getting rid of unecessary bit masksi8kt9gloh

* making changes based on comments

* Formatting the file after the upstream sync

* Adding L1 CDF creation

* Forgot to commit poetry changes

* Making changes based on Greg's comments,
as well as issues 1-8 from SPDF

* Fixing import statements

* Changing version into a string

* Adding more descriptions, units, and labels in the attributes

* Fixing the way strings are inserted into the CDF

* Fixing issues with white spice in the text of attrs

* Files are finally ISTP compliant!

* Fixing a couple things for ISTP

* Had the contents of the 32 bit things reversed

* Fixing the packet definition for the CCSDS file

* More fixes to ensure ISTP compliance

* Updating the file name

* touching finishes on the L1 CDF

* Adding fixture for temp directory in tests

* Fixing one of the ruff errors by adding a namedtuple

* Adding new "write_cdf" function that writed cdfs based on attributes

* Getting rid of a few more magic numbers

* Updating the packet definition

* initial hit l1a decom (#164)

* initial hit l1a decom

* using defaultdict, add docs

* fixed hit l1a docstrings

* fixed hit l1a docstring formatting

* doc updates and minor fixes

* minor docstring update

* minor comment update

* rebased with dev

* fixed ruff issues

* regenerated xtce

* fixed unit test

* github doc gen fix attempt

* added class docstring attributes

* fixed more doc issues

* another doc fix attempt

* doc fix attempt

* doc fix attempt

* changed doc python v to 3.10

* I-ALiRT: L0 parsing (#246)

* adding ialirt

* CDF documentation update (#273)

* CDF documentation update

* Adding the cdf_guide.rst to the toc

* Formatting update!

* Updating the docs for sections I missed

* Formatting a little nicer

* Adding links enabled in the docs

* Fixing duplicate headers

* Getting rid of duplicate headers

* Fixing more documentation

* Responding to a few code review comments

* Breaking up the page into sub-pages

* Adding a link to xarray_to_cdf page

* Adding more links to different sections

* fixing image link

* Editting some of the wording now that we're on different pages

* Typos and format fixed

* Another format error

* Create CDF class based system (#272)

* Updating IDEX CDF attrs

* remove comments, create constants class

* Updating names and addressing PR comments

* Updating docs

* Adding tests

* Updating types of L0 to properly read bytearrays

* Updating tests

* GLOWS direct events compressed and uncompressed

* updating decom dataclasses

* Updating GLOWS L0

* L1A tests

* removing unneeded code

* updating tests and fixing minor bug

* fixing autodocs

* addressing comments from code review

* Updating constants to a frozen dataclass

* Updating direct events packet reading to correctly read packet data, updating test data, adding tests

* Updating docstrings

---------

Co-authored-by: Tenzin Choedon <tenzin.choedon@lasp.colorado.edu>
Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com>
Co-authored-by: GLOWS team <maro@cbk.waw.pl>
Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>
Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com>
Co-authored-by: Gabriel M <104743000+GFMoraga@users.noreply.github.com>
Co-authored-by: Bryan Harter <41062454+bryan-harter@users.noreply.github.com>
Co-authored-by: Sean Hoyt <sean.hoyt@lasp.colorado.edu>
Co-authored-by: Laura Sandoval <46567335+laspsandoval@users.noreply.github.com>
laspsandoval added a commit to laspsandoval/imap_processing that referenced this pull request Apr 2, 2024
* update telemetry_generator to be more generic

* Updates to telemetry generator

* updating docs

* updating to remove local-specific testing changes

* first pass at glows xtce

* update way of checking if description exists or not. update generator template.

* updated generator to use correct IntegerParameterType

* First pass at GLOWS decom

* updating telemetry generator

* Updating glows decom to properly process data into dictionary

* Finished histogram decom

* Updating decom to include direct events

* Finalizing code, adding smaller test data file and combined XML, updating documentation and tests

* Updating test file path

* Fixing test runs

* updating docs

* Updating data classes to use attributes, and changing names

* Removing keys from data class and getting attribute names from the packet

* Move INT to SINT in telemetry_generator

Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com>

* Addressing code review comments

* Fixing test failure

* Review updates

* fixing typo

* Updating IntEnum to Enum to fix automatic documentation

* Move L0 data classes to a dataclass type with specific attributes

* Initial GLOWS L1A data classes

* Adding required attributes to L0 data class

* Adding L1A histogram data class

* fixing test failures

* Finishing ccsds dataclass

* Minor updates

* Finishing GLOWS L1A histograms

* Updating to include direct events data

* Updating glows l1a

* Adding useful GLOWS code from GLOWS team and Marek Strumik

* Adding direct event information

Co-authored-by: GLOWS team <maro@cbk.waw.pl>

* Updates for direct event creation

* Updates to L1A direct events

* Updated release workflow to reflect removal of "main" branch and use of version branches

* Reduced the filesize of git-workflow.png

* Glows L0 changes to data class to accomodate L1A needs (IMAP-Science-Operations-Center#264)

* Glows L0 changes to data class to accomodate L1A needs

* DOC: Add intersphinx to documentation for external references

* DOC: Add a clean option to the documentation Makefile build command

* DOC: Update glows types in dataclasses

* DOC: Change glows currentmodule references

* DOC: Add ccsds header to glows documentation

* DOC: Add inherited member description for glows

This adds the inherited attributes from the dataclass directly
into the subclasses. There may be a better way to do this, so
perhaps some setting/configuration parameter needs to be updated?
But this at least fixes the warnings for now.

* FIX: Run black on glows test

* PR updates

---------

Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com>

* Housekeeping data testing and validation for CoDICE l0 (IMAP-Science-Operations-Center#176)

Validated housekeeping data with tests plus minor updates

---------

Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>

* L1 cdf creation (IMAP-Science-Operations-Center#179)

* Extracting and adding new data to the IDEX L1

* Adding some of the values to the XML, rather than shifting bits in the code
(work in progress, need to do a few more variables)

* Finishing up having space packet parser decom

* Getting rid of unecessary bit masksi8kt9gloh

* making changes based on comments

* Formatting the file after the upstream sync

* Adding L1 CDF creation

* Forgot to commit poetry changes

* Making changes based on Greg's comments,
as well as issues 1-8 from SPDF

* Fixing import statements

* Changing version into a string

* Adding more descriptions, units, and labels in the attributes

* Fixing the way strings are inserted into the CDF

* Fixing issues with white spice in the text of attrs

* Files are finally ISTP compliant!

* Fixing a couple things for ISTP

* Had the contents of the 32 bit things reversed

* Fixing the packet definition for the CCSDS file

* More fixes to ensure ISTP compliance

* Updating the file name

* touching finishes on the L1 CDF

* Adding fixture for temp directory in tests

* Fixing one of the ruff errors by adding a namedtuple

* Adding new "write_cdf" function that writed cdfs based on attributes

* Getting rid of a few more magic numbers

* Updating the packet definition

* initial hit l1a decom (IMAP-Science-Operations-Center#164)

* initial hit l1a decom

* using defaultdict, add docs

* fixed hit l1a docstrings

* fixed hit l1a docstring formatting

* doc updates and minor fixes

* minor docstring update

* minor comment update

* rebased with dev

* fixed ruff issues

* regenerated xtce

* fixed unit test

* github doc gen fix attempt

* added class docstring attributes

* fixed more doc issues

* another doc fix attempt

* doc fix attempt

* doc fix attempt

* changed doc python v to 3.10

* I-ALiRT: L0 parsing (IMAP-Science-Operations-Center#246)

* adding ialirt

* CDF documentation update (IMAP-Science-Operations-Center#273)

* CDF documentation update

* Adding the cdf_guide.rst to the toc

* Formatting update!

* Updating the docs for sections I missed

* Formatting a little nicer

* Adding links enabled in the docs

* Fixing duplicate headers

* Getting rid of duplicate headers

* Fixing more documentation

* Responding to a few code review comments

* Breaking up the page into sub-pages

* Adding a link to xarray_to_cdf page

* Adding more links to different sections

* fixing image link

* Editting some of the wording now that we're on different pages

* Typos and format fixed

* Another format error

* Create CDF class based system (IMAP-Science-Operations-Center#272)

* Updating IDEX CDF attrs

* remove comments, create constants class

* Updating names and addressing PR comments

* Updating docs

* Adding tests

* Updating types of L0 to properly read bytearrays

* Updating tests

* GLOWS direct events compressed and uncompressed

* updating decom dataclasses

* Updating GLOWS L0

* L1A tests

* removing unneeded code

* updating tests and fixing minor bug

* fixing autodocs

* addressing comments from code review

* Updating constants to a frozen dataclass

* Updating direct events packet reading to correctly read packet data, updating test data, adding tests

* Updating docstrings

---------

Co-authored-by: Tenzin Choedon <tenzin.choedon@lasp.colorado.edu>
Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com>
Co-authored-by: GLOWS team <maro@cbk.waw.pl>
Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu>
Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com>
Co-authored-by: Gabriel M <104743000+GFMoraga@users.noreply.github.com>
Co-authored-by: Bryan Harter <41062454+bryan-harter@users.noreply.github.com>
Co-authored-by: Sean Hoyt <sean.hoyt@lasp.colorado.edu>
Co-authored-by: Laura Sandoval <46567335+laspsandoval@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ins: CoDICE Related to the CoDICE instrument Level: L1 Level 1 processing
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants