Glows L1A Processing Algorithm (IMAP-Science-Operations-Center#287)

* update telemetry_generator to be more generic * Updates to telemetry generator * updating docs * updating to remove local-specific testing changes * first pass at glows xtce * update way of checking if description exists or not. update generator template. * updated generator to use correct IntegerParameterType * First pass at GLOWS decom * updating telemetry generator * Updating glows decom to properly process data into dictionary * Finished histogram decom * Updating decom to include direct events * Finalizing code, adding smaller test data file and combined XML, updating documentation and tests * Updating test file path * Fixing test runs * updating docs * Updating data classes to use attributes, and changing names * Removing keys from data class and getting attribute names from the packet * Move INT to SINT in telemetry_generator Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> * Addressing code review comments * Fixing test failure * Review updates * fixing typo * Updating IntEnum to Enum to fix automatic documentation * Move L0 data classes to a dataclass type with specific attributes * Initial GLOWS L1A data classes * Adding required attributes to L0 data class * Adding L1A histogram data class * fixing test failures * Finishing ccsds dataclass * Minor updates * Finishing GLOWS L1A histograms * Updating to include direct events data * Updating glows l1a * Adding useful GLOWS code from GLOWS team and Marek Strumik * Adding direct event information Co-authored-by: GLOWS team <maro@cbk.waw.pl> * Updates for direct event creation * Updates to L1A direct events * Updated release workflow to reflect removal of "main" branch and use of version branches * Reduced the filesize of git-workflow.png * Glows L0 changes to data class to accomodate L1A needs (IMAP-Science-Operations-Center#264) * Glows L0 changes to data class to accomodate L1A needs * DOC: Add intersphinx to documentation for external references * DOC: Add a clean option to the documentation Makefile build command * DOC: Update glows types in dataclasses * DOC: Change glows currentmodule references * DOC: Add ccsds header to glows documentation * DOC: Add inherited member description for glows This adds the inherited attributes from the dataclass directly into the subclasses. There may be a better way to do this, so perhaps some setting/configuration parameter needs to be updated? But this at least fixes the warnings for now. * FIX: Run black on glows test * PR updates --------- Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> * Housekeeping data testing and validation for CoDICE l0 (IMAP-Science-Operations-Center#176) Validated housekeeping data with tests plus minor updates --------- Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> * L1 cdf creation (IMAP-Science-Operations-Center#179) * Extracting and adding new data to the IDEX L1 * Adding some of the values to the XML, rather than shifting bits in the code (work in progress, need to do a few more variables) * Finishing up having space packet parser decom * Getting rid of unecessary bit masksi8kt9gloh * making changes based on comments * Formatting the file after the upstream sync * Adding L1 CDF creation * Forgot to commit poetry changes * Making changes based on Greg's comments, as well as issues 1-8 from SPDF * Fixing import statements * Changing version into a string * Adding more descriptions, units, and labels in the attributes * Fixing the way strings are inserted into the CDF * Fixing issues with white spice in the text of attrs * Files are finally ISTP compliant! * Fixing a couple things for ISTP * Had the contents of the 32 bit things reversed * Fixing the packet definition for the CCSDS file * More fixes to ensure ISTP compliance * Updating the file name * touching finishes on the L1 CDF * Adding fixture for temp directory in tests * Fixing one of the ruff errors by adding a namedtuple * Adding new "write_cdf" function that writed cdfs based on attributes * Getting rid of a few more magic numbers * Updating the packet definition * initial hit l1a decom (IMAP-Science-Operations-Center#164) * initial hit l1a decom * using defaultdict, add docs * fixed hit l1a docstrings * fixed hit l1a docstring formatting * doc updates and minor fixes * minor docstring update * minor comment update * rebased with dev * fixed ruff issues * regenerated xtce * fixed unit test * github doc gen fix attempt * added class docstring attributes * fixed more doc issues * another doc fix attempt * doc fix attempt * doc fix attempt * changed doc python v to 3.10 * I-ALiRT: L0 parsing (IMAP-Science-Operations-Center#246) * adding ialirt * CDF documentation update (IMAP-Science-Operations-Center#273) * CDF documentation update * Adding the cdf_guide.rst to the toc * Formatting update! * Updating the docs for sections I missed * Formatting a little nicer * Adding links enabled in the docs * Fixing duplicate headers * Getting rid of duplicate headers * Fixing more documentation * Responding to a few code review comments * Breaking up the page into sub-pages * Adding a link to xarray_to_cdf page * Adding more links to different sections * fixing image link * Editting some of the wording now that we're on different pages * Typos and format fixed * Another format error * Create CDF class based system (IMAP-Science-Operations-Center#272) * Updating IDEX CDF attrs * remove comments, create constants class * Updating names and addressing PR comments * Updating docs * Adding tests * Updating types of L0 to properly read bytearrays * Updating tests * GLOWS direct events compressed and uncompressed * updating decom dataclasses * Updating GLOWS L0 * L1A tests * removing unneeded code * updating tests and fixing minor bug * fixing autodocs * addressing comments from code review * Updating constants to a frozen dataclass * Updating direct events packet reading to correctly read packet data, updating test data, adding tests * Updating docstrings --------- Co-authored-by: Tenzin Choedon <tenzin.choedon@lasp.colorado.edu> Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> Co-authored-by: GLOWS team <maro@cbk.waw.pl> Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> Co-authored-by: Gabriel M <104743000+GFMoraga@users.noreply.github.com> Co-authored-by: Bryan Harter <41062454+bryan-harter@users.noreply.github.com> Co-authored-by: Sean Hoyt <sean.hoyt@lasp.colorado.edu> Co-authored-by: Laura Sandoval <46567335+laspsandoval@users.noreply.github.com>
laspsandoval · Jan 8, 2024 · b7cd2c8 · b7cd2c8
1 parent 6d96076
commit b7cd2c8
Show file tree

Hide file tree

Showing 11 changed files with 1,066 additions and 62 deletions.
diff --git a/docs/source/code-documentation/glows.rst b/docs/source/code-documentation/glows.rst
@@ -18,7 +18,6 @@ GLOWS (Global Solar Wind Structure)
 This is the GLOWS Instrument module, which contains the code for processing
 data from the GLOWS instrument.
 
-The L0 code to decommutate the CCSDS packet data can be found below.
 
 .. autosummary::
     :toctree: generated/
@@ -27,3 +26,5 @@ The L0 code to decommutate the CCSDS packet data can be found below.
 
     l0.decom_glows
     l0.glows_l0_data
+    l1.glows_l1a_data
+    utils.constants
diff --git a/imap_processing/glows/l0/decom_glows.py b/imap_processing/glows/l0/decom_glows.py
@@ -7,6 +7,7 @@
 from space_packet_parser import parser, xtcedef
 
 from imap_processing import imap_module_directory
+from imap_processing.ccsds.ccsds_data import CcsdsData
 from imap_processing.glows import version
 from imap_processing.glows.l0.glows_l0_data import DirectEventL0, HistogramL0
 
@@ -66,11 +67,28 @@ def decom_packets(
                 apid = packet.header["PKT_APID"].derived_value
                 # Do something with the packet data
                 if apid == GlowsParams.HIST_APID.value:
-                    hist_l0 = HistogramL0(packet, version, filename)
+                    values = [
+                        item.derived_value
+                        if item.derived_value is not None
+                        else item.raw_value
+                        for item in packet.data.values()
+                    ]
+                    hist_l0 = HistogramL0(
+                        version, filename, CcsdsData(packet.header), *values
+                    )
                     histdata.append(hist_l0)
 
                 if apid == GlowsParams.DE_APID.value:
-                    de_l0 = DirectEventL0(packet, version, filename)
+                    values = [
+                        item.derived_value
+                        if item.derived_value is not None
+                        else item.raw_value
+                        for item in packet.data.values()
+                    ]
+
+                    de_l0 = DirectEventL0(
+                        version, filename, CcsdsData(packet.header), *values
+                    )
                     dedata.append(de_l0)
 
         except ReadError as e:

diff --git a/imap_processing/glows/l0/glows_l0_data.py b/imap_processing/glows/l0/glows_l0_data.py
@@ -1,6 +1,6 @@
 """Contains data classes to support GLOWS L0 processing."""
 
-from dataclasses import dataclass, fields
+from dataclasses import dataclass
 
 from imap_processing.ccsds.ccsds_data import CcsdsData
 
@@ -112,37 +112,17 @@ class HistogramL0(GlowsL0):
     ELAVG: int
     ELVAR: int
     EVENTS: int
-    HISTOGRAM_DATA: bytes
-
-    def __init__(self, packet, software_version: str, packet_file_name: str):
-        """Initialize data class with a packet of histogram data.
-
-        Parameters
-        ----------
-        packet
-            Packet generated from space_packet_parser. Should be a type NamedTuple
-            with header and data fields.
-        software_version: str
-            The version of the ground software being used
-        packet_file_name: str
-            The filename of the packet
-        """
-        super().__init__(software_version, packet_file_name, CcsdsData(packet.header))
-
-        attributes = [field.name for field in fields(self)]
-
-        # For each item in packet, assign it to the matching attribute in the class.
-        for key, item in packet.data.items():
-            value = (
-                item.derived_value if item.derived_value is not None else item.raw_value
-            )
-            if key in attributes:
-                setattr(self, key, value)
-            else:
-                raise KeyError(
-                    f"Did not find matching attribute in Histogram data class for "
-                    f"{key}"
+    HISTOGRAM_DATA: bytearray
+
+    def __post_init__(self):
+        """Convert HISTOGRAM_DATA attribute from string to bytearray if needed."""
+        if isinstance(self.HISTOGRAM_DATA, str):
+            # Convert string output from space_packet_parser to bytearray
+            self.HISTOGRAM_DATA = bytearray(
+                int(self.HISTOGRAM_DATA, 2).to_bytes(
+                    len(self.HISTOGRAM_DATA) // 8, "big"
                 )
+            )
 
 
 @dataclass
@@ -154,50 +134,62 @@ class DirectEventL0(GlowsL0):
     MET : int
         CCSDS Packet Time Stamp (coarse time)
     SEC : int
-        Data timestamp, seconds counter.
+        Data IMAP timestamp, seconds counter.
     LEN : int
         Number of packets in data set.
     SEQ : int
         Packet sequence in data set.
+    DE_DATA : bytearray
+        Raw direct event data (compressed)
     ground_sw_version : str
         Ground software version
     packet_file_name : str
         File name of the source packet
     ccsds_header : CcsdsData
         CCSDS header data
+
+    Methods
+    -------
+    within_same_sequence
+        Compare two DirectEventL0 objects and see if MET and LEN are equal (meaning they
+        are in the same sequence)
+
     """
 
     MET: int
     SEC: int
     LEN: int
     SEQ: int
+    DE_DATA: bytearray
+
+    def __post_init__(self):
+        """Convert from string to bytearray if DE_DATA is a string of ones and zeros."""
+        if isinstance(self.DE_DATA, str):
+            # Convert string output from space_packet_parser to bytearray
+            self.DE_DATA = bytearray(
+                int(self.DE_DATA, 2).to_bytes(len(self.DE_DATA) // 8, "big")
+            )
+
+    def within_same_sequence(self, other):
+        """
+        Compare fields for L0 which should be the same for packets within one sequence.
 
-    def __init__(self, packet, software_version: str, packet_file_name: str):
-        """Initialize data class with a packet of direct event data.
+        This method compares the IMAP time (MET) and packet length (LEN) fields.
 
         Parameters
         ----------
-        packet
-            Packet generated from space_packet_parser. Should be a type NamedTuple
-            with header and data fields.
-        software_version: str
-            The version of the ground software being used
-        packet_file_name: str
-            The filename of the packet
+        other
+            Another instance of DirectEventL0 to compare to.
+
+        Returns
+        -------
+        True if the MET and LEN fields match, False otherwise.
         """
-        super().__init__(software_version, packet_file_name, CcsdsData(packet.header))
+        if not isinstance(other, DirectEventL0):
+            return False
 
-        attributes = [field.name for field in fields(self)]
+        # Time and overall packet length should match
+        if self.MET == other.MET and self.LEN == other.LEN:
+            return True
 
-        # For each item in packet, assign it to the matching attribute in the class.
-        for key, item in packet.data.items():
-            value = (
-                item.derived_value if item.derived_value is not None else item.raw_value
-            )
-            if key in attributes:
-                setattr(self, key, value)
-            else:
-                raise KeyError(
-                    f"Did not find matching attribute in Direct events data class for "
-                    f"{key}"
-                )
+        return False