Added eu analog conversion to test_decom #274

GFMoraga · 2023-11-08T17:06:07Z

Change Summary

This is a WIP draft. The new test does not pass right now, so will be fixing and updating. Just wanted to share the progress for the ticket #247

Overview

This PR will apply analog conversions to validation_data and test against the eu csv.

New Files

idle_export_eu.COD_NHK_20230822_122700 2.csv
- validation eu csv

Updated Files

test_decom.py
- added analog conversions
- added function to call eu csv.

Testing

The new test function def test_unit_conversion will not pass! TODO: update and get feedback on how to pass this.

closes #247

GFMoraga · 2023-11-08T17:09:15Z

imap_processing/codice/tests/test_decom.py

+    assert engineering_data.shape == validation_data.shape
+
+    # Define a tolerance for floating-point precision
+    tolerance = 1e-6  # Adjust this tolerance as needed


This can be deleted. Put it there as an "option" if we need something like this.

I think you might be able to do a numpy array comparison with allclose for your tolerance:
https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html#numpy.testing.assert_allclose

If you don't need a tolerance you should remove it though so we don't accidentally introduce some math slop somewhere.

I don't think it is needed. I was just thinking about the decimal places because they aren't whole numbers. Def do not want math slop. I will look into the links.

GFMoraga · 2023-11-08T17:29:54Z

imap_processing/codice/tests/test_decom.py

+    )
+
+    for mnemonic in validation_data.columns:
+        matching_row = filtered_analog[filtered_analog["mnemonic"] == mnemonic]


I need to update filtered_analog. Right now it doesn't exist..sry

greglucas · 2023-11-09T18:49:54Z

imap_processing/codice/tests/test_decom.py

+            ].values[0]
+
+            raw_data = validation_data[mnemonic]
+            engineering_unit_data = (


I think @tech3371 was using Numpy's Polynomial class for this, are you able to do the same?
https://numpy.org/doc/stable/reference/generated/numpy.polynomial.polynomial.Polynomial.html#numpy.polynomial.polynomial.Polynomial

np.polynomial.Polynomial(coefficients)(raw_data)

@tech3371 did mention this to me in person and I saw it in her PR. I will look at her code again and look into this.

greglucas · 2023-11-09T18:51:58Z

imap_processing/codice/tests/test_decom.py

+    assert engineering_data.shape == validation_data.shape
+
+    # Define a tolerance for floating-point precision
+    tolerance = 1e-6  # Adjust this tolerance as needed


I think you might be able to do a numpy array comparison with allclose for your tolerance:
https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html#numpy.testing.assert_allclose

If you don't need a tolerance you should remove it though so we don't accidentally introduce some math slop somewhere.

greglucas · 2023-11-09T18:53:42Z

imap_processing/codice/tests/test_decom.py

+    # Loop through each column and each row to compare the values
+    for column in validation_data.columns:
+        for index, row in validation_data.iterrows():
+            # Compare each value in the column
+            assert abs(engineering_data.at[index, column] - row[column]) < tolerance


Can you compare the arrays directly without needing to loop over the quantities?

np.testing.assert_allclose(engineering_data, validation_data, rtol=1e-6)

GFMoraga · 2023-11-22T16:54:59Z

@tech3371 can you review this an make suggestions please. There is an issue in the last test test_derived_eu_data and can you look over the convert_raw_to_eu function. I thought I made good adjustments based on yesterdays convo.

GFMoraga · 2023-11-22T16:55:57Z

@bourque can you review, and see what I need to change. What did I miss from yesterday's convo?

tech3371 · 2023-11-27T16:43:53Z

imap_processing/codice/tests/test_decom.py

@@ -28,13 +31,13 @@ def decom_test_data():


 @pytest.fixture(scope="session")
-def validation_data():
+def raw_data():


I would revert that name to what you had before. It made more sense

tech3371 · 2023-11-27T16:51:11Z

imap_processing/codice/tests/test_decom.py

+    dataset = xr.Dataset()
+    dataset["First_mnemonic"] = xr.DataArray(np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))


You are doing right by creating dataset as input to convert_raw_to_eu() but looks like we are doing at the wrong place. when we call convert_raw_to_eu() from imap_processing/utils.py we assume, we have unpacked raw packets data and saved in l1a cdf file. Therefore, the input is in xr.Dataset. I am not sure what's best way to do it here. I see why you are doing it here. They gave you raw unpacked data in csv file. I feel like you would want store unpacked packets data into xr.Dataset and then keep that csv data as validation still. Then use that dataset as input to the function that converts raw to engineering unit.

tech3371 · 2023-11-27T16:55:47Z

imap_processing/codice/tests/test_decom.py

+    return raw_data
+
+
+def convert_raw_to_eu(dataset: xr.Dataset, packet_name):


I know I guided you to copy the function for development. Will rebasing with latest change conflict with your work? If not, I would rebase and then import that function into test and use that.

This function looks similar to what we have in imap_processing/utils.py but you are missing last operation which converts raw data to EU value.

tech3371 · 2023-11-27T17:02:19Z

imap_processing/codice/tests/test_decom.py


    # Return the data
-    return validation_data
+    return raw_data


you could do something similar to what I did here create_dataset(). If you look at references, you can do same thing here. Its input is packet list which you already have from decom_test_data(). create_dataset() gives back xr.Dataset which you can then use as input to convert_raw_to_eu().

https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/swe/utils/swe_utils.py

bourque · 2023-11-28T16:48:06Z

imap_processing/codice/tests/data/idle_export_eu.COD_NHK_20230822_122700 2.csv

I suggest to rename this to use underscore instead of a whitespace (i.e. `COD_NHK_20230822_122700_2.csv). Things can get really messy in Python trying to deal with whitespaces in filenames.

bourque · 2023-11-30T22:09:33Z

@GFMoraga (I think) I successfully rebased this branch with upstream/dev after my merge of #288

…analog_eu

tech3371 · 2023-12-08T21:18:00Z

imap_processing/codice/tests/test_codice_l0.py

+    dataset = xr.Dataset()
+    dataset["First_mnemonic"] = xr.DataArray(np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
+


remove this?

tech3371 · 2023-12-08T21:22:13Z

imap_processing/codice/tests/test_codice_l0.py

+    # Determine the number of CCSDS header fields
+    num_ccsds_header_fields = 7
+
+    # Compare EU values of housekeeping data, skipping CCSDS header fields
+    for idx, field in enumerate(eu_hk_data):
+        # Skip the first num_ccsds_header_fields fields
+        if idx < num_ccsds_header_fields:
+            continue
+
+        # Extract the specific EU values for the field
+        eu_values = eu_hk_data[field].data
+
+        # Extract the corresponding validation data for the field
+        validation_values = validation_data[field].values
+
+        # Compare each individual element
+        for eu_val, validation_val in zip(eu_values, validation_values):
+            assert round(eu_val, 5) == round(validation_val, 5)
+


Really nice work!

You could move all above and this new changes into a new function because it's testing engineering unit of housekeeping. so you could call it like test_eu_hk_data() or something in that manner.

GFMoraga · 2023-12-08T23:00:50Z

I made final updates and cleaned up based on @tech3371 review. I am going to merge for now and wrap it up.

…ter#274) * added eu analog conversion to test_decom

added eu analog conversion to test_decom

4b9ec18

GFMoraga added the Ins: CoDICE Related to the CoDICE instrument label Nov 8, 2023

GFMoraga requested review from bourque, greglucas, tech3371 and maxinelasp November 8, 2023 17:06

GFMoraga self-assigned this Nov 8, 2023

GFMoraga commented Nov 8, 2023

View reviewed changes

greglucas reviewed Nov 9, 2023

View reviewed changes

GFMoraga changed the title ~~Added eu analog conversion to test_decom WIP~~ [WIP ]Added eu analog conversion to test_decom Nov 10, 2023

GFMoraga added 2 commits November 20, 2023 15:56

progress on testing analog units

13847f9

updates made on eu function and added eu csv-needs work

e802d66

tech3371 reviewed Nov 27, 2023

View reviewed changes

bourque reviewed Nov 28, 2023

View reviewed changes

bourque mentioned this pull request Nov 28, 2023

Initial CoDICE L1A Processing Algorithm #288

Merged

GFMoraga added 2 commits November 30, 2023 15:03

Rebased branch with upstream/dev

bbe4098

updates made on eu function and added eu csv-needs work

96a91e3

GFMoraga added 3 commits December 6, 2023 10:32

Merge branch 'analog_eu' of github.com:GFMoraga/imap_processing into …

d585cbc

…analog_eu

added test for comparing converted values and cleaned up

55452b8

passing tests updates

1a4a6cf

GFMoraga marked this pull request as ready for review December 8, 2023 18:43

GFMoraga changed the title ~~[WIP ]Added eu analog conversion to test_decom~~ Added eu analog conversion to test_decom Dec 8, 2023

GFMoraga requested review from bourque and tech3371 December 8, 2023 18:49

tech3371 approved these changes Dec 8, 2023

View reviewed changes

final clean-up

1785bcb

Merge branch 'dev' into analog_eu

51e6cfc

GFMoraga merged commit 4cf062b into IMAP-Science-Operations-Center:dev Dec 8, 2023
14 checks passed

GFMoraga deleted the analog_eu branch December 8, 2023 23:04

bourque mentioned this pull request Dec 19, 2023

Convert CoDICE Raw Housekeeping to Engineering Units #197

Closed

laspsandoval pushed a commit to laspsandoval/imap_processing that referenced this pull request Apr 2, 2024

Added eu analog conversion to test_decom (IMAP-Science-Operations-Cen…

1360141

…ter#274) * added eu analog conversion to test_decom

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added eu analog conversion to test_decom #274

Added eu analog conversion to test_decom #274

GFMoraga commented Nov 8, 2023 •

edited by sdhoyt

Loading

GFMoraga Nov 8, 2023

greglucas Nov 9, 2023

GFMoraga Nov 10, 2023

GFMoraga Nov 8, 2023

greglucas Nov 9, 2023

GFMoraga Nov 10, 2023

greglucas Nov 9, 2023

greglucas Nov 9, 2023

GFMoraga commented Nov 22, 2023

GFMoraga commented Nov 22, 2023

tech3371 Nov 27, 2023

tech3371 Nov 27, 2023

tech3371 Nov 27, 2023

tech3371 Nov 27, 2023

bourque Nov 28, 2023

bourque commented Nov 30, 2023

tech3371 Dec 8, 2023

tech3371 Dec 8, 2023

GFMoraga commented Dec 8, 2023

		dataset = xr.Dataset()
		dataset["First_mnemonic"] = xr.DataArray(np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))

		return raw_data


		def convert_raw_to_eu(dataset: xr.Dataset, packet_name):

Added eu analog conversion to test_decom #274

Added eu analog conversion to test_decom #274

Conversation

GFMoraga commented Nov 8, 2023 • edited by sdhoyt Loading

Change Summary

Overview

New Files

Updated Files

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GFMoraga commented Nov 22, 2023

GFMoraga commented Nov 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bourque commented Nov 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GFMoraga commented Dec 8, 2023

GFMoraga commented Nov 8, 2023 •

edited by sdhoyt

Loading