updates to utils create_dateset function #532
Conversation
…cess housekeeping
tech3371
added
enhancement
tech3371
requested review from
bourque,
sdhoyt,
greglucas,
subagonsouth,
vmartinez-cu,
laspsandoval and
maxinelasp
and removed request for
a team
| @@ -101,17 +101,22 @@ def convert_raw_to_eu(dataset: xr.Dataset, conversion_table_path, packet_name): | |||
| return dataset | |||
|
|
|||
|
|
|||
| def create_dataset(packets: list[Packet], met_name="shcoarse") -> xr.Dataset: | |||
| def create_dataset( | |||
There was a problem hiding this comment.
Ooooo. I didn't know about this function. Love that it is being made more generic. If I am not mistaken it can be used for any product that is generated directly from the packet storing everything in one-dimensional, 12-bit CDF variables? Some details about usage restrictions should be added to the docstring.
I wonder if functionality could be added to allow for custom unpacking of certain keys in the input packets as well as CDF variables that are not type "I12"?
There was a problem hiding this comment.
I see now that I was misinterpreting the format="I12" and now understand that this is to display an integer with a field width of 12. Ignore that part of my comment.
There was a problem hiding this comment.
Nice improvements to this function!
There was a problem hiding this comment.
@subagonsouth let me know if you like clarification on this.
yes. I was trying to make this function more generic as I go along.
| include_header: bool, Optional | ||
| Whether to include CCSDS header data in the dataset | ||
| skip_keys: list, Optional | ||
| Keys to skip in the metadata |
There was a problem hiding this comment.
What is metadata in this context? I think it means that matching fields in the input packets are not included in the output dataset?
There was a problem hiding this comment.
Here, I am using metadata as keys from packet definition after CCSDS header keys. I appreciate other term suggestion.
imap_processing/utils.py
Outdated
| Parameters | ||
| ---------- | ||
| packets : list[Packet] | ||
| packet list | ||
| met_name : str | ||
| metadata name to use as epoch time | ||
| met_name : str, Optional |
There was a problem hiding this comment.
Do some packets contain a value that is valid to directly use as the epoch coordinate? The few examples that I have seen require a function that converts what is the packet to what is desired to be stored in the epoch coordinate. For example, a SPICE call to convert CCSDS_MET to a datetime object?
There was a problem hiding this comment.
In this phase, I am not converting because some uses this exact key to sort data by this time. May be we should change time here in the format the CDF desire(datetime64 object). I could do that in upcoming PR.
| @@ -101,17 +101,22 @@ def convert_raw_to_eu(dataset: xr.Dataset, conversion_table_path, packet_name): | |||
| return dataset | |||
|
|
|||
|
|
|||
| def create_dataset(packets: list[Packet], met_name="shcoarse") -> xr.Dataset: | |||
| def create_dataset( | |||
There was a problem hiding this comment.
Nice improvements to this function!
|
I am going to merge this so that others can use it sooner. We can update this more in future PR as we go along. |
tech3371
merged commit 0eee306
into
IMAP-Science-Operations-Center:dev
Change Summary
Overview
Updated
imap_processing/utils.py's create_dataset function to be more generic and updated IMAP-Hi Housekeeping processing to use it.Updated Files
Two main files that was changed