CDF documentation update #273
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
greglucas
reviewed
Nov 7, 2023
There was a problem hiding this comment.
Overall I think this is a nice overview. I personally like linking out to external places where possible and only adding explanatory text/relevant text to the mission. Right now this page is huge with a lot of repeated text/images from the main sites, and I don't think people like landing on long docs pages because they get intimidated. You could break this up into a few different sub-pages / sections.
Brief CDF introduction (lots of linking out to the main SPDF / CDF pages)
cdflib / xarray section (some linking out to cdflib documentation)
IMAP standards (more text specifically relating to this mission, possibly linking to our cdf module)
There was a problem hiding this comment.
Can you link out to these images instead, or point people to the landing page for more description: I think a similar image is here: https://spdf.gsfc.nasa.gov/istp_guide/1d_spectrogram.GIF
There was a problem hiding this comment.
I don't think this is a very descriptive image... I find it hard to understand what this means or what the variables are referring to. Might help to link out to a different image on the CDF website for this?
There was a problem hiding this comment.
I figured the text below it would help describe it?
There was a problem hiding this comment.
I guess I just don't personally think it adds a lot to the context. I think people can grasp shape/dimensions of arrays, and if they don't I am not so sure this image helps at all with that. It isn't stacked in a nice way to show what that means. xarray's logo is a good example of something that might be a little more descriptive and informative for an image:
but alas, CDF likes their 1980s images :-p
|
|
||
| ############################## | ||
| Generating CDF Files | ||
| ############################## |
There was a problem hiding this comment.
Preferably, all underlines should be the same length as the headings for consistency.
|
|
||
|
|
||
| ############################## | ||
| Generating CDF Files |
There was a problem hiding this comment.
Generating CDF files should be a section below CDF Files Introduction I think?
Maybe make the landing heading "CDF Guide" to correspond to what you called the file.
| ****************************** | ||
| CDF Files Introduction | ||
| ****************************** | ||
| CDF files are binary files which require special libraries to create and manipulate. Goddard Space Flight Center (GSFC) maintains these CDF libraries, and is the shepherd of the file format. |
There was a problem hiding this comment.
What does CDF stand for? I don't think is mentioned anywhere here.
| cdflib.xarray_to_cdf | ||
| ***************************** | ||
|
|
||
| The Python library `cdflib <https://github.com/MAVENSDC/cdflib>`_ is a pure-python implementation of the CDF libraries. Of particular note is the function `xarray_to_cdf <https://cdflib.readthedocs.io/en/latest/api/cdflib.xarray.xarray_to_cdf.html#cdflib.xarray.xarray_to_cdf>`_. |
There was a problem hiding this comment.
CDF specification, not libraries?
| cdflib.xarray_to_cdf | ||
| ***************************** | ||
|
|
||
| The Python library `cdflib <https://github.com/MAVENSDC/cdflib>`_ is a pure-python implementation of the CDF libraries. Of particular note is the function `xarray_to_cdf <https://cdflib.readthedocs.io/en/latest/api/cdflib.xarray.xarray_to_cdf.html#cdflib.xarray.xarray_to_cdf>`_. |
There was a problem hiding this comment.
why is xarray_to_cdf of particular note? Maybe mention something about the IMAP SDC using xarray for all internal data representations.
|
|
||
| .. note:: the netCDF file format was created by NOAA a few years after the CDF file format to plug in some of the shortfallings of the CDF file format. These days, the orginal reasons for the split between the two formats have largely disappeared. | ||
|
|
||
| xarray_to_cdf |
There was a problem hiding this comment.
cdflib.xarray_to_cdf ? Link to your code here as well if there is documentation for this method already?
|
|
||
| * This code does not check the values *within* the attributes (except VAR_TYPE and DEPEND_{i}), only that the attributes exist! | ||
| * This function does not put the "Epoch" variable as the first thing in the file, which was recommended (but not required) by the SPDF | ||
| * This code creates variables in a row-major format. Column-major is recommended (but not required) by the SPDF. |
There was a problem hiding this comment.
Since we are just getting started, could we change to what they recommend?
There was a problem hiding this comment.
I can if I have time or if someone else wants to open a pull request on cdflib, but:
- cdflib doesn't currently write column-major
- xarray_to_cdf loops through the variables in the xarray and writes them in whatever order depending on what it reads out of the dictionary first
I was somewhat surprised that those were recommended, since the software libraries abstract that stuff away.
| ================ | ||
| This attribute identifies the name of the instrument or sensor that collected the data. Both a long name and a short name are given. For any data file, only a single value is allowed. | ||
|
|
||
| For IMAP, the following are valid - |
There was a problem hiding this comment.
valid, or what must be used?
bryan-harter
requested review from
a team,
bourque,
sdhoyt,
greglucas,
tech3371,
laspsandoval,
GFMoraga and
maxinelasp
and removed request for
a team
|
So I think I addressed most of your comments thus far, I just need to break up the document into sub-docs. But here is a list of things I included in the document that are things we are not currently doing (or at least I didn't do on IDEX):
|
|
Oh additionally I added the sphinx mod "sphinx.ext.autosectionlabel", so we can just link straight to the titles of sections themselves without including all those ".. _insert_title_here:" things in the rst files. I'm happy to change it back though if there was a reason we weren't using that though. |
greglucas
approved these changes
Nov 9, 2023
| IMAP is archiving its science data with the `Space Physics Data Facility (SPDF) <https://spdf.gsfc.nasa.gov/>`_, which requires that data be in the CDF file format. | ||
|
|
||
| CDF Ecosystem | ||
| =============== |
There was a problem hiding this comment.
Most of the headings are still overflowing the titles by one or two characters.
| .. note:: | ||
| The L0->L2 processing will be using cdflib to generate CDF files on IMAP. For more information about cdflib and the xarray_to_cdf function, see :ref:`cdflib.xarray_to_cdf` | ||
|
|
||
| The other main python library is `**pycdf** <https://spacepy.github.io/pycdf.html>`_, which is a part of **spacepy**. Pycdf wraps the C libraries, and therefore it requires downloading the C libraries. Because it is a part of Spacepy, Pycdf requires downloading and installing Spacepy. Pycdf includes various bells and whistles that help make the files more python friendly, and includes functions for checking for ISTP compliance. |
There was a problem hiding this comment.
This shows up as **pycdf** in the built docs. I think you might be able to bold it by moving the double star outside the link target. But I don't think you even need to do that and could just let the link target choose how it wants to render since they are usually already italicized or something else.
|
|
||
| The way these are astored in the file are somewhat complex, but that information is abstracted away from the end user. The CDF libraries have functions to easily list and extract the variables and attributes within the CDF files. | ||
|
|
||
| The internal format of CDF files are described `cdf specification <https://cdaweb.gsfc.nasa.gov/pub/software/cdf/doc/cdf391/cdf39ifd.pdf>`_. |
There was a problem hiding this comment.
Suggested change
| The internal format of CDF files are described `cdf specification <https://cdaweb.gsfc.nasa.gov/pub/software/cdf/doc/cdf391/cdf39ifd.pdf>`_. | |
| The internal format of CDF files are described in the `cdf specification <https://cdaweb.gsfc.nasa.gov/pub/software/cdf/doc/cdf391/cdf39ifd.pdf>`_. |
| Attributes | ||
| ----------- | ||
|
|
||
| Attributes are essentially key:value pairs inside the CDF, similar to a python dictionary object. For example - Mission=IMAP, or Instrument=IDEX. |
There was a problem hiding this comment.
Suggested change
| Attributes are essentially key:value pairs inside the CDF, similar to a python dictionary object. For example - Mission=IMAP, or Instrument=IDEX. | |
| Attributes are essentially (key=value) pairs inside the CDF, similar to a python dictionary object. For example - Mission=IMAP, or Instrument=IDEX. |
or put colons in the latter examples:
Suggested change
| Attributes are essentially key:value pairs inside the CDF, similar to a python dictionary object. For example - Mission=IMAP, or Instrument=IDEX. | |
| Attributes are essentially key:value pairs inside the CDF, similar to a python dictionary object. For example - Mission:IMAP, or Instrument:IDEX. |
| * Name | ||
| * Variables are given a unique name | ||
| * Shape | ||
| * The the number and the size of the dimensions. In the picture above, "zVariable 1" is 5x5, and "zVariable 2" is 5x2. |
There was a problem hiding this comment.
Suggested change
| * The the number and the size of the dimensions. In the picture above, "zVariable 1" is 5x5, and "zVariable 2" is 5x2. | |
| * The number and the size of the dimensions. In the picture above, "zVariable 1" is 5x5, and "zVariable 2" is 5x2. |
| @@ -0,0 +1,343 @@ | |||
| ##################### | |||
| CDF File Requirements | |||
There was a problem hiding this comment.
IMAP-Specific CDF Requirements ?
Do we want to be explicit that these are IMAP specific requirements since you say these are necessary for this mission in the next sentence?
|
|
||
| ISTP Compliant Global Attributes are listed here: `https://spdf.gsfc.nasa.gov/istp_guide/gattributes.html <https://spdf.gsfc.nasa.gov/istp_guide/gattributes.html>`_, and notes about how they are used on IMAP are below - | ||
|
|
||
| Data_type |
There was a problem hiding this comment.
Would this be easier to follow as a table?
Something like three rows: Attribute Name, Valid Options, Description
It is always a pain to make rst tables though, so this doesn't have to happen now and can be a follow-up if someone does want to do that later.
|
|
||
|
|
||
| ****************************** | ||
| File Naming Convention |
There was a problem hiding this comment.
I'm wondering if some of this is getting into the style-guide territory? I'm not sure where I would expect things to live though.
There was a problem hiding this comment.
We could add a link to this section the Naming Conventions doc in the style guide.
| SPDF Validation | ||
| ****************************** | ||
|
|
||
| As a final validation step, the SPDF will review all completed data produts. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `https://cdaweb.gsfc.nasa.gov/cdfx/ <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. |
There was a problem hiding this comment.
Suggested change
| As a final validation step, the SPDF will review all completed data produts. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `https://cdaweb.gsfc.nasa.gov/cdfx/ <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. | |
| As a final validation step, the SPDF will review all completed data produts. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `CDFx <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. |
There was a problem hiding this comment.
This seems like awesome documentation for the cdflib, so I would look into moving it into there if you get the chance. Then we can cross-reference that content from IMAP. But this is more generally useful than just imap_processing IMO.
maxinelasp
reviewed
Nov 10, 2023
| =============== | ||
| **CDF** refers to more than just the files themselves, but the entire toolkit surrounding those files. The official toolkits are available in IDL, C, Fortran and MATLAB. | ||
|
|
||
| For python implementations, there is `**cdflib** <https://github.com/MAVENSDC/cdflib>`_, which is a pure-python implementation of the CDF specification. **cdflib** contains the functions "xarray_to_cdf" and "cdf_to_xarray" to convert between xarray Dataset objects in python and the CDF file formats. |
There was a problem hiding this comment.
Suggested change
| For python implementations, there is `**cdflib** <https://github.com/MAVENSDC/cdflib>`_, which is a pure-python implementation of the CDF specification. **cdflib** contains the functions "xarray_to_cdf" and "cdf_to_xarray" to convert between xarray Dataset objects in python and the CDF file formats. | |
| For python implementations, there is `cdflib <https://github.com/MAVENSDC/cdflib>`_, which is a pure-python implementation of the CDF specification. **cdflib** contains the functions "xarray_to_cdf" and "cdf_to_xarray" to convert between xarray Dataset objects in python and the CDF file formats. |
| .. note:: | ||
| The L0->L2 processing will be using cdflib to generate CDF files on IMAP. For more information about cdflib and the xarray_to_cdf function, see :ref:`cdflib.xarray_to_cdf` | ||
|
|
||
| The other main python library is `**pycdf** <https://spacepy.github.io/pycdf.html>`_, which is a part of **spacepy**. Pycdf wraps the C libraries, and therefore it requires downloading the C libraries. Because it is a part of Spacepy, Pycdf requires downloading and installing Spacepy. Pycdf includes various bells and whistles that help make the files more python friendly, and includes functions for checking for ISTP compliance. |
There was a problem hiding this comment.
Suggested change
| The other main python library is `**pycdf** <https://spacepy.github.io/pycdf.html>`_, which is a part of **spacepy**. Pycdf wraps the C libraries, and therefore it requires downloading the C libraries. Because it is a part of Spacepy, Pycdf requires downloading and installing Spacepy. Pycdf includes various bells and whistles that help make the files more python friendly, and includes functions for checking for ISTP compliance. | |
| The other main python library is `pycdf <https://spacepy.github.io/pycdf.html>`_, which is a part of **spacepy**. Pycdf wraps the C libraries, and therefore it requires downloading the C libraries. Because it is a part of Spacepy, Pycdf requires downloading and installing Spacepy. Pycdf includes various bells and whistles that help make the files more python friendly, and includes functions for checking for ISTP compliance. |
|
|
||
| The other main python library is `**pycdf** <https://spacepy.github.io/pycdf.html>`_, which is a part of **spacepy**. Pycdf wraps the C libraries, and therefore it requires downloading the C libraries. Because it is a part of Spacepy, Pycdf requires downloading and installing Spacepy. Pycdf includes various bells and whistles that help make the files more python friendly, and includes functions for checking for ISTP compliance. | ||
|
|
||
| Because the L0->L2 processing on IMAP will be making heavy use of "xarray", it will also be using "cdflib.xarray_to_cdf" to create the ".cdf" files. |
There was a problem hiding this comment.
Suggested change
| Because the L0->L2 processing on IMAP will be making heavy use of "xarray", it will also be using "cdflib.xarray_to_cdf" to create the ".cdf" files. |
I think this line is unneeded and kind of confusing
|
|
||
| To archive at the SPDF, files must be ISTP compliant. The SPDF will reject files that do not conform to their standards. | ||
|
|
||
| A guide to SPDF can be found here: `https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html <https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html>`_ |
There was a problem hiding this comment.
Suggested change
| A guide to SPDF can be found here: `https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html <https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html>`_ | |
| A guide to SPDF can be found `on the SPDF website. <https://spdf.gsfc.nasa.gov/istp_guide/istp_guide.html>`_ |
bourque
requested changes
Nov 10, 2023
There was a problem hiding this comment.
Awesome work here @bryan-harter, this is all very detailed and helpful! Most of my comments are just spelling and formatting related, but the contents look great.
| * Records | ||
| * Variables can span across multiple records. This is essentially another dimension of the data, but the idea is that each record represents 1 unique physical measurement from an instrument. In the above diagram, both variables have 3 records shown. | ||
|
|
||
| .. note:: For IMAP instruments, each "Record" is always a unit of time. If a varible does not vary over time, it will only span 1 record. |
There was a problem hiding this comment.
Suggested change
| .. note:: For IMAP instruments, each "Record" is always a unit of time. If a varible does not vary over time, it will only span 1 record. | |
| .. note:: For IMAP instruments, each "Record" is always a unit of time. If a variable does not vary over time, it will only span 1 record. |
|
|
||
| There are two parts of a CDF file that a user of the file ultimately cares about: Variables and Attributes. | ||
|
|
||
| The way these are astored in the file are somewhat complex, but that information is abstracted away from the end user. The CDF libraries have functions to easily list and extract the variables and attributes within the CDF files. |
There was a problem hiding this comment.
Suggested change
| The way these are astored in the file are somewhat complex, but that information is abstracted away from the end user. The CDF libraries have functions to easily list and extract the variables and attributes within the CDF files. | |
| The way these are stored in the file are somewhat complex, but that information is abstracted away from the end user. The CDF libraries have functions to easily list and extract the variables and attributes within the CDF files. |
Comment on lines
11
to
13
| ****************************** | ||
| Global Attributes | ||
| ****************************** |
There was a problem hiding this comment.
Suggested change
| ****************************** | |
| Global Attributes | |
| ****************************** | |
| ***************** | |
| Global Attributes | |
| ***************** |
Comment on lines
25
to
26
| Data_version | ||
| ================ |
There was a problem hiding this comment.
Suggested change
| Data_version | |
| ================ | |
| Data_version | |
| ============ |
Comment on lines
29
to
30
| Descriptor | ||
| ================ |
There was a problem hiding this comment.
Suggested change
| Descriptor | |
| ================ | |
| Descriptor | |
| ========== |
Comment on lines
17
to
19
| ****************************** | ||
| SPDF Validation | ||
| ****************************** |
There was a problem hiding this comment.
Suggested change
| ****************************** | |
| SPDF Validation | |
| ****************************** | |
| **************** | |
| SPDF Validation | |
| **************** |
| SPDF Validation | ||
| ****************************** | ||
|
|
||
| As a final validation step, the SPDF will review all completed data produts. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `https://cdaweb.gsfc.nasa.gov/cdfx/ <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. |
There was a problem hiding this comment.
Suggested change
| As a final validation step, the SPDF will review all completed data produts. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `https://cdaweb.gsfc.nasa.gov/cdfx/ <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. | |
| As a final validation step, the SPDF will review all completed data products. They will run the CDF file through the SKTeditor as a first pass, and also ensure that the auto-generated plots look nice on CDAWeb using the IDL tool `https://cdaweb.gsfc.nasa.gov/cdfx/ <https://cdaweb.gsfc.nasa.gov/cdfx/>`_. |
Comment on lines
39
to
43
| release-workflow | ||
| style-guide/style-guide | ||
| tools/index | ||
| CDFs/index | ||
| poetry |
There was a problem hiding this comment.
Since we now have quite a growing list here, maybe we should just put these in alphabetical order?
| .. _Pertaining-to-the-code: | ||
| .. _contributing-to-the-code: |
There was a problem hiding this comment.
Nice, I like this change!
| "These are used by the variable imap_jim_counts_(mode)_(description)" | ||
| "Ideally we would actually have additional variables that describe " | ||
| "the minimum and the maximum values in each of these bins, but for our " | ||
| "purposes of creating a minimal example, we'll leave extra varaibles out. " |
There was a problem hiding this comment.
Suggested change
| "purposes of creating a minimal example, we'll leave extra varaibles out. " | |
| "purposes of creating a minimal example, we'll leave extra variables out. " |
bourque
approved these changes
Nov 10, 2023
tech3371
reviewed
Nov 10, 2023
| IMAP xarray_to_cdf Example | ||
| *************************** | ||
|
|
||
| Suppose there is an instrument on IMAP named JIM that took 3 samples of ions at 3 different energies, such that the data looks like the following - |
There was a problem hiding this comment.
I stole the idea from Maxine, who gave Heather a test for something involving a JIM satellite
There was a problem hiding this comment.
I stole the idea from Maxine, who gave Heather a test to design an SDC for a JIM instrument
Comment on lines
+228
to
+229
| "Epoch": epoch_xr, | ||
| "Energy": energy_xr |
There was a problem hiding this comment.
Does coordinates look up by 'name' and not the dims? The reason I ask is I noticed that the coordinate's name and xr.DataArray's dims name did not match but it match with xr.DataArray's name.
There was a problem hiding this comment.
In general, the coordinate's name and the dimension's name are separate things. You could name them whatever you wanted for our purposes, as long as you are consistent within the xarray object. The actual dimension names ("energy_dim" in this case) doesn't show up anywhere inside the CDF file, so it's purely for the xarray Dataset to link the two variables together.
I admit it's a bit confusing how I set it up, because I just named one dimension "Epoch", but I added a "_dim" to "Energy". But my goal in doing this was to show that it doesn't ultimately matter what you name the dimensions.
There was a problem hiding this comment.
Thank you for clarifying this!
maxinelasp
pushed a commit
to maxinelasp/imap_processing
that referenced
this pull request
Nov 16, 2023
* CDF documentation update * Adding the cdf_guide.rst to the toc * Formatting update! * Updating the docs for sections I missed * Formatting a little nicer * Adding links enabled in the docs * Fixing duplicate headers * Getting rid of duplicate headers * Fixing more documentation * Responding to a few code review comments * Breaking up the page into sub-pages * Adding a link to xarray_to_cdf page * Adding more links to different sections * fixing image link * Editting some of the wording now that we're on different pages * Typos and format fixed * Another format error
maxinelasp
added a commit
that referenced
this pull request
Jan 8, 2024
* update telemetry_generator to be more generic * Updates to telemetry generator * updating docs * updating to remove local-specific testing changes * first pass at glows xtce * update way of checking if description exists or not. update generator template. * updated generator to use correct IntegerParameterType * First pass at GLOWS decom * updating telemetry generator * Updating glows decom to properly process data into dictionary * Finished histogram decom * Updating decom to include direct events * Finalizing code, adding smaller test data file and combined XML, updating documentation and tests * Updating test file path * Fixing test runs * updating docs * Updating data classes to use attributes, and changing names * Removing keys from data class and getting attribute names from the packet * Move INT to SINT in telemetry_generator Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> * Addressing code review comments * Fixing test failure * Review updates * fixing typo * Updating IntEnum to Enum to fix automatic documentation * Move L0 data classes to a dataclass type with specific attributes * Initial GLOWS L1A data classes * Adding required attributes to L0 data class * Adding L1A histogram data class * fixing test failures * Finishing ccsds dataclass * Minor updates * Finishing GLOWS L1A histograms * Updating to include direct events data * Updating glows l1a * Adding useful GLOWS code from GLOWS team and Marek Strumik * Adding direct event information Co-authored-by: GLOWS team <maro@cbk.waw.pl> * Updates for direct event creation * Updates to L1A direct events * Updated release workflow to reflect removal of "main" branch and use of version branches * Reduced the filesize of git-workflow.png * Glows L0 changes to data class to accomodate L1A needs (#264) * Glows L0 changes to data class to accomodate L1A needs * DOC: Add intersphinx to documentation for external references * DOC: Add a clean option to the documentation Makefile build command * DOC: Update glows types in dataclasses * DOC: Change glows currentmodule references * DOC: Add ccsds header to glows documentation * DOC: Add inherited member description for glows This adds the inherited attributes from the dataclass directly into the subclasses. There may be a better way to do this, so perhaps some setting/configuration parameter needs to be updated? But this at least fixes the warnings for now. * FIX: Run black on glows test * PR updates --------- Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> * Housekeeping data testing and validation for CoDICE l0 (#176) Validated housekeeping data with tests plus minor updates --------- Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> * L1 cdf creation (#179) * Extracting and adding new data to the IDEX L1 * Adding some of the values to the XML, rather than shifting bits in the code (work in progress, need to do a few more variables) * Finishing up having space packet parser decom * Getting rid of unecessary bit masksi8kt9gloh * making changes based on comments * Formatting the file after the upstream sync * Adding L1 CDF creation * Forgot to commit poetry changes * Making changes based on Greg's comments, as well as issues 1-8 from SPDF * Fixing import statements * Changing version into a string * Adding more descriptions, units, and labels in the attributes * Fixing the way strings are inserted into the CDF * Fixing issues with white spice in the text of attrs * Files are finally ISTP compliant! * Fixing a couple things for ISTP * Had the contents of the 32 bit things reversed * Fixing the packet definition for the CCSDS file * More fixes to ensure ISTP compliance * Updating the file name * touching finishes on the L1 CDF * Adding fixture for temp directory in tests * Fixing one of the ruff errors by adding a namedtuple * Adding new "write_cdf" function that writed cdfs based on attributes * Getting rid of a few more magic numbers * Updating the packet definition * initial hit l1a decom (#164) * initial hit l1a decom * using defaultdict, add docs * fixed hit l1a docstrings * fixed hit l1a docstring formatting * doc updates and minor fixes * minor docstring update * minor comment update * rebased with dev * fixed ruff issues * regenerated xtce * fixed unit test * github doc gen fix attempt * added class docstring attributes * fixed more doc issues * another doc fix attempt * doc fix attempt * doc fix attempt * changed doc python v to 3.10 * I-ALiRT: L0 parsing (#246) * adding ialirt * CDF documentation update (#273) * CDF documentation update * Adding the cdf_guide.rst to the toc * Formatting update! * Updating the docs for sections I missed * Formatting a little nicer * Adding links enabled in the docs * Fixing duplicate headers * Getting rid of duplicate headers * Fixing more documentation * Responding to a few code review comments * Breaking up the page into sub-pages * Adding a link to xarray_to_cdf page * Adding more links to different sections * fixing image link * Editting some of the wording now that we're on different pages * Typos and format fixed * Another format error * Create CDF class based system (#272) * Updating IDEX CDF attrs * remove comments, create constants class * Updating names and addressing PR comments * Updating docs * Adding tests * Updating types of L0 to properly read bytearrays * Updating tests * GLOWS direct events compressed and uncompressed * updating decom dataclasses * Updating GLOWS L0 * L1A tests * removing unneeded code * updating tests and fixing minor bug * fixing autodocs * addressing comments from code review * Updating constants to a frozen dataclass * Updating direct events packet reading to correctly read packet data, updating test data, adding tests * Updating docstrings --------- Co-authored-by: Tenzin Choedon <tenzin.choedon@lasp.colorado.edu> Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> Co-authored-by: GLOWS team <maro@cbk.waw.pl> Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> Co-authored-by: Gabriel M <104743000+GFMoraga@users.noreply.github.com> Co-authored-by: Bryan Harter <41062454+bryan-harter@users.noreply.github.com> Co-authored-by: Sean Hoyt <sean.hoyt@lasp.colorado.edu> Co-authored-by: Laura Sandoval <46567335+laspsandoval@users.noreply.github.com>
|
@all-contributors please add @bryan-harter for documentation |
|
I've put up a pull request to add @bryan-harter! 🎉 |
laspsandoval
pushed a commit
to laspsandoval/imap_processing
that referenced
this pull request
Apr 2, 2024
* CDF documentation update * Adding the cdf_guide.rst to the toc * Formatting update! * Updating the docs for sections I missed * Formatting a little nicer * Adding links enabled in the docs * Fixing duplicate headers * Getting rid of duplicate headers * Fixing more documentation * Responding to a few code review comments * Breaking up the page into sub-pages * Adding a link to xarray_to_cdf page * Adding more links to different sections * fixing image link * Editting some of the wording now that we're on different pages * Typos and format fixed * Another format error
laspsandoval
added a commit
to laspsandoval/imap_processing
that referenced
this pull request
Apr 2, 2024
* update telemetry_generator to be more generic * Updates to telemetry generator * updating docs * updating to remove local-specific testing changes * first pass at glows xtce * update way of checking if description exists or not. update generator template. * updated generator to use correct IntegerParameterType * First pass at GLOWS decom * updating telemetry generator * Updating glows decom to properly process data into dictionary * Finished histogram decom * Updating decom to include direct events * Finalizing code, adding smaller test data file and combined XML, updating documentation and tests * Updating test file path * Fixing test runs * updating docs * Updating data classes to use attributes, and changing names * Removing keys from data class and getting attribute names from the packet * Move INT to SINT in telemetry_generator Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> * Addressing code review comments * Fixing test failure * Review updates * fixing typo * Updating IntEnum to Enum to fix automatic documentation * Move L0 data classes to a dataclass type with specific attributes * Initial GLOWS L1A data classes * Adding required attributes to L0 data class * Adding L1A histogram data class * fixing test failures * Finishing ccsds dataclass * Minor updates * Finishing GLOWS L1A histograms * Updating to include direct events data * Updating glows l1a * Adding useful GLOWS code from GLOWS team and Marek Strumik * Adding direct event information Co-authored-by: GLOWS team <maro@cbk.waw.pl> * Updates for direct event creation * Updates to L1A direct events * Updated release workflow to reflect removal of "main" branch and use of version branches * Reduced the filesize of git-workflow.png * Glows L0 changes to data class to accomodate L1A needs (IMAP-Science-Operations-Center#264) * Glows L0 changes to data class to accomodate L1A needs * DOC: Add intersphinx to documentation for external references * DOC: Add a clean option to the documentation Makefile build command * DOC: Update glows types in dataclasses * DOC: Change glows currentmodule references * DOC: Add ccsds header to glows documentation * DOC: Add inherited member description for glows This adds the inherited attributes from the dataclass directly into the subclasses. There may be a better way to do this, so perhaps some setting/configuration parameter needs to be updated? But this at least fixes the warnings for now. * FIX: Run black on glows test * PR updates --------- Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> * Housekeeping data testing and validation for CoDICE l0 (IMAP-Science-Operations-Center#176) Validated housekeeping data with tests plus minor updates --------- Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> * L1 cdf creation (IMAP-Science-Operations-Center#179) * Extracting and adding new data to the IDEX L1 * Adding some of the values to the XML, rather than shifting bits in the code (work in progress, need to do a few more variables) * Finishing up having space packet parser decom * Getting rid of unecessary bit masksi8kt9gloh * making changes based on comments * Formatting the file after the upstream sync * Adding L1 CDF creation * Forgot to commit poetry changes * Making changes based on Greg's comments, as well as issues 1-8 from SPDF * Fixing import statements * Changing version into a string * Adding more descriptions, units, and labels in the attributes * Fixing the way strings are inserted into the CDF * Fixing issues with white spice in the text of attrs * Files are finally ISTP compliant! * Fixing a couple things for ISTP * Had the contents of the 32 bit things reversed * Fixing the packet definition for the CCSDS file * More fixes to ensure ISTP compliance * Updating the file name * touching finishes on the L1 CDF * Adding fixture for temp directory in tests * Fixing one of the ruff errors by adding a namedtuple * Adding new "write_cdf" function that writed cdfs based on attributes * Getting rid of a few more magic numbers * Updating the packet definition * initial hit l1a decom (IMAP-Science-Operations-Center#164) * initial hit l1a decom * using defaultdict, add docs * fixed hit l1a docstrings * fixed hit l1a docstring formatting * doc updates and minor fixes * minor docstring update * minor comment update * rebased with dev * fixed ruff issues * regenerated xtce * fixed unit test * github doc gen fix attempt * added class docstring attributes * fixed more doc issues * another doc fix attempt * doc fix attempt * doc fix attempt * changed doc python v to 3.10 * I-ALiRT: L0 parsing (IMAP-Science-Operations-Center#246) * adding ialirt * CDF documentation update (IMAP-Science-Operations-Center#273) * CDF documentation update * Adding the cdf_guide.rst to the toc * Formatting update! * Updating the docs for sections I missed * Formatting a little nicer * Adding links enabled in the docs * Fixing duplicate headers * Getting rid of duplicate headers * Fixing more documentation * Responding to a few code review comments * Breaking up the page into sub-pages * Adding a link to xarray_to_cdf page * Adding more links to different sections * fixing image link * Editting some of the wording now that we're on different pages * Typos and format fixed * Another format error * Create CDF class based system (IMAP-Science-Operations-Center#272) * Updating IDEX CDF attrs * remove comments, create constants class * Updating names and addressing PR comments * Updating docs * Adding tests * Updating types of L0 to properly read bytearrays * Updating tests * GLOWS direct events compressed and uncompressed * updating decom dataclasses * Updating GLOWS L0 * L1A tests * removing unneeded code * updating tests and fixing minor bug * fixing autodocs * addressing comments from code review * Updating constants to a frozen dataclass * Updating direct events packet reading to correctly read packet data, updating test data, adding tests * Updating docstrings --------- Co-authored-by: Tenzin Choedon <tenzin.choedon@lasp.colorado.edu> Co-authored-by: Tenzin Choedon <36522642+tech3371@users.noreply.github.com> Co-authored-by: GLOWS team <maro@cbk.waw.pl> Co-authored-by: Matthew Bourque <matthew.bourque@lasp.colorado.edu> Co-authored-by: Greg Lucas <greg.m.lucas@gmail.com> Co-authored-by: Gabriel M <104743000+GFMoraga@users.noreply.github.com> Co-authored-by: Bryan Harter <41062454+bryan-harter@users.noreply.github.com> Co-authored-by: Sean Hoyt <sean.hoyt@lasp.colorado.edu> Co-authored-by: Laura Sandoval <46567335+laspsandoval@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Summary
Added CDF file Documentation
Overview
Added new documentation regarding the creation of CDF files. This goes over the required attributes to make an ISTP compliant CDF file, and how xarray_to_cdf works.
New Files
cdf_guide.rst