Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cdf attribute tests #637

Merged
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
24cc329
Writing initial TODO comments.
anamanica Jun 13, 2024
f7d0262
Troubleshooting
anamanica Jun 13, 2024
484d838
Schema tests passed
anamanica Jun 14, 2024
56389b2
I think I understand instrument_id
anamanica Jun 14, 2024
af3f48f
Fixing test file
anamanica Jun 17, 2024
64a3672
Var attribute tests
anamanica Jun 17, 2024
d336f90
First draft
anamanica Jun 17, 2024
6258bca
Deleting file
anamanica Jun 17, 2024
34c7a98
Test
anamanica Jun 17, 2024
fbceece
Fixing pulled errors
anamanica Jun 17, 2024
5348fb7
Quick
anamanica Jun 17, 2024
8dc2b82
Trying different things to pass pre-checks
anamanica Jun 18, 2024
1d56ef9
Testing
anamanica Jun 18, 2024
9e14421
TEST
anamanica Jun 18, 2024
b1c5209
Fixing failed pre-check
anamanica Jun 18, 2024
2423921
Test2
anamanica Jun 18, 2024
4e1990f
Fixing pre-commit issues
anamanica Jun 18, 2024
79f3ba9
Fixing prechecks
anamanica Jun 18, 2024
a6426c0
Removing test statements
anamanica Jun 18, 2024
6936749
Moving files
anamanica Jun 18, 2024
33bd956
Fixing PR draft comments
anamanica Jun 18, 2024
4c3c4f2
Adding additional tests
anamanica Jun 18, 2024
4f6e3fb
Breaking up test functions
anamanica Jun 18, 2024
9e43ad8
Adding more tests
anamanica Jun 18, 2024
3d797d1
Getting closer to final draft
anamanica Jun 18, 2024
0bb6b88
Finishing touches
anamanica Jun 18, 2024
d2fe904
Codcov check
anamanica Jun 20, 2024
4045f94
Codcov check
anamanica Jun 20, 2024
84b78a9
Else Testing
anamanica Jun 20, 2024
c454d3c
Hopefully done
anamanica Jun 20, 2024
be0d064
Fixing PR comments
anamanica Jun 20, 2024
f446db3
More work done
anamanica Jun 21, 2024
67271cc
Cleaning up
anamanica Jun 21, 2024
0bff645
Figured out Path issue
anamanica Jun 21, 2024
e1644bf
Final touches (again)
anamanica Jun 21, 2024
1029e2d
Comment mods
anamanica Jun 24, 2024
db68128
Pre-commit check
anamanica Jun 24, 2024
53cbb04
Pre-commit numpydoc
anamanica Jun 24, 2024
da12644
Pre
anamanica Jun 24, 2024
4a7712a
Final Final Commit
anamanica Jun 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Finishing touches
  • Loading branch information
anamanica committed Jun 25, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 0bb6b88074d426427846801c4bb97a0fa01f885c
8 changes: 6 additions & 2 deletions imap_processing/cdf/imap_cdf_manager.py
Original file line number Diff line number Diff line change
@@ -13,9 +13,13 @@
class ImapCdfAttributes(CdfAttributeManager):
"""Contains IMAP specific tools and settings for CDF management."""

def __init__(self):
def __init__(self, source_dir_input=None):
anamanica marked this conversation as resolved.
Show resolved Hide resolved
"""Set the path to the config directory."""
super().__init__(Path(__file__).parent / "config")
if source_dir_input is None:
super().__init__(Path(__file__).parent / "config")
else:
super().__init__(Path(__file__).parent / "config")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should pass the source_dir_input into the super() init function because there are steps in __init__ that depend on the source_dir and it would be confusing to have it set to different values for the init vs for the rest of processing.

Then you can remove line 21

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has actually been causing me a lot of weird struggles with testing this class. In CdfAttributeManager , the default schema and IMAPS default variables are in the "config" file path and not guaranteed in the source_dir_input file path. I can't get the tests to run correctly without loading in that default info first, and then switching the file path... Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In those tests, you can pass in the "config" path (or pass in nothing and use the default) and then override it. This way it will still get the default schemas and global variables from "config" but you can also access your test files in the other file path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hahaha, happy to say I just got it. Thank you so much! I see now.

self.source_dir = Path(__file__).parent / source_dir_input

def add_instrument_global_attrs(self, instrument: str):
"""
24 changes: 24 additions & 0 deletions imap_processing/cdf/tests/imap_instrument2_global_cdf_attrs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
instrument_base: &instrument_base
Descriptor: SWE>Solar Wind Electron
TEXT: >
The IMAP Solar Wind Electron (SWE) instrument measures the solar wind electrons
in the heliosphere. SWE will contribute to our understanding of the acceleration
and transport of charged particles in the heliosphere.
SWE design and assembly is led by the Princeton Plasma Physics Laboratory. See
https://imap.princeton.edu/instruments/swe for more details.
Instrument_type: "Particles (space)"

imap_swe_l1a_sci:
<<: *instrument_base
# NOTE: Right now, this Data_level is required to produce valid CDF
anamanica marked this conversation as resolved.
Show resolved Hide resolved
Data_level: 1A
Data_type: L1A_SCI>Level-1A Science data
Logical_source: imap_swe_l1a_sci
Logical_source_description: SWE Instrument Level-1A Science Data

imap_swe_l1b_sci:
<<: *instrument_base
Data_level: 1A
Data_type: L1B_SCI>Level-1B Science data
Logical_source: imap_swe_l1b_sci
Logical_source_description: SWE Instrument Level-1B Science Data
310 changes: 310 additions & 0 deletions imap_processing/cdf/tests/imap_instrument2_level2_variable_attrs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
default_attrs: &default
# Assumed values for all variable attrs unless overwritten
DEPEND_0: epoch
DISPLAY_TYPE: time_series
FILLVAL: -9223372036854775808
FORMAT: I19
VALIDMIN: 0
VALIDMAX: 9223372036854775807
VAR_TYPE: data

epoch:
<<: *default
CATDESC: Time, number of nanoseconds since J2000 with leap seconds included
FIELDNAM: epoch
LABLAXIS: epoch
UNITS: ns
VAR_TYPE: support_data
SCALETYP: linear
MONOTON: INCREASE
TIME_BASE: J2000
TIME_SCALE: Terrestrial Time
REFERENCE_POSITION: Rotating Earth Geoid

x_front:
<<: *default
CATDESC: x front position
FIELDNAM: Event x-position on front foil
LABLAXIS: x front position
# TODO: come back to format
UNITS: mm

y_front:
<<: *default
CATDESC: y front position
FIELDNAM: Event y-position on front foil
LABLAXIS: y front position
# TODO: come back to format
UNITS: mm

x_back:
<<: *default
CATDESC: x back position
FIELDNAM: x_back
LABLAXIS: x back position
# TODO: come back to format
UNITS: mm

y_back:
<<: *default
CATDESC: y back position
FIELDNAM: y_back
LABLAXIS: y back position
# TODO: come back to format
UNITS: mm

x_coin:
<<: *default
CATDESC: x coincidence position
FIELDNAM: x_coin
LABLAXIS: x coincidence position
# TODO: come back to format
UNITS: mm

tof_start_stop:
<<: *default
CATDESC: Particle time of flight from start to stop
FIELDNAM: tof_start_stop
LABLAXIS: tof start stop
# TODO: come back to format
UNITS: ns

tof_stop_coin:
<<: *default
CATDESC: Particle time of flight from stop to coincidence
FIELDNAM: tof_stop_coin
LABLAXIS: tof stop coin
# TODO: come back to format
UNITS: ns

tof_corrected:
<<: *default
CATDESC: Corrected time of flight
FIELDNAM: tof_corrected
LABLAXIS: tof corrected
# TODO: come back to format
UNITS: ns

eventtype:
<<: *default
CATDESC: Eventtype (1-2 top and bottom stop types; 8-15 ssd stop types)
FIELDNAM: eventtype
LABLAXIS: eventtype
# TODO: come back to format
UNITS: " "

vx_ultra:
<<: *default
CATDESC: Normalized component of the velocity vector in x direction in the instrument frame.
FIELDNAM: vx_ultra
LABLAXIS: vx ultra
# TODO: come back to format
UNITS: " "

vy_ultra:
<<: *default
CATDESC: Normalized component of the velocity vector in y direction in the instrument frame.
FIELDNAM: vy_ultra
LABLAXIS: vy ultra
# TODO: come back to format
UNITS: " "

vz_ultra:
<<: *default
CATDESC: Normalized component of the velocity vector in z direction in the instrument frame.
FIELDNAM: vz_ultra
LABLAXIS: vz ultra
# TODO: come back to format
UNITS: " "

energy:
<<: *default
CATDESC: Energy measured using the pulse height from the stop anode (DN) or Energy measured using the SSD (keV) depending on eventtype.
FIELDNAM: energy
LABLAXIS: energy
# TODO: come back to format
UNITS: keV

species:
<<: *default
CATDESC: Species bin.
FIELDNAM: species
LABLAXIS: species
# TODO: come back to format
UNITS: " "

event_efficiency:
<<: *default
CATDESC: Estimated event efficiency for this path through the instrument.
FIELDNAM: event_efficiency
LABLAXIS: event efficiency
# TODO: come back to format
UNITS: " "

vx_sc:
<<: *default
CATDESC: Normalized component of the velocity vector in x direction in the spacecraft frame.
FIELDNAM: vx_sc
LABLAXIS: vx sc
# TODO: come back to format
UNITS: " "

vy_sc:
<<: *default
CATDESC: Normalized component of the velocity vector in y direction in the spacecraft frame.
FIELDNAM: vy_sc
LABLAXIS: vy sc
# TODO: come back to format
UNITS: " "

vz_sc:
<<: *default
CATDESC: Normalized component of the velocity vector in z direction in the spacecraft frame.
FIELDNAM: vz_sc
LABLAXIS: vz sc
# TODO: come back to format
UNITS: " "

vx_dps_sc:
<<: *default
CATDESC: Normalized component of the velocity vector in x direction in the DPS frame at rest WRT spacecraft (velocity w/ spacecraft frame subtracted)
FIELDNAM: vx_dps_sc
LABLAXIS: vx dps sc
# TODO: come back to format
UNITS: " "

vy_dps_sc:
anamanica marked this conversation as resolved.
Show resolved Hide resolved
<<: *default
CATDESC: Normalized component of the velocity vector in y direction in the DPS frame at rest WRT spacecraft (velocity w/ spacecraft frame subtracted)
FIELDNAM: vy_dps_sc
LABLAXIS: vy dps sc
# TODO: come back to format
UNITS: " "

vz_dps_sc:
<<: *default
CATDESC: Normalized component of the velocity vector in z direction in the DPS frame at rest WRT spacecraft (velocity w/ spacecraft frame subtracted)
FIELDNAM: vz_dps_sc
LABLAXIS: vz dps sc
# TODO: come back to format
UNITS: " "

vx_dps_helio:
<<: *default
CATDESC: Normalized component of the velocity vector in x direction in the DPS frame at rest WRT heliosphere.
FIELDNAM: vx_dps_helio
LABLAXIS: vx dps helio
# TODO: come back to format
UNITS: " "

vy_dps_helio:
<<: *default
CATDESC: Normalized component of the velocity vector in y direction in the DPS frame at rest WRT heliosphere.
FIELDNAM: vy_dps_helio
LABLAXIS: vy dps helio
# TODO: come back to format
UNITS: " "

vz_dps_helio:
<<: *default
CATDESC: Normalized component of the velocity vector in z direction in the DPS frame at rest WRT heliosphere.
FIELDNAM: vz_dps_helio
LABLAXIS: vz dps helio
# TODO: come back to format
UNITS: " "

eventtimes:
<<: *default
CATDESC: Event times calculated from event and universal spin table.
FIELDNAM: eventtimes
LABLAXIS: event times
# TODO: come back to format
UNITS: " "

spin_number:
<<: *default
CATDESC: Spin number from Universal Spin Table.
FIELDNAM: spin_number
LABLAXIS: spin_number
# TODO: come back to format
UNITS: " "

spin_start_time:
<<: *default
CATDESC: Spin start time from Universal Spin Table.
FIELDNAM: spin_start_time
LABLAXIS: spin start time
# TODO: come back to format
UNITS: s

avg_spin_period:
<<: *default
CATDESC: Average spin period from Universal Spin Table.
FIELDNAM: avg_spin_period
LABLAXIS: avg_spin_period
# TODO: come back to format
UNITS: s

rate_start_pulses:
<<: *default
CATDESC: Rate of start pulses (/s).
FIELDNAM: rate_start_pulses
LABLAXIS: rate start pulses
# TODO: come back to format
UNITS: 1/s

rate_stop_pulses:
<<: *default
CATDESC: Rate of stop pulses (/s).
FIELDNAM: rate_stop_pulses
LABLAXIS: rate stop pulses
# TODO: come back to format
UNITS: 1/s

rate_coin_pulses:
<<: *default
CATDESC: Rate of coincidence pulses (/s).
FIELDNAM: rate_coin_pulses
LABLAXIS: rate coincidence pulses
# TODO: come back to format
UNITS: 1/s

rate_processed_events:
<<: *default
CATDESC: Rate of processed events (/s).
FIELDNAM: rate_processed_events
LABLAXIS: rate processed events
# TODO: come back to format
UNITS: 1/s

rate_rejected_events:
<<: *default
CATDESC: Rate of rejected events (/s).
FIELDNAM: rate_rejected_events
LABLAXIS: rate rejected events
# TODO: come back to format
UNITS: 1/s

quality_hk:
<<: *default
CATDESC: Spin filter derived from Ultra housekeeping. Bitwise flagging used to filter the spin.
FIELDNAM: quality_hk
LABLAXIS: quality hk
# TODO: come back to format
UNITS: " "

quality_attitude:
<<: *default
CATDESC: Spin filter derived from IMAP attitude. Bitwise flagging used to filter the spin.
FIELDNAM: quality_attitude
LABLAXIS: quality attitude
# TODO: come back to format
UNITS: " "

quality_instruments:
<<: *default
CATDESC: Spin filter derived from instruments other than Ultra. Bitwise flagging used to filter the spin.
FIELDNAM: quality_instruments
LABLAXIS: quality instruments
# TODO: come back to format
UNITS: " "
Loading