Updating CLI to fix MAG #404
Conversation
maxinelasp
requested review from
a team,
bourque,
sdhoyt,
greglucas,
subagonsouth,
tech3371,
vmartinez-cu and
laspsandoval
and removed request for
a team
| version = f"v{int(dataset.attrs['Data_version']):03d}" # vXXX | ||
| r = re.compile(r"v\d{3}") | ||
| if r.match(dataset.attrs["Data_version"]) is None: | ||
| version = f"v{int(dataset.attrs['Data_version']):03d}" # vXXX |
There was a problem hiding this comment.
The original version was throwing errors because batch starter provides the version as "vXXX"
There was a problem hiding this comment.
I see. can you add this comment in the code?
| @@ -78,6 +84,11 @@ def process_and_write_data( | |||
| if not packet_data: | |||
| return [] | |||
|
|
|||
| # TODO: Rework attrs to be better | |||
| raw_attrs["Data_version"] = data_version | |||
There was a problem hiding this comment.
This is a temporary fix, I wanted to run L1A in the cloud today for MAG but I have rewritten the way these attributes work in my WIP branch.
| version = f"v{int(dataset.attrs['Data_version']):03d}" # vXXX | ||
| r = re.compile(r"v\d{3}") | ||
| if r.match(dataset.attrs["Data_version"]) is None: | ||
| version = f"v{int(dataset.attrs['Data_version']):03d}" # vXXX |
There was a problem hiding this comment.
I see. can you add this comment in the code?
| raw_attrs: dict, | ||
| mago_attrs: dict, | ||
| magi_attrs: dict, | ||
| data_version: str, |
There was a problem hiding this comment.
It is unclear to me which part of these changes is a temporary. If the addition of data_version to this function signature is not temporary, please add it to the docstring.
There was a problem hiding this comment.
All of it is temporary, including data_version in this change. I need to add a bunch of global attributes so I'm reworking the way this works.
| logger.info(f"Uploading file: {filename_burst}") | ||
| imap_data_access.upload(filename_burst) | ||
| if len(output_files) == 0: | ||
| print("No files to upload.") |
There was a problem hiding this comment.
Should these prints be logging calls?
There was a problem hiding this comment.
When they were logging calls, the AWS batch runner wouldn't print them into the cloudwatch logs
imap_processing/cdf/utils.py
Outdated
| @@ -66,7 +67,11 @@ def write_cdf(dataset: xr.Dataset): | |||
| start_time = np.datetime_as_string(dataset["epoch"].values[0], unit="D").replace( | |||
| "-", "" | |||
| ) | |||
| version = f"v{int(dataset.attrs['Data_version']):03d}" # vXXX | |||
| r = re.compile(r"v\d{3}") | |||
| if r.match(dataset.attrs["Data_version"]) is None: | |||
There was a problem hiding this comment.
r.match will raise a TypeError if dataset.attrs["Data_version"] is not a string. Is this a possibility?
There was a problem hiding this comment.
It is a possibility but I think it would be an incorrect usage of dataset attributes. But, I can add a catch here if it's an int.
maxinelasp
merged commit b248149
into
IMAP-Science-Operations-Center:dev
Change Summary
Missed this section during my refactor. Just a minor change to remove old filename management.