Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asdf-Support #708

Merged
merged 35 commits into from
Aug 22, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
39faa60
initial commit
ViciousEagle03 May 20, 2024
9644bbb
version alignment
ViciousEagle03 May 20, 2024
af7dccb
Remove method to remove any delay and
ViciousEagle03 May 22, 2024
67b8805
Update tag and
ViciousEagle03 May 22, 2024
b62ac94
enable schema test
ViciousEagle03 May 22, 2024
e2928ea
revert small change
ViciousEagle03 May 22, 2024
c2b240f
Add the converters for ExtraCoords and TimeTableCoordinate class
ViciousEagle03 May 30, 2024
94f2065
Add ExtraCoords and TimeTableCoord Schema
ViciousEagle03 May 30, 2024
cc65cc3
Update manifest and entry_point.py
ViciousEagle03 May 30, 2024
5ffb493
Add the QuantityTableCoordinate and SkyCoordTableCoordinate converters
ViciousEagle03 May 31, 2024
695f8f6
Add the schema for QuantityTableCoordinate and SkyCoordTableCoordinate
ViciousEagle03 May 31, 2024
9e38352
Update manifest and entry_point.py
ViciousEagle03 May 31, 2024
a9c5868
Add validation for schema and manifests
ViciousEagle03 Jun 4, 2024
835c5c7
Merge branch 'asdf-support' into asdf_basic_support
Cadair Jun 5, 2024
3bb70e4
pre-commit
Cadair Jun 5, 2024
04409b5
Update the tox.ini and CI workflow
ViciousEagle03 Jun 5, 2024
76cc27e
Update schemas
ViciousEagle03 Jun 5, 2024
3ea5e65
Add the converter and schema for GlobalCoords
ViciousEagle03 Jun 17, 2024
a31f5fc
Update converters
ViciousEagle03 Jun 17, 2024
f313395
Update entry_points,schemas and manifest
ViciousEagle03 Jun 17, 2024
3d9d9e8
minor change
ViciousEagle03 Jun 19, 2024
d031c35
lowercase schema URIs and use wildcards
ViciousEagle03 Jun 21, 2024
171dcd6
Add tests and GWCS objects
ViciousEagle03 Jul 1, 2024
0c7135f
merge from upstream/asdf-support
ViciousEagle03 Jul 4, 2024
0a84f8b
revert small change
ViciousEagle03 Jul 4, 2024
cd31d95
apply suggestions from code review
ViciousEagle03 Jul 6, 2024
7c04d35
Remove mesh as a property validator
ViciousEagle03 Jul 11, 2024
266f88e
Update the dependencies version
ViciousEagle03 Jul 11, 2024
67358b4
Style changes and add warnings to NDCube converter
ViciousEagle03 Jul 19, 2024
6e7a338
env name update
ViciousEagle03 Jul 19, 2024
c1a6b49
Add asdf as an optional dep
ViciousEagle03 Aug 4, 2024
0546cad
Remove asdf as an optional dependency
ViciousEagle03 Aug 5, 2024
e072a9a
Add support for meta in converter and schema
ViciousEagle03 Aug 6, 2024
5cfa4f9
Add mask as a validator property
ViciousEagle03 Aug 21, 2024
e8268fc
Merge branch 'asdf-support' into asdf_basic_support
Cadair Aug 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added ndcube/asdf/__init__.py
Empty file.
29 changes: 29 additions & 0 deletions ndcube/asdf/converters/extracoords_converter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from asdf.extension import Converter


class ExtraCoordsConverter(Converter):
tags = ["tag:sunpy.org:ndcube/extra_coords/extra_coords/ExtraCoords-*"]
types = ["ndcube.extra_coords.extra_coords.ExtraCoords"]

def from_yaml_tree(self, node, tag, ctx):
from ndcube.extra_coords.extra_coords import ExtraCoords
extra_coords = ExtraCoords()
extra_coords._wcs = node.get("wcs")
Cadair marked this conversation as resolved.
Show resolved Hide resolved
extra_coords._mapping = node.get("mapping")
if "lookup_tables" in node:
extra_coords._lookup_tables = node.get("lookup_tables")
extra_coords._dropped_tables = node.get("dropped_tables")
extra_coords._ndcube = node.get("ndcube")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@braingram @ViciousEagle03 is asdf smart enough to handle this circular reference and not save out the ndcube object twice? i.e. I save an NDCube which has an ExtraCoords which references the same NDCube?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically yes but I believe the converter will need to be updated.
https://asdf.readthedocs.io/en/latest/asdf/extending/converters.html#reference-cycles
That seems like a good test case.

Copy link
Member Author

@ViciousEagle03 ViciousEagle03 Jul 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@braingram, for the current implementation I believe it is still not storing the NDCube object twice, right?
And also in the current from_yaml_tree method, it doesn't have yield but it can still deserialize the extra_coords object, that is, when the extra_coords converter is called the extra_coords.ndcube shows _PendingValue and when it gets picked up in the NDCube converter and the ndcube.extra_coords is initialized the reference to the ndcube is properly mapped in the ndcube.extra_coords._ndcube.

We can see the deserialized NDCube object and the NDCube reference address for the ExtraCoords associated is the same for the below.

(Pdb) ndcube.extra_coords._ndcube
<ndcube.ndcube.NDCube object at 0x7c84fe60c2d0>
NDCube
------
Shape: (10, 10)
Physical Types of Axes: [('em.energy', 'time', 'pos.eq.ra', 'pos.eq.dec'), ('time', 'custom:CUSTOM')]
Unit: None
Data Type: float64
(Pdb) ndcube
<ndcube.ndcube.NDCube object at 0x7c84fe60c2d0>
NDCube
------
Shape: (10, 10)
Physical Types of Axes: [('em.energy', 'time', 'pos.eq.ra', 'pos.eq.dec'), ('time', 'custom:CUSTOM')]
Unit: None
Data Type: float64
(Pdb) 

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for testing this out. This looks to be working because __set__ overwrites _ndcube when _extra_coords is assigned:

value._ndcube = obj

I am not sure what's going on with __set__. @Cadair is there ever an instance where:

  • the extra_coords attached to a NDCube will reference a different NDCube?
  • extra_coords will exist without a NDCube?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the extra_coords attached to a NDCube will reference a different NDCube?

No, that should be explicitly prohibited by the descriptor.

extra_coords will exist without a NDCube?

I think this might be possible yes.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra_coords will exist without a NDCube?

I think this might be possible yes.

Is there a minimal example that would show this? Perhaps it can be used for a test case for the converter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can just instantiate one standalone and then add some coordinates to it with the same API as you can if it's attached to the cube.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

turns out you can't: #753

return extra_coords

def to_yaml_tree(self, extracoords, tag, ctx):
node = {}
if extracoords._wcs is not None:
node["wcs"] = extracoords._wcs
if extracoords._mapping is not None:
node["mapping"] = extracoords._mapping
if extracoords._lookup_tables:
node["lookup_tables"] = extracoords._lookup_tables
node["dropped_tables"] = extracoords._dropped_tables
node["ndcube"] = extracoords._ndcube
return node
25 changes: 25 additions & 0 deletions ndcube/asdf/converters/ndcube_converter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from asdf.extension import Converter
import numpy as np


class NDCubeConverter(Converter):
tags = ["tag:sunpy.org:ndcube/ndcube/NDCube-*"]
types = ["ndcube.ndcube.NDCube"]

def from_yaml_tree(self, node, tag, ctx):
from ndcube.ndcube import NDCube

data = np.asanyarray(node["data"])
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
wcs = node["wcs"]
ndcube = NDCube(data, wcs)
ndcube._extra_coords = node["extra_coords"]

return ndcube

def to_yaml_tree(self, ndcube, tag, ctx):
node = {}
node["data"] = np.asarray(ndcube.data)
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
node["wcs"] = ndcube.wcs
node["extra_coords"] = ndcube.extra_coords

return node
85 changes: 85 additions & 0 deletions ndcube/asdf/converters/tablecoord_converter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
from asdf.extension import Converter


class TimeTableCoordConverter(Converter):
tags = ["tag:sunpy.org:ndcube/extra_coords/table_coord/TimeTableCoordinate-*"]
types = ["ndcube.extra_coords.table_coord.TimeTableCoordinate"]

def from_yaml_tree(self, node, tag, ctx):
from ndcube.extra_coords.table_coord import TimeTableCoordinate

table = node.get("table")
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
names = node.get("names")
physical_types = node.get("physical_types")
reference_time = node.get("reference_time")
timetablecoordinate = TimeTableCoordinate(
table, names=names, physical_types=physical_types, reference_time=reference_time)
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved

return timetablecoordinate

def to_yaml_tree(self, timetablecoordinate, tag, ctx):
node = {}
node["table"] = timetablecoordinate.table
node["names"] = timetablecoordinate.names
node["mesh"] = timetablecoordinate.mesh
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
if timetablecoordinate.physical_types is not None:
node["physical_types"] = timetablecoordinate.physical_types
node["reference_time"] = timetablecoordinate.reference_time

return node


class QuantityTableCoordinateConverter(Converter):
tags = ["tag:sunpy.org:ndcube/extra_coords/table_coord/QuantityTableCoordinate-*"]
types = ["ndcube.extra_coords.table_coord.QuantityTableCoordinate"]

def from_yaml_tree(self, node, tag, ctx):
from ndcube.extra_coords.table_coord import QuantityTableCoordinate

unit = node.get("unit")
table = node.get("table")
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
names = node.get("names")
mesh = node.get("mesh")
physical_types = node.get("physical_types")
quantitytablecoordinate = QuantityTableCoordinate(*table,
names=names, physical_types=physical_types,)
quantitytablecoordinate.unit = unit
quantitytablecoordinate.mesh = mesh
return quantitytablecoordinate

def to_yaml_tree(self, quantitytablecoordinate, tag, ctx):
node = {}
node["unit"] = quantitytablecoordinate.unit
node["table"] = quantitytablecoordinate.table
node["names"] = quantitytablecoordinate.names
node["mesh"] = quantitytablecoordinate.mesh
if quantitytablecoordinate.physical_types is not None:
node["physical_types"] = quantitytablecoordinate.physical_types

return node


class SkyCoordTableCoordinateConverter(Converter):
tags = ["tag:sunpy.org:ndcube/extra_coords/table_coord/SkyCoordTableCoordinate-*"]
types = ["ndcube.extra_coords.table_coord.SkyCoordTableCoordinate"]

def from_yaml_tree(self, node, tag, ctx):
from ndcube.extra_coords.table_coord import SkyCoordTableCoordinate

table = node.get("table")
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
names = node.get("names")
mesh = node.get("mesh")
physical_types = node.get("physical_types")
skycoordinatetablecoordinate = SkyCoordTableCoordinate(table, mesh=mesh,
names=names, physical_types=physical_types)
return skycoordinatetablecoordinate

def to_yaml_tree(self, skycoordinatetablecoordinate, tag, ctx):
node = {}
node["table"] = skycoordinatetablecoordinate.table
node["names"] = skycoordinatetablecoordinate.names
node["mesh"] = skycoordinatetablecoordinate.mesh
if skycoordinatetablecoordinate.physical_types is not None:
node["physical_types"] = skycoordinatetablecoordinate.physical_types

return node
44 changes: 44 additions & 0 deletions ndcube/asdf/entry_points.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
"""
This file contains the entry points for asdf.
"""
import importlib.resources as importlib_resources
from asdf.extension import ManifestExtension
from asdf.resource import DirectoryResourceMapping


def get_resource_mappings():
"""
Get the resource mapping instances for myschemas
and manifests. This method is registered with the
asdf.resource_mappings entry point.

Returns
-------
list of collections.abc.Mapping
"""
from ndcube.asdf import resources
resources_root = importlib_resources.files(resources)
return [

Check warning on line 21 in ndcube/asdf/entry_points.py

View check run for this annotation

Codecov / codecov/patch

ndcube/asdf/entry_points.py#L19-L21

Added lines #L19 - L21 were not covered by tests
DirectoryResourceMapping(
resources_root / "schemas", "asdf://sunpy.org/ndcube/schemas/"),
DirectoryResourceMapping(
resources_root / "manifests", "asdf://sunpy.org/ndcube/manifests/"),
]


def get_extensions():
"""
Get the list of extensions.
"""
from ndcube.asdf.converters.ndcube_converter import NDCubeConverter
from ndcube.asdf.converters.extracoords_converter import ExtraCoordsConverter
from ndcube.asdf.converters.tablecoord_converter import TimeTableCoordConverter
from ndcube.asdf.converters.tablecoord_converter import QuantityTableCoordinateConverter
from ndcube.asdf.converters.tablecoord_converter import SkyCoordTableCoordinateConverter

Check warning on line 37 in ndcube/asdf/entry_points.py

View check run for this annotation

Codecov / codecov/patch

ndcube/asdf/entry_points.py#L33-L37

Added lines #L33 - L37 were not covered by tests

ndcube_converters = [NDCubeConverter(),ExtraCoordsConverter(),TimeTableCoordConverter(),QuantityTableCoordinateConverter(),SkyCoordTableCoordinateConverter()]
_manifest_uri = "asdf://sunpy.org/ndcube/manifests/ndcube-0.1.0"

Check warning on line 40 in ndcube/asdf/entry_points.py

View check run for this annotation

Codecov / codecov/patch

ndcube/asdf/entry_points.py#L39-L40

Added lines #L39 - L40 were not covered by tests

return [

Check warning on line 42 in ndcube/asdf/entry_points.py

View check run for this annotation

Codecov / codecov/patch

ndcube/asdf/entry_points.py#L42

Added line #L42 was not covered by tests
ManifestExtension.from_uri(_manifest_uri, converters=ndcube_converters)
]
22 changes: 22 additions & 0 deletions ndcube/asdf/resources/manifests/ndcube-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
%YAML 1.1
---
id: asdf://sunpy.org/ndcube/manifests/ndcube-0.1.0
extension_uri: asdf://sunpy.org/extensions/ndcube-0.1.0
title: NDCube ASDF Manifest
description: ASDF schemas and tags for NDCube classes.

tags:
- tag_uri: "tag:sunpy.org:ndcube/ndcube/NDCube-0.1.0"
schema_uri: "asdf://sunpy.org/ndcube/schemas/NDCube-0.1.0"

- tag_uri: "tag:sunpy.org:ndcube/extra_coords/extra_coords/ExtraCoords-0.1.0"
schema_uri: "asdf://sunpy.org/ndcube/schemas/ExtraCoords-0.1.0"

- tag_uri: "tag:sunpy.org:ndcube/extra_coords/table_coord/TimeTableCoordinate-0.1.0"
schema_uri: "asdf://sunpy.org/ndcube/schemas/TimeTableCoordinate-0.1.0"

- tag_uri: "tag:sunpy.org:ndcube/extra_coords/table_coord/QuantityTableCoordinate-0.1.0"
schema_uri: "asdf://sunpy.org/ndcube/schemas/QuantityTableCoordinate-0.1.0"

- tag_uri: "tag:sunpy.org:ndcube/extra_coords/table_coord/SkyCoordTableCoordinate-0.1.0"
schema_uri: "asdf://sunpy.org/ndcube/schemas/SkyCoordTableCoordinate-0.1.0"
34 changes: 34 additions & 0 deletions ndcube/asdf/resources/schemas/ExtraCoords-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://sunpy.org/ndcube/schemas/ExtraCoords-0.1.0"
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved

title:
Represents the ndcube ExtraCoords object

description:
Represents the ndcube ExtraCoords object

type: object
properties:
wcs:
tag: "tag:stsci.edu:gwcs/wcs-1.*"
mapping:
type: array
lookup_tables:
Cadair marked this conversation as resolved.
Show resolved Hide resolved
type: array
items:
type: array
properties:
index:
type: array
table:
$ref: "asdf://sunpy.org/ndcube/schemas/TimeTableCoordinate-0.1.0"
dropped_tables:
type: array
ndcube:
$ref: "asdf://sunpy.org/ndcube/schemas/NDCube-0.1.0"

required: [ndcube]
allowAdditionalProperties: False
...
23 changes: 23 additions & 0 deletions ndcube/asdf/resources/schemas/NDCube-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://sunpy.org/ndcube/schemas/NDCube-0.1.0"

title:
Represents the ndcube NDCube object

description:
Represents the ndcube NDCube object

type: object
properties:
data:
tag: "tag:stsci.edu:asdf/core/ndarray-1.*"
wcs:
tag: "tag:stsci.edu:gwcs/wcs-1.*"
extra_coords:
$ref: "asdf://sunpy.org/ndcube/schemas/ExtraCoords-0.1.0"

required: [data , wcs]
allowAdditionalProperties: False
Cadair marked this conversation as resolved.
Show resolved Hide resolved
...
29 changes: 29 additions & 0 deletions ndcube/asdf/resources/schemas/QuantityTableCoordinate-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://sunpy.org/ndcube/schemas/QuantityTableCoordinate-0.1.0"

title:
Represents the QuantityTableCoords class

description:
Represents the QuantityTableCoords class

type: object
properties:
unit:
tag: "tag:stsci.edu:asdf/unit/unit-*"
table:
type: array
properties:
tag: "tag:stsci.edu:asdf/unit/quantity-*"
names:
type: array
mesh:
type: boolean
physical_types:
type: array

required: ["table"]
ViciousEagle03 marked this conversation as resolved.
Show resolved Hide resolved
additionalProperties: False
...
25 changes: 25 additions & 0 deletions ndcube/asdf/resources/schemas/SkyCoordTableCoordinate-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://sunpy.org/ndcube/schemas/SkyCoordTableCoordinate-0.1.0"

title:
Represents the SkyCoordTableCoordinate class

description:
Represents the SkyCoordTableCoordinate class

type: object
properties:
table:
tag: "tag:astropy.org:astropy/coordinates/skycoord-*"
names:
type: array
mesh:
type: boolean
physical_types:
type: array

required: ["table"]
additionalProperties: False
...
27 changes: 27 additions & 0 deletions ndcube/asdf/resources/schemas/TimeTableCoordinate-0.1.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://sunpy.org/ndcube/schemas/TimeTableCoordinate-0.1.0"

title:
Represents the TimeTableCoords class

description:
Represents the TimeTableCoords class

type: object
properties:
table:
tag: "tag:stsci.edu:asdf/time/time-1*"
names:
type: array
mesh:
type: boolean
physical_types:
type: array
reference_time:
tag: "tag:stsci.edu:asdf/time/time-1*"

required: ["table"]
additionalProperties: False
...
10 changes: 10 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,16 @@ exclude = ["ndcube._dev*"]
[tool.setuptools_scm]
write_to = "ndcube/_version.py"

[tool.pytest.ini_options]
asdf_schema_root = 'ndcube/asdf/resources/schemas'
asdf_schema_tests_enabled = true

[project.entry-points."asdf.resource_mappings"]
ndcube = "ndcube.asdf.entry_points:get_resource_mappings"

[project.entry-points."asdf.extensions"]
ndcube = "ndcube.asdf.entry_points:get_extensions"

Cadair marked this conversation as resolved.
Show resolved Hide resolved
[tool.towncrier]
package = "ndcube"
filename = "CHANGELOG.rst"
Expand Down
Loading