Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow converter deferral and move Reference to a converter #1561

Merged
merged 7 commits into from
Jul 17, 2023

Conversation

braingram
Copy link
Contributor

@braingram braingram commented Jun 9, 2023

Following the ASDF tag-up discussion here is another option for moving Reference to a converter based on the suggestions from @eslavich

asdf.reference.Reference serializes to an untagged dictionary.

asdf/asdf/reference.py

Lines 45 to 46 in b0c9a50

class Reference(_types._AsdfType):
yaml_tag = "tag:yaml.org,2002:map"

asdf/asdf/reference.py

Lines 108 to 111 in b0c9a50

@classmethod
def to_tree(cls, data, ctx):
uri = generic_io.relative_uri(ctx.uri, data._uri) if ctx.uri is not None else data._uri
return {"$ref": uri}

This type of conversion is unsupported in the new extension API (using Converters) in asdf main. To allow Reference to be moved to a Converter this PR introduces the ability to define converters that defer conversion to other converters (or in the case of Reference to un-tagged serialization to a yaml map).

This PR changes the public API of Converter.select_tag (in a backwards compatible way) allowing it to return None when asdf should convert the object being serialized with a different converter. When select_tag returns None, the deferring Converter.to_yaml_tree is called on the object being serialized to convert the object instance from one type (the one that triggered deferral) to another type (which can either then defer to another converter or return a valid tag and serialize the object).

In the case of Reference the new ReferenceConverter.select_tag returns None and ReferenceConverter.to_yaml_tree returns a 'vanilla' dictionary which will be converted to an untagged yaml map.

This converter deferral can be used to support subclasses that can be serialized as the superclass (which was done automatically with the old extension API) and docs were updated to describe options for how subclasses can be supported. The changes in this PR provide a migration path for user code that previously relied on ndarray subclass conversion as described in #1537

@braingram braingram added the development No backport required label Jun 9, 2023
@github-actions github-actions bot added this to the 3.0.0 milestone Jun 9, 2023
@braingram braingram marked this pull request as ready for review June 12, 2023 15:26
@braingram braingram requested a review from a team as a code owner June 12, 2023 15:26
@braingram braingram requested review from eslavich and perrygreenfield and removed request for a team June 12, 2023 15:26
@braingram
Copy link
Contributor Author

py38-devdeps failures are expected and addressed in: #1556

by Converter.select_tag returning None asdf
can use Converter.to_yaml_tree to convert to a new
type which asdf will then use to search for a new
converter (used converters are tracked to avoid
conversion cycles)
@braingram braingram merged commit d189cfc into asdf-format:main Jul 17, 2023
@braingram braingram deleted the select_tag_none branch July 17, 2023 19:37
braingram added a commit to braingram/asdf that referenced this pull request Jul 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Jul 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Jul 31, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Aug 2, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
@braingram braingram mentioned this pull request Aug 4, 2023
braingram added a commit to braingram/asdf that referenced this pull request Aug 16, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Aug 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Aug 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Aug 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Aug 18, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Sep 8, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Sep 11, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
braingram added a commit to braingram/asdf that referenced this pull request Sep 11, 2023
Prior to asdf-format#1561 the types supported by a converter that
doesn't support any tags that are supported by the extnesion
were ignored. The modifications in that PR checked
if the converter implemented 'select_tag' and if not
ignored the types. However if the converter implemented
'select_tag' the converter types were indexed (as the converter
could return None from 'select_tag' and defer to a new
converter).

If a converter inherits from Converter it gains the
select_tag method from the parent Converter class
(which provides documentation of the method but doesn't
implement functionality that is otherwise covered by
ConverterProxy). This creates an issue when indexing
the converter tags/types if a converter is included
in an extension that doesn't list support for the
tags supported by the converter.

The changes included here check if 'select_tag'
is implemented by a subclass of Converter. If overwritten,
the types of the converter are indexed (to allow the
converter to defer conversion). However, if 'select_tag'
is merely inherited by Converter, the types supported by
the converter are ignored (agreeing with behavior prior to asdf-format#1561).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development No backport required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants