PyXB is not longer under development and is preventing us from upgrading from Python 3.8 #40

makaylas · 2020-10-01T22:22:42Z

rdflib is pinned to v4.2.2 which has a DeprecationWarning

rdflib/plugins/sparql/compat.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping, MutableMapping  # was added in 2.6

-- Docs: https://docs.pytest.org/en/latest/warnings.html

rdflib should be upgraded to v5.0.0

The text was updated successfully, but these errors were encountered:

vchendrix · 2021-02-11T20:55:12Z

Actually it seems that rdf has fixed this problem but the PyXB library has this same problem. It also seems that PyXB is not currenlty under development. The last commit was in 2017. Do you have any plans to address this?

../../anaconda3/envs/essdive-toolset-py8/lib/python3.8/site-packages/PyXB-1.2.6-py3.8.egg/pyxb/binding/content.py:807
  /Users/val/anaconda3/envs/essdive-toolset-py8/lib/python3.8/site-packages/PyXB-1.2.6-py3.8.egg/pyxb/binding/content.py:807: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    class _PluralBinding (collections.MutableSequence):

-- Docs: https://docs.pytest.org/en/stable/warnings.html

rogerdahl · 2021-02-12T00:30:06Z

The author of PyXB is looking for someone to take over maintenance. pabigot/pyxb#100

gothub · 2021-04-12T19:54:05Z

@mbjones @datadavev @csjx After building a local version of d1_python with the PyXB-X fork (of pyxb), I confirmed that the collections.abc issue has been resolved as described here.
Note that most of the other 40 forks of pyxb are either even or behind the main pyxb repo.

Since the main pyxb repo will not be continuing forward, these are a couple of options:

build and release d1_python with renalreg/PyXB-B fork
create and maintain our own DataONEorg fork
implement a solution other than pyxb

Are there other options to consider?

mbjones · 2021-04-13T23:38:03Z

I think migrating to PyxB-X makes sense, as it is simply a better-maintained version of the same library. We may at some point in the future have to jettison PyxB altogether, but it would be a large undertaking. The other alternative is to stick to versions of Python that the old PyxB still works on, as there isn't a very large practical difference in the capabilities of the python versions involved. But if we must use the latest python, PyxB-X seems like a reasonable path to me. Have all of the tests passed for both PyxB-X and our d1-python stack?

rogerdahl · 2021-04-14T04:56:01Z

I don't think PyXB has any automated testing. d1_python has all tests passing on master. https://travis-ci.org/github/DataONEorg/d1_python/builds/716449984. A diff between PyXB and PyXB-X shows more changes than I expected. Looks like someone is putting in more than just the minimal effort required for keeping it alive. But, while old PyXB gets 40,000 downloads per month on PyPI (which seems like a healthy number to me), PyXB-X gets only 100. Combine that with the lack of tests, and that there's so little to gain by moving to latest Python, I think I'd just stay on old PyXB. Or, if looking for another version of PyXB, maybe it's better to use one that makes only the minimal changes required for keeping it working. https://pypistats.org/packages/pyxb https://pypistats.org/packages/pyxb-x Since it would be a good indicator, I thought I'd do a quick check to see how many Python packages have PyXB as a dependency, but that's apparently still not possible due to Python's subpar package management infrastructure. d1_python does a lot of the XML processing without PyXB, just with XPath and ElementTree (a sort of lightweight DOM). PyXB handles serialization, deserialization and validation only for the D1 XML types. That's convenient, but the types aren't complex, and they're limited in number (on the MN side anyway), so moving that part over to ElementTree and separate validation may not be so bad. On 4/13/21 5:38 PM, Matt Jones wrote: I think migrating to PyxB-X makes sense, as it is simply a better-maintained version of the same library. We may at some point in the future have to jettison PyxB altogether, but it would be a large undertaking. The other alternative is to stick to versions of Python that the old PyxB still works on, as there isn't a very large practical difference in the capabilities of the python versions involved. But if we must the latest python, PyxB-X seems like a reasonable path to me. Have all of the tests passed for both PyxB-X and our d1-python stack? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#40 (comment)", "url": "#40 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

mbjones · 2021-04-14T20:15:39Z

Thanks @rogerdahl that is really helpful. But it does muddy the water as to our choice a little. Seems like we have the following options:

Option 1: Stay with python 3.8 and continue using pyxb until a security issue causes a need to move
- Pros: simple, minimal work, uses stable version that most others are using
- Cons: prevents clients from using recent python 3.9 versions and later, highly unlikely to get security and other critical updates
Option 2: Migrate to pyxb-x
- Pros: supports multiple versions of python
- Cons: few people using the package per se, unclear maintenance and feature trajectory
Option 3: Refactor to remove PyxB dependency and use ElementTree
- Pros: supports multiple versions of python, removes dependency on abandoned package
- Cons: requires unfunded development work, means manually maintaining DataONE type classes

Are there other options, or pros/cons I missed? I think I still prefer option 2, but any of those could work. And we could change our mind if the situation shifts. I would love to get people's votes/input, especially @datadavev and other users of the python library like @vchendrix. Would ESS-DIVE want to support this maintenance work?

amoeba · 2021-04-14T20:28:39Z

I'll think? these are also valid options, correct me if I'm wrong:

Option 4: Fork pyxb under DataONEorg, patch and maintain
- Pros: Fewest PyXB code changes from the upstream, full control over changes
- Cons: Increased maintenance burden on us, hard for other projects to install/depend on PyXB as it's not published
Option 5: Vendor pyxb in d1_python
- Pros: Fewest PyXB code changes from the upstream, full control over changes
- Cons: Increased maintenance burden on us

I'll add that the con to both approaches (increased maintenance burden on us) is already on us as we've having repeated conversations about PyXB and have been for quite some time.

vchendrix · 2022-06-27T23:31:22Z

After reading all of the comments, I think removing the PyXB dependency from d1_python all together seems to be better for the long term. Especially, if the only thing it is being used for is a few D1 xml data structures. Our only dependency on d1_python is to build SystemMetacata and ResourceMap XML.

mbjones · 2022-06-28T02:31:36Z

Thanks for the thoughts, @vchendrix. One note is that its not just a few XML data structures, the types library generates 61 of them under Java using jaxb, and 61 type classes under python using PyxB. That's a fair number of manually maintained classes (each needing a serializer/deserializer and validity checking in both directions). See the listings below.

JaxB classes from d1_common

.
├── v1
│   ├── AccessPolicy.java
│   ├── AccessRule.java
│   ├── Checksum.java
│   ├── ChecksumAlgorithmList.java
│   ├── DescribeResponse.java
│   ├── Event.java
│   ├── Group.java
│   ├── Identifier.java
│   ├── Log.java
│   ├── LogEntry.java
│   ├── MonitorInfo.java
│   ├── MonitorList.java
│   ├── Node.java
│   ├── NodeList.java
│   ├── NodeReference.java
│   ├── NodeReplicationPolicy.java
│   ├── NodeState.java
│   ├── NodeType.java
│   ├── ObjectFactory.java
│   ├── ObjectFormat.java
│   ├── ObjectFormatIdentifier.java
│   ├── ObjectFormatList.java
│   ├── ObjectInfo.java
│   ├── ObjectList.java
│   ├── ObjectLocation.java
│   ├── ObjectLocationList.java
│   ├── Permission.java
│   ├── Person.java
│   ├── Ping.java
│   ├── Replica.java
│   ├── ReplicationPolicy.java
│   ├── ReplicationStatus.java
│   ├── Schedule.java
│   ├── Service.java
│   ├── ServiceMethodRestriction.java
│   ├── Services.java
│   ├── Session.java
│   ├── Slice.java
│   ├── Subject.java
│   ├── SubjectInfo.java
│   ├── SubjectList.java
│   ├── Synchronization.java
│   ├── SystemMetadata.java
│   └── TypeFactory.java
├── v1_1
│   ├── ObjectFactory.java
│   ├── QueryEngineDescription.java
│   ├── QueryEngineList.java
│   └── QueryField.java
└── v2
    ├── Log.java
    ├── LogEntry.java
    ├── MediaType.java
    ├── MediaTypeProperty.java
    ├── Node.java
    ├── NodeList.java
    ├── ObjectFactory.java
    ├── ObjectFormat.java
    ├── ObjectFormatList.java
    ├── OptionList.java
    ├── Property.java
    ├── SystemMetadata.java
    └── TypeFactory.java

PyxB classes from d1_common

class ChecksumAlgorithm(pyxb.binding.datatypes.string):
class CrontabEntry(pyxb.binding.datatypes.token):
class CrontabEntrySeconds(pyxb.binding.datatypes.token):
class Event(pyxb.binding.datatypes.string, pyxb.binding.basis.enumeration_mixin):
class NodeState(pyxb.binding.datatypes.NMTOKEN, pyxb.binding.basis.enumeration_mixin):
class NodeType(pyxb.binding.datatypes.NMTOKEN, pyxb.binding.basis.enumeration_mixin):
class NonEmptyString(pyxb.binding.datatypes.string):
class Permission(pyxb.binding.datatypes.string, pyxb.binding.basis.enumeration_mixin):
class ReplicationStatus(
class ObjectFormatIdentifier(NonEmptyString):
class NonEmptyString800(NonEmptyString):
class ServiceName(NonEmptyString):
class ServiceVersion(NonEmptyString):
class NonEmptyNoWhitespaceString800(NonEmptyString800):
class AccessPolicy(pyxb.binding.basis.complexTypeDefinition):
class AccessRule(pyxb.binding.basis.complexTypeDefinition):
class ChecksumAlgorithmList(pyxb.binding.basis.complexTypeDefinition):
class Group(pyxb.binding.basis.complexTypeDefinition):
class LogEntry(pyxb.binding.basis.complexTypeDefinition):
class NodeReplicationPolicy(pyxb.binding.basis.complexTypeDefinition):
class NodeList(pyxb.binding.basis.complexTypeDefinition):
class ObjectFormat(pyxb.binding.basis.complexTypeDefinition):
class ObjectInfo(pyxb.binding.basis.complexTypeDefinition):
class ObjectLocation(pyxb.binding.basis.complexTypeDefinition):
class ObjectLocationList(pyxb.binding.basis.complexTypeDefinition):
class Person(pyxb.binding.basis.complexTypeDefinition):
class Ping(pyxb.binding.basis.complexTypeDefinition):
class Replica(pyxb.binding.basis.complexTypeDefinition):
class ReplicationPolicy(pyxb.binding.basis.complexTypeDefinition):
class Services(pyxb.binding.basis.complexTypeDefinition):
class Session(pyxb.binding.basis.complexTypeDefinition):
class Slice(pyxb.binding.basis.complexTypeDefinition):
class Synchronization(pyxb.binding.basis.complexTypeDefinition):
class SubjectInfo(pyxb.binding.basis.complexTypeDefinition):
class SubjectList(pyxb.binding.basis.complexTypeDefinition):
class SystemMetadata(pyxb.binding.basis.complexTypeDefinition):
class Checksum(pyxb.binding.basis.complexTypeDefinition):
class Log(Slice):
class Node(pyxb.binding.basis.complexTypeDefinition):
class NodeReference(pyxb.binding.basis.complexTypeDefinition):
class ObjectFormatList(Slice):
class ObjectList(Slice):
class ServiceMethodRestriction(SubjectList):
class Schedule(pyxb.binding.basis.complexTypeDefinition):
class Subject(pyxb.binding.basis.complexTypeDefinition):
class Service(pyxb.binding.basis.complexTypeDefinition):
class Identifier(pyxb.binding.basis.complexTypeDefinition):
class QueryEngineDescription(pyxb.binding.basis.complexTypeDefinition):
class QueryEngineList(pyxb.binding.basis.complexTypeDefinition):
class QueryField(pyxb.binding.basis.complexTypeDefinition):
class MediaTypeProperty(pyxb.binding.basis.complexTypeDefinition):
class MediaType(pyxb.binding.basis.complexTypeDefinition):
class SystemMetadata(_ImportedBinding_dataoneTypes_v1.SystemMetadata):
class NodeList(pyxb.binding.basis.complexTypeDefinition):
class Node(_ImportedBinding_dataoneTypes_v1.Node):
class Property(pyxb.binding.basis.complexTypeDefinition):
class ObjectFormat(_ImportedBinding_dataoneTypes_v1.ObjectFormat):
class ObjectFormatList(_ImportedBinding_dataoneTypes_v1.Slice):
class Log(_ImportedBinding_dataoneTypes_v1.Slice):
class LogEntry(pyxb.binding.basis.complexTypeDefinition):
class OptionList(pyxb.binding.basis.complexTypeDefinition):

iannesbitt · 2022-12-22T23:21:49Z

I'll add a point here and say that if we want to support running d1_python on the Apple M1 chipset out-of-the-box, we need to be able to remove the python==3.7 pin for this package, as versions python<=3.9 are not built for arm64.

vchendrix · 2023-02-15T21:40:05Z

Circling back to see if this issue has any traction.

mbjones · 2023-02-15T21:58:58Z

Traction but no resources for it. Did you have any comments on the 61 classes that would need to be manually rewritten and maintained if we eliminated PyxB? We have a similar situation on the java side with JaxB.

vchendrix · 2023-02-16T00:38:50Z

Traction but no resources for it. Did you have any comments on the 61 classes that would need to be manually rewritten and maintained if we eliminated PyxB? We have a similar situation on the java side with JaxB.

The 61 classes is overwhelming to me. We currently use all the classes related to SystemMetadata and ResourceMaps which I appreciate and don't what do have to implement. There is a lot of logic there.

Maybe the following option mentioned by @amoeba? I think this means that pyxb would be forked and added as a submodule to d1_python which would get packaged together and published in pypi?

Option 5: Vendor pyxb in d1_python

Pros: Fewest PyXB code changes from the upstream, full control over changes
Cons: Increased maintenance burden on us

rogerdahl · 2023-02-16T01:02:08Z

Now that there is a real need to move to a more recent version of Python, I think the way to go would be to fork the last version of PyXB, and update it. When I looked at this a few years ago, only a few one-line changes were required in order to get PyXB working on the current version of Python, probably 3.9. I'd take a look through the many forks of PyXB to see which tweaks may be required: https://github.com/pabigot/pyxb/network/members

datadavev · 2023-04-05T12:48:16Z

The GeoScienceAustralia for of PyXB-X looks promising. https://github.com/GeoscienceAustralia/PyXB-X
Evaluating along with related updates for python 3.9+

srstsavage · 2023-11-29T06:48:50Z

Is this fixed by #78?

mbjones · 2023-11-29T23:33:20Z

yes, @srstsavage -- it is. On my superficial look, it seems like it was not tagged as a release, so we need to update the docs and tag the release as 3.6.0, and publish to pypi. cc @iannesbitt @rogerdahl @datadavev

srstsavage · 2024-04-23T21:08:54Z

@mbjones We're looking at reinvigorating the effort to dockerize GMN. Any chance of getting that 3.6.0 release in the near future?

rogerdahl · 2024-04-24T03:01:26Z

@mbjones I've tagged the 3.5.2 release on GH now. This release is already on PyPI. I think you're good to go on containerizing, @srstsavage.

mbjones · 2024-05-03T19:06:49Z

thanks @rogerdahl -- that's so appreciated.

In terms of docker, I'm sure there would be others that would be interested in running this via docker, so @srstsavage please do consider contributing such changes. We should track dockerization work in issue #77

gothub self-assigned this Apr 2, 2021

rogerdahl changed the title ~~Upgrade rdflib to remove DeprecationWarning~~ PyXB is not longer under development and is preventing us from upgrading from Python 3.8 Dec 30, 2021

Xarthisius mentioned this issue Feb 24, 2023

Drop support for DataONE publishing whole-tale/whole-tale#149

Open

Xarthisius mentioned this issue Mar 13, 2023

Drop references to D1 publishing whole-tale/wt-design-docs#175

Open

This was referenced Apr 4, 2023

Update d1_python dependencies to work with python 3.8+ #76

Closed

Support GMN as a Docker container #77

Open

rogerdahl closed this as completed Apr 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PyXB is not longer under development and is preventing us from upgrading from Python 3.8 #40

PyXB is not longer under development and is preventing us from upgrading from Python 3.8 #40

makaylas commented Oct 1, 2020

vchendrix commented Feb 11, 2021

rogerdahl commented Feb 12, 2021

gothub commented Apr 12, 2021

mbjones commented Apr 13, 2021 •

edited

Loading

rogerdahl commented Apr 14, 2021 via email

mbjones commented Apr 14, 2021

amoeba commented Apr 14, 2021

vchendrix commented Jun 27, 2022

mbjones commented Jun 28, 2022

iannesbitt commented Dec 22, 2022 •

edited

Loading

vchendrix commented Feb 15, 2023

mbjones commented Feb 15, 2023

vchendrix commented Feb 16, 2023

rogerdahl commented Feb 16, 2023

datadavev commented Apr 5, 2023

srstsavage commented Nov 29, 2023

mbjones commented Nov 29, 2023

srstsavage commented Apr 23, 2024

rogerdahl commented Apr 24, 2024

mbjones commented May 3, 2024

PyXB is not longer under development and is preventing us from upgrading from Python 3.8 #40

PyXB is not longer under development and is preventing us from upgrading from Python 3.8 #40

Comments

makaylas commented Oct 1, 2020

vchendrix commented Feb 11, 2021

rogerdahl commented Feb 12, 2021

gothub commented Apr 12, 2021

mbjones commented Apr 13, 2021 • edited Loading

rogerdahl commented Apr 14, 2021 via email

mbjones commented Apr 14, 2021

amoeba commented Apr 14, 2021

vchendrix commented Jun 27, 2022

mbjones commented Jun 28, 2022

iannesbitt commented Dec 22, 2022 • edited Loading

vchendrix commented Feb 15, 2023

mbjones commented Feb 15, 2023

vchendrix commented Feb 16, 2023

rogerdahl commented Feb 16, 2023

datadavev commented Apr 5, 2023

srstsavage commented Nov 29, 2023

mbjones commented Nov 29, 2023

srstsavage commented Apr 23, 2024

rogerdahl commented Apr 24, 2024

mbjones commented May 3, 2024

mbjones commented Apr 13, 2021 •

edited

Loading

iannesbitt commented Dec 22, 2022 •

edited

Loading