Compatibility with Pydantic >= 2 through pydantic.v1

[SarahG-579462]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Recently, pygeoapi's pydantic was downgraded from 2.0-compatible to pydantic < 1. This removed compatibility with two of our libraries, xscen and xclim, for climate data processing, due to intake-esm's current requirement on pydantic > 2

Because we require these in the use of our pygeoapi process plugins, this change forced us to use either very outdated versions of xscen (and thus xarray, pandas, xclim, etc.), or a very outdated version of pygeoapi.

Thankfully, a simple fix is available by the use of pydantic 2's backwards-compatible pydantic.v1 module.

It is not entirely clear to me why pydantic was downgraded, the best explanation I can see in the discussion was due to "versionitis".

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Pydantic issues:
#1573
#1341
#1584

[ridoo]

Seems that the dev team has decided to "decouple" more from pydantic in favor of using native dataclass and Protocols (PEP-544). @francbartoli mentioned them in #1547. Not sure how deep this was discussed before, but Protocols has been mentioned before the versionitis term in the last code sprint meeting:

• Protocol / duck typing
  • introduce after api.py refactor in core to allows extensibility
  • do not touch plugins atm
• pydantic
  • master is broke because of versionitis
  • p1 PR to downgrade pydantic 2 --> 1
  • release thereafter
  • should NOT affect rebasing during api refactor

This does not answer your question, why pygeoapi isn't switching to v2 and using pydantic.v1 module. I only can guess, that there are other parts using pydantic which deserve some refactoring first.

@francbartoli @tomkralidis do you have strong arguments not using pydantic anymore? As far I have read things on v2, it got a lot faster. Is this versionitis (sorry for re-calling this term again :)) a temporary issue or do some other problems keep you from resolving this?

[tomkralidis]

Given RFC2, and that our current base OS is Ubuntu jammy, we are bound to OS specific versions. AFAICT, Pydantic is at version 1.8.2 on jammy.

[francbartoli]

@SarahG-579462 @ridoo Pydantic was downgraded to adhere with RFC2 as @tomkralidis mentioned.

Also, the direction of the project is to prefer the simplicity of plain standard dictionaries over the translation from Pydantic models when needed.
However, I recognise the robustness/easiness of a pydantic data validation/serialisation and I'm in the same condition in some downstream projects that I maintain.
That's why I was trying to design a more abstract structure that allows:

• firstly, to remove entirely Pydantic from the code base
• secondly to easily replace, with a consolidated pattern like duck typing, a standard dataclass model with a Pydantic one using a new pygeoapi plugin module that can be eventually declared in the configuration

Hopefully, this might happen soon but most probably after the completion of v2 ubuntu packaging

I Hope this clarifies further.

[ridoo]

@francbartoli thanks for giving more background! BTW, I got notified that #1547 got selected to be closed by github-actions.

[francbartoli]

@ridoo @SarahG-579462 python3-pydantic-core is now available as a package in Debian. Accordingly to RFC2, now I'm going to revert #1584 #1596 and will add the package to the CI docker image (cc @tomkralidis)

[tomkralidis]

Thanks @francbartoli. @kalxas let us know if this works/any issues.

[kalxas]

The package seems to be in Unstable: https://packages.debian.org/sid/python3-pydantic-core

[kalxas]

Lets wait until this ends up in a Debian/Ubuntu stable release

[github-actions]

This Issue has been inactive for 90 days. As per RFC4, in order to manage maintenance burden, it will be automatically closed in 7 days.

[kalxas]

Not yet available in Noble

[francbartoli]

@kalxas is not quite enough to have it available in Jammy https://github.com/geopython/pygeoapi/blob/master/Dockerfile#L37 for us?

[kalxas]

@francbartoli it is not available for jammy, so I looked at noble to see if it exists there

[francbartoli]

@kalxas yes, it is available only in Oracular https://launchpad.net/ubuntu/+source/pydantic (Testing) atm

[ridoo]

one general question: Why pygeoapi depends on python packages managed by the OS? What about creating a venv and install whatever package version you want?

[francbartoli]

@ridoo please have a look at this RFC2 as the reference

[ridoo]

Thanks and sorry for the noise