Support for Pydantic V2 [Planned for Q1 2024]

[ravi-kumar-pilla]

Description

• This will bring pydantic v2 features to Kedro Viz

Context

• Pydantic v2 has changed its implementation to use Rust under the hood which makes it faster

Possible Implementation

https://docs.pydantic.dev/2.0/migration/

Timelines - We will work on the migration to pydantic v2 next year (Q1, 2024)

Possible Alternatives

NA

Checklist

• Include labels so that we can categorise your feature request

[antonymilne]

I think this is not the right way to do things (or at least not for a long time). While the addition of model_post_init in pydantic v2 is very nice and would make it very easy to convert from dataclasses __post_init__, doing this would mean dropping support for pydantic v1, which I don't think we want to do for e.g. 3-6 months at least. The performance benefit of pydantic v2 will really be insignificant for kedro-viz and we don't especially need any of the features that pydantic v2 brings.

Besides, many of the things that are doing in __post_init__ in dataclasses are actually better suited to validators (note v1 documentation). Let me leave a few comments on #1565 to illustrate.

So what I would recommend is to support both pydantic v1 and v2 but use only the v1 API, as in #1529. Then in due course convert from pydantic v1 to v2 API, which means you can then use model_post_init where useful, and drop support for pydantic v1 entirely.

[astrojuanlu]

This is because apache-airflow has moved to pydantic v2 in 2.8.0

kedro-org/kedro-plugins#511 (comment)

There is more pressure from upstream dependencies to support Pydantic v2.

[astrojuanlu]

Also to make things clear, supporting Pydantic v2 is not about performance (which for Kedro-Viz will have a negligible impact), but about allowing Kedro-Viz to coexist with Pydantic v2 in the same environment.

[butterlyn]

@astrojuanlu Hi, big fan of Kedro (and your latest YouTube video series), trying to convince my organisation to adopt the platform in day-to-day operations.

Is it possible for this to be released earlier in 2024 (perhaps Q2 or late Q1?)? Even in a non-stable development version? 2025 Q2 timeframe is a deal-breaker since our tech stack heavily uses Pydantic v2, and if I say to upper management that we'll need to wait to 2025 then I'll be fighting a losing battle when pitching Kedro. Thanks

[astrojuanlu]

Hi @butterlyn , thanks for your kind words! I'm almost sure 2025 is a typo, we definitely want to tackle this in 2024. Watch this space.

[ravi-kumar-pilla]

Hi @butterlyn , Thank you for your patience. As @astrojuanlu pointed out, we will be prioritizing pydantic v2 support in Q1, 2024.

[butterlyn]

@ravi-kumar-pilla Got the green light to trial Kedro after saying this feature was planned for Q1 2024. Thank you!! 😀 Will prepare to migrate in anticipation of the update

[antonymilne]

FYI vizro now supports pydantic 1 and 2 and uses the v1 API throughout - see mckinsey/vizro#189.

Don't be put off by the size of the diff in the PR. It was actually very easy to do 🙂 Although we did not have any complications with having FastAPI as a dependency which I know was an issue on kedro-viz.

[astrojuanlu]

@MarkusSagen also proposed migrating to Pydantic v2 in #1719

[MarkusSagen]

Anything I can help with to speed up the timeline for this?
Would you accept PRs for this?

@astrojuanlu

[astrojuanlu]

@rashidakanchwala any ongoing work that would be blocking this?

@MarkusSagen we'd be happy to have an external contribution for this! be mindful that there are several aspects to consider (FastAPI compatibility among them). you can read past discussion in #1529, #1634

[astrojuanlu]

In any case, let's wait for confirmation from the team - there are a couple of refactors ongoing and it might not make sense for you to open a PR against current main @MarkusSagen

[MarkusSagen]

Thanks for letting me know!

[rashidakanchwala]

Hi @MarkusSagen -- we will be doing this work in our next sprint which is Monday onwards. So hopefully we shld get this merged by 3rd or 4th week of Feb.

[noklam]

For the record, we decide to support V2 only going forward. For pydantic v1 user they should be able to use the older version of kedro-viz.