chore: Move test requirements to optional dependencies in pyproject.toml

[sbrugman]

Description

Fixes open todos:

• Migrate testing requirements to optional dependencies
• Use fully static metadata for kedro-datasets (using recursive optional dependencies)

Development notes

Closes kedro-org/kedro#1498 and closes #169

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

This is fantastic, thanks a lot @sbrugman!

The biggest change is going from our big dynamic setup.py in kedro-datasets to having all optional dependencies using self-referential requirements. I am personally a fan of this approach, but I am deferring to the engineering team to do the final approval.

[merelcht]

This is super neat! ⭐ Thank you so much @sbrugman ! I left one question, but other than that I'm happy to get this merged.

I don't think it fully closes kedro-org/kedro#1498 yet because we'll need to remove any dataset related dependencies from the core repo for the 0.19.0 release, but it's definitely reduced the task.

[sbrugman]

Rebased the changes on top of the pre-commit.

@merelcht Could we merge this and then take on purging related dependencies from the core repo for the 0.19.0 release separately?

(Fully moving to pyproject.toml and supporting python 3.11 in both projects are chores that we can get done soon)

[merelcht]

@merelcht Could we merge this and then take on purging related dependencies from the core repo for the 0.19.0 release separately?

(Fully moving to pyproject.toml and supporting python 3.11 in both projects are chores that we can get done soon)

Yes absolutely. We are not working on any 0.19.0 changes yet, to ensure that the main and develop branches won't diverge too much. Likely we'll kick that all off end of summer.