Pin pyproject dependencies

[jemrobinson]

✅ Checklist

• You have given your pull request a meaningful title (e.g. Enable foobar integration rather than 515 foobar).
• You are targeting the appropriate branch. If you're not certain which one this is, it should be develop.
• Your branch is up-to-date with the target branch (it probably was when you started, but it may have changed since then).

🚦 Depends on

n/a

⤴️ Summary

• Pin Python package dependencies in pyproject.toml
• Drop custom pre-installation of pinned versions
• Drop custom updating scripts in favour of dependabot
• Use recommended structure for environment requirements: group them into optional feature groups which means they can (if desired) be installed with e.g. pip install ".[test]"

Justification:

• "Should I pin my dependencies?" discussion here and here
  • summary: yes, if you're writing an application that isn't expecting other packages to depend on it

Potential issues:

• This only pins direct dependencies, not the things that those packages depend on
  • I think this is OK, as this shouldn't affect the functions we're calling. If it does, then we probably have a (hidden) direct dependency and should probably add that to pyproject.toml
  • Solved by using hatch-pip-compile which is a built-in version of what we were previously doing manually
• Dependabot might try to update the pinned requirements files
  • Not sure whether this is a problem or not

🌂 Related issues

Part of #2084

🔬 Tests

Tested on a fork of this repository - dependabot is updating pyproject.toml as expected.

[github-actions]

Coverage report

This PR does not seem to contain any modification to coverable code.

[JimMadge]

I don't think this is the way I want to go.

We loose the pinned dependencies here as we only have direct dependencies. That would make debugging more difficult and docs builds unreproducible.

I think there is an advantage to having the minimal dependencies for development (for example, when we had to set a minimum patch version for pulumi-azure-native) and the full pinned dependencies. pip-tools feels like a good, hosting agnostic way to do that.

A better solution might be to figure out how to support a requirements.in with the full dependencies going to pyproject.toml rather than requirement.txt

[jemrobinson]

In principle pinning direct dependencies should be OK. If any of the functions from our directly-called libraries change their behaviour because of the specific version of their dependencies then we actually depend on that library too and we should be explicit about that.

I disagree that having minimal dependencies listed in pyproject.toml is helpful. We actually have no idea whether the set of constraints in this file were correct, because we never used them in development - we were always installing from requirements.txt.

[jemrobinson]

@JimMadge : Transitive dependencies now pinned using hatch-pip-compile

[JimMadge]

Happy to merge this as it doesn't look like there is a sensible way to do what we would ideally want.

We can review that again in the future.