Docs build: Add upper bounds to transitive dependencies

[AlexWaygood]

Our immediate requirements for the docs build are already pinned pretty tightly in our Doc/requirements.txt file. However, our transitive dependencies are currently entirely unpinned. This means our CI can be unexpectedly broken by a new release of a transitive dependency, as recently happened when Pygments==2.15.0 was released: see #103421.

Previously it wasn't so much of a big deal if the Docs build got broken briefly. However, with the recent move to use GitHub automerge instead of @miss-islington's merging capabilities, it's now important that we make the Docs workflow a "required" check, so that PRs with broken ReST syntax don't accidentally get automerged into the main branch. Making the docs workflow a "required" check means it is more important to make sure that the Docs build is very resilient.

This PR adds upper bounds to various transitive dependencies by using a constraints.txt file.

[CAM-Gerlach]

LGTM, thanks @AlexWaygood . We should presumably backport this as well, so backport PRs that are automerged can be treated similarly

[AlexWaygood]

We should presumably backport this as well, so backport PRs that are automerged can be treated similarly

Yes, that SGTM!

[AA-Turner]

Seems a good idea in principle -- my questions would be how we ensure that new transitive dependencies are always added (including dependencies of dependencies, etc), and when we consider it appropriate to update the upper-bounds (security certainly, new feature releases -- maybe?).

A

[AlexWaygood]

my questions would be how we ensure that new transitive dependencies are always added (including dependencies of dependencies, etc), and when we consider it appropriate to update the upper-bounds (security certainly, new feature releases -- maybe?).

I don't have fully formed answers to these questions, but I agree that we will need to work these things out :)

What's the current process for updating our Doc/requirements.txt file? I'd imagine that the constraints.txt file would generally only get updated when the requirements.txt file is updated -- which seems fine to me?

[AA-Turner]

What's the current process for updating our Doc/requirements.txt file?

@JulienPalard is probably the man for the 'policy' answer here, though my guess is ad-hoc PRs as needed (as with many other things!). In which case pragmatism seems the best approach as you suggest -- hopefully reviewers notice / remember the constraints file, but fundamentally if we make a mistake it's just a quick PR to fix!

Thanks for the quick response -- more appreciated than usual as it's past midnight British time!

A

[CAM-Gerlach]

At this point, maybe worth just using something like pip-tools (pip-compile) to just generate a pseudo-lockfile and use that (only) in the CIs, and schedule periodic (e.g. monthly) CI runs to update it? That's a lot more maintainable than just relying on people manually maintaining (somehow...) at irregular intervals.

Also, is this going to get picked up by the build server infra?

[AlexWaygood]

At this point, maybe worth just using something like pip-tools (pip-compile) to just generate a pseudo-lockfile and use that (only) in the CIs, and schedule periodic (e.g. monthly) CI runs to update it? That's a lot more maintainable than just relying on people manually maintaining (somehow...) at irregular intervals.

I'd be okay with that. I also feel like the maintainability here isn't really going to be a problem though. We'll probably only need to update it whenever requirements.txt is updated, which won't be very often. I'd advocate for merging something like this now, and maybe investigating a more elaborate solution using pip-tools and a periodic GitHub Action job as a followup issue.

Also, is this going to get picked up by the build server infra?

No idea, I don't know anything about the build server infra 😃 Who do I ask about that?

[AA-Turner]

Also, is this going to get picked up by the build server infra?

No idea, I don't know anything about the build server infra 😃 Who do I ask about that?

It will be! See docsbuild-scripts:

https://github.com/python/docsbuild-scripts/blob/main/build_docs.py#L105

and

https://github.com/python/docsbuild-scripts/blob/main/build_docs.py#L123

A

[willingc]

LGTM. I would go ahead and merge and we can adjust as needed. @pradyunsg may have some additional insights on dependencies.

[AlexWaygood]

Great, thanks everybody! Feels like we have consensus that this is a good idea and that this is good enough for now, so I'll go ahead and merge. Followup PRs to improve our workflow around this are welcome :)

[miss-islington]

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[miss-islington]

Sorry @AlexWaygood, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 81cf94c4426b3bb949be8a0bb26ae0adccdbc88c 3.11

[miss-islington]

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-103887 is a backport of this pull request to the 3.11 branch.