Pull request preview builds for online docs

[EliahKagan]

Read the Docs supports pull request builds, where pull requests trigger preview builds that are published to a different domain and marked with a warning banner to prevent confusion, and where the build status is reported as one of the CI checks for the PR.

I recommend enabling this for GitPython. Unlike some CI checks that are configured in GitHub Actions workflow files, my understanding is that Read the Docs pull request builds, including their status reporting as CI checks, do not use GitHub Actions. Instead, pull request builds are triggered via webhook (as are other Read the Docs builds), and the check status is reported to GitHub by authenticating with OAuth and posting it with (I think) the GitHub API.

For this reason, I don't believe I can propose the necessary changes by pull request. Instead, if you choose to enable pull request builds, you can do so by following these official instructions.

This should make it easier to verify the correctness of documentation changes in pull requests, as well as make it easier for contributors (including me) to become aware of, investigate, and attempt to improve problems in Read the Docs builds.

My immediate motivation for recommending this change is the hope that it would enable me to submit and test a fix for #1840. However, even if it turns out that the effect of changes to .readthedocs.yaml are not fully previewed in pull request builds--I think they are but I am not sure--it seems to me that this would be a worthwhile addition to the project's CI checks.

Currently the pythonpackage.yml workflow does test if a local documentation build is able to succeed, but this does not test if a remote (Read the Docs) build succeeds, and more importantly it does not produce a hosted preview of the built documentation that can be easily inspected.

[Byron]

Thanks for researching this! I have enabled pull-request builds for GitPython and will be waiting for your confirmation that it actually works. The setup didn't involve more than ticking a checkbox.

[EliahKagan]

That's great--I'll open a pull request shortly and we'll see what happens. Thanks!

[EliahKagan]

Partial success:

• #1843 did trigger an RTD pull request build, and the fix in that PR of installing GitPython and its documentation build dependencies repopulated the API reference (so the PR will fix #1840) as well as enabling the theme to work again.
• But there is no CI check, which is to say that the build status is not being reported to GitHub. Instead, I had to find the build in the recent builds page. Fortunately this troubleshooting section covers this kind of problem. I am hoping you may be able to use the guidance in the "Build status is not being reported to your Git provider" subsection to fix that.

Although this should make no difference for the CI check, which is attempted in response to an RTD pull request build happening, and not in response to status changes on the PR other than opening it and pushing commits, I am marking that PR ready for review. That should verify that its draft status is not a factor here. It may also make it easier for you to merge it if you choose, though I suggest holding off on merging it until the status reporting is fixed.

After a change is made that might fix status reporting, it may be necessary to push to the PR to test that. This can be done by amending without changing any content or even the commit message and force-pushing. (Assuming I am available, I can do that after an attempted fix.)

Update: As expected, marking the PR non-draft had no RTD-related effect (no new checks).

[Byron]

That's awesome, this reads as full success to me :).

• I am hoping you may be able to use the guidance in the "Build status is not being reported to your Git provider" subsection to fix that.

I think I will pass on this back-channel in favor of keeping the amount of authorized oauth-apps at zero in the organization. My hope is that the current setup will suffice once changes are intentionally made, that other PRs don't make changes so they wouldn't accidentally break anything.

That way it's also a bit cumbersome to find PR-builds and review the docs that way, to me the greatest benefit of having the check visible, but I allow myself to wait and see.

Now heading over to the PR merging - thanks again, the docs look great!

[EliahKagan]

I think I will pass on this back-channel in favor of keeping the amount of authorized oauth-apps at zero in the organization.

Understood.

There may be alternatives that further improve reporting while still satisfying the requirement that no such authorizations be added. I'll look into it and comment again with my findings.

[EliahKagan]

Although a nicer solution could be coded up as a new custom action, the options that I believe would be pretty quick and easy to put in place, while still not requiring any apps to be authorized, are:

1. Do nothing at this time. We have pull request builds and know where to find them. This issue could simply be closed as completed.
2. Add a GitHub Actions job for this, to edit a link to the preview. An official RTD action to do this, which may or may not be suitable for use here, already exists as readthedocs/actions/preview. The disadvantage is the possibility of overlapping edits. Fortunately the more common case would be the documentation link disappearing because of the PR author editing the description soon after submitting it; the more serious problem that new material in the description would disappear (albeit remaining accessible in the edit history) should be rarer and is perhaps acceptably unlikely. The official action has not, at this time, been upgraded to use Node 20.
3. Add a GitHub Actions job for this, to post a comment with a link to the preview. The disadvantage of this is that PR authors would always get comment notifications immediately after opening the PR.

Approach (2) requires that the workflow have the pull_request_target trigger with write access to the PR, and therefore would have to be written carefully, ensuring to avoid pitfalls like unnecessarily checking out anything from the repository or allowing injection attacks from metadata of the PR. Unlike the pull_request trigger which generally runs jobs with no special access to the repository, the pull_request_target trigger causes jobs to run "as you."

I am unsure about approach (3), but I think it requires that, too. Though an alternative could be to make a bot that runs as another account, rather than using a GitHub Actions job. The bot could listen to a web hook but would not have to have special privileges. This is less trivial to implement, though!

The link contains the PR number but not the commit, so further commits do not invalidate it. However, it would also be useful to have a link to the build page, which is per commit. I don't know of a way to do that easily (without the oauth authorization).

It could be done by writing a custom workflow or custom action that attempts to crawl a tiny bit of the site or to use the Read the Docs API. The former approach, forgoing the API, might be undesirable since it might entail going against robots.txt in a situation where a smiled-upon alternative is available, and if this is done often from GitHub Actions runners then it might be automatically blocked. The latter approach of using the API sounds good, except it looks like the non-deprecated API requires an API key that is per-user and might offer elevated access to Read the Docs, which would be unnecessary and unwise for this. I am uncertain, though; perhaps a limited key can be created. This would still entail adding a GitHub Actions secret, though.

[Byron]

Thanks for investigating this issue!

To me, picking option 1) seems like the easiest solution while, in the common case, not loosing out on anything. Those PRs that deal specifically with adjusting documentation can link to the respective builds by hand. Generally I don't feel great about elevating permissions of PRs to achieve a better integration, even if it was super-easy.

And even though it would be very nice to have such integration for reviewing, I think if it really matters the reviewer can generate the documentation locally as well.

Thus, I am closing the issue as suggested in 1), and thank you again for evaluating the matter in such great detail.