Proposal: Shift changelog content generation away from releaser onto PR author

[thejcannon]

Proposal

In order to help automate the release process, and shift the burden of changelog "news fragments" from the releaser onto the PR author, I propose we shift our process to have the PR which proposes a change also update the changelog.

Manually curating the changelog is a breeding ground for merge conflicts, so I suggest we leverage towncrier to aid us as well.

Towncrier is able to generate changelog entries from in-repo "news fragments" which exist as markdown files on disk with a very short summary of the change. This essentially eliminates changelog.py. It is customizable enough to adapt to our current categorizations.

---

How

PRs

• PR authors, in addition to making code changes (and possibly docs changes) will also add a file whose name is the category (unless it is "internal") with suffix ".md" in a new directory under src/python/pants/notes. If the PR requires a cherrypick, the directory will be the name of the milestone (e.g. 2.15.x) otherwise the directory will be main. E.g. store stdout/stderr output from tests as bytes #19184 would've added bugfix.md to src/python/pants/notes/main. Optimize Target and FieldSet operations #18917 would've added performance.md to src/python/pants/notes/2.15.x.
  • CI will validate that the file has been added and matches the category label. It will also validate the directory is correct based on the milestone.
• CI will immediately rename the file to include the PR number as the prefix and push to the branch (CI will skip testing). This serves two purposes:
  • towncrier requires all news fragments start with a unique slug (this avoids merge conflicts). The PR is a unique identifier for the change.
  • When generating the changelog, this is used to reference the PR which introduced the change.
• (Cherry-picking happens normally)
  • If a cherry-pick is determined to be needed after merging. A new PR should be made that moves the file into the relevant subdirectory from main. This PR MUST be cherry-picked, and must be picked after the original cherry-pick is merged.
    • (This can be automated)

Branching

• When main is bumped to .a0, the directory for the feature branch is created (e.g. 2.17.x), and the relevant towncrier configs are added/updated.
  • Now cherry-pick PRs can target 2.17.x with their docs
  • Open PRs not label for cherrypick will be using the directory main, which will point to the next version when they are released
  • (This can be automated)

Releasing

• When preparing a release for a feature-branch, run towncrier build --config src/python/pants/notes/<branch>/towncrier.toml --version <new version> --date '<human date>'
  • (This will be automated)
  • This will delete the news fragments on disk and update the relevant changelog .md.
  • Make a PR which should be labeled for cherry-pick targeting the right branch
    • (alternatively we could make a similar change to the release branch without picking. That would alleviate the need to pick the notes moving PR if we cherry picked something after merge)
    • this is really only so that main has a master copy of all release notes
• When preparing a .devN release, the only differences are:
  • Use the config for main
  • Don't label for cherry-pick

---

(We can decide after the fact if we want to keep the PR labels, or just use the fragment name. I consider that out of scope for this proposal though. The category labels will continue to be required and must match the filename.)

[benjyw]

This is interesting. I am all-in on automating the release process further.

My concern here is that this doesn't appear to shift a burden so much as create a new one. This requires more human work overall, and a lot of redundant files. If the content of this new file is the same as the PR title then why have the former? And if it isn't then why not? We would expect the PR description to be, er, descriptive. If it isn't sufficiently so then we should require that it is. Having two places where a change description lives seems unnecessary.

Today we already know how to generate the changelog content automatically, using just the category: labels and the PR description. Wouldn't trivially automating the silly thing we do today of manually copying the output of ./pants run build-support/bin/changelog.py into the changelog give us effectively the same result but with fewer moving parts and no further imposition on PR authors?

[kaos]

[...] and not merely a recital of every PR

I feel this missed the point with the news fragment file. It would not be merely a recital. Any single PR would be able to have an entire chapter in as much detail as needed/desired for the what's new section, while other PRs may not have any. So it's a curated list with additional context/content not merely a recital.

[benjyw]

In a sequence of changes relating to some feature, which one(s) would have the summary? It might not be possible to know when you're authoring a PR whether it is the magic one.

[kaos]

That's a fair point. I think it'll be pretty clear, but of course there may be cases when it isn't and that has to work out as well.

[benjyw]

Overall this feels to me like using a sledgehammer to crack a nut. But I don't want to harp on it any longer - I've expressed my opinion, I respect that you differ. Let's hear from others.

[thejcannon]

Just adding that towncrier also merges multiple fragments with the same content on the same line (e.g. multiple PRs that go into a single change)

[huonw]

I think it'd be useful to clarify how these snippets are expected to be shown: I think there's, like, four levels of "changelog":

1. "commits": every single git commit
2. "exhaustive": the actual PRs that merge (for Pants, with squash-and-merge, this is the same as 1)
3. "summary": aggregated interesting features and bug fixes, the sort of thing that appears in "what's new" (e.g. if one feature is split across four PRs, they'll only get one listing, or one PR implements multiple (related, hopefully) changes, it might have multiple, and some changes won't rate a mention (internal only, or niche bugfix, or similar))
4. "highlights": what one might put in a blog post, with extra explanation, or even have dedicated posts about

It feels like automation is great for 1 and 2, and indeed, AIUI, Pants already has acceptable(?) automation here?

Given this, maybe this is trying to target 3? That is, making a "what's new"-style section automatic/easier to write?

Alternatively, is it trying to make it easier to write better text for 2?

[huonw]

It feels like automation is great for 1 and 2, and indeed, AIUI, Pants already has acceptable(?) automation here?

Ah, to answer my own question: build-support/bin/changelog.py will do git magic and hit APIs and so on, but it'll just print the result for a human to wrangle, rather than writing it to the appropriate changelog file.

I could imagine this current approach could be made more automatically, e.g. a manually triggered "start release" workflow, that runs that, updates the files, commits the results and opens a PR, with a human to make any updates like typo fixes or whatever, and then merge once it's valid...

(Of course, as evidenced by not knowing the details of changelog.py, I've never run a Pants release, so I can only guess at what the highest value areas to focus on are, I don't actually know. 🙃 )

[benjyw]

Yep, the hard part is already solved by that script. The remainder is to automatically prepend that text to the changelog file. IMO for a first pass, the highest bang for our bucks is to fully automate our existing process. Then we can consider changing it by seeing what extra benefit any changes bring.

[huonw]

IMO for a first pass, the highest bang for our bucks is to fully automate our existing process

I've run with that idea in #19279

[huonw]

We've made a step towards this with #20888