Maintain a semantic changelog

[michaeltlombardi]

Summary of the new feature / enhancement

As a user of DSC, I want to be able to quickly review changes to DSC itself so I can understand the changes in recent releases and how they apply to my context.

As a resource author, I want to be able to review changes to DSC and its schemas so I can understand how leverage any improvements or comply with new requirements.

As an integrating developer, I want to be able to review changes to DSC and its contracts so I can ensure that my code effectively uses DSC's features and manage handling across versions.

In the current release model, there's a single set of release notes for each tag of the project. These release notes include maintenance changes, bug fixes, documentation updates, and new features in a way that requires me to parse the full set of notes to understand what changes are important to me. Because the entries are pulled from the PR titles, I also usually need to follow the links to the PRs and often the issues those PRs relate to in order to get the full context.

Proposed technical implementation details (optional)

Keep a semantic changelog for the project that breaks down changes to each version across the kind of change. Write the changelog entries for non-maintainers - resource authors, end users, and integrating vendors in particular - and link to the documentation for new features or changes as well as the relevant PRs and issues.

Consider this comparison between the auto-generated release notes and an intentionally crafted changelog for alpha.2:

Release Notes

What's Changed

• (MAINT) Fix the registry find command options by @michaeltlombardi in (MAINT) Fix the registry find command options #163
• Dependson by @SteveL-MSFT in Dependson #175
• Export scenario support by @anmenaga in Export scenario support #171

Changelog

v3.0.0-alpha.2 - 2023-09-05

Added

• Implemented functionality for the dependsOn property of resource instances in configuration documents, enabling resource instances to depend on the successful processing of one or more other instances in the document.

  Related Work Items

  • Issues: Implement dependsOn processing for config #45
  • PRs: Dependson #175

• The export property to the resource manifest schema, indicating that the resource is exportable and defining how DSC can retrieve the current state for every instance of the resource.

  Related Work Items

  • Issues: Need a way to support export scenario #73
  • PRs: Export scenario support #171

• The dsc config export command to convert an input configuration document defining a list of resource types into a usable configuration document that defines the current state for every instance of those resources.

  Related Work Items

  • Issues: Need a way to support export scenario #73
  • PRs: Export scenario support #171

• The dsc resource export command to generate a usable configuration document that defines the current state for every instance of a specified resource.

  Related Work Items

  • Issues: Need a way to support export scenario #73
  • PRs: Export scenario support #171

• The --all option for the dsc resource get command, enabling users to retrieve the current state for every instance of an exportable resource with a single command.

  Related Work Items

  • Issues:
    • Need a way to support export scenario #73
    • Support dsc resource get --all for prefetching instances #174
  • PRs: Export scenario support #171

Fixed

• Using the registry find command no longer raises a panic error due to conflicting option definitions on the command.

  Related Work Items

  • PRs: (MAINT) Fix the registry find command options #163

Considerations

At first glance, maintaining a high-quality changelog is substantial effort compared to just relying on the auto-generation for the release notes. I think there's a way forward that uses a combination of auto-generation and hand-crafting for this purpose, such as a transformer over YAML data, which I'd be happy to investigate and prototype for the team.

My primary concern is that end users, resource authors, and integrating developers have a useful document that describes the changes in a release. In the short term, I'd be happy to fully hand-craft this document.

In my experience, there are a few concerns that get brought up when discussing maintaining a changelog by hand instead of using automated tooling:

1. It requires more work.

   In my experience, this only applies when the existing entries aren't very useful. There are changelog tools that generate the semantic structure and entries using a combination of PR titles and labels. In those cases, the work is actually the same - you need to write your PRs like changelog entries and remember to label your PRs appropriately. This is fundamentally the same work as adding a list item in the correct section of the changelog.

2. Writing and reviewing the changelog at release time is error prone.

   I agree with this entirely. One common solution to this problem is to maintain an Unreleased section at the top of the changelog that accumulates every change since the last version. This serves a few purposes. It ensures that changelog entries are written and updated as close to the implementation of the change as possible, meaning everyone working on the changelog has the freshest and most accurate perspective on the change that they're going to have. From that point on, our memories of the change and context only degrade with time. It also enables curious readers to check the changelog for how the next release will affect them, giving them an opportunity to file work items for themselves or the team before the code ships in a release.

3. It can lead to merge conflicts when two PRs both modify the same section of the changelog and are merged near the same time.

   There's not really a good way around this, except to be nindful that addressing the conflict may require rebasing and a bit of work, but adjusting changelog entries is never as complicated as addressing merge conflicts for code in my experience.

Beyond those points, auto-generated changelogs that rely on PRs to build the changelog have a few pitfalls that I think are fundamental issues:

1. Pull Requests are maintainer-facing documentation, not user, author, or integrating vendor documentation. Pull requests are about the internal context leading to a change. The title and body of a PR are intended to provide context to the reviewers and maintainers of the repository. That context is not always parseable for external folks, especially general users. When we conflate maintainer-facing documentation with external-facing documentation, we need to make compromises for one group or the other.

2. When changelogs are generated from pull requests, we only get one entry in the changelog per PR. This means that either:

   • Only the primary purpose of the PR gets listed in the changelog, like a new feature. If the PR also fixes bug, that information is lost and only retrievable by exploring the pull request.
   • The PR title becomes long and difficult to parse, requiring the title to enumerate every external-facing change and context.

   One solution to this problem is to require PRs to be fully atomic: each PR only makes exactly one external-facing change that would merit a changelog entry. This vastly increases the overhead for maintainers and contributors, requiring folks to split PRs or accept the flattening of the changelog or release notes.

3. When changelogs are generated from pull requests directly, the only documentation we can associate with those entries is the PR (maintainer facing documentation) and related issues (contextual to why the change happened but not necessarily descriptive of how the change affects the project).