Breaking change doc tool

[ericstj]

This adds a workflow to run a script when a PR is labeled as needs-breaking-change-doc-created,

The script collects details about the PR, the breaking change issue template, and sample issues then asks an LLM to generate an issue based on that.

The issue content is then encoded into a clickable link which is then added as a comment to the PR.

I'm doing this from a dotnet branch so that I can hopefully create a PR off of it to demonstrate the workflow in action.

[Copilot]

Pull Request Overview

This PR introduces an automated workflow for creating breaking change documentation. When a PR is labeled needs-breaking-change-doc-created, a PowerShell script analyzes the PR details and uses an LLM to generate documentation content. The generated content is then encoded into a clickable GitHub issue creation link, which is posted as a comment on the PR.

Key Changes

• PowerShell script that collects PR data, analyzes it with LLM, and generates breaking change documentation
• GitHub Actions workflow that triggers on label addition to run the automation
• Configuration file for customizing LLM provider, repository settings, and other parameters

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
eng/breakingChanges/config.ps1	Configuration file defining LLM settings, GitHub repository paths, issue template settings, and rate limiting parameters
eng/breakingChanges/breaking-change-doc.ps1	Main PowerShell script implementing the breaking change documentation workflow with data collection, LLM analysis, and issue generation
eng/breakingChanges/README.md	Documentation explaining the automation tool's features, setup instructions, usage examples, and configuration options
.github/workflows/breaking-change-doc.yml	GitHub Actions workflow that triggers the script when a PR receives the needs-breaking-change-doc-created label

[ericstj]

This is testing the workflow - #120988 Looks like it's failing to fetch the issues from dotnet/docs. I'll see if I can get it issue read permission to the docs repo.

[ericstj]

Now seeing Error: unexpected response from the server: 403 Forbidden

Need to find out how to use models from a workflow.

[rzikm]

Looks great, I look forward to using this once the technical hurdles get ironed out.

[ericstj]

This works file even without the workflow, so it can be used by directly running the script after installing the github CLI. Worse case we'll just use that and occasionally run it.

The summary of the problems with the workflow is that our workflows in the dotnet org don't have access to github models, and that's currently not a technical problem. I can unblock that by storing a secret in the repo as a token for another LLM provider, but I don't like storing/needing-to-update a secret.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

[ericstj]

I don't expect this to work, but also trying copilot CLI provider.

[ericstj]

Ok, this is working now. Ready for review.

[ericstj]

@jeffhandley @dotnet/runtime-infrastructure can I get a review? If folks are worried about the workflow + secret used here I can remove that part and just keep the scripts, but I think the automation adds value.

[jeffhandley]

What's the scenario for wanting to determine feature areas without area labels? I've found it's pretty time-consuming for the GH API to get the list of files from PRs, especially for PRs older that then newest 100 (presumably due to caching optimizations), so if we can avoid collecting the file names that would be beneficial.

[ericstj]

Yeah, I can remove it, but I'm going to have the list of files regardless since I feed this into the model that generates the doc. The perf hasn't been too bad.

[jeffhandley]

Got it; thanks. I found the dotnet-api-docs repo as a good stress test for crawling PRs, and other repos where hundreds of files are edited at once on PRs commonly also exhibit the perf cliff I've seen. In order for labeler-train.yml (dotnet/dotnet-api-docs) to work, it can only consume 1 pull request at a time and even with that, GitHub's API sometimes times out.

[ericstj]

I see. You are worried about being rate limited when fetching details of a single PR. I will see if I can give this script the largest PRs I can find to determine if it will get rate limited in a way that blocks progress.

[ericstj]

it looks like the changes to URL escaping broke the formatting of newlines. I'll need to see when that happened and undo it.

[ericstj]

I actually broke it when refactoring how we call the gh CLI. I also noticed some concern with passing prompts on the commandline. We should avoid this - instead passing a prompt as a file.