[ty] Add workflow to comment conformance tests diff #19555
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhruvmanila
added
ci
dhruvmanila
force-pushed
the
dhruv/conformance-tests-diff
branch
2 times, most recently
from
e4cb4d5 to
3a0a354
Compare
dhruvmanila
force-pushed
the
dhruv/conformance-tests-diff
branch
from
3a0a354 to
d781a6a
Compare
dhruvmanila
added a commit
that referenced
this pull request
Jul 25, 2025
…19556) ## Summary This PR adds a workflow to comment the diff of diagnostics when running ty between `main` and a pull request on the [typing conformance test suite](https://github.com/python/typing/tree/main/conformance/tests). The main workflow is introduced in #19555 which this workflow depends on. This workflow is similar to the [mypy primer comment](https://github.com/astral-sh/ruff/blob/d781a6ab3f44afe5dd1afe1d06fb58fd3739fab5/.github/workflows/mypy_primer_comment.yaml) workflow. ## Test Plan I cannot test this workflow without merging it on `main` unless anyone knows a way to do this.
dhruvmanila
force-pushed
the
dhruv/conformance-tests-diff
branch
from
d781a6a to
e8fee63
Compare
Contributor
Diagnostic diff on typing conformance testsNo changes detected when running ty on typing conformance tests ✅ |
This reverts commit d0db993.
Contributor
|
Member
Author
|
The mypy-primer workflow got triggered as I made a change in the Rust code which added the diff, but it did not get triggered once I reverted the Rust code change so the diff has persisted. |
AlexWaygood
pushed a commit
that referenced
this pull request
Jul 25, 2025
…19556) ## Summary This PR adds a workflow to comment the diff of diagnostics when running ty between `main` and a pull request on the [typing conformance test suite](https://github.com/python/typing/tree/main/conformance/tests). The main workflow is introduced in #19555 which this workflow depends on. This workflow is similar to the [mypy primer comment](https://github.com/astral-sh/ruff/blob/d781a6ab3f44afe5dd1afe1d06fb58fd3739fab5/.github/workflows/mypy_primer_comment.yaml) workflow. ## Test Plan I cannot test this workflow without merging it on `main` unless anyone knows a way to do this.
sharkdp
approved these changes
Jul 25, 2025
| echo "new commit" | ||
| git checkout -b new_commit "${{ github.event.pull_request.head.sha }}" | ||
| git rev-list --format=%s --max-count=1 new_commit | ||
| cargo build --release --bin ty |
Contributor
There was a problem hiding this comment.
I guess we could experiment with whether or not a release build is faster here (longer build time vs faster check time)
|
|
||
| echo "Running ty on old commit (merge base)" | ||
| cd conformance/tests | ||
| ../../../ruff/ty-old check --color=never --output-format=concise . > ../../../old-output.txt 2>&1 || true |
Contributor
There was a problem hiding this comment.
Elsewhere, we fail the build if the exit code indicates a panic. But I guess it's fine if we see it in the diff?
Member
Author
There was a problem hiding this comment.
Yeah, we can experiment, not sure what's the right approach is here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #19556, this PR adds the workflow that computes the diagnostic diff which the workflow introduced in the linked PR will add as a comment.
This workflow is similar to the ty ecosystem-analyzer workflow.
Closes: astral-sh/ty#212
Test Plan
Use the comment history to look at the diff output where the order of the history corresponds to the steps mentioned above in reverse order i.e., the edit in the middle will contain the diff output: