Ecosystem testing

[mosuem]

Workflow for testing a package upgrade against the ecosystem.

See for example mosuem/i18n#2 (comment)

---

• I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

• See our contributor guide for general expectations for PRs.
• Larger or significant changes should be discussed in an issue before creating a PR.
• Contributions to our repos should follow the Dart style guide and use dart format.
• Most changes should add an entry to the changelog and may need to rev the pubspec package version.
• Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

[github-actions]

PR Health

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ✔️

File	Coverage

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbols

License Headers ✔️

// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

[devoncarew]

A few in-line comments, mostly to help me get context on the PR.

I hadn't considered this mechanism - a re-usable workflow. With this, any repo could set up a mini-corpus to test what a package upgrade might do to their apps under test.

I think another - perhaps less flexible solution - would be thuswise:

• have a standalone repo for testing package upgrades (this would be useful for testing highly used packages, but not general packages)
• have a corpus of apps defined declaratively
• have a PR somehow indicate which package upgrade to test (modifying a specific file?) the system would likely use this to generate dependency override files
• attempt to shard the PR's workflow into multiple jobs, one per app in the corpus. then failure of a specific corpus app is easier to investigate
• have the workflow collate all job status info into a markdown table that it appends to the PR; this might show info like which apps changed state from 'pub get' working to 'pub get' failing; similarly for analysis and test

The markdown table lets you see at a glance which apps were adversely affected by the package upgrade.

Thoughts?

[mosuem]

A few in-line comments, mostly to help me get context on the PR.

Yeah, sorry for the complete lack of documentation - this is not ready for review yet.

[mosuem]

I hadn't considered this mechanism - a re-usable workflow. With this, any repo could set up a mini-corpus to test what a package upgrade might do to their apps under test.

Yes, that's the idea. This way, any package might have "their" set of packages they care about, so we could

I think another - perhaps less flexible solution - would be thuswise:

• have a standalone repo for testing package upgrades (this would be useful for testing highly used packages, but not general packages)
• have a corpus of apps defined declaratively
• have a PR somehow indicate which package upgrade to test (modifying a specific file?) the system would likely use this to generate dependency override files

So I would have to file a PR with that repo in addition to the PR in the repo of the package-to-be-upgraded?

• attempt to shard the PR's workflow into multiple jobs, one per app in the corpus. then failure of a specific corpus app is easier to investigate

I originally had in mind to create multiple jobs, but could not find a way to collapse the list of jobs in the Github action view, the"Some checks were not successful"-thing. If run in a separate repository as you are proposing here, that is fine, but running it together with other checks would not work, as it would produce dozens of checks.

• have the workflow collate all job status info into a markdown table that it appends to the PR; this might show info like which apps changed state from 'pub get' working to 'pub get' failing; similarly for analysis and test

Yes, that would also be added here, kind of instead of having multiple Github jobs. That way the user still gets to see which app failed easily. For debugging, we could output the logs to different workflow artifacts, and link those - that would make it very easy to investigate.

The markdown table lets you see at a glance which apps were adversely affected by the package upgrade.

Thoughts?

I think (IIUC) that filing a separate PR in another repo might be more work than running this reusable workflow "on command", either by hand or by labeling the PR with something like ecosystem-<packagename>-upgrade.

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:firehose	0.10.0	ready to publish	firehose-v0.10.0
package:dart_flutter_team_lints	3.2.1	already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[devoncarew]

lgtm, though I would verify that all references to mosuem/ecosystem and such are intentional (not holdovers from where this tool was prototyped).