Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run our full automation test suite on all new changes to the develop branch #4101

Closed
guarani opened this issue Oct 11, 2021 · 3 comments · Fixed by #4114
Closed

Run our full automation test suite on all new changes to the develop branch #4101

guarani opened this issue Oct 11, 2021 · 3 comments · Fixed by #4114
Labels
[Pri] High Testing Anything related to automated tests

Comments

@guarani
Copy link
Contributor

guarani commented Oct 11, 2021

Is your feature request related to a problem? Please describe.

We found in the 1.63.0 release that Dependabot caught an incoming Gutenberg breakage after the fact (i.e. after a previous PR had update the Gutenberg submodule ref). Dependabot was able to catch the breakage while the previous PR did not, because Dependabot runs optional tests by default, but regular PRs do not (they are optional tests that must be manually triggered).

Describe the solution you'd like

Dependabot runs daily, so to reduce the amount of time between a breakage being introduced and it potentially being caught by Dependabot, this issue proposes to have all updates to the Gutenberg Mobile develop branch trigger the full suite of tests. I think this is similar to the main apps which run optional tests after PRs have merged to develop (e.g. here for WPiOS, and here for WPAndroid).

Describe alternatives you've considered

Alternatives include:

  • making Gutenberg Mobile optional tests not optional anymore (would slow down our workflow for all developers)
  • leaving things as-is since this proposal only shortens the time between an incoming Gutenberg breakage being introduced and it being detected, it doesn't avoid detection happening after the fact
  • Moving optional tests to the Gutenberg repo and possibly running them there on all updates to trunk (would slow down the workflow for all mobile/web developers)
@guarani guarani added the Testing Anything related to automated tests label Oct 11, 2021
@guarani
Copy link
Contributor Author

guarani commented Oct 12, 2021

Dependabot might not be the right tool to achieve the goal of running optional tests on every update to develop, since this task won't involve updating dependencies.

@mchowning
Copy link
Contributor

mchowning commented Oct 12, 2021

👍 I like this idea!

@guarani guarani changed the title Make Dependabot run on all new changes that land in the develop branch Run our full automation test suite on all new changes to the develop branch Oct 14, 2021
@guarani
Copy link
Contributor Author

guarani commented Oct 14, 2021

👋 Hey @mchowning, I've updated the issue now that I think a solution to this involves CircleCI, not Dependabot. I've also added you as a reviewer here: #4114
We could definitely discuss this as a team before moving ahead with it (since it's a change that affects our shared workflow), but I'm keen to hear what you think at this early stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Pri] High Testing Anything related to automated tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants