Run full e2e test suite on dependabot submodule update PRs

[ceyhun]

Run full e2e test suite on dependabot submodule update PRs and post failures to slack

To test: Check if Test iOS on Device - Submodule Update and Test Android on Device - Submodule Update are run without any approval.

PR submission checklist:

• I have considered adding unit tests where possible.
• I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary.

[hypest]

Nice tweak, thanks Ceyhun! Makes total sense to have the full set of tests run automatically on these hash-update PRs!

While we're at it I'm thinking: Dependabot doesn't offer PR grouping yet, to have 1 single PR with the Gutenberg and the Jetpack submodule for example, leading to too many tests running automatically like that, let alone not being super sure that the two updates are successfully stacked unless we manually ask Dependabot to rebase after we merge one of the two. So, to save time and tests CI occupancy, what do you think about setting https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates#open-pull-requests-limit to 1 so there will be only 1 outstanding PR? Idea is that we'll let either the Gutenberg or the Jetpack hash update PR finish and get merged first and the Dependabot will open the other.

[ceyhun]

Thanks for the idea @hypest. I have 2 things I'm not sure about this:

1. What if it doesn't open another PR until next day, because the interval is daily?
2. What if it opens another gutenberg PR instead of a jetpack one?

We have to try and see I guess?

[hypest]

Yeah, I thought of case No2 (which should be kinda OK to favor Gutenberg updates more often than Jetpack's) but case No1 might still favor Gutenberg's hash and that'd be too much.

Let's stick with the current limit (5 PRs by default) and we'll see. The time these PRs will open (5am UTC) are perhaps early enough that it won't be rush hour anyway.

[hypest]

With the full testsuite now running automatically on the hash-update Dependabot PRs, I think we can also experiment with auto-merging. I've enabled the option in our account and if you agree Ceyhun, we can add the support via this PR. What we need to do is update the dependabot config with something like this:

version: 2
updates:
  - package-ecosystem: "gitsubmodule" # See documentation for possible values
    directory: "/" # Location of package manifests
    schedule:
      interval: "daily"
+   automerged_updates:
+     - match:
+         dependency_name: "gutenberg"
+     - match:
+         dependency_name: "jetpack"

What do you think?

[ceyhun]

Sounds good to me, will update the config 👍

[hypest]

Ouch, I should have looked deeper. It looks like auto-marging is not supported in the GitHub-native Dependabot :( dependabot/dependabot-core#1973 (comment)

[ceyhun]

Indeed :( got an error in the checks saying Your .github/dependabot.yml contained invalid details. Will revert this change.

[hypest]

Yeah, sorry for the time lost there Ceyhun :(

On a different note, I wonder if the peril prompt to run the full testsuite might be confusing to people. I didn't see it popup in the first full run of the tests so I thought it wouldn't trigger, but obviously it does. 🤔

[ceyhun]

I think it was checking for the job named Optional UI Tests: https://github.com/Automattic/peril-settings/blob/685783686a07f5a09ee8d65acbf96335f7b0b146/org/pr/gb-mobile-ui-tests.ts#L7
Ignored it for submodule update branches.

[hypest]

LGTM and it it will be very useful to have in place, thanks Ceyhun. If the Peril prompt persists we'll address it in a separate PR 👍