[HOLD for payment 2024-02-19] [$500] Create Automation for when main fails in /App

[blimpich]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Problem:

We have a Github Action defined in App/.github/workflows/preDeploy.yml that can fail without being noticed. It runs every time a PR is merged into our main branch. When it fails there is no process for notifying engineers of its failure other than an email to the person who merged the PR, which can easily get lost or ignored. Previous examples of this happening and being handled manually are here and here.

Allowing this test suite to break and go unnoticed can make it difficult to diagnose what caused the issue in the first place, and opens us up to performance regressions that could negatively impact the user experience.

Solution:

Create an automation that produces a new Github issue when main fails. It should only create an issue if there isn't already an open issue for the associated breakage already. The issue should be labelled as daily and should automatically assign both the the author of the PR that contains the commit that kicked off the failed workflow run and the person who merged the PR into main. The issue should also link to the PR that is associated with the failed run.

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~018b564dd37aa88e42
• Upwork Job ID: 1744445606630244352
• Last Price Increase: 2024-01-08
• Automatic offers:
• jjcoffee | Reviewer | 28091182
• rayane-djouah | Contributor | 28091183

Issue Owner

Current Issue Owner: @muttmuure

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~018b564dd37aa88e42

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @jjcoffee (External)

[rayane-djouah]

Proposal

Please re-state the problem that we are trying to solve in this issue.

The current Process new code merged to main GitHub Action lacks a notification system for workflow failures, risking unnoticed issues and impacting user experience.

What is the root cause of that problem?

The absence of automated alerts delays issue detection as notifications rely on emails, often overlooked or lost.

What changes do you think we should make in order to solve the problem?

Implement a new workflow, failureNotifier.yml, triggered on Process new code merged to main completion. Create a GitHub issue on failure if there is no open failure notifier issue for the failed workflow, labeling it 'daily' and assigning it to the author and PR merger.

What alternative solutions did you explore? (Optional)

[jjcoffee]

@rayane-djouah's proposal LGTM! I'm guessing we'll dig into the details on the PR.

🎀👀🎀 C+ reviewed

[melvin-bot]

Triggered auto assignment to @pecanoro, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[pecanoro]

The solution is pretty vague, but I am hoping you guys will handle it in the PR. Assigning @rayane-djouah

[melvin-bot]

📣 @jjcoffee 🎉 An offer has been automatically sent to your Upwork account for the Reviewer role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job

[melvin-bot]

📣 @rayane-djouah 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

[melvin-bot]

Triggered auto assignment to @muttmuure (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[pecanoro]

I added the bug label so we can get someone from the Bug Zero team to handle payments later

[ishpaul777]

Hey @pecanoro i know i am late to party, would that be ok if i add a Proposal with implementation detail now, as selected proposal shares no Plan of action or any details on the solution

[rayane-djouah]

I started to work on it already. The PR will be up soon

[ishpaul777]

Sorry to say, but the proposal seems incomplete and I think the general guidelines is to share a Plan of action for a fair selection. But thats ok if you started working on a fix. Good luck!

[jjcoffee]

@blimpich Sorry, I should've clarified this earlier but I realised I'm not 100% on the expected result. Should a new issue be created for any job that's part of the Process new code merged to main workflow failing? I see that we already have a Slack notification for the build job, for example:

App/.github/workflows/preDeploy.yml

Lines 26 to 27 in 964548a

	- name: Announce failed workflow in Slack
	if: ${{ needs.typecheck.result == 'failure' || needs.lint.result == 'failure' || needs.test.result == 'failure' }}

But no notification for the E2E tests, which seems to be the main issue here from the examples you've linked to.

[rayane-djouah]

@blimpich, yes, it will create a single issue for the e2ePerformanceTests job failure because both of the failed jobs (Build apk from latest release as a baseline and Build apk from delta ref) are part of the e2ePerformanceTests job in the Process new code merged to main workflow. Do you think we should create separate issues for them?

[blimpich]

No I like the way it currently works, just confirming 🙂👍

[melvin-bot]

This issue has not been updated in over 15 days. @pecanoro, @jjcoffee, @muttmuure, @rayane-djouah eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

[rayane-djouah]

updating PR by EOD

[muttmuure]

I think we're just waiting for the above PR to pass the regression period? Or is there more to do here? @blimpich

[pecanoro]

The issue will get updated when the PR gets deployed to prod, it's still on staging

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.39-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] Create Automation for when main fail #34262
• [No QA] Fix permission error for assigning the new issue for main failure notification #36042

If no regressions arise, payment will be issued on 2024-02-19. 🎊

For reference, here are some details about the assignees on this issue:

• @jjcoffee requires payment automatic offer (Reviewer)
• @rayane-djouah requires payment automatic offer (Contributor)

[jjcoffee]

@muttmuure Friendly bump for payment 🙇

[blimpich]

Just wanted to say I say this working for the first time in the wild today (link). Nice job!! 🙂

[melvin-bot]

@pecanoro, @jjcoffee, @muttmuure, @rayane-djouah Whoops! This issue is 2 days overdue. Let's get this updated quick!

[muttmuure]

Everyone is paid up