-
Notifications
You must be signed in to change notification settings - Fork 118
Description
PR #1316 introduced a new Gradle task launching full suite integration tests ./gradlew integrationTests
. Those are in addition to already implemented IT, not a replacement. Current IT (let's call them sanity IT) do their job as quick check if flank's jar works correctly and nothing unplanned causes flank to crash with a simple run. Sanity ITs are also cheap.
Given mentioned I think it's fine to run them automatically.
Full suite IT requires several times more devices (both virtual and physical). flank-open-source
project has a very high quota limit, yet it's not unlimited. With full suite IT running automatically (similar to sanity IT) we could burn daily quota very quickly, as result PR's would be blocked and general work on repo would be difficult, at least until the end of the day.
Requirements for trigger (my proposals in parenthesis):
- GH action should run ONLY full suite ITs (separated flow/gh action?)
- we can use a similar trigger that was implemented in ci: Run integration tests on comment or manual trigger #1342 (@flank-it-fs)
- IT results should be presented to the user, there are some possibilities:
- PR's checklist (might be tough due to missing PR context for issue comment event)
- update/change PR description
- add a comment (separate for each trigger invocation)
- consider the form of presented results: text, a summary table, etc (I think the easiest would be to print gradle build scan URL, similar to ubuntu-workflow)
- consider a limitation of who can trigger full suite IT
- instant feedback, we should inform a user about the current status of IT/action. (This can be achieved via reactions), for instance:
- 👀
noticed
- gh action was triggered - 🚀
IT started
- full suite ITs were started and are in progress - 🎉 - all tests finished successfully
- 👎 - there are some failed tests
- 👀
- sanity IT (currently running) should remain
- Add/update PR's check (need investigations if it is possible)
Additional info:
- as mentioned earlier issue comment is missing PR context (it's due to the fact GH considers PRs as normal issues)
- we don't need to implement super fancy flow/gh action from the beginning, Let's start with the easiest implementation that fulfills requirements