Smoke Test Dev Review

[michelkaporin]

It would be great if each of you can familiarise yourself with smoke test area that you are responsible of (this involves code review, general automated smoke test operation). Smoke test areas are defined as here - https://github.com/Microsoft/vscode/wiki/Smoke-Test.

This is required to ensure we reduce any potential flaws in our smoke tests. Moreover, going further you may need to implement some tests if you add new feature to VS Code, or redesign some of existing smoke test parts due to any UI changes. Hence, it is important that you are familiar with the way automated smoke test is working now.

Here is a general readme on the code - https://github.com/Microsoft/vscode/tree/master/test/smoke.

When reviewing your area make sure you understand:

• Code architecture
• What to do when smoke test fails (looking it up on VSTS website, downloading test artifacts after build has finished, debugging)
• Weak points

Also think of the way you would improve the smoke test, and write your thoughts here. Please check yourself off once you became familiar with it.

• @alexandrudima
• @weinand
• @chrmarti
• @Tyriar
• @dbaeumer
• @bpasero
• @egamma
• @isidorn
• @joaomoreno
• @jrieken
• @kieferrm
• @aeschli
• @rebornix
• @ramya-rao-a
• @roblourens
• @sandy081
• @mjbvz

All feedback is welcome and appreciated, skepticism should be transformed into suggestions together with contribution to automated smoke test implementation, ensuring it is always reliable! 🙂

[michelkaporin]

I haven't mentioned it anywhere else, hence adding it here.

If you see test failure that you cannot reproduce locally, you might want to check out screenshots generated by smoke test. You can do it on VSTS websites, by going to build and dowloading build artifacts.

[bpasero]

Multi root smoke test: #31059

[roblourens]

Why do we often send a 'NULL' key? https://github.com/Microsoft/vscode/blob/master/test/smoke/src/areas/search.ts#L20

[michelkaporin]

@roblourens it indicates KEYUP event to the webdriver.

[roblourens]

I see, a helper for that would be useful

[egamma]

Closing - the internship of @michelkaporin is over and we are now on our own 😄

[weinand]

Some smoke tests depend on each other. e.g. a git test depends on an exact number of created or modified files. If you disable a test that creates or modifies files, you will have to adapt the git test, other wise it will fail.

[michelkaporin]

@weinand this is correct, as the automated smoke test was implemented with the same scenario as in manual smoke test we did before. The latter one had the same dependency as you've mentioned above. If we split each area not to be dependent, we lose end-to-end scenario, thus reducing scope of testing. As a solution every test can become self-contained, but the time it would take to perform such tests would be much bigger.