Community testing
Element is released every two weeks to bring all the latest features to our users. The release is normally made on a Monday and the release candidate (RC) is available for testing from the previous Tuesday (that’s almost a week to test!). RCs are deployed to https://staging.element.io, where they can be tried and tested by everyone.
Testing can feel overwhelming when doing it for the first time: where should I start? What should I test? Do I need to test what others already tested? We have set-up a Matrix room for community testing of Element, and fortnightly testing session to do it together. Note that you can take part in the testing session without joining the call!
We are currently running the user testing initiative on Web and we hope to expand it to include Android and iOS.
Join our Matrix room for community testing of Element and have a look at the room description. It should contain the link to the spreadsheet we are going to use for this session. You will need to request edit access to the spreadsheet in the Matrix room.
When you have edit access to the spreadsheet, the first thing you need to do is find an empty column and fill in your Matrix ID (so we can reach out to you if we have any questions). Below your MXID, please fill in your operating system, browser and browser version. We need this information as many issues are platform specific so it will help us reproduce any issues that you find.
You can now open the RC for the next Element release in private browsing mode: https://staging.element.io
You’re now ready to test the staging version of Element! To do so, you need to follow the test cases in the lower left frame of the spreadsheet. In our example, the first step is to create an account on Matrix.org. Click on the test case name, and it will bring you to a document with all the steps you need to follow.
Please mark test cases as "done" in your column in the spreadsheet by filling them with a green colour for pass or red for fail. You can also use orange if there is something which needs attention.
If you find any issues, head to the Issues found tab of the spreadsheet:
You should see a sheet which looks like this:
When walking through the steps of the test case, you may notice oddities, unsettling behaviour of the application, unexpected screens, confusing messages, or even failures. If one of these occurs, please check if the defect has already been recorded in the spreadsheet, and if not, take a screenshot or a screen recording of the issue. When you're ready to make a note of the problem, fill in a new line in the sheet:
- Defect is when a feature or interaction is broken, while an Enhancement is when a feature or interaction could be improved.
- Who found it: if you test alone, list yourself so we can follow up with you for more information; if you test with other people, you can list them as well if that is useful.
- Which test cases you were following: copy and paste the name of the test case you were following. It must be one of the tests listed on the
Overviewtab unless you have gone off script. - Description: a short description of what happened.
- Issue already exists: have a quick search through existing issues to see if it’s been filed before.
- How does it affect you: write a sentence or two to explain why this issue is a problem for you.
- Steps to reproduce: list detailed steps to explain how someone else can reproduce the issue. Make sure to mention where you are starting, which part of the screen you click (e.g. “the + button in the bottom right corner”), what to expect in between interactions. Use Alt+Return to go to a new line in the cell.
- Do you have a screenshot or a screen recording: screenshots and recordings are worth a thousand words. If you have any relevant ones, we can reach out to you to get copies of them.
- Link to the issue on GitHub: if you are filing your own issues or if the issue already exists, you can add a link here so that the issue isn’t duplicated.