Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cypress][docu] tests/System/README.md revised in 5 points #44660

Open
wants to merge 3 commits into
base: 5.2-dev
Choose a base branch
from
Open

[Cypress][docu] tests/System/README.md revised in 5 points #44660

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
da3307e
tests/System/README.md revised in 5 points
muhme Dec 23, 2024
c8d9bb4
Linked to the official Joomla Docker containers
muhme Dec 24, 2024
e11c14e
Revert to title case in capitalisation
muhme Dec 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 58 additions & 27 deletions tests/System/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# System Tests in Joomla
# Joomla System Tests

The Joomla CMS System Tests are end-to-end (E2E) tests and executed in real browsers
The **Joomla System Tests** are end-to-end (E2E) tests and executed in real browsers
using the [Cypress](https://www.cypress.io) test automation tool.
These tests simulate real user interactions, clicking and navigating like a human would,
to ensure the entire application works as expected.
Expand All @@ -10,14 +10,14 @@ and details how to execute and extend tests.
It concludes with solutions for common failure situations in the troubleshooting chapter.


## Software Architecture – Overview
## Software architecture overview

The following software architecture diagram illustrates the Joomla System Tests architecture and provides an overview.
It is simplified to offer an initial understanding. Detailed explanations follow later in this document.

![System Tests Architecture](images/system-tests.svg)
![Joomla System Tests Architecture](images/system-tests.svg)

On the left, **Cypress** is running as a [Node.js](https://nodejs.org/) application. The file **`cypress.config.mjs`** is used to configure settings and preferences for running the System Tests in your environment.
On the left, **Cypress** is running as a [Node.js](https://nodejs.org/) application. The file **`cypress.config.mjs`** is used to configure settings and preferences for running the Joomla System Tests in your environment.

In the middle, the **Cypress Test Runner** controls a **Browser** with the **Joomla** application running HTML, CSS, and JavaScript. Also running in the browser context are the **Database Commands**, the **API commands** and the npm packages **[joomala-cypress](https://github.com/joomla-projects/joomla-cypress/)** and
**[smtp-tester](https://www.npmjs.com/package/smtp-tester)**.
Expand All @@ -26,19 +26,32 @@ The **Joomla** CMS server software is depicted on the right. It runs with PHP on
several key components: the public **User-Frontend**, the administrator **Admin-Backend**, the **API**, and the **Web-Installer**. These components and their interactions will be detailed later in the document.
The file **`configuration.php`** is used to configure settings for Joomla server software.

Joomla uses a **Database**, and the System Tests do as well.
Joomla uses a **Database**, and the Joomla System Tests do as well.


## Installation

Joomla System Tests work on different operating systems such as macOS, desktop Linux distributions like Ubuntu,
and Windows (using WSL 2 or a framework like Laragon). They also work well with Docker containers.
You will need a Web Server and a database like MariaDB running.
Joomla System Tests are compatible with various operating systems, including macOS, desktop Linux distributions like Ubuntu, and Windows (using WSL 2 or frameworks like Laragon).
They also integrate seamlessly with Docker containers (see e.g. [JBT](https://github.com/muhme/joomla-branches-tester)).
muhme marked this conversation as resolved.
Show resolved Hide resolved

To run these tests, you will need a web server (e.g. Apache) and a database (e.g. MariaDB).
Ensure the web server is configured to load the `rewrite_module` and allow directory overrides. For example, in Apache:

```apache
# Load the rewrite module
LoadModule rewrite_module lib/httpd/modules/mod_rewrite.so

# Configure directory overrides
<Directory "...">
AllowOverride All
</Directory>
```

Before getting started, please ensure you have the following prerequisites installed:
[PHP](https://www.php.net/), [Git](https://git-scm.com/), [npm](https://www.npmjs.com/),
and [Composer](https://getcomposer.org/).

The following steps are required for the installation of the CMS System Tests.
The following steps are required for the installation of the Joomla System Tests.

1. Clone Joomla into an empty folder (e.g. `/var/www/html`) where it can be served by a Web Server
```
Expand All @@ -58,7 +71,7 @@ cp cypress.config.dist.mjs cypress.config.mjs
5. Adapt the env variables in the file `cypress.config.mjs`, they should point to the site, user data and database environment. Ensure that the `smtp_port` is not in use on your system.


## Running System Tests
## Running Joomla System Tests

After installation, you can start the Joomla System Tests with headless Cypress. The test suite starts with Joomla Web-Installer as the first test step.
```
Expand Down Expand Up @@ -86,34 +99,35 @@ npx cypress run --spec 'tests/System/integration/{administrator,site,api,plugins
> npm run cypress:open
> ```

If you are running System Tests, you will see `console.log()` outputs from Cypress Tasks in the Node.js environment. If you would like to see `console.log()` output from the browser in headless mode as well, you can use the Electron web browser and set the following environment variable:
If you are running Joomla System Tests, you will see `console.log()` outputs from Cypress Tasks in the Node.js environment. If you would like to see `console.log()` output from the browser in headless mode as well, you can use the Electron web browser and set the following environment variable:
```
export ELECTRON_ENABLE_LOGGING=1
npm run cypress:run --browser electron
```


## Software Architecture – More Detailed
## Software architecture details

Since many interactions in System Tests are involved, the following image illustrates 10 simplified interactions,
Since Joomla System Tests involve many interactions, the following image illustrates 10 simplified interactions,
which are numbered and described below.

![System Tests Architecture with 10 Interactions](images/system-tests-interactions.svg)
![Joomla System Tests Architecture with 10 Interactions](images/system-tests-interactions.svg)

1. **Cypress** starts the **Browser** and runs **Cypress Test Runner** to control Joomla running in the browser and access the DOM.
2. **Joomla** software running in the browser sends requests to the **Web Server** and receives responses just as it would during normal use, even without tests.
3. The Cypress custom **API commands** (described later) interact with the Joomla **API** on the Web Server.
4. Cypress **Tasks** are used to execute code within the Cypress Node.js context. These tasks are triggered by the Cypress Test Runner, which runs in the browser, and are typically used for operations like
interacting with the file system.
5. Joomla on the Web Server interacts with the **Database** as it normally would, without running any tests.
6. System Tests has Cypress custom **Database Commands** (described later) to interact with the database.
7. The file `cypress.config.mjs` is read by **Cypress** and used to configure settings and preferences for running the System Tests in your environment.
6. Joomla System Tests has Cypress custom **Database Commands** (described later) to interact with the database.
7. The file `cypress.config.mjs` is read by **Cypress** and used to configure settings and
preferences for running the Joomla System Tests in your environment.
8. The Joomla installation is initiated by the test spec [Installation.cy.js](integration/install/Installation.cy.js),
which is the first test executed in the overall test suite.
This test spec deletes the Joomla configuration file, and since the `configuration.php` file no longer exists,
the following Joomla Web-Installer call starts the installation process, including database creation.
To ensure that this initial test spec runs correctly, the `installation` folder must not be deleted,
allowing the System Tests to be executed multiple times.
allowing the Joomla System Tests to be executed multiple times.
After the Joomla Web-Installer completes, [Installation.cy.js](integration/install/Installation.cy.js)
modifies some parameters in the `configuration.php` file, such as SMTP settings or the API `$secret`.
9. Joomla Web-Installer creates `configuration.php` file. For security reasons, the file mask is set to read-only (444).
Expand All @@ -132,7 +146,7 @@ This server accepts connections, receives emails, and provides the capability to
> Welcome to the async land of Node.js. :sweat_smile:


## Create New Tests
## Create new tests

To create new tests, create a `cy.js` file in a new folder which matches the following pattern
(replace *foo* with the extension name to test):
Expand All @@ -146,7 +160,7 @@ To create new tests, create a `cy.js` file in a new folder which matches the fol
> Probably the easiest way is to copy an existing file and adapt it to the extension which should be tested.


## Test Development
## Test development

Tests should be:
- Repeatable
Expand Down Expand Up @@ -178,7 +192,7 @@ The following code in a test executes the writing file task with parameters:
### Commands

We are using [custom commands](https://docs.cypress.io/api/cypress-api/custom-commands) to enhance Cypress
with reusable code snippets for the System Tests.
with reusable code snippets for the Joomla System Tests.
These commands can be used to create objects in the database or to call the API.
Since Cypress doesn't support namespaces for commands, we prefix them in the function name.
Therefore, a Database Command always starts with `db_`, and an API command with `api_`.
Expand All @@ -190,7 +204,7 @@ cy.db_createArticle({ title: 'automated test article' }).then((id) => { ... })`
These commands are executed in the browser where the `cy.` namespace is available.


#### Database Commands
#### Database commands

The Database Commands create items in the database like articles or users. They are asynchronous and must be chained like:
```JavaScript
Expand Down Expand Up @@ -240,9 +254,26 @@ The following commands are available and are served by the file
- **api_responseContains** – Checks if the given attribute in the response contains the specified value


### Developer Tips
#### Config commands

There is a single config command provided by the file
[tests/System/support/commands/config.mjs](/tests/System/support/commands/config.mjs):

- **config_setParameter** - Sets a parameter in `configuration.php` file

Even tough `config_setParameter` is a Cypress command running in the browser context,
it uses `cy.readFile()` and the Cypress task `writeRelativeFile` to delegate file system operations to Node.js,
as browsers cannot access the filesystem directly.

This command is asynchronous and must be chained, as shown below:
```JavaScript
cy.config_setParameter('sef', false).then(() => { ... })`
```


### Developer tips

#### Running a Single Test
#### Running a single test

If you wish to run only one single test from a test spec file for debugging, you can add `.only` to the test function:
```JavaScript
Expand All @@ -253,7 +284,7 @@ it.only('running only this test', () => {
For more details, see the [Cypress docu, Excluding and Including Tests](https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests#Excluding-and-Including-Tests).


#### Wait After Action
#### Wait after action

When developing tests with Cypress, it can be helpful to insert delays for debugging, allowing you to observe the status.
For example:
Expand Down Expand Up @@ -310,9 +341,9 @@ If the used SMTP server port is already in use you will see an error like:
:point_right: If you use `npx` instead of `npm`, you may see `Your configFile threw an error from: cypress.config.js`,
but you still need to configure `cypress.config.mjs` file.

### Timeout Error on Slow Machines
### Timeout error on slow machines

If you encounter the following error while running the System Tests on slow machines:
If you encounter the following error while running the Joomla System Tests on slow machines:

```
AssertionError: Timed out retrying after 4000ms: Expected to find element
Expand Down