chore(docs): Reflect the Transactions to Test Suites #3051

xoscar · 2023-08-09T21:07:25Z

This PR updates the documentation to reflect the naming change from Transactions to Test Suites

Changes

Updates the file names
Updates texts and images
Keeps the same overall message as the logic is the same but with a different name

Checklist

tested locally
added new dependencies
updated the docs
added a test

https://www.loom.com/share/791c34547af447dcbec4ac8dfaef1501

…ite Changes

jfermi · 2023-08-09T21:18:31Z

docs/docs/cli/creating-test-suites.md

+This page showcases how to create and edit Test Suites with the CLI.
+
+:::tip
+[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)


Suggested change

[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)

[To read more about Test Suites check out the Test Suites concepts page.](../concepts/test-suites.md)

jfermi · 2023-08-09T21:18:45Z

docs/docs/cli/creating-test-suites.md

+[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)
+:::
+
+Just like other structures of Tracetest, you can also manage your test suites using the CLI and definition files.


Suggested change

Just like other structures of Tracetest, you can also manage your test suites using the CLI and definition files.

Just like other structures of Tracetest, you can also manage your Test Suites using the CLI and definition files.

jfermi · 2023-08-09T21:18:58Z

docs/docs/cli/creating-test-suites.md

+
+Just like other structures of Tracetest, you can also manage your test suites using the CLI and definition files.
+
+A definition file for a test suite looks like the following:


Suggested change

A definition file for a test suite looks like the following:

A definition file for a Test Suite looks like the following:

jfermi · 2023-08-09T21:19:15Z

docs/docs/cli/creating-test-suites.md

+    - testID # you can also reference tests by their ids instead of referencing the definition file
+```
+
+In order to apply this test suite to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:


Suggested change

In order to apply this test suite to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:

In order to apply this Test Suite to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:

jfermi · 2023-08-09T21:19:34Z

docs/docs/cli/creating-test-suites.md

+tracetest apply testsuite -f <testsuite.yaml>
+```
+
+> If the file contains the property `spec.id`, the operation will be considered a test suite udpate.


Suggested change

> If the file contains the property `spec.id`, the operation will be considered a test suite udpate.

> If the file contains the property `spec.id`, the operation will be considered a Test Suite update.

jfermi · 2023-08-09T21:20:23Z

docs/docs/cli/running-test-suites.md


-Once you have created a transaction, whether from the Tracetest UI or via a text editor, you will need the capabity to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.
+Once you have created a test suite, whether from the Tracetest UI or via a text editor, you will need the capabity to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.


Suggested change

Once you have created a test suite, whether from the Tracetest UI or via a text editor, you will need the capabity to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.

Once you have created a Test Suite, whether from the Tracetest UI or via a text editor, you will need the capability to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.

jfermi · 2023-08-09T21:20:40Z

docs/docs/cli/running-test-suites.md


-The command to run a transaction is the same as running a test from the CLI.
+The command to run a test suite is the same as running a test from the CLI.


Suggested change

The command to run a test suite is the same as running a test from the CLI.

The command to run a Test Suite is the same as running a test from the CLI.

jfermi · 2023-08-09T21:21:15Z

docs/docs/cli/running-test-suites.md


-To run a transaction, give the path to the transaction definition file with the `'-f'` option. This will launch a transaction, providing us with a link to the created transaction run.
+To run a test suite, give the path to the test suite definition file with the `'-f'` option. This will launch a test suite, providing us with a link to the created test suite run.


Suggested change

To run a test suite, give the path to the test suite definition file with the `'-f'` option. This will launch a test suite, providing us with a link to the created test suite run.

To run a Test Suite, give the path to the Test Suite definition file with the `'-f'` option. This will launch a Test Suite, providing us with a link to the created Test Suite run.

jfermi · 2023-08-09T21:21:32Z

docs/docs/cli/running-test-suites.md


-There are two ways of referencing a variable set when running a transaction.
+There are two ways of referencing a variable set when running a test suite.


Suggested change

There are two ways of referencing a variable set when running a test suite.

There are two ways of referencing a variable set when running a Test Suite.

jfermi · 2023-08-09T21:21:54Z

docs/docs/cli/running-test-suites.md


 You can reference an existing variable set using its id. For example, given this defined variable set with an id of `'testenv'`:

 ![testenv](../img/show-environment-definition.png)

-We can run a transaction and specify that variable set with this command:
+We can run a test suite and specify that variable set with this command:


Suggested change

We can run a test suite and specify that variable set with this command:

We can run a Test Suite and specify that variable set with this command:

jfermi · 2023-08-09T21:22:24Z

docs/docs/cli/undefined-variables.md

@@ -1,6 +1,6 @@
 # Undefined Variables

-When a user runs a test or a transaction, any variables that will be needed but are not defined will be prompted for:
+When a user runs a test or a test suite, any variables that will be needed but are not defined will be prompted for:


Suggested change

When a user runs a test or a test suite, any variables that will be needed but are not defined will be prompted for:

When a user runs a Test or a Test Suite, any variables that will be needed but are not defined will be prompted for:

jfermi · 2023-08-09T21:22:53Z

docs/docs/cli/undefined-variables.md

@@ -13,7 +13,7 @@ POKEID:
 POKENAME:
 ```

-Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the test or transaction by passing it into the test run command.
+Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the test or test suite by passing it into the test run command.


Suggested change

Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the test or test suite by passing it into the test run command.

Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the Test or Test Suite by passing it into the `test run` command.

jfermi · 2023-08-09T21:23:27Z

docs/docs/concepts/ad-hoc-testing.md

@@ -14,15 +14,15 @@ This page showcases use-cases for undefined variables and how to enable ad-hoc t

 ### **Supply Variable Value at Runtime**

-A user wants a test or transaction they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the test or transaction, they will be prompted for the unspecified variables.
+A user wants a test or test suite they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the test or test suite, they will be prompted for the unspecified variables.


Suggested change

A user wants a test or test suite they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the test or test suite, they will be prompted for the unspecified variables.

A user wants a test or Test Suite they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the Test or Test Suite, they will be prompted for the unspecified variables.

jfermi · 2023-08-09T21:23:47Z

docs/docs/concepts/ad-hoc-testing.md


 ### **Supply Variable Value from a Previous Test**

-A user wants to define 3 tests as part of a transaction. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test. 
+A user wants to define 3 tests as part of a test suite. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test. 


Suggested change

A user wants to define 3 tests as part of a test suite. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test.

A user wants to define 3 tests as part of a Test Suite. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test.

jfermi · 2023-08-09T21:24:13Z

docs/docs/concepts/ad-hoc-testing.md

@@ -40,10 +40,10 @@ In Tracetest, undefined variables can be used in both the UI and CLI.

 ![Test Output](../img/test-output.png)

-5. Now, you can create a transaction with the two tests - first, add the list test, then the add test, and then the list test again:
+5. Now, you can create a test suite with the two tests - first, add the list test, then the add test, and then the list test again:


Suggested change

5. Now, you can create a test suite with the two tests - first, add the list test, then the add test, and then the list test again:

5. Now, you can create a Test Suite with the two tests - first, add the list test, then the add test, and then the list test again:

jfermi · 2023-08-09T21:24:56Z

docs/docs/concepts/test-suites.md

+Most End-to-End tests are not simple to run. They require some setup before the actual test is run. Actions like creating a new user, removing all items from a cart, etc. It is important that you can execute multiple steps as part of your test suite. Tracetest introduces the concept of **Test Suites** to achieve this goal.
+
+## What is a Test Suite?
+A test suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.


Suggested change

A test suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.

A Test Suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.

jfermi · 2023-08-09T21:25:11Z

docs/docs/concepts/test-suites.md

+A test suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.
+
+## Chaining Tests
+The main benefit of using test suites is to chain tests together and use values obtained from a test in a subsequent test.


Suggested change

The main benefit of using test suites is to chain tests together and use values obtained from a test in a subsequent test.

The main benefit of using Test Suites is to chain tests together and use values obtained from a test in a subsequent test.

jfermi · 2023-08-09T21:25:38Z

docs/docs/concepts/test-suites.md

+The main benefit of using test suites is to chain tests together and use values obtained from a test in a subsequent test.
+
+### How Values are Shared by Tests
+When a test suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the test suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.


Suggested change

When a test suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the test suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.

When a Test Suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the Test Suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.

jfermi · 2023-08-09T21:26:14Z

docs/docs/concepts/test-suites.md

+### How Values are Shared by Tests
+When a test suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the test suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.
+
+When a test is executed within a test suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the test suite will be able to reference those values.


Suggested change

When a test is executed within a test suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the test suite will be able to reference those values.

When a test is executed within a Test Suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the Test Suite will be able to reference those values.

jfermi · 2023-08-09T21:26:38Z

docs/docs/concepts/test-suites.md

+
+When a test is executed within a test suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the test suite will be able to reference those values.
+
+> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the test suite run context object.


Suggested change

> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the test suite run context object.

> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the Test Suite run context object.

jfermi · 2023-08-09T21:27:02Z

docs/docs/concepts/test-suites.md

+
+> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the test suite run context object.
+
+Consider you have 3 tests within a test suite: A, B, and C. Tests A and B generate outputs called A_OUTPUT and B_OUTPUT, respectively. When running the test suite, we provide a variable set which contains a `HOST` variable. The execution of test A would only be able to reference `env:HOST`. B would be able to reference `env:HOST`, and `env:A_OUTPUT`. While C would be able to reference all three variables: `env:HOST`, `env:A_OUTPUT`, `env:B_OUTPUT`.


Suggested change

Consider you have 3 tests within a test suite: A, B, and C. Tests A and B generate outputs called A_OUTPUT and B_OUTPUT, respectively. When running the test suite, we provide a variable set which contains a `HOST` variable. The execution of test A would only be able to reference `env:HOST`. B would be able to reference `env:HOST`, and `env:A_OUTPUT`. While C would be able to reference all three variables: `env:HOST`, `env:A_OUTPUT`, `env:B_OUTPUT`.

Consider you have 3 tests within a Test Suite: A, B, and C. Tests A and B generate outputs called A_OUTPUT and B_OUTPUT, respectively. When running the Test Suite, we provide a variable set which contains a `HOST` variable. The execution of test A would only be able to reference `env:HOST`. B would be able to reference `env:HOST`, and `env:A_OUTPUT`. While C would be able to reference all three variables: `env:HOST`, `env:A_OUTPUT`, `env:B_OUTPUT`.

jfermi · 2023-08-09T21:27:49Z

docs/docs/concepts/test-suites.md

+      expression: attr:tracetest.time.start
+```
+
+This would create an output called `TIME_CANCEL_SUBSCRIPTION_MESSAGE_OBTAINED` that is obtained by reading the attribute `tracetest.time.start` from the span with `name` equal to `Process request from cancel subscription queue`. This value would then be injected into the variables of that test suite to be accessed by other tests within the same test suite run.


Suggested change

This would create an output called `TIME_CANCEL_SUBSCRIPTION_MESSAGE_OBTAINED` that is obtained by reading the attribute `tracetest.time.start` from the span with `name` equal to `Process request from cancel subscription queue`. This value would then be injected into the variables of that test suite to be accessed by other tests within the same test suite run.

This would create an output called `TIME_CANCEL_SUBSCRIPTION_MESSAGE_OBTAINED` that is obtained by reading the attribute `tracetest.time.start` from the span with `name` equal to `Process request from cancel subscription queue`. This value would then be injected into the variables of that Test Suite to be accessed by other tests within the same Test Suite run.

jfermi · 2023-08-09T21:28:10Z

docs/docs/concepts/test-suites.md

+
+### Test Suites Execution Flow
+
+Test Suite steps are executed sequentially. A next step is only executed after the previous step finishes executing successfully. A successful step is one which managed to trigger an operation and received a trace back from the data store. Failing assertions do not stop a test suite from executing the next steps.


Suggested change

Test Suite steps are executed sequentially. A next step is only executed after the previous step finishes executing successfully. A successful step is one which managed to trigger an operation and received a trace back from the data store. Failing assertions do not stop a test suite from executing the next steps.

Test Suite steps are executed sequentially. A next step is only executed after the previous step finishes executing successfully. A successful step is one which managed to trigger an operation and received a trace back from the data store. Failing assertions do not stop a Test Suite from executing the next steps.

jfermi · 2023-08-09T21:28:51Z

docs/docs/live-examples/opentelemetry-store/use-cases/user-purchasing-products.md


-We can do that by creating the tests and transactions through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the transaction with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.
+We can do that by creating the tests and test suites through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the test suite with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.


Suggested change

We can do that by creating the tests and test suites through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the test suite with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.

We can do that by creating the Tests and Test Suites through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the Test Suite with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.

jfermi · 2023-08-09T21:29:31Z

docs/docs/live-examples/opentelemetry-store/use-cases/user-purchasing-products.md

@@ -172,16 +172,16 @@ spec:
  - ./checkout.yaml
 ```

-By having the test, transaction and environment files in the same directory, we can call the CLI and execute this transaction:
+By having the test, test suite and environment files in the same directory, we can call the CLI and execute this test suite:


Suggested change

By having the test, test suite and environment files in the same directory, we can call the CLI and execute this test suite:

By having the Test, Test Suite and Environment Files in the same directory, we can call the CLI and execute this Test Suite:

jfermi · 2023-08-09T21:30:38Z

docs/docs/web-ui/creating-test-suites.md

+This page showcases how to create and edit Test Suites in the Web UI.
+
+:::tip
+[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)


Suggested change

[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)

[To read more about Test Suites check out the Test Suites concepts page.](../concepts/test-suites.md)

jfermi · 2023-08-09T21:30:52Z

docs/docs/web-ui/creating-test-suites.md

+
+![Create a Test Button](../img/create-button-0.11.png)
+
+Give your test suite a name, and click **Next**:


Suggested change

Give your test suite a name, and click **Next**:

Give your Test Suite a name, and click **Next**:

jfermi · 2023-08-09T21:31:04Z

docs/docs/web-ui/creating-test-suites.md

+
+![Name the Test Suite](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685712802/docs/beta.tracetest.io__page_1_jynf6o.png)
+
+Next, select which tests to run in the test suite and click **Create & Run**:


Suggested change

Next, select which tests to run in the test suite and click **Create & Run**:

Next, select which tests to run in the Test Suite and click **Create & Run**:

jfermi · 2023-08-09T21:31:15Z

docs/docs/web-ui/creating-test-suites.md

+
+![Select Tests in Test Suite](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685712954/docs/beta.tracetest.io__page_1_1_agjvg0.png)
+
+The test suite will start:


Suggested change

The test suite will start:

The Test Suite will start:

jfermi · 2023-08-09T21:31:30Z

docs/docs/web-ui/creating-test-suites.md

+
+![Running Test Suite](../img/running-testsuite.png)
+
+ On the automate tab, you find methods to automate the current test suite, including the YAML test file and the CLI command for Tracetest.


Suggested change

On the automate tab, you find methods to automate the current test suite, including the YAML test file and the CLI command for Tracetest.

On the automate tab, you find methods to automate the current Test Suite, including the YAML test file and the CLI command for Tracetest.

jfermi · 2023-08-09T21:31:45Z

docs/docs/web-ui/creating-test-suites.md

+
+ With all of the toggles `Off`, each criteria is tested. Toggle individual criteria `On`, to select on certain checks for the selected test.
+
+When the test suite is finished, you will get the following result:


Suggested change

When the test suite is finished, you will get the following result:

When the Test Suite is finished, you will get the following result:

jfermi · 2023-08-09T21:31:59Z

docs/docs/web-ui/creating-test-suites.md

+
+![Finished Test Suite](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685713712/docs/demo.tracetest.io__x0o1gu.png)
+
+You can now view individual [Test Results](test-results.md) executed by the test suite by clicking on any of the tests in the list.


Suggested change

You can now view individual [Test Results](test-results.md) executed by the test suite by clicking on any of the tests in the list.

You can now view individual [Test Results](test-results.md) executed by the Test Suite by clicking on any of the tests in the list.

jfermi · 2023-08-09T21:32:15Z

docs/docs/web-ui/undefined-variables.md

@@ -1,10 +1,10 @@
 # Undefined Variables

-When a user runs a test or a transaction, any variables that will be needed but are not defined will be prompted for:
+When a user runs a test or a test suite, any variables that will be needed but are not defined will be prompted for:


Suggested change

When a user runs a test or a test suite, any variables that will be needed but are not defined will be prompted for:

When a user runs a Test or a Test Suite, any variables that will be needed but are not defined will be prompted for:

jfermi · 2023-08-09T21:32:30Z

docs/docs/web-ui/undefined-variables.md


 ![Undefined Variables Modal](../img/undefined-variables-modal.png)

-Undefined variables are dependent on the environment selected and whether or not the variable is defined in the current environment. Select the environment to run the test or transaction in from the dropdown list at the top right of the page:
+Undefined variables are dependent on the environment selected and whether or not the variable is defined in the current environment. Select the environment to run the test or test suite in from the dropdown list at the top right of the page:


Suggested change

Undefined variables are dependent on the environment selected and whether or not the variable is defined in the current environment. Select the environment to run the test or test suite in from the dropdown list at the top right of the page:

Undefined variables are dependent on the environment selected and whether or not the variable is defined in the current environment. Select the environment to run the Test or Test Suite in from the dropdown list at the top right of the page:

jfermi

Some editing suggestions. I capitalized Test and Test Suite when referring to the Tracetest concepts that the doc is describing for clarity, but you can reject those if you don't think that is necessary.

adnanrahic · 2023-08-16T12:38:57Z

@xoscar Can you add redirects to the links?

I see a lot of pages were renamed. We should add redirects in the docusaurus.config.js for everything that was changed.

kdhamric · 2023-08-16T17:27:02Z

docs/docs/cli/creating-test-suites.md

+tracetest apply testsuite -f <testsuite.yaml>
+```
+
+> If the file contains the property `spec.id`, the operation will be considered a Test Suite udpate.


update is not spelled correctly on this last line.

kdhamric · 2023-08-16T17:31:54Z

docs/docs/concepts/test-suites.md

+
+> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the Test Suite run context object.
+
+Consider you have 3 tests within a Test Suite: A, B, and C. Tests A and B generate outputs called A_OUTPUT and B_OUTPUT, respectively. When running the Test Suite, we provide a variable set which contains a `HOST` variable. The execution of test A would only be able to reference `env:HOST`. B would be able to reference `env:HOST`, and `env:A_OUTPUT`. While C would be able to reference all three variables: `env:HOST`, `env:A_OUTPUT`, `env:B_OUTPUT`.


should we be referencing var:HOST, var:A_OUTPUT, etc here?

kdhamric

Looks great. Added a couple comments.

chore(docs): Updating the Docs to reflect the Transactions to Test Su…

fca0613

…ite Changes

xoscar self-assigned this Aug 9, 2023

xoscar requested review from adnanrahic, schoren, jfermi and kdhamric and removed request for schoren August 9, 2023 21:07

xoscar added the documentation Improvements or additions to documentation label Aug 9, 2023

xoscar marked this pull request as ready for review August 9, 2023 21:13

xoscar added the do not merge label Aug 9, 2023

jfermi reviewed Aug 9, 2023

View reviewed changes

xoscar added 3 commits August 16, 2023 09:29

Merge branch 'main' into chore/transactions-to-test-suites-docs

1ef47cc

PR review comments and redirects

d23fbf7

fixing typos

151a367

kdhamric reviewed Aug 16, 2023

View reviewed changes

kdhamric approved these changes Aug 16, 2023

View reviewed changes

cleanup

f5471d9

xoscar merged commit 7eebb04 into main Aug 16, 2023
30 checks passed

xoscar deleted the chore/transactions-to-test-suites-docs branch August 16, 2023 19:52

	[To read more about test suites check out test suites concepts.](../concepts/test-suites.md)
	[To read more about Test Suites check out the Test Suites concepts page.](../concepts/test-suites.md)

	Just like other structures of Tracetest, you can also manage your test suites using the CLI and definition files.
	Just like other structures of Tracetest, you can also manage your Test Suites using the CLI and definition files.


		Just like other structures of Tracetest, you can also manage your test suites using the CLI and definition files.

		A definition file for a test suite looks like the following:

	In order to apply this test suite to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:
	In order to apply this Test Suite to your Tracetest instance, make sure to have your [CLI configured](./configuring-your-cli.md) and run:

	> If the file contains the property `spec.id`, the operation will be considered a test suite udpate.
	> If the file contains the property `spec.id`, the operation will be considered a Test Suite update.


		Once you have created a transaction, whether from the Tracetest UI or via a text editor, you will need the capabity to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.
		Once you have created a test suite, whether from the Tracetest UI or via a text editor, you will need the capabity to run it via the Command Line Interface (CLI) to integrate it into your CI/CD process or your local development workflow.


		The command to run a transaction is the same as running a test from the CLI.
		The command to run a test suite is the same as running a test from the CLI.


		To run a transaction, give the path to the transaction definition file with the `'-f'` option. This will launch a transaction, providing us with a link to the created transaction run.
		To run a test suite, give the path to the test suite definition file with the `'-f'` option. This will launch a test suite, providing us with a link to the created test suite run.


		There are two ways of referencing a variable set when running a transaction.
		There are two ways of referencing a variable set when running a test suite.

	We can run a test suite and specify that variable set with this command:
	We can run a Test Suite and specify that variable set with this command:

	When a user runs a test or a test suite, any variables that will be needed but are not defined will be prompted for:
	When a user runs a Test or a Test Suite, any variables that will be needed but are not defined will be prompted for:

	Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the test or test suite by passing it into the test run command.
	Undefined variables are dependent on the variable set selected and whether or not the variable is defined in the current variable set. Select the variable set to run the Test or Test Suite by passing it into the `test run` command.

	A user wants a test or test suite they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the test or test suite, they will be prompted for the unspecified variables.
	A user wants a test or Test Suite they can run on a particular user, order id, etc. that is configurable at run time. This makes running an adhoc test in an environment, even production, very easy and convenient. In this case, the user references the variable, but doesn't add it to the environment. Each time they run the Test or Test Suite, they will be prompted for the unspecified variables.

	A user wants to define 3 tests as part of a test suite. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test.
	A user wants to define 3 tests as part of a Test Suite. The first test has an output variable and this output is used by the second test. They define the first test. They then define the second test and reference the variable value that is output from the first test.

	5. Now, you can create a test suite with the two tests - first, add the list test, then the add test, and then the list test again:
	5. Now, you can create a Test Suite with the two tests - first, add the list test, then the add test, and then the list test again:

	A test suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.
	A Test Suite is defined as a group of steps that are executed in the defined order and can access information exported by previous step executions. Each step is a test.

	The main benefit of using test suites is to chain tests together and use values obtained from a test in a subsequent test.
	The main benefit of using Test Suites is to chain tests together and use values obtained from a test in a subsequent test.

	When a test suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the test suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.
	When a Test Suite is run, a context object is created with information about that specific run. One of those pieces of information is a `variable set` object, which is empty by default. If the Test Suite is run when referencing an [variable set](./variable-sets.md), all values from the selected variable sets will be copied to the `variable set` object.

	When a test is executed within a test suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the test suite will be able to reference those values.
	When a test is executed within a Test Suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the Test Suite will be able to reference those values.


		When a test is executed within a test suite, if it generates any outputs, its outputs will be injected into the test suite context variable set object. After the outputs are injected, all subsequent tests to be run within the test suite will be able to reference those values.

		> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the test suite run context object.

chore(docs): Reflect the Transactions to Test Suites #3051

chore(docs): Reflect the Transactions to Test Suites #3051

Conversation

xoscar commented Aug 9, 2023 • edited Loading

Changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfermi left a comment

Choose a reason for hiding this comment

adnanrahic commented Aug 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdhamric left a comment

Choose a reason for hiding this comment

xoscar commented Aug 9, 2023 •

edited

Loading


		> :information_source: Outputs generated by steps don't modify the selected [variable set](./variable-sets.md). It only modifies the test suite run context object.

		Consider you have 3 tests within a test suite: A, B, and C. Tests A and B generate outputs called A_OUTPUT and B_OUTPUT, respectively. When running the test suite, we provide a variable set which contains a `HOST` variable. The execution of test A would only be able to reference `env:HOST`. B would be able to reference `env:HOST`, and `env:A_OUTPUT`. While C would be able to reference all three variables: `env:HOST`, `env:A_OUTPUT`, `env:B_OUTPUT`.

	This would create an output called `TIME_CANCEL_SUBSCRIPTION_MESSAGE_OBTAINED` that is obtained by reading the attribute `tracetest.time.start` from the span with `name` equal to `Process request from cancel subscription queue`. This value would then be injected into the variables of that test suite to be accessed by other tests within the same test suite run.
	This would create an output called `TIME_CANCEL_SUBSCRIPTION_MESSAGE_OBTAINED` that is obtained by reading the attribute `tracetest.time.start` from the span with `name` equal to `Process request from cancel subscription queue`. This value would then be injected into the variables of that Test Suite to be accessed by other tests within the same Test Suite run.


		### Test Suites Execution Flow

		Test Suite steps are executed sequentially. A next step is only executed after the previous step finishes executing successfully. A successful step is one which managed to trigger an operation and received a trace back from the data store. Failing assertions do not stop a test suite from executing the next steps.


		We can do that by creating the tests and transactions through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the transaction with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.
		We can do that by creating the tests and test suites through the Web UI or using the CLI. In this example, we will use the CLI to create a Variable Set and then create the test suite with all tests needed. The [assertions](../../../concepts/assertions.md) that we will check are the same for every single test.

	By having the test, test suite and environment files in the same directory, we can call the CLI and execute this test suite:
	By having the Test, Test Suite and Environment Files in the same directory, we can call the CLI and execute this Test Suite:


		![Create a Test Button](../img/create-button-0.11.png)

		Give your test suite a name, and click Next:


		![Name the Test Suite](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685712802/docs/beta.tracetest.io__page_1_jynf6o.png)

		Next, select which tests to run in the test suite and click Create & Run:


		![Select Tests in Test Suite](https://res.cloudinary.com/djwdcmwdz/image/upload/v1685712954/docs/beta.tracetest.io__page_1_1_agjvg0.png)

		The test suite will start:


		![Running Test Suite](../img/running-testsuite.png)

		On the automate tab, you find methods to automate the current test suite, including the YAML test file and the CLI command for Tracetest.


		With all of the toggles `Off`, each criteria is tested. Toggle individual criteria `On`, to select on certain checks for the selected test.

		When the test suite is finished, you will get the following result: