Merge pull request #26 from scality/docs/tests_management

docs(tests): describe full testing procedure

CONTRIBUTING.md

@@ -135,31 +135,7 @@ submitted.
 
 #### The different types of tests
 
-Three types of tests can be identified: Unit tests, Integration tests and
-Performance tests.
-
-Unit tests are tests that only involve the code expressing the logic of the
-program, confined outside of any external influence (no third party library, no
-systems-related component). At most, it tolerates tests using in-memory
-backends. The role of this type of test is to validate the logic of a written
-piece of code. The shall cover expected usage as well as as many unexpected
-usages as possible.
-
-Integration tests are tests that involve testing any set of logic and
-functionalities together as one system, in order to get as close as possible
-to a real deployment situation. Those tests shall attempt to use the whole
-system in a multitude of different situations. They shall also generate errors
-in a number of different components, in order to prove the reliability of the
-code even in a failing situation.
-
-Performance tests are tests that involve long run times, that shall collect
-datapoints along the test, in order to generate statistics files (.csv or
-anything easily usable), and visual graphics (through gnuplot or similar
-tools). They are used to evaluate the performance of an implementation, and
-evaluate the evolution of the performance over time and over usage of the
-software. This data can be used to detect unreliable or insufficient
-performance within a component, as well as allow the detection of performance
-regressions.
+Please see our [testing guidelines](TESTING.md)
 
 #### Documenting the Integration Test Plan
 

TESTING.md

@@ -0,0 +1,60 @@
+# Test layout in IronMan projects
+
+Guidelines surrounding our testing setup
+
+## Types of tests
+
+During the growth of our project, we identified four different kind of testing.
+These vary in size, but they can be separated in two categories: *small tests*
+and **big tests**.
+
+### Small tests
+
+They should be easy to run, using simple npm scripts, to allow for a quick local
+testing. They are launched by a vanilla Circle-CI every time you push a branch
+and will not require [IronMan-CI]. That means they are expected to be found in
+the `tests` directory of the related project.
+
+#### Unit tests
+
+Those are the smallest tests possible. They do not require to run an instance
+of the actual program, and should assess the good behaviour of every function.
+They can be run using the `npm test` command, and are written using mocha.
+
+The role of this type of test is to validate the logic of a written piece of
+code. They cover expected usage as well as as many unexpected uses as possible.
+
+Belong to the `tests/unit` directory.
+
+#### Functional tests
+
+These tests encompass the whole program, thus need a running instance of it.
+They can be launched with the `npm run ft_test` command, and will require the
+program to be started using `npm start` beforehand. They may require a few
+dependencies (S3 Clients, supervisord, ...).
+
+Belong to the `tests/functional` directory.
+
+### Big tests
+
+They are meant to test heavily a single component, or test the entire project
+(end-to-end testing). Those are not trivial to set-up, and are only started
+on an @ironman-machine invocation. That invocation starts a build using
+[IronMan-Integration] master branch, using the branch related to the ongoing PR
+for each project. They are thus expected to be found in [IronMan-Integration].
+
+#### Integration tests
+
+These tests are still only assessing a single module of the project. Being
+non-trivial, they require a more advanced setup usually covered by [IronMan-CI]
+using `Dockerfile`s and `docker-compose`.
+
+#### End-to-end tests
+
+These tests ensure no breakage in the chain, using our deployment setup,
+[IronMan-Federation]. A small dependency patcher will be used to ensure testing
+of a supposedly non-breaking setup.
+
+[IronMan-CI]: https://github.com/scality/IronMan-CI
+[IronMan-Integration]: https://github.com/scality/IronMan-Integration
+[IronMan-Federation]: https://github.com/scality/IronMan-Federation