Merge pull request #679 from tempesta-tech/ik-func-tests

Testing framework

tempesta_fw/t/functional/.gitignore

@@ -0,0 +1 @@
+tests_config.ini

tempesta_fw/t/functional/README.md

@@ -0,0 +1,215 @@
+# Functional Tests for TempestaFW
+
+## Recommended configuration
+
+Running tests during development process can cause crashes to TempestaFW.
+Since TempestaFW is implemented as a set of kernel modules it is not convenient
+to run testing framework on the same host. It is recommended to run testing
+framework on a separated host.
+
+Recommended test-beds:
+
+- Local testing. All parts of the testing framework are running on the same
+host. The simpliest configuration to check that current revision of TempestaFW
+passes all the functional tests. It is default configuration.
+```
+    ┌─────────────────────────────────────────────┐
+    │ Testing Framework + TempestaFW + Web Server │
+    └─────────────────────────────────────────────┘
+```
+
+- With isolated testing framework. This preset more helpful for development
+process, since testing framework itself is isolated from possible kernel
+crashes or hangs. This configuration is recommended for TempestaFW developers.
+```
+    ┌───────────────────┐
+    │ Testing Framework ├────┐
+    └──────┬────────────┘    │ Management over SSH
+           │              ┌──┴──────────────────────┐
+           │              │ TempestaFW + Web Server │
+           │              └───────────────┬─────────┘
+           └──────────────────────────────┘
+          Separated network for test traffic
+```
+
+- Fully distributed. 3 different hosts with their own roles are used. This
+configuration isolates traffic generated by benchmark utilities and traffic
+generators in test network. Handy for stress and performance testing but require
+a lot of resources.
+```
+    ┌───────────────────┐
+    │ Testing Framework ├────┐
+    └──────┬────────────┘    │ Management over SSH
+           │                 ├────────────────────┐
+           │          ┌──────┴─────┐        ┌─────┴──────┐
+           │          │ TempestaFW │        │ Web Server │
+           │          └──────┬─────┘        └─────┬──────┘
+           └─────────────────┴────────────────────┘
+              Separated network for test traffic
+```
+
+There is two different models of tests: workload tests and pure functional
+tests. Workload tests uses fully functional HTTP benchmark programs (ab, siege,
+wrk) and HTTP servers (Apache, nginx) to check TempestaFW behaviour. This type
+of tests is used for schedulers, stress and performance testing.
+
+Pure functional tests check internal logic. Here combined HTTP client-server
+server is used. It sends HTTP messages to TempestaFW, analyses how they are
+forwarded to server, and vice versa, which server connections are used.
+
+
+## Requirements
+
+- Host for testing framework: `Python2`, `python2-paramiko`,
+`python-configparser`, `python-subprocess32`, `wrk`, `ab`, `siege`
+- All hosts except previous one: `sftp-server`
+- Host for running TempestaFW: Linux kernel with Tempesta, TempestaFW sources
+- Host for running server: `nginx`, web content directory accessible by nginx
+
+`wrk` is an HTTP benchmarking tool, available from [Github](https://github.com/wg/wrk).
+
+`ab` is Apache benchmark tool, that can be found in `apache2-utils` package in
+Debian or `httpd-tools` in CentOS.
+
+`siege` is an HTTP benchmarking tool, available in `siege` package in Debian
+and `siege` in [EPEL repository](https://dl.fedoraproject.org/pub/epel/7/x86_64/s/siege-4.0.2-2.el7.x86_64.rpm)
+in CentOS.
+
+Unfortunately, CentOS does not have `python-subprocess32` package, but it can be
+downloaded from [CentOS CBS](https://cbs.centos.org/koji/buildinfo?buildID=10904)
+
+Testing framework manages other hosts via SSH protocol, so the host running
+testing framework must be able to be authenticated on other hosts by the key.
+That can be done using `ssh-copy-id`.
+
+
+## Run tests
+
+### Configuration
+
+Testing framework is configured via `tests_config.ini' file. Example
+configuration is described in `tests_config.ini.sample' file.
+You can also create default tests configuration by calling:
+
+```sh
+$ ./run_tests.py -d
+```
+
+There is 4 sections in configuration: `General`, `Client`, `Tempesta`, `Server`.
+
+#### General Section
+
+`General` section describes the options related to testing framework itself.
+
+`verbose`: verbose level of output:
+- `0` - quiet mode, result of each test is shown by symbols. `.` - passed, `F` -
+failed, `u` - unexpected success, `x` - expected failure. `s` - skipped;
+- `1` - Show test names and doc strings;
+- `2` - Show tests names and performance counters;
+- `3` - Full debug output.
+
+`Duration` option controls duration in seconds of each workload test. Use small
+values to obtain results quickly add large for more heavy stress tests. Default
+is `10` seconds.
+
+This group of options can be overridden by command line options, for more
+information run tests with `-h` key.
+```sh
+$ ./run_tests.py -h
+```
+
+#### Client Section
+
+`workdir` - directory to place temporary files (configs, pidfiles, etc.) on the
+host. R/W access is required, must be absolute path.
+
+`ab`, `siege` and `wrk` - absolute path to corresponding binaries.
+
+#### Tempesta Section
+
+`ip` - IPv4/IPv6 address of the host in test network. Used to run `wrk`,
+`tempesta`, `nginx` and others with right parameters. Default is `127.0.0.1`.
+
+`hostname`, `port`, `user` - this options describes "management" interface of
+the host. Testing framework uses this fields to connect to each test node.
+
+Options above are common for hosts `Server` and `Tempesta` and present in each
+section.
+
+`workdir` - Directory with TempestaFW sources. Must be absolute path.
+
+#### Server Section
+
+Options listed in [Tempesta Section](#tempesta-section): `ip`, `hostname`,
+`port`, `user` are also applied to this section.
+
+`workdir` - directory to place temporary files (configs, pidfiles, etc.) on the
+host. R/W access is required, must be absolute path.
+
+`nginx` - absolute path to corresponding binary.
+
+`resourses` - absolute path to directory with sample web pages. Must be
+reachable by `nginx`.
+
+
+### Run tests
+
+To run all the tests simply run:
+```sh
+$ ./run_tests.py
+```
+
+The unittest module can be used from the command line to run tests from modules,
+classes or even individual test methods:
+```sh
+$ python2 -m unittest test_module1 test_module2
+$ python2 -m unittest test_module.TestClass
+$ python2 -m unittest test_module.TestClass.test_method
+```
+Next command will run all tests from specified directory:
+```sh
+$ python2 -m unittest discover <directory>
+```
+In this case verbosity of the tests names is controlled separately from
+configuration flie:
+```sh
+$ python2 -m unittest -v test_module.TestClass
+```
+For a list of all the command-line options:
+```sh
+$ python2 -m unittest -h
+```
+
+
+## Adding new tests
+
+Adding new tests is easy. First, create new Python file in the new Python module
+(directory) or existing one.
+Name of the file must be started with `test_`
+```sh
+$ mkdir my_test
+$ touch my_test/test_some_feature.py
+$ echo "__all__ = [ 'test_some_feature' ]" >> my_test/__init.py__
+```
+
+Import module `unittest`, and derive you test class from `stress.StressTest`
+or `functional.FunctionalTest`
+class from `testers` module. Add functions started with `test_`. Test class may
+have any name. Here is example of `my_test/test_some_feature.py`:
+```python
+import unittest
+from testers import stress
+
+class MyTester(stress.StressTest):
+    """ Test class documentation. """
+
+    tempesta_defconfig = 'cache 0;\n'
+
+    def test_some_featue(self):
+        """ Test documentation. """
+        self.generic_test_routine(self.tempesta_defconfig)
+```
+
+Tests can be skipped or marked as expected to fail.
+More info at [Python documentation](https://docs.python.org/3/library/unittest.html).
+

tempesta_fw/t/functional/cache/__init__.py

@@ -0,0 +1 @@
+__all__ = ['check_cache']

tempesta_fw/t/functional/cache/test_cache.py

@@ -0,0 +1,100 @@
+"""Functional tests of caching responses."""
+
+from __future__ import print_function
+import unittest
+from helpers import deproxy, tf_cfg, tempesta
+from testers import functional
+
+__author__ = 'Tempesta Technologies, Inc.'
+__copyright__ = 'Copyright (C) 2017 Tempesta Technologies, Inc.'
+__license__ = 'GPL2'
+
+# TODO: add tests for RFC compliance
+
+class TestCacheDisabled(functional.FunctionalTest):
+
+    messages = 10
+
+    # Disable caching
+    cache_mode = 0
+
+    def chain(self, uri='/', cache_alowed=True):
+        if self.cache_mode == 0:
+            cache_alowed = False
+        if cache_alowed:
+            return cache_chains(self.messages, uri=uri)
+        return proxy_chains(self.messages, uri=uri)
+
+    def test_cache_fulfill_all(self):
+        config = ('cache %d;\n'
+                  'cache_fulfill * *;\n' % self.cache_mode)
+        self.generic_test_routine(config, self.chain(cache_alowed=True))
+
+    def test_cache_bypass_all(self):
+        config = ('cache %d;\n'
+                  'cache_bypass * *;\n' % self.cache_mode)
+        self.generic_test_routine(config, self.chain(cache_alowed=False))
+
+    def mixed_config(self):
+        return ('cache %d;\n'
+                'cache_fulfill suffix ".jpg" ".png";\n'
+                'cache_bypass suffix ".avi";\n'
+                'cache_bypass prefix "/static/dynamic_zone/";\n'
+                'cache_fulfill prefix "/static/";\n'
+                % self.cache_mode)
+
+    def test_cache_fulfill_suffix(self):
+        self.generic_test_routine(
+            self.mixed_config(),
+            self.chain(cache_alowed=True, uri='/picts/bear.jpg'))
+
+    def test_cache_fulfill_suffix_2(self):
+        self.generic_test_routine(
+            self.mixed_config(),
+            self.chain(cache_alowed=True, uri='/jsnfsjk/jnd.png'))
+
+    def test_cache_bypass_suffix(self):
+        self.generic_test_routine(
+            self.mixed_config(),
+            self.chain(cache_alowed=False, uri='/howto/film.avi'))
+
+    def test_cache_bypass_prefix(self):
+        self.generic_test_routine(
+            self.mixed_config(),
+            self.chain(cache_alowed=False,
+                       uri='/static/dynamic_zone/content.html'))
+
+    def test_cache_fulfill_prefix(self):
+        self.generic_test_routine(
+            self.mixed_config(),
+            self.chain(cache_alowed=True, uri='/static/content.html'))
+
+
+class TestCacheSharding(TestCacheDisabled):
+
+    # Sharding mode.
+    cache_mode = 1
+
+class TestCacheReplicated(TestCacheDisabled):
+
+    # Replicated mode.
+    cache_mode = 2
+
+
+def cache_chain(uri):
+    cached_chain = functional.base_message_chain(uri=uri)
+    cached_chain.no_forward()
+    return cached_chain
+
+def proxy_chain(uri):
+    return functional.base_message_chain(uri=uri)
+
+def cache_chains(count, uri='/'):
+    chains = [proxy_chain(uri)]
+    chain = cache_chain(uri)
+    cached_chains = [chain for i in range (1, count)]
+    return chains + cached_chains
+
+def proxy_chains(count, uri='/'):
+    chain = proxy_chain(uri)
+    return [chain for i in range (count)]

tempesta_fw/t/functional/helpers/__init__.py

@@ -1 +1 @@
-__all__ = [ 'be', 'cli', 'tfw' ]
+__all__ = ['tf_cfg', 'deproxy', 'nginx', 'tempesta', 'siege', 'error']