Testing framework #679
Testing framework #679
Conversation
vankoven
force-pushed
the
ik-func-tests
branch
from
6a9a4ec to
ba78a8e
Compare
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| ## Requirements | ||
|
|
||
| - Host for testing framework: `Pyhon3` and `pyhon3-paramiko` |
There was a problem hiding this comment.
Do we really need Python3? Current CentOS 7 doesn't have, so installation of the newer Python might be an issue.
tempesta_fw/t/functional/README.md
Outdated
| achieved by populating ARP/Neighbour tables on each host. In that case all hosts | ||
| except one running testing framework must be in one subnet. See fully | ||
| distributed configuration. Possible options: `True` or `False`. Default is | ||
| `False`. |
There was a problem hiding this comment.
Only several ARP packets are required to fully fill the ARP cache for the largest test installation. Why ARP cache population is actually needed and how does it reduce load onto TCP stack?
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| `dir` - Directory with TempestaFW sources. Must be absolute path. | ||
|
|
||
| #### Tempesta Section |
There was a problem hiding this comment.
Seems should be "Server Section"
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| `hostname`, `port`, `user` - this options describes "management" interface of | ||
| the host. Testing framework uses this fields to connect to each test node. | ||
|
|
There was a problem hiding this comment.
What is workdir in Client section?
| ```sh | ||
| $ ./run_tests.py | ||
| ``` | ||
|
|
There was a problem hiding this comment.
There is some issue with the test suit. I can't run it on my CentOS 7 VM, please see the trace at the below. Probably the issue can be easily fixed or just analyzed at least. However, the fundamental problem with the test suite is improper exceptions handling: it prints inadequate call trace instead of printing some user friendly message. The call trace is familiar to the test developer only.
[root@localhost functional]# ./run_tests.py -f
----------------------------------------------------------------------
Running functional tests...
----------------------------------------------------------------------
E
======================================================================
ERROR: sched.test_sched_hash (unittest.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib64/python3.4/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib64/python3.4/unittest/case.py", line 618, in run
testMethod()
File "/usr/lib64/python3.4/unittest/loader.py", line 33, in testFailure
raise self._exception
ImportError: Failed to import test module: sched.test_sched_hash
Traceback (most recent call last):
File "/usr/lib64/python3.4/unittest/loader.py", line 323, in _find_tests
module = self._get_module_from_name(name)
File "/usr/lib64/python3.4/unittest/loader.py", line 301, in _get_module_from_name
__import__(name)
File "/root/tempesta/tempesta_fw/t/functional/sched/test_sched_hash.py", line 2, in <module>
from helpers import tfw_test, tempesta
File "/root/tempesta/tempesta_fw/t/functional/helpers/tfw_test.py", line 2, in <module>
from . import tf_cfg, control, tempesta
File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 4, in <module>
from . import tf_cfg, remote, nginx, tempesta, siege
File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 194, in <module>
assert(setter.fill_arp())
AssertionError
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
[root@localhost functional]# cat tests_config.ini
[Tempesta]
mac = ff:ff:ff:ff:ff:ff
dir = /root/tempesta
hostname = localhost
ip = 127.0.0.1
user = root
port = 22
[Server]
nginx = nginx
mac = ff:ff:ff:ff:ff:ff
workdir = /opt/nginx-1.11.3/
hostname = localhost
ip = 127.0.0.1
user = root
resourses = /var/www/html/
port = 22
[General]
arp = True
duration = 10
verbose = 0
concurent_connections = -1
[Client]
ab = ab
wrk = wrk
mac = ff:ff:ff:ff:ff:ff
workdir = /root/client
hostname = localhost
ip = 127.0.0.1
user = root
siege = siege
port = 22
[root@localhost functional]# pwd
/root/tempesta/tempesta_fw/t/functional
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| Recommended test-beds: | ||
|
|
||
| - Fully distributed. 4 different hosts with their own roles are used. This |
There was a problem hiding this comment.
I think in most cases the test suite will be run on the same VM, just to verify that Tempesta FW passes all the tests. The distributed test is certainly useful, but for development tasks only. Thus, it's better to use single VM installation as a default configuration.
tempesta_fw/t/functional/README.md
Outdated
| Or run next command to show an example of configuration: | ||
| ```sh | ||
| $ ./run_tests.py -e | ||
| ``` |
There was a problem hiding this comment.
The option is good, but a user still needs to read the README to configure the test suite. And that's not comfortable at all. It's better to add a default configuration file with comments for all the options.
| 'port': '22', | ||
| 'nginx': 'nginx', | ||
| 'workdir': '/root/nginx', | ||
| 'resourses': '/srv/http/'} |
There was a problem hiding this comment.
It's better to use /var/www/html/ as a default directory for Apache HTTPD or Nginx
|
|
||
| To run all the tests simply run: | ||
| ```sh | ||
| $ ./run_tests.py |
There was a problem hiding this comment.
Probably we need the file only. Now run_all_tests.sh is just broken, so it's better to remove it at all
vankoven
force-pushed
the
ik-func-tests
branch
2 times, most recently
from
e29f344 to
d2178d4
Compare
…as client wrk closes connections before Tempesta finish working with it's requests. That can cause "other errors". In all current tests ignoring the errors is not a problem since other counters gives pretty accurate information.
Remove to simplify testing framework
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| #### Server Section | ||
|
|
||
| Options listed in [Temoesta Section](#tempesta-section): `ip`, `hostname`, |
There was a problem hiding this comment.
"Temoesta" -> "Tempesta"
tempesta_fw/t/functional/README.md
Outdated
|
|
||
| ## Requirements | ||
|
|
||
| - Host for testing framework: `Pyhon2`, `pyhon2-paramiko`, |
There was a problem hiding this comment.
Python2 and python2-paramiko
tempesta_fw/t/functional/README.md
Outdated
| ## Requirements | ||
|
|
||
| - Host for testing framework: `Pyhon2`, `pyhon2-paramiko`, | ||
| `python-configparser`, `python-subprocess32`, `wrk`, `ab`, `siege` |
There was a problem hiding this comment.
It'd be good to mention wrk website and apache2-utils for Debian or httpd-tools for CentOS. pythom-subprocess32 isn't presented in CentOS 7 packages, so please refer https://cbs.centos.org/koji/buildinfo?buildID=10904 where a user can download the package.
| Functional tests for TempestaFW. | ||
|
|
||
| Test Framework Configuration is stored in 'tests_config.ini', Use '-e' option | ||
| to get example. Normally 3 machines are used to run tests: one to run HTTP |
There was a problem hiding this comment.
The option doesn't work any more
tempesta_fw/t/functional/README.md
Outdated
| $ touch my_test/test_some_feature.py | ||
| $ echo "__all__ = [ 'test_some_feature' ]" >> my_test/__init.py__ | ||
| ``` | ||
| 2. Import module `unittest`, and derive you test class from `tfw_test.Loader` |
There was a problem hiding this comment.
The enumeration doesn't work in resulting file
Here is framework for functional, stress and performance testing of TempestaFW.
Readmeprovide more information.