tls.test_tls_stress read errors

[krizhanovsky]

Check that you have the latest wrk:

# /root/wrk/wrk --version|head -1
wrk 4.1.0-4-g0896020 [epoll] Copyright (C) 2012 Will Glozer

since today I observed that older wrk not always produce actual load (check fix tempesta-tech/tempesta-test#105 which makes corresponding assertion).

I have following config:

# cat ./tests_config.ini
[Client]
ip = 127.0.0.1
hostname = localhost
ab = ab
#wrk = /usr/bin/wrk
wrk = /root/wrk/wrk
workdir = /tmp/client

[Tempesta]
workdir = /root/tempesta
ip = 127.0.0.1
hostname = localhost
user = root
config = /tmp/tempesta.conf
port = 22

[Server]
aliases_interface = eth0
workdir = /tmp/nginx
ip = 127.0.0.1
aliases_base_ip = 192.168.10.1
nginx = nginx
user = root
port = 22
max_workers = 16
hostname = localhost
keepalive_timeout = 60
keepalive_requests = 100
resources = /var/www/html/

[General]
duration = 10
concurrent_connections = 10
verbose = 3
log_file = tests_log.log

And I see following failure of the test (@i-rinat also observed this, but not always):

test_tls (tls.test_tls_stress.StressTls) ... 
	Init test case...
	Starting Nginx on 127.0.0.1:8000
	Starting TempestaFW on localhost
	Running 1 HTTP clients on localhost
Running 10s test @ https://127.0.0.1/
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.86ms    6.74ms  35.80ms   74.63%
    Req/Sec    27.71     14.47   134.00     75.44%
  540 requests in 10.10s, 137.11KB read
  Socket errors: connect 0, read 2639, write 0, timeout 0
Requests/sec:     53.46
Transfer/sec:     13.57KB
---- RESULTS --------
Status 200 : 540 times
---- END ------------
 
WARNING! Socket errors on wrk. Too many concurrent connections?
	Client: errors: 2639, requests: 540, rate: 53
	Clients in total: errors: 2639, requests: 540, rate: 53
errors 500: 0
errors 502: 0
errors 504: 0
errors connect: 0
errors read: 2639
errors write: 0
errors timeout: 0
FAIL
	Stopping TempestaFW on localhost
	Stopping Nginx on 127.0.0.1:8000
	Removing Nginx config for 127.0.0.1:8000

======================================================================
FAIL: test_tls (tls.test_tls_stress.StressTls)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta-test/tls/test_tls_stress.py", line 39, in test_tls
    self.generic_test_routine(config)
  File "/root/tempesta-test/testers/stress.py", line 287, in generic_test_routine
    self.generic_asserts_test()
  File "/root/tempesta-test/testers/stress.py", line 277, in generic_asserts_test
    self.assert_clients()
  File "/root/tempesta-test/testers/stress.py", line 187, in assert_clients
    self.assert_client(req, err, statuses)
  File "/root/tempesta-test/testers/stress.py", line 171, in assert_client
    self.assertEqual(err, e_500 + e_502 + e_504 + e_connect, msg=msg)
AssertionError: HTTP client detected 2639/540 errors. Results: {200: 540, 'write_error': 0, 'timeout_error': 0, 'connect_error': 0, 'read_error': 2639}

----------------------------------------------------------------------
Ran 1 test in 10.918s

FAILED (failures=1)

Need to understand the reason for read errors, use BPF or at least strace or systemtap to research the nature of the errors. When we understand the reason, we'll need to fix the test.

The test is written using the deprecated API, so please rewrite it using the new framework.

[i-rinat]

As @avbelov23 mentioned in off-github discussion, the issue is highly likely caused by TLS encryption that garbles data pages from the pagecache.

Indeed, I can reproduce the same error on Tempesta from master branch, but code from #1264 which makes sure not to overwrite pages from cache, makes test failures go away. I also tried to turn off sendfile Nginx option in tests, and that also made the test pass.