Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel panic during pipelined test #926

Closed
vladtcvs opened this issue Feb 26, 2018 · 4 comments
Closed

Kernel panic during pipelined test #926

vladtcvs opened this issue Feb 26, 2018 · 4 comments
Assignees
Milestone

Comments

@vladtcvs
Copy link
Contributor

[  512.195192] ------------[ cut here ]------------
[  512.201033] WARNING: CPU: 0 PID: 3 at net/core/skbuff.c:4628 skb_try_coalesce+0x41b/0x440
[  512.201829] Modules linked in: tfw_sched_ratio(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) ntfs vfat msdos fat xfs fuse edac_mce_amd edac_core crct10dif_pclmul e1000 ppdev ghash_clmulni_intel psmouse sr_mod aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd cdrom battery parport_pc ac pcspkr sg button parport i2c_piix4 ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache serio_raw sd_mod [last unloaded: tempesta_tls]
[  512.246117] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           O    4.9.35-kdump+ #17
[  512.246916] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  512.247739]  ffff9939c032b920 ffffffffb84bb0aa 0000000000000000 0000000000000000
[  512.248131]  ffff9939c032b960 ffffffffb8262121 00001214c032ba40 ffff8b1bc83ecf00
[  512.248131]  ffff8b1bdc2c3f00 ffff9939c032b9bc 0000000000000373 0000000000000273
[  512.248131] Call Trace:
[  512.248131]  [<ffffffffb84bb0aa>] dump_stack+0x4d/0x63
[  512.248131]  [<ffffffffb8262121>] __warn+0xc1/0xe0
[  512.248131]  [<ffffffffb8262248>] warn_slowpath_null+0x18/0x20
[  512.248131]  [<ffffffffb865e76b>] skb_try_coalesce+0x41b/0x440
[  512.248131]  [<ffffffffb86c6a2e>] tcp_try_coalesce+0x5e/0xc0
[  512.248131]  [<ffffffffb86cdbe5>] tcp_data_queue+0xb55/0xd40
[  512.248131]  [<ffffffffb86cdffd>] tcp_rcv_established+0x22d/0x6f0
[  512.248131]  [<ffffffffb868be2b>] ? sk_filter_trim_cap+0x2b/0x260
[  512.248131]  [<ffffffffb86d8a50>] tcp_v4_do_rcv+0x140/0x200
[  512.248131]  [<ffffffffb86da272>] tcp_v4_rcv+0x912/0xa10
[  512.248131]  [<ffffffffb86b85a6>] ? ip_finish_output+0x126/0x1c0
[  512.248131]  [<ffffffffc0641848>] ? tdb_rec_get+0x38/0x80 [tempesta_db]
[  512.248131]  [<ffffffffb86b31bd>] ip_local_deliver_finish+0x9d/0x1d0
[  512.248131]  [<ffffffffb86b34aa>] ip_local_deliver+0x6a/0xe0
[  512.248131]  [<ffffffffb86abd00>] ? nf_iterate+0x50/0x60
[  512.248131]  [<ffffffffb86b2e43>] ip_rcv_finish+0x113/0x3f0
[  512.248131]  [<ffffffffb86b37a6>] ip_rcv+0x286/0x3d0
[  512.248131]  [<ffffffffb86b2d30>] ? inet_del_offload+0x40/0x40
[  512.248131]  [<ffffffffb8672512>] __netif_receive_skb_core+0x4d2/0xa30
[  512.248131]  [<ffffffffb8658200>] ? skb_page_frag_refill+0x60/0xc0
[  512.248131]  [<ffffffffb86c6dfc>] ? tcp_rearm_rto.part.61+0x4c/0x80
[  512.248131]  [<ffffffffb86cca7f>] ? tcp_rearm_rto+0x1f/0x30
[  512.248131]  [<ffffffffc0680ff3>] ? tfw_wq_pop_ticket+0x83/0x1090 [tempesta_fw]
[  512.248131]  [<ffffffffb8672a83>] __netif_receive_skb+0x13/0x60
[  512.248131]  [<ffffffffb8673a68>] process_backlog+0x88/0x120
[  512.248131]  [<ffffffffb8673252>] net_rx_action+0x202/0x2f0
[  512.248131]  [<ffffffffb8266f2d>] __do_softirq+0xed/0x200
[  512.248131]  [<ffffffffb8282c50>] ? smpboot_update_cpumask_percpu_thread+0x130/0x130
[  512.248131]  [<ffffffffb8267057>] run_ksoftirqd+0x17/0x30
[  512.248131]  [<ffffffffb8282d55>] smpboot_thread_fn+0x105/0x180
[  512.248131]  [<ffffffffb827f459>] kthread+0xf9/0x110
[  512.248131]  [<ffffffffb827f360>] ? kthread_park+0x60/0x60
[  512.248131]  [<ffffffffb8772742>] ret_from_fork+0x22/0x30
[  512.489768] ---[ end trace 9e44a8cb407bdaa2 ]---
[  514.296321] ------------[ cut here ]------------
[  514.302289] WARNING: CPU: 0 PID: 3 at net/ipv4/tcp_input.c:3266 tcp_ack+0x1380/0x15b0
[  514.347943] Modules linked in: tfw_sched_ratio(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) ntfs vfat msdos fat xfs fuse edac_mce_amd edac_core crct10dif_pclmul e1000 ppdev ghash_clmulni_intel psmouse sr_mod aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd cdrom battery parport_pc ac pcspkr sg button parport i2c_piix4 ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache serio_raw sd_mod [last unloaded: tempesta_tls]
[  514.352719] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G        W  O    4.9.35-kdump+ #17
[  514.356057] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  514.356057]  ffff9939c032b8e0 ffffffffb84bb0aa 0000000000000000 0000000000000000
[  514.356057]  ffff9939c032b920 ffffffffb8262121 00000cc2c032b908 ffff8b1c0713ed80
[  514.356057]  ffff9939c032b9e0 0000000000000000 000000007b1337ed 0000000000000004
[  514.356057] Call Trace:
[  514.356057]  [<ffffffffb84bb0aa>] dump_stack+0x4d/0x63
[  514.356057]  [<ffffffffb8262121>] __warn+0xc1/0xe0
[  514.356057]  [<ffffffffb8262248>] warn_slowpath_null+0x18/0x20
[  514.356057]  [<ffffffffb86cc330>] tcp_ack+0x1380/0x15b0
[  514.356057]  [<ffffffffb86cdedd>] tcp_rcv_established+0x10d/0x6f0
[  514.356057]  [<ffffffffb868be2b>] ? sk_filter_trim_cap+0x2b/0x260
[  514.356057]  [<ffffffffb86d8a50>] tcp_v4_do_rcv+0x140/0x200
[  514.356057]  [<ffffffffb86da272>] tcp_v4_rcv+0x912/0xa10
[  514.356057]  [<ffffffffc0641848>] ? tdb_rec_get+0x38/0x80 [tempesta_db]
[  514.356057]  [<ffffffffb86b31bd>] ip_local_deliver_finish+0x9d/0x1d0
[  514.356057]  [<ffffffffb86b34aa>] ip_local_deliver+0x6a/0xe0
[  514.356057]  [<ffffffffb86abd00>] ? nf_iterate+0x50/0x60
[  514.356057]  [<ffffffffb86b2e43>] ip_rcv_finish+0x113/0x3f0
[  514.356057]  [<ffffffffb86b37a6>] ip_rcv+0x286/0x3d0
[  514.356057]  [<ffffffffb86b2d30>] ? inet_del_offload+0x40/0x40
[  514.356057]  [<ffffffffb8672512>] __netif_receive_skb_core+0x4d2/0xa30
[  514.356057]  [<ffffffffb86d1e00>] ? tcp_write_xmit+0x300/0xef0
[  514.356057]  [<ffffffffb8672a83>] __netif_receive_skb+0x13/0x60
[  514.356057]  [<ffffffffb8673a68>] process_backlog+0x88/0x120
[  514.356057]  [<ffffffffb8673252>] net_rx_action+0x202/0x2f0
[  514.356057]  [<ffffffffb8266f2d>] __do_softirq+0xed/0x200
[  514.356057]  [<ffffffffb8282c50>] ? smpboot_update_cpumask_percpu_thread+0x130/0x130
[  514.356057]  [<ffffffffb8267057>] run_ksoftirqd+0x17/0x30
[  514.356057]  [<ffffffffb8282d55>] smpboot_thread_fn+0x105/0x180
[  514.356057]  [<ffffffffb827f459>] kthread+0xf9/0x110
[  514.356057]  [<ffffffffb827f360>] ? kthread_park+0x60/0x60
[  514.356057]  [<ffffffffb8772742>] ret_from_fork+0x22/0x30
[  514.697300] ---[ end trace 9e44a8cb407bdaa3 ]---
[  514.880590] BUG: unable to handle kernel NULL pointer dereference at 0000000000000035
[  514.884088] IP: [<ffffffffb86cc72a>] tcp_enter_loss+0x17a/0x370
[  514.884088] PGD 0 [  514.884088] 
[  514.884088] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[  514.884088] Modules linked in: tfw_sched_ratio(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) ntfs vfat msdos fat xfs fuse edac_mce_amd edac_core crct10dif_pclmul e1000 ppdev ghash_clmulni_intel psmouse sr_mod aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd cdrom battery parport_pc ac pcspkr sg button parport i2c_piix4 ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache serio_raw sd_mod [last unloaded: tempesta_tls]
[  514.884088] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G        W  O    4.9.35-kdump+ #17
[  514.884088] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  514.884088] task: ffff8b1c1ca742c0 task.stack: ffff9939c0328000
[  514.884088] RIP: 0010:[<ffffffffb86cc72a>]  [<ffffffffb86cc72a>] tcp_enter_loss+0x17a/0x370
[  514.884088] RSP: 0018:ffff9939c032bc98  EFLAGS: 00010207
[  514.884088] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffff8b1bc89db070
[  514.884088] RDX: 000000007b1337ed RSI: 0000000000000000 RDI: 0000000000000004
[  514.884088] RBP: ffff9939c032bcb0 R08: 0000000000000032 R09: 0000000000000100
[  514.884088] R10: 000000000007170d R11: 0000000000038b86 R12: ffffffffb8c88cc0
[  514.884088] R13: ffff8b1bc89daf00 R14: ffffffffb8c88cc0 R15: 000000000000000f
[  514.884088] FS:  0000000000000000(0000) GS:ffff8b1c3f800000(0000) knlGS:0000000000000000
[  514.884088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  514.884088] CR2: 0000000000000035 CR3: 000000005c1b5000 CR4: 00000000000406f0
[  514.884088] Stack:
[  514.884088]  ffff8b1bc89daf00 ffff8b1bc89db070 ffffffffb8c88cc0 ffff9939c032bcf0
[  514.884088]  ffffffffb86d565d ffffffff00000000 0000000000000000 ffff8b1bc89daf00
[  514.884088]  ffff8b1bc89daf88 ffffffffb86d5c90 ffff8b1bc89daf00 ffff9939c032bd18
[  514.884088] Call Trace:
[  514.884088]  [<ffffffffb86d565d>] tcp_retransmit_timer+0x33d/0x780
[  514.884088]  [<ffffffffb86d5c90>] ? tcp_write_timer_handler+0x1f0/0x1f0
[  514.884088]  [<ffffffffb86d5b37>] tcp_write_timer_handler+0x97/0x1f0
[  514.884088]  [<ffffffffb86d5cfb>] tcp_write_timer+0x6b/0x70
[  514.884088]  [<ffffffffb86d5c90>] ? tcp_write_timer_handler+0x1f0/0x1f0
[  514.884088]  [<ffffffffb82c4937>] call_timer_fn.isra.29+0x17/0x70
[  514.884088]  [<ffffffffb82c4b17>] run_timer_softirq+0x187/0x380
[  514.884088]  [<ffffffffb829b42c>] ? pick_next_task_fair+0x44c/0x510
[  514.884088]  [<ffffffffb8213718>] ? __switch_to+0x338/0x5a0
[  514.884088]  [<ffffffffb8772289>] ? _raw_spin_unlock_irq+0x9/0x10
[  514.884088]  [<ffffffffb8266f2d>] __do_softirq+0xed/0x200
[  514.884088]  [<ffffffffb8282c50>] ? smpboot_update_cpumask_percpu_thread+0x130/0x130
[  514.884088]  [<ffffffffb8267057>] run_ksoftirqd+0x17/0x30
[  514.884088]  [<ffffffffb8282d55>] smpboot_thread_fn+0x105/0x180
[  514.884088]  [<ffffffffb827f459>] kthread+0xf9/0x110
[  514.884088]  [<ffffffffb827f360>] ? kthread_park+0x60/0x60
[  514.884088]  [<ffffffffb8772742>] ret_from_fork+0x22/0x30
[  514.884088] Code: f6 49 3b 85 c8 02 00 00 74 68 83 e6 01 eb 1c 40 84 f6 75 20 83 e2 f9 88 50 35 48 8b 00 48 39 c8 74 50 49 3b 85 c8 02 00 00 74 47 <0f> b6 50 35 f6 c2 01 75 db f6 c2 04 0f 84 0a 01 00 00 89 d7 83 
[  514.884088] RIP  [<ffffffffb86cc72a>] tcp_enter_loss+0x17a/0x370
[  514.884088]  RSP <ffff9939c032bc98>
[  514.884088] CR2: 0000000000000035
[  514.884088] ---[ end trace 9e44a8cb407bdaa4 ]---
[  514.884088] Kernel panic - not syncing: Fatal exception in interrupt
[  514.884088] Kernel Offset: 0x37200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  514.884088] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
@vladtcvs
Copy link
Contributor Author

Reproducing: 10000 connections, regression.test_stress_pipeline.Pipeline.test_pipelined_requests

@krizhanovsky
Copy link
Contributor

Related patch which isn't in 4.9.x and corresponding LKML discussion. The patch

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index a0d768fd..4733379c 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1417,9 +1417,9 @@ EXPORT_SYMBOL(__pskb_copy_fclone);
 int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
                     gfp_t gfp_mask)
 {
-       int i;
+       int i, osize = skb_end_offset(skb);
        u8 *data;
-       int size = nhead + skb_end_offset(skb) + ntail;
+       int size = osize + nhead + ntail;
        long off;
 
        BUG_ON(nhead < 0);
@@ -1494,6 +1494,13 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
        skb->hdr_len  = 0;
        skb->nohdr    = 0;
        atomic_set(&skb_shinfo(skb)->dataref, 1);
+
+if (!skb->sk || skb->destructor == sock_edemux)
+       skb->truesize += size - osize;
+BUG_ON(skb->truesize - (skb_headlen(skb)
+                        ? SKB_DATA_ALIGN(sizeof(struct sk_buff))
+                        : SKB_TRUESIZE(skb_end_offset(skb))) < skb->len);
+
        return 0;
 
 nofrags:

removes the problem. Note the BUG_ON() statement which fails w/o the skb->truesize adjustment. However, with the patch, I hit warnings in sk_stream_kill_queues() WARN_ON(sk->sk_wmem_queued), just the same as in #8 and #60 .

The BUG_ON() fails in pskb_expand_head() with call trace

[  276.212013] Call Trace:
[  276.212013]  [<ffffffff88920530>] ? tcp_write_timer_handler+0x1f0/0x1f0
[  276.212013]  [<ffffffff8891e32f>] __tcp_retransmit_skb+0x56f/0x660
[  276.212013]  [<ffffffff88920002>] ? tcp_retransmit_timer+0x442/0x780
[  276.212013]  [<ffffffff88920530>] ? tcp_write_timer_handler+0x1f0/0x1f0
[  276.212013]  [<ffffffff8891e577>] tcp_send_loss_probe+0x157/0x200
[  276.212013]  [<ffffffff889203c5>] tcp_write_timer_handler+0x85/0x1f0
[  276.212013]  [<ffffffff889205e8>] tcp_write_timer+0xb8/0xc0
[  276.212013]  [<ffffffff88920530>] ? tcp_write_timer_handler+0x1f0/0x1f0
[  276.212013]  [<ffffffff884d37e5>] call_timer_fn+0x75/0x100
[  276.212013]  [<ffffffff884d3770>] ? process_timeout+0x10/0x10
[  276.212013]  [<ffffffff884d3bc5>] run_timer_softirq+0x195/0x3b0
[  276.212013]  [<ffffffffc054f890>] ? ss_tcp_data_ready+0xa0/0xa0 [tempesta_fw]
[  276.212013]  [<ffffffff8846726e>] __do_softirq+0x11e/0x250
[  276.212013]  [<ffffffff88487de0>] ? sort_range+0x20/0x20
[  276.212013]  [<ffffffff884673ba>] run_ksoftirqd+0x1a/0x40

, i.e. the skb belongs to a socket (skb->destructor = sock_edemux is set by skb_set_owner_w() in ip_finish_output2()) and we retransmit part of the unacknowledged data. So the patch doesn't adjust sk->sk_wmem_queued for the new skb->truesize and that seems wrong.

@krizhanovsky
Copy link
Contributor

The problem is fixed in https://github.com/tempesta-tech/linux-4.9.35-tfw/pull/5 , but I still get test errors (while there are no any oopses or crashes):

test_hm_del_add_srvs (reconf.test_stress_health_monitor.HealthMonitorChangedAutoCustomSg) ... ERROR
test_502_resp_fault (regression.test_stress_pipeline.PipelineFaultInjection) ... ERROR
Exception in stoping Tempesta: Remote error: Cannot remove stap from Tempesta., stdout = , stderr = rmmod: ERROR: Module stap_tempesta is not currently loaded

ERROR
test (mixed_requests.test_mixed.BigPostRequests) ... FAIL
test (mixed_requests.test_mixed.EmptyPostRequests) ... FAIL
test (mixed_requests.test_mixed.GetPostRequests) ... FAIL
test (mixed_requests.test_mixed.GetRequests) ... FAIL
test (mixed_requests.test_mixed.HeadGetRequests) ... WARNING! Socket errors on wrk. Too many concurrent connections?
FAIL
test (mixed_requests.test_mixed.RarelyUsedRequests) ... WARNING! Socket errors on wrk. Too many concurrent connections?
FAIL
test (mixed_requests.test_mixed.RealRequest) ... FAIL
test (mixed_requests.test_mixed.RealRequest2) ... FAIL
test (mixed_requests.test_mixed.RealRequestPipeline) ... FAIL
test (mixed_requests.test_mixed.SmallPostRequests) ... FAIL
test_purge (cache.test_purge.TestPurge) ... FAIL
tcp_connection.test_connection_close (unittest.loader.ModuleImportFailure) ... ERROR
======================================================================
ERROR: test_hm_del_add_srvs (reconf.test_stress_health_monitor.HealthMonitorChangedAutoCustomSg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/reconf/test_stress_health_monitor.py", line 48, in test_hm_del_add_srvs
    self.configure_srvs_del_add)
  File "/root/tempesta/tempesta_fw/t/functional/reconf/reconf_stress.py", line 82, in stress_reconfig_generic
    control.servers_start(self.servers)
  File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 455, in servers_start
    pool.map(Nginx.start, servers)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
Exception: Remote error: Can't start Nginx on 127.0.0.1:8000, stdout = , stderr = None

======================================================================
ERROR: test_502_resp_fault (regression.test_stress_pipeline.PipelineFaultInjection)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/regression/test_stress_pipeline.py", line 78, in test_502_resp_fault
    self.generic_test_routine('cache 0;\n')
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 164, in generic_test_routine
    self.tempesta.start()
  File "/root/tempesta/tempesta_fw/t/functional/helpers/stateful.py", line 29, in start
    self.run_start()
  File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 363, in run_start
    self.inject()
  File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 348, in inject
    ('inject', 'into')))
  File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 80, in run_cmd
    (err_msg, stdout, stderr))
Exception: Remote error: Cannot inject stap into Tempesta., stdout = , stderr = semantic error: while resolving probe point: identifier 'module' at /root/tempesta/tempesta_fw/t/functional/systemtap/resp_alloc_err.stp:31:7
        source: probe module("tempesta_fw").function("tfw_http_msg_alloc_err_resp").return
                      ^

semantic error: no match (similar functions: tfw_http_msg_alloc_light, tfw_http_msg_alloc, __tfw_http_msg_alloc, tfw_http_msg_grow_hdr_tbl, tfw_http_msg_process)

Pass 2: analysis failed.  [man error::pass2]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.


======================================================================
ERROR: test_502_resp_fault (regression.test_stress_pipeline.PipelineFaultInjection)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 81, in tearDown
    raise Exception("Error during stopping tempesta")
Exception: Error during stopping tempesta

======================================================================
ERROR: tcp_connection.test_connection_close (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: tcp_connection.test_connection_close
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/root/tempesta/tempesta_fw/t/functional/tcp_connection/test_connection_close.py", line 7, in <module>
    from helpers import analyzer, deproxy, chains
  File "/root/tempesta/tempesta_fw/t/functional/helpers/analyzer.py", line 7, in <module>
    from scapy.all import *
ImportError: No module named scapy.all


======================================================================
FAIL: test_tls (tls.test_tls_stress.StressTls)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/tls/test_tls_stress.py", line 38, in test_tls
    self.generic_test_routine(config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.BigPostRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.EmptyPostRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.GetPostRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.GetRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.HeadGetRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.RarelyUsedRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.RealRequest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.RealRequest2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.RealRequestPipeline)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test (mixed_requests.test_mixed.SmallPostRequests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/mixed_requests/mixed_test.py", line 17, in test
    self.generic_test_routine(self.config)
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 172, in generic_test_routine
    self.assert_clients()
  File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 114, in assert_clients
    msg='Client received non 2xx or 3xx responses')
AssertionError: Client received non 2xx or 3xx responses

======================================================================
FAIL: test_purge (cache.test_purge.TestPurge)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/tempesta/tempesta_fw/t/functional/cache/test_purge.py", line 40, in test_purge
    self.generic_test_routine(self.config, self.chains())
  File "/root/tempesta/tempesta_fw/t/functional/testers/functional.py", line 152, in generic_test_routine
    self.tester.run()
  File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 733, in run
    self.check_expectations()
  File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 744, in check_expectations
    % (message, recieved.msg, expected.msg))
AssertionError: Received message (response) does not suit expected one!

	Receieved:
<<<<<|
HTTP/1.1 200 OK
Date: Wed, 14 Mar 2018 17:47:53 GMT
Content-Length: 0
Server: Tempesta FW/0.5.0
Connection: keep-alive

|>>>>>
	Expected:
<<<<<|
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 0
Date: Wed, 14 Mar 2018 17:47:50 GMT

|>>>>>


----------------------------------------------------------------------
Ran 227 tests in 2129.650s

FAILED (failures=12, errors=4, skipped=1, expected failures=4)

@krizhanovsky
Copy link
Contributor

Some of the tests fixed, some of them aren't, see #956

openvz-integrator pushed a commit to OpenVZ/vzkernel that referenced this issue May 19, 2020
Slava Shwartsman reported a warning in skb_try_coalesce(), when we
detect skb->truesize is completely wrong.

In his case, issue came from IPv6 reassembly coping with malicious
datagrams, that forced various pskb_may_pull() to reallocate a bigger
skb->head than the one allocated by NIC driver before entering GRO
layer.

Current code does not change skb->truesize, leaving this burden to
callers if they care enough.

Blindly changing skb->truesize in pskb_expand_head() is not
easy, as some producers might track skb->truesize, for example
in xmit path for back pressure feedback (sk->sk_wmem_alloc)

We can detect the cases where it should be safe to change
skb->truesize :

1) skb is not attached to a socket.
2) If it is attached to a socket, destructor is sock_edemux()

My audit gave only two callers doing their own skb->truesize
manipulation.

I had to remove skb parameter in sock_edemux macro when
CONFIG_INET is not set to avoid a compile error.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Slava Shwartsman <slavash@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

https://jira.sw.ru/browse/PSBM-102386
tempesta-tech/tempesta#926

(cherry picked from commit 158f323)
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>

Backport notes: ieee80211_data_from_8023() hunk has been dropped,
seems the functional has been removed by RedHat, anyway, not used since 2012.
openvz-integrator pushed a commit to OpenVZ/vzkernel that referenced this issue Jun 9, 2020
Slava Shwartsman reported a warning in skb_try_coalesce(), when we
detect skb->truesize is completely wrong.

In his case, issue came from IPv6 reassembly coping with malicious
datagrams, that forced various pskb_may_pull() to reallocate a bigger
skb->head than the one allocated by NIC driver before entering GRO
layer.

Current code does not change skb->truesize, leaving this burden to
callers if they care enough.

Blindly changing skb->truesize in pskb_expand_head() is not
easy, as some producers might track skb->truesize, for example
in xmit path for back pressure feedback (sk->sk_wmem_alloc)

We can detect the cases where it should be safe to change
skb->truesize :

1) skb is not attached to a socket.
2) If it is attached to a socket, destructor is sock_edemux()

My audit gave only two callers doing their own skb->truesize
manipulation.

I had to remove skb parameter in sock_edemux macro when
CONFIG_INET is not set to avoid a compile error.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Slava Shwartsman <slavash@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

https://jira.sw.ru/browse/PSBM-102386
tempesta-tech/tempesta#926

(cherry picked from commit 158f323)
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>

Backport notes: ieee80211_data_from_8023() hunk has been dropped,
seems the functional has been removed by RedHat, anyway, not used since 2012.
openvz-integrator pushed a commit to OpenVZ/vzkernel that referenced this issue Sep 23, 2023
Slava Shwartsman reported a warning in skb_try_coalesce(), when we
detect skb->truesize is completely wrong.

In his case, issue came from IPv6 reassembly coping with malicious
datagrams, that forced various pskb_may_pull() to reallocate a bigger
skb->head than the one allocated by NIC driver before entering GRO
layer.

Current code does not change skb->truesize, leaving this burden to
callers if they care enough.

Blindly changing skb->truesize in pskb_expand_head() is not
easy, as some producers might track skb->truesize, for example
in xmit path for back pressure feedback (sk->sk_wmem_alloc)

We can detect the cases where it should be safe to change
skb->truesize :

1) skb is not attached to a socket.
2) If it is attached to a socket, destructor is sock_edemux()

My audit gave only two callers doing their own skb->truesize
manipulation.

I had to remove skb parameter in sock_edemux macro when
CONFIG_INET is not set to avoid a compile error.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Slava Shwartsman <slavash@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

https://jira.sw.ru/browse/PSBM-102386
tempesta-tech/tempesta#926

(cherry picked from commit 158f323)
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>

Backport notes: ieee80211_data_from_8023() hunk has been dropped,
seems the functional has been removed by RedHat, anyway, not used since 2012.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants