Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stack-out-of-bounds in tdb_htrie_create_rec() #690

Closed
keshonok opened this issue Feb 28, 2017 · 2 comments · Fixed by #745
Closed

stack-out-of-bounds in tdb_htrie_create_rec() #690

keshonok opened this issue Feb 28, 2017 · 2 comments · Fixed by #745
Assignees
@vankoven
Labels
bug duplicate
Milestone
0.5 alpha

Comments

@keshonok
Copy link
Contributor

The setup:

  • Tempesta's current master at 56b0ecb;
  • Tempesta kernel 4.8.15 built with KASAN support;
  • Both Tempesta and Nginx run in the same VM;
  • Simplest configuration with cache explicitly turned on:
cache 2;
cache_fulfill * * *;
server 127.0.0.1:8080
  • A simple wget -d -S http://192.168.10.230/ to the VM with Tempesta and Nginx.

The kernel (KASAN) reports the following:

[ 3379.438729] ==================================================================
[ 3379.439706] BUG: KASAN: stack-out-of-bounds in tdb_htrie_create_rec+0x1c6/0x260 [tempesta_db] at addr ffff8801f6e06f88
[ 3379.439706] Read of size 4016 by task swapper/0/0
[ 3379.439706] page:ffffea0007db8180 count:1 mapcount:0 mapping:          (null) index:0x0
[ 3379.439706] flags: 0x1000000000000400(reserved)
[ 3379.439706] page dumped because: kasan: bad access detected
[ 3379.439706] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G    B      O    4.8.15-ab+ #2
[ 3379.439706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 3379.439706]  ffff8801f6e06f88 ffff8801f6e06a68 ffffffff8199f263 ffff8801f6e06af8
[ 3379.439706]  ffff8801f6e06f88 ffff8801f6e06ae8 ffffffff8151c33e ffffffff84492c20
[ 3379.439706]  ffffffffa03a57a0 0000000000000001 0000000000000292 ffff8801f6e06aa8
[ 3379.439706] Call Trace:
[ 3379.439706]  <IRQ>  [<ffffffff8199f263>] dump_stack+0x67/0x94
[ 3379.439706]  [<ffffffff8151c33e>] kasan_report_error+0x4ae/0x4e0
[ 3379.439706]  [<ffffffff812c85f5>] ? is_module_address+0x15/0x30
[ 3379.439706]  [<ffffffff8120eb53>] ? static_obj+0x33/0x50
[ 3379.439706]  [<ffffffff8151c7c9>] kasan_report+0x39/0x40
[ 3379.439706]  [<ffffffffa039a900>] ? tdb_alloc_data+0x1d0/0x380 [tempesta_db]
[ 3379.439706]  [<ffffffffa039a0a6>] ? tdb_htrie_create_rec+0x1c6/0x260 [tempesta_db]
[ 3379.439706]  [<ffffffff8151b05e>] check_memory_region+0x13e/0x1a0
[ 3379.439706]  [<ffffffff8151b523>] memcpy+0x23/0x50
[ 3379.439706]  [<ffffffffa039a0a6>] tdb_htrie_create_rec+0x1c6/0x260 [tempesta_db]
[ 3379.439706]  [<ffffffffa039c9af>] tdb_htrie_insert+0x5ef/0x1050 [tempesta_db]
[ 3379.439706]  [<ffffffff81a32630>] ? perf_trace_swiotlb_bounced+0x6c0/0x6c0
[ 3379.439706]  [<ffffffffa0215b2b>] ? e1000_xmit_frame+0xf1b/0x57c0 [e1000]
[ 3379.439706]  [<ffffffff81f87f25>] ? consume_skb+0xc5/0x320
[ 3379.439706]  [<ffffffffa039c3c0>] ? tdb_htrie_extend_rec+0x410/0x410 [tempesta_db]
[ 3379.439706]  [<ffffffff82047bdf>] ? sch_direct_xmit+0x11f/0x590
[ 3379.439706]  [<ffffffffa03a1324>] tdb_entry_create+0x44/0xa0 [tempesta_db]
[ 3379.439706]  [<ffffffffa2b208f7>] __cache_add_node+0x437/0x1bc0 [tempesta_fw]
[ 3379.439706]  [<ffffffff81fcb5b0>] ? __dev_queue_xmit+0x840/0x1ed0
[ 3379.439706]  [<ffffffff81fcb5dc>] ? __dev_queue_xmit+0x86c/0x1ed0
[ 3379.439706]  [<ffffffff8121699d>] ? trace_hardirqs_on+0xd/0x10
[ 3379.439706]  [<ffffffffa2b204c0>] ? tfw_cache_ipi+0x20/0x20 [tempesta_fw]
[ 3379.439706]  [<ffffffff81fcb605>] ? __dev_queue_xmit+0x895/0x1ed0
[ 3379.439706]  [<ffffffff81fccc50>] ? dev_queue_xmit+0x10/0x20
[ 3379.439706]  [<ffffffff8151b296>] ? kasan_unpoison_shadow+0x36/0x50
[ 3379.439706]  [<ffffffff8151b588>] ? kasan_alloc_pages+0x38/0x40
[ 3379.439706]  [<ffffffff8142da65>] ? get_page_from_freelist+0x4f5/0x1d70
[ 3379.439706]  [<ffffffff81fcad70>] ? netdev_pick_tx+0x290/0x290
[ 3379.439706]  [<ffffffffa2b319ee>] ? tfw_http_req_cache_cb+0x28e/0x3b0 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b2fef0>] ? tfw_http_adjust_resp+0x3e0/0x3e0 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b30bd0>] ? tfw_http_resp_cache+0x290/0x290 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b223e6>] tfw_cache_add+0x366/0x410 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b30bd0>] ? tfw_http_resp_cache+0x290/0x290 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b25111>] tfw_cache_process+0x141/0x800 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b24fd0>] ? tfw_cache_msg_cacheable+0x70/0x70 [tempesta_fw]
[ 3379.439706]  [<ffffffff822e3c87>] ? _raw_spin_unlock+0x27/0x40
[ 3379.439706]  [<ffffffffa2b30b19>] tfw_http_resp_cache+0x1d9/0x290 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b30940>] ? tfw_http_msg_create_sibling.isra.9+0x1e0/0x1e0 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b31bb3>] ? tfw_http_resp_gfsm+0xa3/0x150 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b32464>] tfw_http_msg_process+0x804/0x1300 [tempesta_fw]
[ 3379.439706]  [<ffffffff81f7acb7>] ? __pg_skb_alloc+0x6f7/0x950
[ 3379.439706]  [<ffffffff8121699d>] ? trace_hardirqs_on+0xd/0x10
[ 3379.439706]  [<ffffffffa2b31c60>] ? tfw_http_resp_gfsm+0x150/0x150 [tempesta_fw]
[ 3379.439706]  [<ffffffff8151b4a1>] ? memset+0x31/0x40
[ 3379.439706]  [<ffffffff81f7b2a7>] ? __alloc_skb+0x397/0x6d0
[ 3379.439706]  [<ffffffffa2b7a332>] ? ss_skb_queue_coalesce_tail+0x8b2/0x1360 [tempesta_fw]
[ 3379.439706]  [<ffffffff81f7ea92>] ? skb_release_all+0x42/0x50
[ 3379.439706]  [<ffffffffa2b31c60>] ? tfw_http_resp_gfsm+0x150/0x150 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b2dcce>] __gfsm_fsm_exec+0x14e/0x2c0 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b7b67b>] ? ss_skb_unroll+0x89b/0xf40 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b2e461>] tfw_gfsm_dispatch+0x41/0x70 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b2c68d>] tfw_connection_recv+0x1d/0x20 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b71d43>] ss_tcp_process_data+0x563/0xf00 [tempesta_fw]
[ 3379.439706]  [<ffffffffa2b717e0>] ? ss_synchronize+0x400/0x400 [tempesta_fw]
[ 3379.439706]  [<ffffffff820f6a6e>] ? tcp_event_data_recv+0x53e/0xbd0
[ 3379.439706]  [<ffffffffa2b7384c>] ss_tcp_data_ready+0xec/0x180 [tempesta_fw]
[ 3379.439706]  [<ffffffff82116b71>] tcp_data_queue+0x13c1/0x4ed0
[ 3379.439706]  [<ffffffff8210f130>] ? tcp_fastretrans_alert+0x2f50/0x2f50
[ 3379.439706]  [<ffffffff821157b0>] ? tcp_fin+0x910/0x910
[ 3379.439706]  [<ffffffff820f9320>] ? tcp_xmit_recovery.part.54+0x100/0x100
[ 3379.439706]  [<ffffffff818a74c4>] ? tempesta_sock_tcp_rcv+0xc4/0x1d0
[ 3379.439706]  [<ffffffff81216ec0>] ? debug_check_no_locks_freed+0x280/0x280
[ 3379.439706]  [<ffffffff8211ad97>] tcp_rcv_established+0x717/0x2de0
[ 3379.439706]  [<ffffffff8200b932>] ? sk_filter_trim_cap+0x2a2/0x700
[ 3379.439706]  [<ffffffff8211a680>] ? tcp_data_queue+0x4ed0/0x4ed0
[ 3379.439706]  [<ffffffff82154480>] ? tcp_v4_rcv+0x1ee0/0x3420
[ 3379.439706]  [<ffffffff8214fdf9>] tcp_v4_do_rcv+0x529/0x8c0
[ 3379.439706]  [<ffffffff821546de>] tcp_v4_rcv+0x213e/0x3420
[ 3379.439706]  [<ffffffffa2b2b285>] ? tfw_classify_ipv4+0x5/0x190 [tempesta_fw]
[ 3379.439706]  [<ffffffff820a3a0b>] ip_local_deliver_finish+0x2cb/0x9b0
[ 3379.439706]  [<ffffffff820a386a>] ? ip_local_deliver_finish+0x12a/0x9b0
[ 3379.439706]  [<ffffffff820a484d>] ip_local_deliver+0x24d/0x330
[ 3379.439706]  [<ffffffff820a4600>] ? ip_call_ra_chain+0x510/0x510
[ 3379.439706]  [<ffffffff8207d2a6>] ? nf_hook_slow+0x186/0x2a0
[ 3379.439706]  [<ffffffff8207d2c5>] ? nf_hook_slow+0x1a5/0x2a0
[ 3379.439706]  [<ffffffff8207d125>] ? nf_hook_slow+0x5/0x2a0
[ 3379.439706]  [<ffffffff820a2469>] ip_rcv_finish+0x599/0x1870
[ 3379.439706]  [<ffffffff820a51cb>] ip_rcv+0x89b/0x11d0
[ 3379.439706]  [<ffffffff8127a4a0>] ? msleep+0xc0/0xc0
[ 3379.439706]  [<ffffffff820a4930>] ? ip_local_deliver+0x330/0x330
[ 3379.439706]  [<ffffffff8127a402>] ? msleep+0x22/0xc0
[ 3379.439706]  [<ffffffff820a1ed0>] ? inet_del_offload+0x40/0x40
[ 3379.439706]  [<ffffffff81f608c8>] ? sk_reset_timer+0x18/0x30
[ 3379.439706]  [<ffffffff820a4930>] ? ip_local_deliver+0x330/0x330
[ 3379.439706]  [<ffffffff81fbdea0>] __netif_receive_skb_core+0x1690/0x2bc0
[ 3379.439706]  [<ffffffff81216ec0>] ? debug_check_no_locks_freed+0x280/0x280
[ 3379.439706]  [<ffffffff81216ec0>] ? debug_check_no_locks_freed+0x280/0x280
[ 3379.439706]  [<ffffffff81fbc810>] ? net_tx_action+0x9d0/0x9d0
[ 3379.439706]  [<ffffffff8121699d>] ? trace_hardirqs_on+0xd/0x10
[ 3379.439706]  [<ffffffff81fbf777>] ? process_backlog+0x217/0x650
[ 3379.439706]  [<ffffffff81fbf42a>] __netif_receive_skb+0x5a/0x190
[ 3379.439706]  [<ffffffff81fbf638>] process_backlog+0xd8/0x650
[ 3379.439706]  [<ffffffff81fbf777>] ? process_backlog+0x217/0x650
[ 3379.439706]  [<ffffffff81fc46e5>] net_rx_action+0x655/0xde0
[ 3379.439706]  [<ffffffff81fc4090>] ? sk_busy_loop+0xb90/0xb90
[ 3379.439706]  [<ffffffff811af802>] ? sched_clock_tick+0x42/0xe0
[ 3379.439706]  [<ffffffff8112932c>] ? irq_exit+0x15c/0x190
[ 3379.439706]  [<ffffffff822e72ac>] __do_softirq+0x22c/0x99f
[ 3379.439706]  [<ffffffff8112970c>] ? __raise_softirq_irqoff+0x12c/0x220
[ 3379.439706]  [<ffffffff8112932c>] irq_exit+0x15c/0x190
[ 3379.439706]  [<ffffffff810b74c0>] smp_call_function_single_interrupt+0x70/0x90
[ 3379.439706]  [<ffffffff822e6249>] call_function_single_interrupt+0x89/0x90
[ 3379.439706]  <EOI>  [<ffffffff81073303>] ? default_idle+0x53/0x3b0
[ 3379.439706]  [<ffffffff81074a4f>] arch_cpu_idle+0xf/0x20
[ 3379.439706]  [<ffffffff811fb18d>] default_idle_call+0x4d/0x60
[ 3379.439706]  [<ffffffff811fb6a2>] cpu_startup_entry+0x502/0x710
[ 3379.439706]  [<ffffffff822d062b>] rest_init+0x15b/0x170
[ 3379.439706]  [<ffffffff82dff5e8>] start_kernel+0x5ea/0x612
[ 3379.439706]  [<ffffffff82dfeffe>] ? thread_stack_cache_init+0x6/0x6
[ 3379.439706]  [<ffffffff82dfe120>] ? early_idt_handler_array+0x120/0x120
[ 3379.439706]  [<ffffffff82dfe29a>] x86_64_start_reservations+0x2a/0x2c
[ 3379.439706]  [<ffffffff82dfe3df>] x86_64_start_kernel+0x143/0x152
[ 3379.439706] Memory state around the buggy address:
[ 3379.439706]  ffff8801f6e06e80: f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f2 f2 f2 f2 00
[ 3379.439706]  ffff8801f6e06f00: f4 f4 f4 f2 f2 f2 f2 00 00 00 00 f2 f2 f2 f2 00
[ 3379.439706] >ffff8801f6e06f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f4 f3
[ 3379.439706]                                                              ^
[ 3379.439706]  ffff8801f6e07000: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 3379.439706]  ffff8801f6e07080: 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00 f3
[ 3379.439706] ==================================================================
@keshonok keshonok added the bug label Feb 28, 2017
@keshonok keshonok added this to the 0.5.0 Web Server milestone Feb 28, 2017
@krizhanovsky krizhanovsky self-assigned this Feb 28, 2017
@krizhanovsky
Copy link
Contributor

Yeah, I saw KASAN reports previously in TDB code. I have TODO to debug them during debugging #516.

@keshonok keshonok mentioned this issue Jun 5, 2017
Closed
@krizhanovsky krizhanovsky assigned vankoven and unassigned krizhanovsky Jun 5, 2017
@krizhanovsky
Copy link
Contributor

Duplicate of #743

vankoven added a commit that referenced this issue Jun 8, 2017
@vankoven
db: add api to create tdb entry without unwanted copying
a90e2d3 
fix #690
@vankoven vankoven mentioned this issue Jun 8, 2017
Merged
vankoven added a commit that referenced this issue Jun 8, 2017
@vankoven
db: add api to create tdb entry without unwanted copying
d9f5442 
fix #690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vankoven vankoven

Labels
bug duplicate
Projects
None yet
Milestone
0.5 alpha
Development

Successfully merging a pull request may close this issue.

Fix serving from cache responses with empty bodies tempesta-tech/tempesta
3 participants
@krizhanovsky @vankoven @keshonok