Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: unable to handle page fault for address. Crash in tfw_http_msg_free() #2285

Open
RomanBelozerov opened this issue Nov 19, 2024 · 1 comment · Fixed by #2288
Open

BUG: unable to handle page fault for address. Crash in tfw_http_msg_free() #2285

RomanBelozerov opened this issue Nov 19, 2024 · 1 comment · Fixed by #2288
Assignees
@EvgeniiMekhanik
Labels
bug h2
Milestone
0.8 - Beta

Comments

@RomanBelozerov
Copy link
Contributor

To Reproduce
http2_general.test_h2_stream_states.TestHalfClosedStreamStateUnexpectedFrames.test_reset_frame_in_half_closed_state with local setup on CI

Configuration file

            listen 443 proto=h2;
            srv_group default {
                server ${server_ip}:8000;
            }
            vhost good {
                frang_limits {http_strict_host_checking false;}
                proxy_pass default;
            }
            tls_certificate ${tempesta_workdir}/tempesta.crt;
            tls_certificate_key ${tempesta_workdir}/tempesta.key;
            tls_match_any_server_name;
            http_max_header_list_size 134217728; #128 KB
            
            block_action attack reply;
            block_action error reply;
            http_chain {
                host == "bad.com"   -> block;
                                    -> good;
            }

Version or commit hash
Tempesta - 4f68ea5
tempesta-test - a5918febfd207ed6f189ed67d97f4e3273d2d96b
kernel - 5.10.35.tfw-cf95567

Stacktrace or debug log

Nov 19 05:32:06 192.168.50.95 [27889.967991] BUG: unable to handle page fault for address: 0000001100000009
Nov 19 05:32:06 192.168.50.95 [27889.968898] #PF: supervisor read access in kernel mode
Nov 19 05:32:06 192.168.50.95 [27889.969750] #PF: error_code(0x0000) - not-present page
Nov 19 05:32:06 192.168.50.95 [27889.970636] PGD 0 P4D 0 
Nov 19 05:32:06 192.168.50.95 [27889.971532] Oops: 0000 [#1] SMP NOPTI
Nov 19 05:32:06 192.168.50.95 [27889.972434] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           OE     5.10.35.tfw-cf95567 #1
Nov 19 05:32:06 192.168.50.95 [27889.973265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014
Nov 19 05:32:06 192.168.50.95 [27889.975061] RIP: 0010:tfw_http_msg_free+0x38/0xe0 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.975904] Code: 55 48 89 e5 53 48 8b 47 38 48 89 fb 48 85 c0 74 10 48 c7 40 38 00 00 00 00 48 c7 47 38 00 00 00 00 48 8b 7b 10 48 85 ff 74 44 <48> 8b 57 08 48 8b 07 48 85 d2 74 7c 48 85 c0 74 77 48 39 f8 74 5f
Nov 19 05:32:06 192.168.50.95 [27889.977730] RSP: 0018:ffffa568c010cb08 EFLAGS: 00010202
Nov 19 05:32:06 192.168.50.95 [27889.978650] RAX: ffff91399039f1f8 RBX: ffff91398fbd0020 RCX: 0000000000000000
Nov 19 05:32:06 192.168.50.95 [27889.979581] RDX: 00000000ffffffff RSI: ffff9139827e1a40 RDI: 0000001100000001
Nov 19 05:32:06 192.168.50.95 [27889.980499] RBP: ffffa568c010cb10 R08: 0000000000000000 R09: 0000000000000000
Nov 19 05:32:06 192.168.50.95 [27889.981384] R10: 0000000000000000 R11: 0000000000000000 R12: ffff91398fbd0020
Nov 19 05:32:06 192.168.50.95 [27889.982291] R13: ffff9138d4489c40 R14: ffff913988eaf5a0 R15: ffff9138c36c2688
Nov 19 05:32:06 192.168.50.95 [27889.983201] FS:  0000000000000000(0000) GS:ffff913ab7d00000(0000) knlGS:0000000000000000
Nov 19 05:32:06 192.168.50.95 [27889.984082] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 05:32:06 192.168.50.95 [27889.984977] CR2: 0000001100000009 CR3: 000000005800a006 CR4: 0000000000770ee0
Nov 19 05:32:06 192.168.50.95 [27889.985899] PKRU: 55555554
Nov 19 05:32:06 192.168.50.95 [27889.986789] Call Trace:
Nov 19 05:32:06 192.168.50.95 [27889.987653]  <IRQ>
Nov 19 05:32:06 192.168.50.95 [27889.988536]  tfw_http_conn_msg_free+0x6d/0xa0 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.989397]  tfw_h2_stream_unlink_lock+0xad/0xc0 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.990296]  tfw_h2_conn_streams_cleanup+0x5a/0x80 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.991195]  tfw_http_conn_drop+0x16a/0x2c0 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.992089]  tfw_tls_conn_drop+0x1e/0x30 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.992993]  tfw_connection_drop+0x25/0x30 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.993893]  tfw_sock_clnt_drop+0x101/0x170 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.994773]  ss_conn_drop_guard_exit+0x34/0x60 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27889.995655]  tcp_done+0xc9/0x140
Nov 19 05:32:06 192.168.50.95 [27889.996515]  tcp_rcv_state_process+0xd9f/0x1120
Nov 19 05:32:06 192.168.50.95 [27889.997349]  ? security_sock_rcv_skb+0x2f/0x50
Nov 19 05:32:06 192.168.50.95 [27889.998168]  ? sk_filter_trim_cap+0xde/0x240
Nov 19 05:32:06 192.168.50.95 [27889.998983]  ? tcp_v4_inbound_md5_hash+0x4e/0x160
Nov 19 05:32:06 192.168.50.95 [27889.999758]  tcp_v4_do_rcv+0xc4/0x200
Nov 19 05:32:06 192.168.50.95 [27890.000555]  tcp_v4_rcv+0xcfd/0xe10
Nov 19 05:32:06 192.168.50.95 [27890.001339]  ip_protocol_deliver_rcu+0x30/0x1b0
Nov 19 05:32:06 192.168.50.95 [27890.002123]  ip_local_deliver_finish+0x48/0x60
Nov 19 05:32:06 192.168.50.95 [27890.002906]  ip_local_deliver+0xfa/0x110
Nov 19 05:32:06 192.168.50.95 [27890.003683]  ? ip_protocol_deliver_rcu+0x1b0/0x1b0
Nov 19 05:32:06 192.168.50.95 [27890.004435]  ip_rcv_finish+0x87/0xa0
Nov 19 05:32:06 192.168.50.95 [27890.005177]  ip_rcv+0xcc/0xe0
Nov 19 05:32:06 192.168.50.95 [27890.005912]  ? ip_rcv_finish_core.isra.0+0x420/0x420
Nov 19 05:32:06 192.168.50.95 [27890.006652]  __netif_receive_skb_one_core+0x88/0xa0
Nov 19 05:32:06 192.168.50.95 [27890.007353]  __netif_receive_skb+0x18/0x60
Nov 19 05:32:06 192.168.50.95 [27890.008068]  process_backlog+0xa9/0x160
Nov 19 05:32:06 192.168.50.95 [27890.008764]  net_rx_action+0x13e/0x390
Nov 19 05:32:06 192.168.50.95 [27890.009460]  __do_softirq+0xd9/0x291
Nov 19 05:32:06 192.168.50.95 [27890.010137]  asm_call_irq_on_stack+0x12/0x20
Nov 19 05:32:06 192.168.50.95 [27890.010810]  </IRQ>
Nov 19 05:32:06 192.168.50.95 [27890.011474]  do_softirq_own_stack+0x3d/0x50
Nov 19 05:32:06 192.168.50.95 [27890.012157]  irq_exit_rcu+0xa4/0xb0
Nov 19 05:32:06 192.168.50.95 [27890.012800]  sysvec_call_function_single+0x3d/0x90
Nov 19 05:32:06 192.168.50.95 [27890.013441]  asm_sysvec_call_function_single+0x12/0x20
Nov 19 05:32:06 192.168.50.95 [27890.014075] RIP: 0010:native_safe_halt+0xe/0x10
Nov 19 05:32:06 192.168.50.95 [27890.014680] Code: 7b ff ff ff eb bd cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 9e 47 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d b6 9e 47 00 fb f4 <c3> cc 0f 1f 44 00 00 55 48 89 e5 53 65 8b 15 3f ed a7 7d 0f 1f 44
Nov 19 05:32:06 192.168.50.95 [27890.015937] RSP: 0018:ffffa568c0093e90 EFLAGS: 00000212
Nov 19 05:32:06 192.168.50.95 [27890.016566] RAX: ffffffff82592690 RBX: 0000000000000002 RCX: ffff913ab7d2cdc0
Nov 19 05:32:06 192.168.50.95 [27890.017206] RDX: 00000000013ccece RSI: 0000195d9a762d0b RDI: 0000000000000082
Nov 19 05:32:06 192.168.50.95 [27890.017848] RBP: ffffa568c0093e98 R08: 000000cd42e4dffb R09: 0000195dc09bcd0b
Nov 19 05:32:06 192.168.50.95 [27890.018443] R10: 000000000000000f R11: 0000000000000000 R12: ffff9139802db000
Nov 19 05:32:06 192.168.50.95 [27890.019089] R13: ffff9139802db000 R14: 0000000000000000 R15: 0000000000000000
Nov 19 05:32:06 192.168.50.95 [27890.019725]  ? __cpuidle_text_start+0x8/0x8
Nov 19 05:32:06 192.168.50.95 [27890.020350]  ? default_idle+0xe/0x20
Nov 19 05:32:06 192.168.50.95 [27890.020977]  arch_cpu_idle+0x15/0x20
Nov 19 05:32:06 192.168.50.95 [27890.021586]  default_idle_call+0x38/0xc0
Nov 19 05:32:06 192.168.50.95 [27890.022154]  do_idle+0x1f8/0x260
Nov 19 05:32:06 192.168.50.95 [27890.022772]  ? do_idle+0x196/0x260
Nov 19 05:32:06 192.168.50.95 [27890.023360]  cpu_startup_entry+0x20/0x30
Nov 19 05:32:06 192.168.50.95 [27890.023940]  start_secondary+0x111/0x150
Nov 19 05:32:06 192.168.50.95 [27890.024508]  secondary_startup_64_no_verify+0xb0/0xbb
Nov 19 05:32:06 192.168.50.95 [27890.025104] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) tcp_diag inet_diag xt_nat xt_tcpudp veth sha256_ssse3 sha512_ssse3 tls vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock xt_conntrack xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo nft_masq xt_addrtype nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc nf_tables nfnetlink overlay sch_fq_codel binfmt_misc kvm_amd ccp kvm input_leds joydev mac_hid serio_raw qemu_fw_cfg netconsole dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ramoops reed_solomon efi_pstore msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd bochs_drm glue_helper drm_vram_helper drm_ttm_helper ttm
Nov 19 05:32:06 192.168.50.95 [27890.025167]  drm_kms_helper syscopyarea sysfillrect sysimgblt psmouse virtio_net fb_sys_fops cec net_failover failover drm virtio_scsi i2c_piix4 pata_acpi floppy [last unloaded: tempesta_lib]
Nov 19 05:32:06 192.168.50.95 [27890.032293] CR2: 0000001100000009
Nov 19 05:32:06 192.168.50.95 [27890.033120] ---[ end trace 143aa23d25ef0e82 ]---
Nov 19 05:32:06 192.168.50.95 [27890.033878] RIP: 0010:tfw_http_msg_free+0x38/0xe0 [tempesta_fw]
Nov 19 05:32:06 192.168.50.95 [27890.034625] Code: 55 48 89 e5 53 48 8b 47 38 48 89 fb 48 85 c0 74 10 48 c7 40 38 00 00 00 00 48 c7 47 38 00 00 00 00 48 8b 7b 10 48 85 ff 74 44 <48> 8b 57 08 48 8b 07 48 85 d2 74 7c 48 85 c0 74 77 48 39 f8 74 5f
Nov 19 05:32:06 192.168.50.95 [27890.036324] RSP: 0018:ffffa568c010cb08 EFLAGS: 00010202
Nov 19 05:32:06 192.168.50.95 [27890.037185] RAX: ffff91399039f1f8 RBX: ffff91398fbd0020 RCX: 0000000000000000
Nov 19 05:32:06 192.168.50.95 [27890.038076] RDX: 00000000ffffffff RSI: ffff9139827e1a40 RDI: 0000001100000001
Nov 19 05:32:06 192.168.50.95 [27890.038986] RBP: ffffa568c010cb10 R08: 0000000000000000 R09: 0000000000000000
Nov 19 05:32:06 192.168.50.95 [27890.039884] R10: 0000000000000000 R11: 0000000000000000 R12: ffff91398fbd0020
Nov 19 05:32:06 192.168.50.95 [27890.040737] R13: ffff9138d4489c40 R14: ffff913988eaf5a0 R15: ffff9138c36c2688
Nov 19 05:32:06 192.168.50.95 [27890.041553] FS:  0000000000000000(0000) GS:ffff913ab7d00000(0000) knlGS:0000000000000000
Nov 19 05:32:06 192.168.50.95 [27890.042478] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 05:32:06 192.168.50.95 [27890.043409] CR2: 0000001100000009 CR3: 000000005800a006 CR4: 0000000000770ee0
Nov 19 05:32:06 192.168.50.95 [27890.044341] PKRU: 55555554
Nov 19 05:32:06 192.168.50.95 [27890.045219] Kernel panic - not syncing: Fatal exception in interrupt
Nov 19 05:32:06 192.168.50.95 [27890.046194] Kernel Offset: 0xa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Nov 19 05:32:06 192.168.50.95 [27890.047144] Rebooting in 1 seconds..
@krizhanovsky krizhanovsky added this to the 0.8 - Beta milestone Nov 19, 2024
@EvgeniiMekhanik EvgeniiMekhanik self-assigned this Nov 20, 2024
EvgeniiMekhanik added a commit that referenced this issue Nov 20, 2024
@EvgeniiMekhanik
Fix double free of stream->msg
4fc1a96 
We should zero `stream->msg` pointer in `tfw_h2_stream_unlink_nolock`
function regardless of whether we delete the message directly in this
function or not. If request is fully parsed it will be deleted later
in `tfw_http_resp_cache_cb->tfw_h2_resp_adjust_fwd` and if we don't
zero `stream->msg` in `tfw_h2_stream_unlink_nolock` we access
already freed request in `test_bit(TFW_HTTP_B_FULLY_PARSED, hmreq->flags)`.

Closes #2285
@EvgeniiMekhanik EvgeniiMekhanik mentioned this issue Nov 20, 2024
Merged
@krizhanovsky
Copy link
Contributor

The original problem should be fixed by #2288 , but there possible existing or future memory leaks and probably we need to improve the architecure, see #2288 (review)

@krizhanovsky krizhanovsky reopened this Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EvgeniiMekhanik EvgeniiMekhanik

Labels
bug h2
Projects
None yet
Milestone
0.8 - Beta
Development

Successfully merging a pull request may close this issue.

Fix double free of stream->msg tempesta-tech/tempesta
3 participants
@krizhanovsky @RomanBelozerov @EvgeniiMekhanik