Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

warn_bad_map on 5.12.0 #196

Closed
darkwrat opened this issue May 25, 2021 · 10 comments
Closed

warn_bad_map on 5.12.0 #196

darkwrat opened this issue May 25, 2021 · 10 comments
Assignees
@pabeni
Labels
bug

Comments

@darkwrat
Copy link
Contributor

darkwrat commented May 25, 2021 •
edited
Loading

A proxy (shadowsocks ss-server) vps running 5.12.0 + patch from #178 (mptcp: fix data stream corruption) got the following splat after two weeks of uptime, inconsistent use and experimentation.

[1085228.981091] ------------[ cut here ]------------
[1085228.981119] Bad mapping: ssn=1239287 map_seq=1188776 map_data_len=49251
[1085228.981244] WARNING: CPU: 0 PID: 0 at net/mptcp/subflow.c:761 mptcp_subflow_data_available (net/mptcp/subflow.c:761 net/mptcp/subflow.c:759 net/mptcp/subflow.c:793 net/mptcp/subflow.c:928 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[1085228.981304] Modules linked in: tcp_illinois binfmt_misc mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag rfkill sunrpc intel_rapl_msr intel_rapl_common joydev virtio_balloon i2c_piix4 tcp_bbr sch_fq fuse zram ip_tables cirrus drm_kms_helper cec drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_net virtio_blk serio_raw net_failover failover ata_generic pata_acpi qemu_fw_cfg pkcs8_key_parser
[1085228.981477] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.12.0-198.local2.fc35.x86_64 #1
[1085228.981497] Hardware name: Vultr VC2, BIOS
[1085228.981501] RIP: 0010:mptcp_subflow_data_available (net/mptcp/subflow.c:761 net/mptcp/subflow.c:759 net/mptcp/subflow.c:793 net/mptcp/subflow.c:928 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[1085228.981535] Code: ff e9 a6 fa ff ff 80 3d b8 18 1a 01 00 0f 85 94 fa ff ff c6 05 ab 18 1a 01 01 41 8b 4e 44 48 c7 c7 08 bc 49 a2 e8 06 4e 01 00 <0f> 0b e9 76 fa ff ff 4c 89 ca 48 c7 c6 d0 bb 49 a2 48 c7 c7 00 f2
All code
========
   0:	ff                   	(bad)  
   1:	e9 a6 fa ff ff       	jmpq   0xfffffffffffffaac
   6:	80 3d b8 18 1a 01 00 	cmpb   $0x0,0x11a18b8(%rip)        # 0x11a18c5
   d:	0f 85 94 fa ff ff    	jne    0xfffffffffffffaa7
  13:	c6 05 ab 18 1a 01 01 	movb   $0x1,0x11a18ab(%rip)        # 0x11a18c5
  1a:	41 8b 4e 44          	mov    0x44(%r14),%ecx
  1e:	48 c7 c7 08 bc 49 a2 	mov    $0xffffffffa249bc08,%rdi
  25:	e8 06 4e 01 00       	callq  0x14e30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	e9 76 fa ff ff       	jmpq   0xfffffffffffffaa7
  31:	4c 89 ca             	mov    %r9,%rdx
  34:	48 c7 c6 d0 bb 49 a2 	mov    $0xffffffffa249bbd0,%rsi
  3b:	48                   	rex.W
  3c:	c7                   	.byte 0xc7
  3d:	c7                   	.byte 0xc7
  3e:	00 f2                	add    %dh,%dl

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	e9 76 fa ff ff       	jmpq   0xfffffffffffffa7d
   7:	4c 89 ca             	mov    %r9,%rdx
   a:	48 c7 c6 d0 bb 49 a2 	mov    $0xffffffffa249bbd0,%rsi
  11:	48                   	rex.W
  12:	c7                   	.byte 0xc7
  13:	c7                   	.byte 0xc7
  14:	00 f2                	add    %dh,%dl
[1085228.981538] RSP: 0018:ffffadb440003a90 EFLAGS: 00010286
[1085228.981542] RAX: 000000000000003b RBX: ffff9b9c44d4d800 RCX: 000000000000083f
[1085228.981544] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
[1085228.981547] RBP: ffff9b9c741371c0 R08: 0000000000000000 R09: ffffadb4400038c0
[1085228.981549] R10: ffffadb4400038b8 R11: ffffffffa2b45ee8 R12: ffff9b9c440e8740
[1085228.981551] R13: ffff9b9c74137290 R14: ffff9b9c45862600 R15: ffff9b9c45862600
[1085228.981553] FS:  0000000000000000(0000) GS:ffff9b9c7ec00000(0000) knlGS:0000000000000000
[1085228.981556] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1085228.981558] CR2: 00007fd08203e010 CR3: 0000000004a00003 CR4: 00000000001706f0
[1085228.981563] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1085228.981565] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[1085228.981567] Call Trace:
[1085228.981585]  <IRQ>
[1085228.981594] subflow_data_ready (net/mptcp/subflow.c:1119 (discriminator 11)) 
[1085228.981599] tcp_data_queue (net/ipv4/tcp_input.c:4993) 
[1085228.981623] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[1085228.981626] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[1085228.981634] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[1085228.981643] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2062) 
[1085228.981647] ? __inet_lookup_established (net/ipv4/inet_hashtables.c:405) 
[1085228.981656] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[1085228.981662] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[1085228.981666] ip_sublist_rcv_finish (./include/net/dst.h:458 net/ipv4/ip_input.c:551) 
[1085228.981668] ip_sublist_rcv (net/ipv4/ip_input.c:610) 
[1085228.981671] ? __build_skb_around (net/core/skbuff.c:193 (discriminator 2)) 
[1085228.981682] ? __alloc_skb (net/core/skbuff.c:441) 
[1085228.981685] ip_list_rcv (net/ipv4/ip_input.c:645) 
[1085228.981690] __netif_receive_skb_list_core (net/core/dev.c:5427 net/core/dev.c:5475) 
[1085228.981707] netif_receive_skb_list_internal (net/core/dev.c:5529 net/core/dev.c:5637) 
[1085228.981719] napi_complete_done (./include/linux/list.h:35 net/core/dev.c:5792 net/core/dev.c:5787 net/core/dev.c:6494) 
[1085228.981723] virtqueue_napi_complete (drivers/net/virtio_net.c:334) virtio_net
[1085228.981735] virtnet_poll (drivers/net/virtio_net.c:1459) virtio_net
[1085228.981740] __napi_poll (net/core/dev.c:6913) 
[1085228.981743] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[1085228.981746] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[1085228.981769] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[1085228.981795] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 14)) 
[1085228.981813]  </IRQ>
[1085228.981815] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[1085228.981823] RIP: 0010:native_safe_halt (./arch/x86/include/asm/irqflags.h:52) 
[1085228.981827] Code: c0 7b 01 00 3e 80 4a 02 20 48 8b 12 83 e2 08 75 c3 e9 7a ff ff ff cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 58 41 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b6 58 41 00 f4 c3 cc cc 0f 1f 44 00
All code
========
   0:	c0 7b 01 00          	sarb   $0x0,0x1(%rbx)
   4:	3e 80 4a 02 20       	orb    $0x20,%ds:0x2(%rdx)
   9:	48 8b 12             	mov    (%rdx),%rdx
   c:	83 e2 08             	and    $0x8,%edx
   f:	75 c3                	jne    0xffffffffffffffd4
  11:	e9 7a ff ff ff       	jmpq   0xffffffffffffff90
  16:	cc                   	int3   
  17:	cc                   	int3   
  18:	cc                   	int3   
  19:	cc                   	int3   
  1a:	cc                   	int3   
  1b:	cc                   	int3   
  1c:	e9 07 00 00 00       	jmpq   0x28
  21:	0f 00 2d c6 58 41 00 	verw   0x4158c6(%rip)        # 0x4158ee
  28:	fb                   	sti    
  29:	f4                   	hlt    
  2a:*	c3                   	retq   		<-- trapping instruction
  2b:	90                   	nop
  2c:	e9 07 00 00 00       	jmpq   0x38
  31:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158ee
  38:	f4                   	hlt    
  39:	c3                   	retq   
  3a:	cc                   	int3   
  3b:	cc                   	int3   
  3c:	0f                   	.byte 0xf
  3d:	1f                   	(bad)  
  3e:	44                   	rex.R
	...

Code starting with the faulting instruction
===========================================
   0:	c3                   	retq   
   1:	90                   	nop
   2:	e9 07 00 00 00       	jmpq   0xe
   7:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158c4
   e:	f4                   	hlt    
   f:	c3                   	retq   
  10:	cc                   	int3   
  11:	cc                   	int3   
  12:	0f                   	.byte 0xf
  13:	1f                   	(bad)  
  14:	44                   	rex.R
	...
[1085228.981830] RSP: 0018:ffffffffa2a03eb0 EFLAGS: 00000202
[1085228.981833] RAX: ffffffffa1bf3d00 RBX: 0000000000000000 RCX: ffff9b9c7ec2b180
[1085228.981835] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b9c7ec1d420
[1085228.981837] RBP: ffffffffa2a1a940 R08: 000000cd42e4dffb R09: 0000000000000076
[1085228.981846] R10: 0000000000000073 R11: 0000000000000000 R12: 0000000000000000
[1085228.981848] R13: 0000000000000000 R14: 000000000000008f R15: 0000000000000000
[1085228.981850] ? __cpuidle_text_start (arch/x86/kernel/process.c:687) 
[1085228.981860] default_idle (./arch/x86/include/asm/paravirt.h:150 arch/x86/kernel/process.c:688) 
[1085228.981864] default_idle_call (./arch/x86/include/asm/paravirt.h:653 kernel/sched/idle.c:120) 
[1085228.981868] do_idle (kernel/sched/idle.c:195 kernel/sched/idle.c:300) 
[1085228.981880] cpu_startup_entry (kernel/sched/idle.c:396 (discriminator 1)) 
[1085228.981883] start_kernel (init/main.c:1066) 
[1085228.981920] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:283) 
[1085228.981941] ---[ end trace 50b9885fa6a311cc ]---
@darkwrat
Copy link
Contributor Author

Here is some nstat from the box.

#kernel
IpInReceives                    11326876           0.0
IpInDelivers                    11326876           0.0
IpOutRequests                   10453470           0.0
IcmpInMsgs                      123277             0.0
IcmpInErrors                    22                 0.0
IcmpInDestUnreachs              1061               0.0
IcmpInTimeExcds                 3                  0.0
IcmpInEchos                     122209             0.0
IcmpInEchoReps                  2                  0.0
IcmpInTimestamps                2                  0.0
IcmpOutMsgs                     123960             0.0
IcmpOutDestUnreachs             1749               0.0
IcmpOutEchoReps                 122209             0.0
IcmpOutTimestampReps            2                  0.0
IcmpMsgInType0                  2                  0.0
IcmpMsgInType3                  1061               0.0
IcmpMsgInType8                  122209             0.0
IcmpMsgInType11                 3                  0.0
IcmpMsgInType13                 2                  0.0
IcmpMsgOutType0                 122209             0.0
IcmpMsgOutType3                 1749               0.0
IcmpMsgOutType14                2                  0.0
TcpActiveOpens                  11646              0.0
TcpPassiveOpens                 90926              0.0
TcpAttemptFails                 1011               0.0
TcpEstabResets                  1361               0.0
TcpInSegs                       11411778           0.0
TcpOutSegs                      16258624           0.0
TcpRetransSegs                  426491             0.0
TcpInErrs                       190                0.0
TcpOutRsts                      913059             0.0
TcpInCsumErrors                 15                 0.0
UdpInDatagrams                  4769               0.0
UdpNoPorts                      2034               0.0
UdpOutDatagrams                 4784               0.0
Ip6InReceives                   217093             0.0
Ip6InDelivers                   217093             0.0
Ip6OutRequests                  138434             0.0
Ip6InMcastPkts                  967                0.0
Ip6OutMcastPkts                 1934               0.0
Ip6InOctets                     3364301129         0.0
Ip6OutOctets                    19117779           0.0
Ip6InMcastOctets                116160             0.0
Ip6OutMcastOctets               181296             0.0
Ip6InNoECTPkts                  191769             0.0
Ip6InECT0Pkts                   26575              0.0
Icmp6InMsgs                     2111               0.0
Icmp6OutMsgs                    3106               0.0
Icmp6InDestUnreachs             17                 0.0
Icmp6InRouterAdvertisements     922                0.0
Icmp6InNeighborSolicits         1019               0.0
Icmp6InNeighborAdvertisements   153                0.0
Icmp6OutRouterSolicits          91                 0.0
Icmp6OutNeighborSolicits        153                0.0
Icmp6OutNeighborAdvertisements  1019               0.0
Icmp6OutMLDv2Reports            1843               0.0
Icmp6InType1                    17                 0.0
Icmp6InType134                  922                0.0
Icmp6InType135                  1019               0.0
Icmp6InType136                  153                0.0
Icmp6OutType133                 91                 0.0
Icmp6OutType135                 153                0.0
Icmp6OutType136                 1019               0.0
Icmp6OutType143                 1843               0.0
TcpExtEmbryonicRsts             830                0.0
TcpExtPruneCalled               2                  0.0
TcpExtOutOfWindowIcmps          1                  0.0
TcpExtTW                        21299              0.0
TcpExtPAWSEstab                 7632               0.0
TcpExtDelayedACKs               43867              0.0
TcpExtDelayedACKLocked          2533               0.0
TcpExtDelayedACKLost            8285               0.0
TcpExtListenDrops               52                 0.0
TcpExtTCPHPHits                 1383795            0.0
TcpExtTCPPureAcks               7098046            0.0
TcpExtTCPHPAcks                 235514             0.0
TcpExtTCPSackRecovery           39376              0.0
TcpExtTCPSACKReneging           60                 0.0
TcpExtTCPSACKReorder            514                0.0
TcpExtTCPRenoReorder            6                  0.0
TcpExtTCPTSReorder              2                  0.0
TcpExtTCPPartialUndo            2                  0.0
TcpExtTCPDSACKUndo              778                0.0
TcpExtTCPLossUndo               1619               0.0
TcpExtTCPLostRetransmit         48941              0.0
TcpExtTCPRenoFailures           3                  0.0
TcpExtTCPSackFailures           22                 0.0
TcpExtTCPLossFailures           233                0.0
TcpExtTCPFastRetrans            393811             0.0
TcpExtTCPSlowStartRetrans       6336               0.0
TcpExtTCPTimeouts               17348              0.0
TcpExtTCPLossProbes             17237              0.0
TcpExtTCPLossProbeRecovery      3401               0.0
TcpExtTCPSackRecoveryFail       2565               0.0
TcpExtTCPRcvCollapsed           48                 0.0
TcpExtTCPBacklogCoalesce        12255              0.0
TcpExtTCPDSACKOldSent           8304               0.0
TcpExtTCPDSACKOfoSent           660                0.0
TcpExtTCPDSACKRecv              32969              0.0
TcpExtTCPDSACKOfoRecv           6                  0.0
TcpExtTCPAbortOnData            1532               0.0
TcpExtTCPAbortOnClose           732                0.0
TcpExtTCPAbortOnTimeout         816                0.0
TcpExtTCPDSACKIgnoredOld        3487               0.0
TcpExtTCPDSACKIgnoredNoUndo     1278               0.0
TcpExtTCPSpuriousRTOs           549                0.0
TcpExtTCPSackShifted            313202             0.0
TcpExtTCPSackMerged             1080647            0.0
TcpExtTCPSackShiftFallback      370867             0.0
TcpExtTCPRcvCoalesce            1092263            0.0
TcpExtTCPOFOQueue               78565              0.0
TcpExtTCPOFOMerge               644                0.0
TcpExtTCPChallengeACK           222                0.0
TcpExtTCPSYNChallenge           178                0.0
TcpExtTCPFastOpenActive         2                  0.0
TcpExtTCPAutoCorking            46941              0.0
TcpExtTCPFromZeroWindowAdv      1920               0.0
TcpExtTCPToZeroWindowAdv        1921               0.0
TcpExtTCPWantZeroWindowAdv      3744               0.0
TcpExtTCPSynRetrans             3412               0.0
TcpExtTCPOrigDataSent           13259503           0.0
TcpExtTCPHystartTrainDetect     4                  0.0
TcpExtTCPHystartTrainCwnd       138                0.0
TcpExtTCPHystartDelayDetect     828                0.0
TcpExtTCPHystartDelayCwnd       38777              0.0
TcpExtTCPACKSkippedSynRecv      17                 0.0
TcpExtTCPACKSkippedPAWS         1799               0.0
TcpExtTCPACKSkippedSeq          909                0.0
TcpExtTCPACKSkippedChallenge    1                  0.0
TcpExtTCPWinProbe               891                0.0
TcpExtTCPDelivered              13301431           0.0
TcpExtTCPDeliveredCE            1                  0.0
TcpExtTCPAckCompressed          28196              0.0
TcpExtTcpTimeoutRehash          14485              0.0
TcpExtTcpDuplicateDataRehash    281                0.0
TcpExtTCPDSACKRecvSegs          32977              0.0
IpExtInOctets                   13594304475        0.0
IpExtOutOctets                  17760997212        0.0
IpExtInNoECTPkts                10271334           0.0
IpExtInECT1Pkts                 11                 0.0
IpExtInECT0Pkts                 1083945            0.0
IpExtInCEPkts                   1                  0.0
MPTcpExtMPCapableSYNRX          11495              0.0
MPTcpExtMPCapableACKRX          11461              0.0
MPTcpExtMPCapableFallbackACK    66229              0.0
MPTcpExtMPTCPRetrans            249559             0.0
MPTcpExtMPJoinNoTokenFound      52                 0.0
MPTcpExtMPJoinSynRx             10138              0.0
MPTcpExtMPJoinAckRx             9498               0.0
MPTcpExtMPJoinAckHMacFailure    126                0.0
MPTcpExtOFOQueueTail            275929             0.0
MPTcpExtOFOQueue                283950             0.0
MPTcpExtOFOMerge                262221             0.0
MPTcpExtNoDSSInWindow           225                0.0
MPTcpExtDuplicateData           10400              0.0
MPTcpExtRmAddr                  13                 0.0

@pabeni
Copy link

pabeni commented May 26, 2021

Hi,

Thank you for the report.

A proxy (shadowsocks ss-server) vps running 5.12.0 + patch from #178 (mptcp: fix data stream corruption)

That patch entered DaveM '-net' tree and should land into vanilla soon (actually is quite strange it's taking so much, but that is a completely different problem ;)

got the following splat after two weeks of uptime, inconsistent use and experimentation.

[1085228.981091] ------------[ cut here ]------------
[1085228.981119] Bad mapping: ssn=1239287 map_seq=1188776 map_data_len=49251

whoops, looks like warn_bad_map() dumps the wrong info in one case, I'll send a patch. Anyhow the bug looks real.

We have a problem with tcp_add_backlog() which could cause the above. I'll send a patch, but I guess will be difficult to validate.

MPTcpExtMPTCPRetrans 249559 0.0

[likely unrelated] this counter is very high, I guess it's time to fix for good #177 ...

@pabeni pabeni added the bug label May 26, 2021
@pabeni pabeni self-assigned this May 26, 2021
@darkwrat
Copy link
Contributor Author

Hi Paolo.

A proxy (shadowsocks ss-server) vps running 5.12.0 + patch from #178 (mptcp: fix data stream corruption)

That patch entered DaveM '-net' tree and should land into vanilla soon (actually is quite strange it's taking so much, but that is a completely different problem ;)

Yeah, I've been following it's fate, after the networking pull request gets through -- I'll move to corresponding 5.13-rcX from
rawhide-nodebug repo.

whoops, looks like warn_bad_map() dumps the wrong info in one case, I'll send a patch. Anyhow the bug looks real.

We have a problem with tcp_add_backlog() which could cause the above. I'll send a patch, but I guess will be difficult to validate.

Thank you! I'm almost there with setting the thing up as a default gateway, so I'll stumble upon it sooner or later.

MPTcpExtMPTCPRetrans 249559 0.0

[likely unrelated] this counter is very high, I guess it's time to fix for good #177 ...

I've been using fq_codel+bbr lately, + one of the links is really lossy at times. Maybe I'll also try out the BBRv2 patch.

@pabeni pabeni mentioned this issue May 27, 2021
Open
@pabeni
Copy link

pabeni commented May 27, 2021

We have a problem with tcp_add_backlog() which could cause the above. I'll send a patch, but I guess will be difficult to validate.

uhmmm... looks I was fooled. It seems that tcp_add_backlog() is safe.

I think the splat could be cause by legit traffic, when the peer falls back to TCP - we don't send infinite mapping - and the first plain TCP packet is coalesced by the TCP stack into the previous one carrying the DSS. Even this is quite hard to test. Possibly the right thing to to is convert the WARN_ONCE into a pr_warn()

@darkwrat
Copy link
Contributor Author

I've captured some splats -- on the client side this time. With
mptcp: let warn_bad_map report relevant values
tcp: ensure that backlog coalescing don't break MPTCP DSS
.. on top of 5.12.0

[167543.588275] ------------[ cut here ]------------
[167543.588284] Bad mapping: ssn=262992234 map_seq=262964514 map_data_len=22824
[167543.588337] WARNING: CPU: 0 PID: 0 at net/mptcp/subflow.c:761 mptcp_subflow_data_available (net/mptcp/subflow.c:761 net/mptcp/subflow.c:759 net/mptcp/subflow.c:793 net/mptcp/subflow.c:928 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[167543.588354] Modules linked in: mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag 8021q garp mrp stp llc iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi intel_powerclamp snd_hda_codec snd_hda_core pcspkr snd_hwdep snd_seq cdc_ether snd_seq_device usbnet snd_pcm i2c_i801 lpc_ich i2c_smbus parport_pc snd_timer snd parport soundcore nft_counter nft_chain_nat xt_REDIRECT nf_nat nft_compat nf_tables nfnetlink it87 hwmon_vid nfsd auth_rpcgss nfs_acl tcp_bbr lockd grace sunrpc nf_conntrack_ftp nfs_ssc ip_tables i915 i2c_algo_bit video drm_kms_helper cec drm uas 8139too serio_raw usb_storage 8139cp ata_generic mii pata_acpi skge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
[167543.588459] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Not tainted 5.12.0-198.local4.fc35.x86_64 #1
[167543.588465] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[167543.588468] RIP: 0010:mptcp_subflow_data_available (net/mptcp/subflow.c:761 net/mptcp/subflow.c:759 net/mptcp/subflow.c:793 net/mptcp/subflow.c:928 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[167543.588475] Code: ff e9 a6 fa ff ff 80 3d 38 18 1a 01 00 0f 85 94 fa ff ff 41 8b 4e 44 48 c7 c7 08 bc 49 af c6 05 20 18 1a 01 01 e8 86 4d 01 00 <0f> 0b e9 76 fa ff ff 4c 89 ca 48 c7 c6 d0 bb 49 af 48 c7 c7 00 f2
All code
========
   0:	ff                   	(bad)  
   1:	e9 a6 fa ff ff       	jmpq   0xfffffffffffffaac
   6:	80 3d 38 18 1a 01 00 	cmpb   $0x0,0x11a1838(%rip)        # 0x11a1845
   d:	0f 85 94 fa ff ff    	jne    0xfffffffffffffaa7
  13:	41 8b 4e 44          	mov    0x44(%r14),%ecx
  17:	48 c7 c7 08 bc 49 af 	mov    $0xffffffffaf49bc08,%rdi
  1e:	c6 05 20 18 1a 01 01 	movb   $0x1,0x11a1820(%rip)        # 0x11a1845
  25:	e8 86 4d 01 00       	callq  0x14db0
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	e9 76 fa ff ff       	jmpq   0xfffffffffffffaa7
  31:	4c 89 ca             	mov    %r9,%rdx
  34:	48 c7 c6 d0 bb 49 af 	mov    $0xffffffffaf49bbd0,%rsi
  3b:	48                   	rex.W
  3c:	c7                   	.byte 0xc7
  3d:	c7                   	.byte 0xc7
  3e:	00 f2                	add    %dh,%dl

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	e9 76 fa ff ff       	jmpq   0xfffffffffffffa7d
   7:	4c 89 ca             	mov    %r9,%rdx
   a:	48 c7 c6 d0 bb 49 af 	mov    $0xffffffffaf49bbd0,%rsi
  11:	48                   	rex.W
  12:	c7                   	.byte 0xc7
  13:	c7                   	.byte 0xc7
  14:	00 f2                	add    %dh,%dl
[167543.588480] RSP: 0018:ffffa859c0003c78 EFLAGS: 00010296
[167543.588484] RAX: 000000000000003f RBX: ffff9195ef299300 RCX: 0000000000000027
[167543.588488] RDX: ffff91961f4185c8 RSI: 0000000000000001 RDI: ffff91961f4185c0
[167543.588490] RBP: ffff9196031c3600 R08: 0000000000000000 R09: ffffa859c0003aa8
[167543.588493] R10: ffffa859c0003aa0 R11: ffffffffafb45ee8 R12: ffff91960c379e00
[167543.588496] R13: ffff9196031c36d0 R14: ffff919603262f00 R15: ffff919603262f00
[167543.588499] FS:  0000000000000000(0000) GS:ffff91961f400000(0000) knlGS:0000000000000000
[167543.588503] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167543.588506] CR2: 000000c000447000 CR3: 0000000002988000 CR4: 00000000000006f0
[167543.588511] Call Trace:
[167543.588515]  <IRQ>
[167543.588521] subflow_data_ready (net/mptcp/subflow.c:1119 (discriminator 11)) 
[167543.588528] tcp_data_queue (net/ipv4/tcp_input.c:4993) 
[167543.588534] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[167543.588539] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[167543.588545] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[167543.588550] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[167543.588555] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[167543.588561] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[167543.588566] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[167543.588571] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[167543.588578] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[167543.588583] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[167543.588597] __napi_poll (net/core/dev.c:6913) 
[167543.588601] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[167543.588606] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[167543.588613] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[167543.588619] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 14)) 
[167543.588669]  </IRQ>
[167543.588673] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167543.588681] RIP: 0010:native_safe_halt (./arch/x86/include/asm/irqflags.h:52) 
[167543.588686] Code: c0 7b 01 00 3e 80 4a 02 20 48 8b 12 83 e2 08 75 c3 e9 7a ff ff ff cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 58 41 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b6 58 41 00 f4 c3 cc cc 0f 1f 44 00
All code
========
   0:	c0 7b 01 00          	sarb   $0x0,0x1(%rbx)
   4:	3e 80 4a 02 20       	orb    $0x20,%ds:0x2(%rdx)
   9:	48 8b 12             	mov    (%rdx),%rdx
   c:	83 e2 08             	and    $0x8,%edx
   f:	75 c3                	jne    0xffffffffffffffd4
  11:	e9 7a ff ff ff       	jmpq   0xffffffffffffff90
  16:	cc                   	int3   
  17:	cc                   	int3   
  18:	cc                   	int3   
  19:	cc                   	int3   
  1a:	cc                   	int3   
  1b:	cc                   	int3   
  1c:	e9 07 00 00 00       	jmpq   0x28
  21:	0f 00 2d c6 58 41 00 	verw   0x4158c6(%rip)        # 0x4158ee
  28:	fb                   	sti    
  29:	f4                   	hlt    
  2a:*	c3                   	retq   		<-- trapping instruction
  2b:	90                   	nop
  2c:	e9 07 00 00 00       	jmpq   0x38
  31:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158ee
  38:	f4                   	hlt    
  39:	c3                   	retq   
  3a:	cc                   	int3   
  3b:	cc                   	int3   
  3c:	0f                   	.byte 0xf
  3d:	1f                   	(bad)  
  3e:	44                   	rex.R
	...

Code starting with the faulting instruction
===========================================
   0:	c3                   	retq   
   1:	90                   	nop
   2:	e9 07 00 00 00       	jmpq   0xe
   7:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158c4
   e:	f4                   	hlt    
   f:	c3                   	retq   
  10:	cc                   	int3   
  11:	cc                   	int3   
  12:	0f                   	.byte 0xf
  13:	1f                   	(bad)  
  14:	44                   	rex.R
	...
[167543.588690] RSP: 0018:ffffffffafa03e40 EFLAGS: 00000246
[167543.588695] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff91961f42b180
[167543.588698] RDX: ffff91961f400000 RSI: ffff9195c1802000 RDI: ffff9195c1802064
[167543.588701] RBP: ffff9195c1802064 R08: ffffffffafc63ba0 R09: 0000000000000018
[167543.588703] R10: 0000000000000208 R11: 0000000000000102 R12: 0000000000000001
[167543.588706] R13: ffffffffafc63c20 R14: 0000000000000001 R15: 0000000000000000
[167543.588711] acpi_idle_do_entry (./arch/x86/include/asm/paravirt.h:150 drivers/acpi/processor_idle.c:111 drivers/acpi/processor_idle.c:517) 
[167543.588716] acpi_idle_enter (drivers/acpi/processor_idle.c:654) 
[167543.588723] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:237) 
[167543.588730] cpuidle_enter (drivers/cpuidle/cpuidle.c:353) 
[167543.588735] do_idle (kernel/sched/idle.c:158 kernel/sched/idle.c:239 kernel/sched/idle.c:300) 
[167543.588740] cpu_startup_entry (kernel/sched/idle.c:396 (discriminator 1)) 
[167543.588744] start_kernel (init/main.c:1066) 
[167543.588752] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:283) 
[167543.588760] ---[ end trace 6b1263c0cf0d261f ]---
[167849.043432] ------------[ cut here ]------------
[167849.043442] WARNING: CPU: 0 PID: 6257 at net/mptcp/protocol.c:614 __mptcp_move_skbs_from_subflow (net/mptcp/protocol.c:614 (discriminator 1)) 
[167849.043460] Modules linked in: mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag 8021q garp mrp stp llc iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi intel_powerclamp snd_hda_codec snd_hda_core pcspkr snd_hwdep snd_seq cdc_ether snd_seq_device usbnet snd_pcm i2c_i801 lpc_ich i2c_smbus parport_pc snd_timer snd parport soundcore nft_counter nft_chain_nat xt_REDIRECT nf_nat nft_compat nf_tables nfnetlink it87 hwmon_vid nfsd auth_rpcgss nfs_acl tcp_bbr lockd grace sunrpc nf_conntrack_ftp nfs_ssc ip_tables i915 i2c_algo_bit video drm_kms_helper cec drm uas 8139too serio_raw usb_storage 8139cp ata_generic mii pata_acpi skge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
[167849.043582] CPU: 0 PID: 6257 Comm: ss-redir Kdump: loaded Tainted: G        W        --------- ---  5.12.0-198.local4.fc35.x86_64 #1
[167849.043588] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[167849.043590] RIP: 0010:__mptcp_move_skbs_from_subflow (net/mptcp/protocol.c:614 (discriminator 1)) 
[167849.043598] Code: 00 00 00 66 85 c0 0f 44 c2 3e 41 01 87 a8 00 00 00 48 89 ef e8 9e cb e3 ff 45 01 e5 44 3b 64 24 0c 0f 86 5d fe ff ff 4d 89 fe <0f> 0b eb 5a 48 83 bb c8 00 00 00 00 0f 85 3b fe ff ff 48 89 ab c8
All code
========
   0:	00 00                	add    %al,(%rax)
   2:	00 66 85             	add    %ah,-0x7b(%rsi)
   5:	c0 0f 44             	rorb   $0x44,(%rdi)
   8:	c2 3e 41             	retq   $0x413e
   b:	01 87 a8 00 00 00    	add    %eax,0xa8(%rdi)
  11:	48 89 ef             	mov    %rbp,%rdi
  14:	e8 9e cb e3 ff       	callq  0xffffffffffe3cbb7
  19:	45 01 e5             	add    %r12d,%r13d
  1c:	44 3b 64 24 0c       	cmp    0xc(%rsp),%r12d
  21:	0f 86 5d fe ff ff    	jbe    0xfffffffffffffe84
  27:	4d 89 fe             	mov    %r15,%r14
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb 5a                	jmp    0x88
  2e:	48 83 bb c8 00 00 00 	cmpq   $0x0,0xc8(%rbx)
  35:	00 
  36:	0f 85 3b fe ff ff    	jne    0xfffffffffffffe77
  3c:	48                   	rex.W
  3d:	89                   	.byte 0x89
  3e:	ab                   	stos   %eax,%es:(%rdi)
  3f:	c8                   	.byte 0xc8

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	eb 5a                	jmp    0x5e
   4:	48 83 bb c8 00 00 00 	cmpq   $0x0,0xc8(%rbx)
   b:	00 
   c:	0f 85 3b fe ff ff    	jne    0xfffffffffffffe4d
  12:	48                   	rex.W
  13:	89                   	.byte 0x89
  14:	ab                   	stos   %eax,%es:(%rdi)
  15:	c8                   	.byte 0xc8
[167849.043603] RSP: 0000:ffffa859c0a37b78 EFLAGS: 00010216
[167849.043607] RAX: 0000000000000001 RBX: ffff9196031c3f00 RCX: ffffffffaf48d450
[167849.043611] RDX: 000000008010000c RSI: ffffffffaeb69467 RDI: fffffa3100d69920
[167849.043613] RBP: ffff9195f5a64d00 R08: 0000000000000001 R09: 0000000000000000
[167849.043616] R10: ffff9195caeaa300 R11: 0000000000000001 R12: 0000000000002760
[167849.043619] R13: 00000000d536587b R14: ffff9195c2774380 R15: ffff9195c2774380
[167849.043622] FS:  00007fdde054e980(0000) GS:ffff91961f400000(0000) knlGS:0000000000000000
[167849.043627] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167849.043631] CR2: 00007fabb2e96010 CR3: 00000000061ea000 CR4: 00000000000006f0
[167849.043636] Call Trace:
[167849.043644] ? mptcp_subflow_data_available (./include/linux/skbuff.h:4286 net/mptcp/subflow.c:931 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[167849.043652] mptcp_data_ready (net/mptcp/protocol.c:693 net/mptcp/protocol.c:736) 
[167849.043666] tcp_data_queue (net/ipv4/tcp_input.c:4993) 
[167849.043673] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[167849.043677] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[167849.043682] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[167849.043687] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[167849.043692] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[167849.043697] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[167849.043702] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[167849.043707] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[167849.043713] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[167849.043718] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[167849.043733] __napi_poll (net/core/dev.c:6913) 
[167849.043737] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[167849.043741] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[167849.043748] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[167849.043753] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 13)) 
[167849.043759] ? asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.043764] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.043769] RIP: 0033:0x7fdddf95d449
[167849.043776] Code: f1 44 31 d9 41 c1 c7 0c c1 c1 10 45 01 f8 44 31 c7 c1 c7 08 41 01 f9 44 31 ef c1 c7 10 45 31 cf 41 01 c9 41 01 fc 41 c1 c7 07 <44> 31 e3 c1 c3 0c 41 01 dd 44 31 ef 41 89 fe 41 c1 c6 08 45 01 f4
All code
========
   0:	f1                   	icebp  
   1:	44 31 d9             	xor    %r11d,%ecx
   4:	41 c1 c7 0c          	rol    $0xc,%r15d
   8:	c1 c1 10             	rol    $0x10,%ecx
   b:	45 01 f8             	add    %r15d,%r8d
   e:	44 31 c7             	xor    %r8d,%edi
  11:	c1 c7 08             	rol    $0x8,%edi
  14:	41 01 f9             	add    %edi,%r9d
  17:	44 31 ef             	xor    %r13d,%edi
  1a:	c1 c7 10             	rol    $0x10,%edi
  1d:	45 31 cf             	xor    %r9d,%r15d
  20:	41 01 c9             	add    %ecx,%r9d
  23:	41 01 fc             	add    %edi,%r12d
  26:	41 c1 c7 07          	rol    $0x7,%r15d
  2a:*	44 31 e3             	xor    %r12d,%ebx		<-- trapping instruction
  2d:	c1 c3 0c             	rol    $0xc,%ebx
  30:	41 01 dd             	add    %ebx,%r13d
  33:	44 31 ef             	xor    %r13d,%edi
  36:	41 89 fe             	mov    %edi,%r14d
  39:	41 c1 c6 08          	rol    $0x8,%r14d
  3d:	45 01 f4             	add    %r14d,%r12d

Code starting with the faulting instruction
===========================================
   0:	44 31 e3             	xor    %r12d,%ebx
   3:	c1 c3 0c             	rol    $0xc,%ebx
   6:	41 01 dd             	add    %ebx,%r13d
   9:	44 31 ef             	xor    %r13d,%edi
   c:	41 89 fe             	mov    %edi,%r14d
   f:	41 c1 c6 08          	rol    $0x8,%r14d
  13:	45 01 f4             	add    %r14d,%r12d
[167849.043780] RSP: 002b:00007ffd79e07870 EFLAGS: 00000a86
[167849.043791] RAX: 00000000e8eb3216 RBX: 000000003120a234 RCX: 0000000086f7b40a
[167849.043794] RDX: 00000000d65fd21e RSI: 00000000a18519a1 RDI: 000000008eeacd86
[167849.043796] RBP: 00000000c9da0fb0 R08: 0000000053b05fb3 R09: 00000000c8d2a0bf
[167849.043798] R10: 00000000981d8646 R11: 00000000bd162bd9 R12: 00000000f8d6cae8
[167849.043801] R13: 00000000b50a2b0d R14: 00000000c94e46ce R15: 00000000a1a32f60
[167849.043835] ---[ end trace 6b1263c0cf0d2620 ]---
[167849.043989] ------------[ cut here ]------------
[167849.043996] WARNING: CPU: 0 PID: 6257 at net/mptcp/protocol.c:617 __mptcp_move_skbs_from_subflow (net/mptcp/protocol.c:617 (discriminator 1)) 
[167849.044010] Modules linked in: mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag 8021q garp mrp stp llc iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi intel_powerclamp snd_hda_codec snd_hda_core pcspkr snd_hwdep snd_seq cdc_ether snd_seq_device usbnet snd_pcm i2c_i801 lpc_ich i2c_smbus parport_pc snd_timer snd parport soundcore nft_counter nft_chain_nat xt_REDIRECT nf_nat nft_compat nf_tables nfnetlink it87 hwmon_vid nfsd auth_rpcgss nfs_acl tcp_bbr lockd grace sunrpc nf_conntrack_ftp nfs_ssc ip_tables i915 i2c_algo_bit video drm_kms_helper cec drm uas 8139too serio_raw usb_storage 8139cp ata_generic mii pata_acpi skge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
[167849.044111] CPU: 0 PID: 6257 Comm: ss-redir Kdump: loaded Tainted: G        W        --------- ---  5.12.0-198.local4.fc35.x86_64 #1
[167849.044116] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[167849.044119] RIP: 0010:__mptcp_move_skbs_from_subflow (net/mptcp/protocol.c:617 (discriminator 1)) 
[167849.044125] Code: 4c 24 20 44 89 44 24 18 e8 9f 94 e3 ff c6 45 7f 00 4c 8b 4c 24 20 44 8b 44 24 18 e9 ab fc ff ff c6 44 24 0b 01 e9 3b fc ff ff <0f> 0b e9 a8 fb ff ff 4d 8b 87 90 05 00 00 0f 1f 44 00 00 4d 39 d0
All code
========
   0:	4c 24 20             	rex.WR and $0x20,%al
   3:	44 89 44 24 18       	mov    %r8d,0x18(%rsp)
   8:	e8 9f 94 e3 ff       	callq  0xffffffffffe394ac
   d:	c6 45 7f 00          	movb   $0x0,0x7f(%rbp)
  11:	4c 8b 4c 24 20       	mov    0x20(%rsp),%r9
  16:	44 8b 44 24 18       	mov    0x18(%rsp),%r8d
  1b:	e9 ab fc ff ff       	jmpq   0xfffffffffffffccb
  20:	c6 44 24 0b 01       	movb   $0x1,0xb(%rsp)
  25:	e9 3b fc ff ff       	jmpq   0xfffffffffffffc65
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	e9 a8 fb ff ff       	jmpq   0xfffffffffffffbd9
  31:	4d 8b 87 90 05 00 00 	mov    0x590(%r15),%r8
  38:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  3d:	4d 39 d0             	cmp    %r10,%r8

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	e9 a8 fb ff ff       	jmpq   0xfffffffffffffbaf
   7:	4d 8b 87 90 05 00 00 	mov    0x590(%r15),%r8
   e:	0f 1f 44 00 00       	nopl   0x0(%rax,%rax,1)
  13:	4d 39 d0             	cmp    %r10,%r8
[167849.044129] RSP: 0000:ffffa859c0a37b78 EFLAGS: 00010292
[167849.044134] RAX: 0000000000005898 RBX: ffff9196031c3f00 RCX: 000000000000fb04
[167849.044137] RDX: 00000000d536311b RSI: 00000000d536311b RDI: ffff9195c2774300
[167849.044140] RBP: ffff9195f5081600 R08: 00000000ffffd8a0 R09: 00000000d5291ad3
[167849.044142] R10: 0000000000001600 R11: 0000000000000001 R12: 00000000d536587b
[167849.044145] R13: 00000000d536311b R14: ffff9195f513bb00 R15: ffff9195c2774380
[167849.044148] FS:  00007fdde054e980(0000) GS:ffff91961f400000(0000) knlGS:0000000000000000
[167849.044152] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167849.044155] CR2: 00007fabb2e96010 CR3: 00000000061ea000 CR4: 00000000000006f0
[167849.044160] Call Trace:
[167849.044167] mptcp_data_ready (net/mptcp/protocol.c:693 net/mptcp/protocol.c:736) 
[167849.044175] tcp_data_queue (net/ipv4/tcp_input.c:4993) 
[167849.044181] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[167849.044185] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[167849.044191] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[167849.044198] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[167849.044202] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[167849.044207] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[167849.044212] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[167849.044216] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[167849.044222] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[167849.044227] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[167849.044238] __napi_poll (net/core/dev.c:6913) 
[167849.044243] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[167849.044247] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[167849.044253] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[167849.044258] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 13)) 
[167849.044263] ? asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.044311] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.044320] RIP: 0033:0x7fdddf95d449
[167849.044327] Code: f1 44 31 d9 41 c1 c7 0c c1 c1 10 45 01 f8 44 31 c7 c1 c7 08 41 01 f9 44 31 ef c1 c7 10 45 31 cf 41 01 c9 41 01 fc 41 c1 c7 07 <44> 31 e3 c1 c3 0c 41 01 dd 44 31 ef 41 89 fe 41 c1 c6 08 45 01 f4
All code
========
   0:	f1                   	icebp  
   1:	44 31 d9             	xor    %r11d,%ecx
   4:	41 c1 c7 0c          	rol    $0xc,%r15d
   8:	c1 c1 10             	rol    $0x10,%ecx
   b:	45 01 f8             	add    %r15d,%r8d
   e:	44 31 c7             	xor    %r8d,%edi
  11:	c1 c7 08             	rol    $0x8,%edi
  14:	41 01 f9             	add    %edi,%r9d
  17:	44 31 ef             	xor    %r13d,%edi
  1a:	c1 c7 10             	rol    $0x10,%edi
  1d:	45 31 cf             	xor    %r9d,%r15d
  20:	41 01 c9             	add    %ecx,%r9d
  23:	41 01 fc             	add    %edi,%r12d
  26:	41 c1 c7 07          	rol    $0x7,%r15d
  2a:*	44 31 e3             	xor    %r12d,%ebx		<-- trapping instruction
  2d:	c1 c3 0c             	rol    $0xc,%ebx
  30:	41 01 dd             	add    %ebx,%r13d
  33:	44 31 ef             	xor    %r13d,%edi
  36:	41 89 fe             	mov    %edi,%r14d
  39:	41 c1 c6 08          	rol    $0x8,%r14d
  3d:	45 01 f4             	add    %r14d,%r12d

Code starting with the faulting instruction
===========================================
   0:	44 31 e3             	xor    %r12d,%ebx
   3:	c1 c3 0c             	rol    $0xc,%ebx
   6:	41 01 dd             	add    %ebx,%r13d
   9:	44 31 ef             	xor    %r13d,%edi
   c:	41 89 fe             	mov    %edi,%r14d
   f:	41 c1 c6 08          	rol    $0x8,%r14d
  13:	45 01 f4             	add    %r14d,%r12d
[167849.044330] RSP: 002b:00007ffd79e07870 EFLAGS: 00000a86
[167849.044335] RAX: 00000000e8eb3216 RBX: 000000003120a234 RCX: 0000000086f7b40a
[167849.044337] RDX: 00000000d65fd21e RSI: 00000000a18519a1 RDI: 000000008eeacd86
[167849.044340] RBP: 00000000c9da0fb0 R08: 0000000053b05fb3 R09: 00000000c8d2a0bf
[167849.044342] R10: 00000000981d8646 R11: 00000000bd162bd9 R12: 00000000f8d6cae8
[167849.044345] R13: 00000000b50a2b0d R14: 00000000c94e46ce R15: 00000000a1a32f60
[167849.044350] ---[ end trace 6b1263c0cf0d2621 ]---
[167849.044538] ------------[ cut here ]------------
[167849.044546] WARNING: CPU: 0 PID: 6257 at net/mptcp/subflow.c:771 mptcp_subflow_data_available (net/mptcp/subflow.c:771 net/mptcp/subflow.c:906 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[167849.044561] Modules linked in: mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag 8021q garp mrp stp llc iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi intel_powerclamp snd_hda_codec snd_hda_core pcspkr snd_hwdep snd_seq cdc_ether snd_seq_device usbnet snd_pcm i2c_i801 lpc_ich i2c_smbus parport_pc snd_timer snd parport soundcore nft_counter nft_chain_nat xt_REDIRECT nf_nat nft_compat nf_tables nfnetlink it87 hwmon_vid nfsd auth_rpcgss nfs_acl tcp_bbr lockd grace sunrpc nf_conntrack_ftp nfs_ssc ip_tables i915 i2c_algo_bit video drm_kms_helper cec drm uas 8139too serio_raw usb_storage 8139cp ata_generic mii pata_acpi skge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
[167849.044669] CPU: 0 PID: 6257 Comm: ss-redir Kdump: loaded Tainted: G        W        --------- ---  5.12.0-198.local4.fc35.x86_64 #1
[167849.044674] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[167849.044676] RIP: 0010:mptcp_subflow_data_available (net/mptcp/subflow.c:771 net/mptcp/subflow.c:906 net/mptcp/subflow.c:989 net/mptcp/subflow.c:1075) 
[167849.044682] Code: 85 c7 fc ff ff e9 50 fb ff ff 49 8b 53 08 48 c7 c6 70 bb 49 af 48 c7 c7 a8 f2 d4 af 88 04 24 e8 f7 4d af ff 0f b6 04 24 eb a1 <0f> 0b e9 e3 fc ff ff 4c 89 e2 48 c7 c6 70 b2 49 af 48 c7 c7 28 f6
All code
========
   0:	85 c7                	test   %eax,%edi
   2:	fc                   	cld    
   3:	ff                   	(bad)  
   4:	ff                   	(bad)  
   5:	e9 50 fb ff ff       	jmpq   0xfffffffffffffb5a
   a:	49 8b 53 08          	mov    0x8(%r11),%rdx
   e:	48 c7 c6 70 bb 49 af 	mov    $0xffffffffaf49bb70,%rsi
  15:	48 c7 c7 a8 f2 d4 af 	mov    $0xffffffffafd4f2a8,%rdi
  1c:	88 04 24             	mov    %al,(%rsp)
  1f:	e8 f7 4d af ff       	callq  0xffffffffffaf4e1b
  24:	0f b6 04 24          	movzbl (%rsp),%eax
  28:	eb a1                	jmp    0xffffffffffffffcb
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	e9 e3 fc ff ff       	jmpq   0xfffffffffffffd14
  31:	4c 89 e2             	mov    %r12,%rdx
  34:	48 c7 c6 70 b2 49 af 	mov    $0xffffffffaf49b270,%rsi
  3b:	48                   	rex.W
  3c:	c7                   	.byte 0xc7
  3d:	c7                   	(bad)  
  3e:	28 f6                	sub    %dh,%dh

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	e9 e3 fc ff ff       	jmpq   0xfffffffffffffcea
   7:	4c 89 e2             	mov    %r12,%rdx
   a:	48 c7 c6 70 b2 49 af 	mov    $0xffffffffaf49b270,%rsi
  11:	48                   	rex.W
  12:	c7                   	.byte 0xc7
  13:	c7                   	(bad)  
  14:	28 f6                	sub    %dh,%dh
[167849.044686] RSP: 0000:ffffa859c0a37bb8 EFLAGS: 00010296
[167849.044690] RAX: 00000000d536fae7 RBX: ffff9195ed829d00 RCX: ffff9195f513bb00
[167849.044693] RDX: 00000000d536311b RSI: 00000000000004ec RDI: 00000000ffff3634
[167849.044695] RBP: ffff9196031c3f00 R08: 000000000000fb04 R09: 98fae94fff1b135f
[167849.044698] R10: 00000000000016a0 R11: ffff9195f521fb10 R12: ffff9195c2774380
[167849.044701] R13: ffff9196031c3fd0 R14: ffff9195f513bb00 R15: ffff9195f513bb00
[167849.044704] FS:  00007fdde054e980(0000) GS:ffff91961f400000(0000) knlGS:0000000000000000
[167849.044707] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167849.044711] CR2: 00007fabb2e96010 CR3: 00000000061ea000 CR4: 00000000000006f0
[167849.044714] Call Trace:
[167849.044721] subflow_data_ready (net/mptcp/subflow.c:1119 (discriminator 11)) 
[167849.044727] tcp_data_queue (net/ipv4/tcp_input.c:4993) 
[167849.044733] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[167849.044737] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[167849.044743] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[167849.044747] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[167849.044752] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[167849.044757] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[167849.044762] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[167849.044766] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[167849.044772] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[167849.044777] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[167849.044788] __napi_poll (net/core/dev.c:6913) 
[167849.044792] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[167849.044797] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[167849.044833] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[167849.044841] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 13)) 
[167849.044846] ? asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.044852] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[167849.044856] RIP: 0033:0x7fdddf95d449
[167849.044862] Code: f1 44 31 d9 41 c1 c7 0c c1 c1 10 45 01 f8 44 31 c7 c1 c7 08 41 01 f9 44 31 ef c1 c7 10 45 31 cf 41 01 c9 41 01 fc 41 c1 c7 07 <44> 31 e3 c1 c3 0c 41 01 dd 44 31 ef 41 89 fe 41 c1 c6 08 45 01 f4
All code
========
   0:	f1                   	icebp  
   1:	44 31 d9             	xor    %r11d,%ecx
   4:	41 c1 c7 0c          	rol    $0xc,%r15d
   8:	c1 c1 10             	rol    $0x10,%ecx
   b:	45 01 f8             	add    %r15d,%r8d
   e:	44 31 c7             	xor    %r8d,%edi
  11:	c1 c7 08             	rol    $0x8,%edi
  14:	41 01 f9             	add    %edi,%r9d
  17:	44 31 ef             	xor    %r13d,%edi
  1a:	c1 c7 10             	rol    $0x10,%edi
  1d:	45 31 cf             	xor    %r9d,%r15d
  20:	41 01 c9             	add    %ecx,%r9d
  23:	41 01 fc             	add    %edi,%r12d
  26:	41 c1 c7 07          	rol    $0x7,%r15d
  2a:*	44 31 e3             	xor    %r12d,%ebx		<-- trapping instruction
  2d:	c1 c3 0c             	rol    $0xc,%ebx
  30:	41 01 dd             	add    %ebx,%r13d
  33:	44 31 ef             	xor    %r13d,%edi
  36:	41 89 fe             	mov    %edi,%r14d
  39:	41 c1 c6 08          	rol    $0x8,%r14d
  3d:	45 01 f4             	add    %r14d,%r12d

Code starting with the faulting instruction
===========================================
   0:	44 31 e3             	xor    %r12d,%ebx
   3:	c1 c3 0c             	rol    $0xc,%ebx
   6:	41 01 dd             	add    %ebx,%r13d
   9:	44 31 ef             	xor    %r13d,%edi
   c:	41 89 fe             	mov    %edi,%r14d
   f:	41 c1 c6 08          	rol    $0x8,%r14d
  13:	45 01 f4             	add    %r14d,%r12d
[167849.044865] RSP: 002b:00007ffd79e07870 EFLAGS: 00000a86
[167849.044869] RAX: 00000000e8eb3216 RBX: 000000003120a234 RCX: 0000000086f7b40a
[167849.044872] RDX: 00000000d65fd21e RSI: 00000000a18519a1 RDI: 000000008eeacd86
[167849.044875] RBP: 00000000c9da0fb0 R08: 0000000053b05fb3 R09: 00000000c8d2a0bf
[167849.044877] R10: 00000000981d8646 R11: 00000000bd162bd9 R12: 00000000f8d6cae8
[167849.044879] R13: 00000000b50a2b0d R14: 00000000c94e46ce R15: 00000000a1a32f60
[167849.044884] ---[ end trace 6b1263c0cf0d2622 ]---

@darkwrat
Copy link
Contributor Author

And here are the counters from the client box.

MPTcpExtMPTCPRetrans            3522410            0.0
MPTcpExtMPJoinSynAckRx          7152               0.0
MPTcpExtMPJoinAckHMacFailure    4                  0.0
MPTcpExtDSSNotMatching          1                  0.0
MPTcpExtOFOQueueTail            1774032            0.0
MPTcpExtOFOQueue                1835261            0.0
MPTcpExtOFOMerge                1660801            0.0
MPTcpExtDuplicateData           118750             0.0

@darkwrat
Copy link
Contributor Author

The client box got stuck on me this morning, same boot as above. But I came prepared, here is dmesg (direct continuation from the [167849.044884] mark above).

[218230.055241] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
[218230.055323] Modules linked in: mptcp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag 8021q garp mrp stp llc iTCO_wdt intel_pmc_bxt ppdev gpio_ich iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi intel_powerclamp snd_hda_codec snd_hda_core pcspkr snd_hwdep snd_seq cdc_ether snd_seq_device usbnet snd_pcm i2c_i801 lpc_ich i2c_smbus parport_pc snd_timer snd parport soundcore nft_counter nft_chain_nat xt_REDIRECT nf_nat nft_compat nf_tables nfnetlink it87 hwmon_vid nfsd auth_rpcgss nfs_acl tcp_bbr lockd grace sunrpc nf_conntrack_ftp nfs_ssc ip_tables i915 i2c_algo_bit video drm_kms_helper cec drm uas 8139too serio_raw usb_storage 8139cp ata_generic mii pata_acpi skge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
[218230.055431] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G        W        --------- ---  5.12.0-198.local4.fc35.x86_64 #1
[218230.055437] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[218230.055440] RIP: 0010:native_queued_spin_lock_slowpath (kernel/locking/qspinlock.c:382 kernel/locking/qspinlock.c:315) 
[218230.055452] Code: 0f ba 2a 08 0f 92 c1 8b 02 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 0f 85 11 01 00 00 85 c0 74 0e 8b 02 84 c0 74 08 f3 90 <8b> 02 84 c0 75 f8 b8 01 00 00 00 66 89 02 c3 8b 37 b9 00 02 00 00
All code
========
   0:	0f ba 2a 08          	btsl   $0x8,(%rdx)
   4:	0f 92 c1             	setb   %cl
   7:	8b 02                	mov    (%rdx),%eax
   9:	0f b6 c9             	movzbl %cl,%ecx
   c:	c1 e1 08             	shl    $0x8,%ecx
   f:	30 e4                	xor    %ah,%ah
  11:	09 c8                	or     %ecx,%eax
  13:	a9 00 01 ff ff       	test   $0xffff0100,%eax
  18:	0f 85 11 01 00 00    	jne    0x12f
  1e:	85 c0                	test   %eax,%eax
  20:	74 0e                	je     0x30
  22:	8b 02                	mov    (%rdx),%eax
  24:	84 c0                	test   %al,%al
  26:	74 08                	je     0x30
  28:	f3 90                	pause  
  2a:*	8b 02                	mov    (%rdx),%eax		<-- trapping instruction
  2c:	84 c0                	test   %al,%al
  2e:	75 f8                	jne    0x28
  30:	b8 01 00 00 00       	mov    $0x1,%eax
  35:	66 89 02             	mov    %ax,(%rdx)
  38:	c3                   	retq   
  39:	8b 37                	mov    (%rdi),%esi
  3b:	b9 00 02 00 00       	mov    $0x200,%ecx

Code starting with the faulting instruction
===========================================
   0:	8b 02                	mov    (%rdx),%eax
   2:	84 c0                	test   %al,%al
   4:	75 f8                	jne    0xfffffffffffffffe
   6:	b8 01 00 00 00       	mov    $0x1,%eax
   b:	66 89 02             	mov    %ax,(%rdx)
   e:	c3                   	retq   
   f:	8b 37                	mov    (%rdi),%esi
  11:	b9 00 02 00 00       	mov    $0x200,%ecx
[218230.055456] RSP: 0018:ffffa859c0003bc0 EFLAGS: 00000202
[218230.055460] RAX: 0000000000000101 RBX: 0000000000000001 RCX: 0000000000000000
[218230.055463] RDX: ffff9195c2772d88 RSI: 0000000000000000 RDI: ffff9195c2772d88
[218230.055466] RBP: ffff9195c2772d00 R08: 00000000000067b0 R09: c6e31da9eb1e44f4
[218230.055469] R10: ffff9195ef379700 R11: ffff9195edb50710 R12: ffff9195c2772d88
[218230.055472] R13: ffff9195f500e3d0 R14: ffff9195ef379700 R15: ffff9195ef379700
[218230.055476] FS:  0000000000000000(0000) GS:ffff91961f400000(0000) knlGS:0000000000000000
[218230.055479] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[218230.055483] CR2: 000000c000407000 CR3: 0000000002988000 CR4: 00000000000006f0
[218230.055487] Call Trace:
[218230.055492]  <IRQ>
[218230.055495] _raw_spin_lock_bh (./arch/x86/include/asm/paravirt.h:554 ./arch/x86/include/asm/qspinlock.h:51 ./include/asm-generic/qspinlock.h:85 ./include/linux/spinlock.h:183 ./include/linux/spinlock_api_smp.h:136 kernel/locking/spinlock.c:175) 
[218230.055503] subflow_error_report (net/mptcp/subflow.c:1164) 
[218230.055513] mptcp_subflow_data_available (net/mptcp/subflow.c:1054 net/mptcp/subflow.c:1075) 
[218230.055519] __mptcp_move_skbs_from_subflow (./arch/x86/include/asm/atomic.h:29 (discriminator 3) ./include/asm-generic/atomic-instrumented.h:28 (discriminator 3) net/mptcp/protocol.c:625 (discriminator 3)) 
[218230.055526] mptcp_data_ready (net/mptcp/protocol.c:693 net/mptcp/protocol.c:736) 
[218230.055533] tcp_data_queue (net/ipv4/tcp_input.c:4966) 
[218230.055539] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[218230.055543] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[218230.055548] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[218230.055553] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[218230.055557] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[218230.055562] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[218230.055567] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[218230.055571] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[218230.055577] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[218230.055582] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[218230.055594] __napi_poll (net/core/dev.c:6913) 
[218230.055598] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[218230.055603] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[218230.055608] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[218230.055613] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 14)) 
[218230.055619]  </IRQ>
[218230.055621] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[218230.055626] RIP: 0010:native_safe_halt (./arch/x86/include/asm/irqflags.h:52) 
[218230.055631] Code: c0 7b 01 00 3e 80 4a 02 20 48 8b 12 83 e2 08 75 c3 e9 7a ff ff ff cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 58 41 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b6 58 41 00 f4 c3 cc cc 0f 1f 44 00
All code
========
   0:	c0 7b 01 00          	sarb   $0x0,0x1(%rbx)
   4:	3e 80 4a 02 20       	orb    $0x20,%ds:0x2(%rdx)
   9:	48 8b 12             	mov    (%rdx),%rdx
   c:	83 e2 08             	and    $0x8,%edx
   f:	75 c3                	jne    0xffffffffffffffd4
  11:	e9 7a ff ff ff       	jmpq   0xffffffffffffff90
  16:	cc                   	int3   
  17:	cc                   	int3   
  18:	cc                   	int3   
  19:	cc                   	int3   
  1a:	cc                   	int3   
  1b:	cc                   	int3   
  1c:	e9 07 00 00 00       	jmpq   0x28
  21:	0f 00 2d c6 58 41 00 	verw   0x4158c6(%rip)        # 0x4158ee
  28:	fb                   	sti    
  29:	f4                   	hlt    
  2a:*	c3                   	retq   		<-- trapping instruction
  2b:	90                   	nop
  2c:	e9 07 00 00 00       	jmpq   0x38
  31:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158ee
  38:	f4                   	hlt    
  39:	c3                   	retq   
  3a:	cc                   	int3   
  3b:	cc                   	int3   
  3c:	0f                   	.byte 0xf
  3d:	1f                   	(bad)  
  3e:	44                   	rex.R
	...

Code starting with the faulting instruction
===========================================
   0:	c3                   	retq   
   1:	90                   	nop
   2:	e9 07 00 00 00       	jmpq   0xe
   7:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158c4
   e:	f4                   	hlt    
   f:	c3                   	retq   
  10:	cc                   	int3   
  11:	cc                   	int3   
  12:	0f                   	.byte 0xf
  13:	1f                   	(bad)  
  14:	44                   	rex.R
	...
[218230.055635] RSP: 0018:ffffffffafa03e40 EFLAGS: 00000246
[218230.055639] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff91961f42b180
[218230.055642] RDX: ffff91961f400000 RSI: ffff9195c1802000 RDI: ffff9195c1802064
[218230.055645] RBP: ffff9195c1802064 R08: ffffffffafc63ba0 R09: 0000000000000008
[218230.055647] R10: 0000000000000211 R11: 000000000000017f R12: 0000000000000001
[218230.055650] R13: ffffffffafc63c20 R14: 0000000000000001 R15: 0000000000000000
[218230.055655] acpi_idle_do_entry (./arch/x86/include/asm/paravirt.h:150 drivers/acpi/processor_idle.c:111 drivers/acpi/processor_idle.c:517) 
[218230.055659] acpi_idle_enter (drivers/acpi/processor_idle.c:654) 
[218230.055666] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:237) 
[218230.055673] cpuidle_enter (drivers/cpuidle/cpuidle.c:353) 
[218230.055677] do_idle (kernel/sched/idle.c:158 kernel/sched/idle.c:239 kernel/sched/idle.c:300) 
[218230.055683] cpu_startup_entry (kernel/sched/idle.c:396 (discriminator 1)) 
[218230.055686] start_kernel (init/main.c:1066) 
[218230.055695] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:283) 
[218230.055704] Kernel panic - not syncing: softlockup: hung tasks
[218230.055756] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G        W    L   --------- ---  5.12.0-198.local4.fc35.x86_64 #1
[218230.055840] Hardware name:  /8I945GZME-RH, BIOS F6 08/29/2006
[218230.055884] Call Trace:
[218230.055906]  <IRQ>
[218230.055924] dump_stack (lib/dump_stack.c:122) 
[218230.055958] panic (kernel/panic.c:239) 
[218230.055987] watchdog_timer_fn.cold (kernel/watchdog.c:421) 
[218230.056024] ? lockup_detector_update_enable (kernel/watchdog.c:343) 
[218230.056067] __hrtimer_run_queues (kernel/time/hrtimer.c:1537 kernel/time/hrtimer.c:1601) 
[218230.056105] hrtimer_interrupt (kernel/time/hrtimer.c:603 kernel/time/hrtimer.c:1666) 
[218230.056140] __sysvec_apic_timer_interrupt (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./arch/x86/include/asm/trace/irq_vectors.h:41 arch/x86/kernel/apic/apic.c:1107) 
[218230.056182] sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1100 (discriminator 13)) 
[218230.056221] asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:632) 
[218230.056263] RIP: 0010:native_queued_spin_lock_slowpath (kernel/locking/qspinlock.c:382 kernel/locking/qspinlock.c:315) 
[218230.056312] Code: 0f ba 2a 08 0f 92 c1 8b 02 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 0f 85 11 01 00 00 85 c0 74 0e 8b 02 84 c0 74 08 f3 90 <8b> 02 84 c0 75 f8 b8 01 00 00 00 66 89 02 c3 8b 37 b9 00 02 00 00
All code
========
   0:	0f ba 2a 08          	btsl   $0x8,(%rdx)
   4:	0f 92 c1             	setb   %cl
   7:	8b 02                	mov    (%rdx),%eax
   9:	0f b6 c9             	movzbl %cl,%ecx
   c:	c1 e1 08             	shl    $0x8,%ecx
   f:	30 e4                	xor    %ah,%ah
  11:	09 c8                	or     %ecx,%eax
  13:	a9 00 01 ff ff       	test   $0xffff0100,%eax
  18:	0f 85 11 01 00 00    	jne    0x12f
  1e:	85 c0                	test   %eax,%eax
  20:	74 0e                	je     0x30
  22:	8b 02                	mov    (%rdx),%eax
  24:	84 c0                	test   %al,%al
  26:	74 08                	je     0x30
  28:	f3 90                	pause  
  2a:*	8b 02                	mov    (%rdx),%eax		<-- trapping instruction
  2c:	84 c0                	test   %al,%al
  2e:	75 f8                	jne    0x28
  30:	b8 01 00 00 00       	mov    $0x1,%eax
  35:	66 89 02             	mov    %ax,(%rdx)
  38:	c3                   	retq   
  39:	8b 37                	mov    (%rdi),%esi
  3b:	b9 00 02 00 00       	mov    $0x200,%ecx

Code starting with the faulting instruction
===========================================
   0:	8b 02                	mov    (%rdx),%eax
   2:	84 c0                	test   %al,%al
   4:	75 f8                	jne    0xfffffffffffffffe
   6:	b8 01 00 00 00       	mov    $0x1,%eax
   b:	66 89 02             	mov    %ax,(%rdx)
   e:	c3                   	retq   
   f:	8b 37                	mov    (%rdi),%esi
  11:	b9 00 02 00 00       	mov    $0x200,%ecx
[218230.056436] RSP: 0018:ffffa859c0003bc0 EFLAGS: 00000202
[218230.056480] RAX: 0000000000000101 RBX: 0000000000000001 RCX: 0000000000000000
[218230.056532] RDX: ffff9195c2772d88 RSI: 0000000000000000 RDI: ffff9195c2772d88
[218230.056583] RBP: ffff9195c2772d00 R08: 00000000000067b0 R09: c6e31da9eb1e44f4
[218230.056634] R10: ffff9195ef379700 R11: ffff9195edb50710 R12: ffff9195c2772d88
[218230.056685] R13: ffff9195f500e3d0 R14: ffff9195ef379700 R15: ffff9195ef379700
[218230.056738] _raw_spin_lock_bh (./arch/x86/include/asm/paravirt.h:554 ./arch/x86/include/asm/qspinlock.h:51 ./include/asm-generic/qspinlock.h:85 ./include/linux/spinlock.h:183 ./include/linux/spinlock_api_smp.h:136 kernel/locking/spinlock.c:175) 
[218230.056772] subflow_error_report (net/mptcp/subflow.c:1164) 
[218230.056808] mptcp_subflow_data_available (net/mptcp/subflow.c:1054 net/mptcp/subflow.c:1075) 
[218230.056850] __mptcp_move_skbs_from_subflow (./arch/x86/include/asm/atomic.h:29 (discriminator 3) ./include/asm-generic/atomic-instrumented.h:28 (discriminator 3) net/mptcp/protocol.c:625 (discriminator 3)) 
[218230.056893] mptcp_data_ready (net/mptcp/protocol.c:693 net/mptcp/protocol.c:736) 
[218230.056926] tcp_data_queue (net/ipv4/tcp_input.c:4966) 
[218230.056959] tcp_rcv_established (./include/linux/skbuff.h:1793 ./include/net/tcp.h:1810 ./include/net/tcp.h:1883 net/ipv4/tcp_input.c:5388 net/ipv4/tcp_input.c:5883) 
[218230.056994] ? tcp_v4_inbound_md5_hash (net/ipv4/tcp_ipv4.c:1401) 
[218230.057033] tcp_v4_do_rcv (net/ipv4/tcp_ipv4.c:1722) 
[218230.057065] tcp_v4_rcv (net/ipv4/tcp_ipv4.c:2063) 
[218230.057095] ? nf_hook_slow (./include/linux/netfilter.h:136 net/netfilter/core.c:589) 
[218230.057127] ip_protocol_deliver_rcu (net/ipv4/ip_input.c:204 (discriminator 1)) 
[218230.057163] ip_local_deliver_finish (./include/linux/rcupdate.h:74 ./include/linux/rcupdate.h:709 net/ipv4/ip_input.c:232) 
[218230.057199] __netif_receive_skb_one_core (net/core/dev.c:5384 (discriminator 4)) 
[218230.057238] netif_receive_skb (net/core/dev.c:5603 net/core/dev.c:5662) 
[218230.057272] rtl8139_poll (drivers/net/ethernet/realtek/8139too.c:2061 drivers/net/ethernet/realtek/8139too.c:2135) 8139too
[218230.057316] __napi_poll (net/core/dev.c:6913) 
[218230.057350] net_rx_action (net/core/dev.c:6982 net/core/dev.c:7067) 
[218230.057382] __do_softirq (./arch/x86/include/asm/jump_label.h:25 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:346) 
[218230.058758] __irq_exit_rcu (kernel/softirq.c:221 kernel/softirq.c:422) 
[218230.060127] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 14)) 
[218230.061483]  </IRQ>
[218230.062814] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:623) 
[218230.064127] RIP: 0010:native_safe_halt (./arch/x86/include/asm/irqflags.h:52) 
[218230.065419] Code: c0 7b 01 00 3e 80 4a 02 20 48 8b 12 83 e2 08 75 c3 e9 7a ff ff ff cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 58 41 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b6 58 41 00 f4 c3 cc cc 0f 1f 44 00
All code
========
   0:	c0 7b 01 00          	sarb   $0x0,0x1(%rbx)
   4:	3e 80 4a 02 20       	orb    $0x20,%ds:0x2(%rdx)
   9:	48 8b 12             	mov    (%rdx),%rdx
   c:	83 e2 08             	and    $0x8,%edx
   f:	75 c3                	jne    0xffffffffffffffd4
  11:	e9 7a ff ff ff       	jmpq   0xffffffffffffff90
  16:	cc                   	int3   
  17:	cc                   	int3   
  18:	cc                   	int3   
  19:	cc                   	int3   
  1a:	cc                   	int3   
  1b:	cc                   	int3   
  1c:	e9 07 00 00 00       	jmpq   0x28
  21:	0f 00 2d c6 58 41 00 	verw   0x4158c6(%rip)        # 0x4158ee
  28:	fb                   	sti    
  29:	f4                   	hlt    
  2a:*	c3                   	retq   		<-- trapping instruction
  2b:	90                   	nop
  2c:	e9 07 00 00 00       	jmpq   0x38
  31:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158ee
  38:	f4                   	hlt    
  39:	c3                   	retq   
  3a:	cc                   	int3   
  3b:	cc                   	int3   
  3c:	0f                   	.byte 0xf
  3d:	1f                   	(bad)  
  3e:	44                   	rex.R
	...

Code starting with the faulting instruction
===========================================
   0:	c3                   	retq   
   1:	90                   	nop
   2:	e9 07 00 00 00       	jmpq   0xe
   7:	0f 00 2d b6 58 41 00 	verw   0x4158b6(%rip)        # 0x4158c4
   e:	f4                   	hlt    
   f:	c3                   	retq   
  10:	cc                   	int3   
  11:	cc                   	int3   
  12:	0f                   	.byte 0xf
  13:	1f                   	(bad)  
  14:	44                   	rex.R
	...
[218230.068120] RSP: 0018:ffffffffafa03e40 EFLAGS: 00000246
[218230.069474] RAX: 0000000000004000 RBX: 0000000000000001 RCX: ffff91961f42b180
[218230.070842] RDX: ffff91961f400000 RSI: ffff9195c1802000 RDI: ffff9195c1802064
[218230.072238] RBP: ffff9195c1802064 R08: ffffffffafc63ba0 R09: 0000000000000008
[218230.073657] R10: 0000000000000211 R11: 000000000000017f R12: 0000000000000001
[218230.075096] R13: ffffffffafc63c20 R14: 0000000000000001 R15: 0000000000000000
[218230.076540] acpi_idle_do_entry (./arch/x86/include/asm/paravirt.h:150 drivers/acpi/processor_idle.c:111 drivers/acpi/processor_idle.c:517) 
[218230.077976] acpi_idle_enter (drivers/acpi/processor_idle.c:654) 
[218230.079403] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:237) 
[218230.080835] cpuidle_enter (drivers/cpuidle/cpuidle.c:353) 
[218230.082230] do_idle (kernel/sched/idle.c:158 kernel/sched/idle.c:239 kernel/sched/idle.c:300) 
[218230.083635] cpu_startup_entry (kernel/sched/idle.c:396 (discriminator 1)) 
[218230.085034] start_kernel (init/main.c:1066) 
[218230.086426] secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:283)

@pabeni
Copy link

pabeni commented Jun 1, 2021

Hello,

The client box got stuck on me this morning, same boot as above. But I came prepared, here is dmesg (direct continuation from the [167849.044884] mark above).

Oh, this looks like an unrelated (meaning: different root cause) issue. Even if the bugged code is likely triggered by the warn bad_map above.

I'll file a different issue to track the problems independently.

I've captured some splats -- on the client side this time. With
[...]
tcp: ensure that backlog coalescing don't break MPTCP DSS

You can drop this one, should not help.

Instead you can try pulling into your tree:

commit dea2b1e
Author: Paolo Abeni pabeni@redhat.com
Date: Thu May 27 16:31:39 2021 -0700

mptcp: do not reset MP_CAPABLE subflow on mapping errors

from Dave'm -net tree

That should not avoid the splat, but avoid unneeded subflow reset on bad mapping events, which in turn should avoid the experienced connections hang.

@pabeni
Copy link

pabeni commented Jun 1, 2021 •
edited
Loading

Oh, this looks like an unrelated (meaning: different root cause) issue. Even if the bugged code is likely triggered by the warn bad_map above.

I'll file a different issue to track the problems independently.

see #199

@pabeni
Copy link

pabeni commented Jun 15, 2021

this should be addressed by the upstream commit:

commit 61e7102
Author: Paolo Abeni pabeni@redhat.com
Date: Thu Jun 10 15:59:42 2021 -0700

mptcp: do not warn on bad input from the network

@pabeni pabeni closed this as completed Jun 15, 2021
@pabeni pabeni mentioned this issue Jun 17, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pabeni pabeni

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@darkwrat @pabeni