-
Notifications
You must be signed in to change notification settings - Fork 20
/
index.html
4565 lines (2435 loc) · 520 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html class="theme-next mist use-motion" lang="zh-Hans">
<head><meta name="generator" content="Hexo 3.9.0">
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<meta http-equiv="Cache-Control" content="no-transform">
<meta http-equiv="Cache-Control" content="no-siteapp">
<link href="/lib/fancybox/source/jquery.fancybox.css?v=2.1.5" rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=Lato:300,300italic,400,400italic,700,700italic&subset=latin,latin-ext" rel="stylesheet" type="text/css">
<link href="/lib/font-awesome/css/font-awesome.min.css?v=4.6.2" rel="stylesheet" type="text/css">
<link href="/css/main.css?v=5.1.1" rel="stylesheet" type="text/css">
<meta name="keywords" content="Hexo, NexT">
<link rel="alternate" href="/atom.xml" title="plantegg" type="application/atom+xml">
<link rel="shortcut icon" type="image/x-icon" href="/favicon.ico?v=5.1.1">
<meta name="description" content="java mysql tcp performance network docker Linux">
<meta property="og:type" content="website">
<meta property="og:title" content="plantegg">
<meta property="og:url" content="https://plantegg.github.io/index.html">
<meta property="og:site_name" content="plantegg">
<meta property="og:description" content="java mysql tcp performance network docker Linux">
<meta property="og:locale" content="zh-Hans">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="plantegg">
<meta name="twitter:description" content="java mysql tcp performance network docker Linux">
<script type="text/javascript" id="hexo.configurations">
var NexT = window.NexT || {};
var CONFIG = {
root: '/',
scheme: 'Mist',
sidebar: {"position":"left","display":"post","offset":12,"offset_float":0,"b2t":false,"scrollpercent":false},
fancybox: true,
motion: true,
duoshuo: {
userId: '0',
author: '博主'
},
algolia: {
applicationID: '',
apiKey: '',
indexName: '',
hits: {"per_page":10},
labels: {"input_placeholder":"Search for Posts","hits_empty":"We didn't find any results for the search: ${query}","hits_stats":"${hits} results found in ${time} ms"}
}
};
</script>
<link rel="canonical" href="https://plantegg.github.io/">
<title>plantegg - java tcp mysql performance network docker Linux</title>
</head>
<body itemscope itemtype="http://schema.org/WebPage" lang="zh-Hans">
<div class="container sidebar-position-left
page-home
">
<div class="headband"></div>
<header id="header" class="header" itemscope itemtype="http://schema.org/WPHeader">
<div class="header-inner"><div class="site-brand-wrapper">
<div class="site-meta custom-logo">
<div class="custom-logo-site-title">
<a href="/" class="brand" rel="start">
<span class="logo-line-before"><i></i></span>
<span class="site-title">plantegg</span>
<span class="logo-line-after"><i></i></span>
</a>
</div>
<h1 class="site-subtitle" itemprop="description">java tcp mysql performance network docker Linux</h1>
</div>
<div class="site-nav-toggle">
<button>
<span class="btn-bar"></span>
<span class="btn-bar"></span>
<span class="btn-bar"></span>
</button>
</div>
</div>
<nav class="site-nav">
<ul id="menu" class="menu">
<li class="menu-item menu-item-home">
<a href="/" rel="section">
<i class="menu-item-icon fa fa-fw fa-home"></i> <br>
首页
</a>
</li>
<li class="menu-item menu-item-categories">
<a href="/categories" rel="section">
<i class="menu-item-icon fa fa-fw fa-categories"></i> <br>
分类
</a>
</li>
<li class="menu-item menu-item-archives">
<a href="/archives" rel="section">
<i class="menu-item-icon fa fa-fw fa-archive"></i> <br>
归档
</a>
</li>
<li class="menu-item menu-item-tags">
<a href="/tags" rel="section">
<i class="menu-item-icon fa fa-fw fa-tags"></i> <br>
标签
</a>
</li>
<li class="menu-item menu-item-sitemap">
<a href="/sitemap.xml" rel="section">
<i class="menu-item-icon fa fa-fw fa-sitemap"></i> <br>
站点地图
</a>
</li>
<li class="menu-item menu-item-about">
<a href="/about" rel="section">
<i class="menu-item-icon fa fa-fw fa-user"></i> <br>
关于
</a>
</li>
</ul>
</nav>
</div>
</header>
<main id="main" class="main">
<div class="main-inner">
<div class="content-wrap">
<div id="content" class="content">
<section id="posts" class="posts-expand">
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="https://plantegg.github.io/2117/06/07/关于本博/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="twitter @plantegg">
<meta itemprop="description" content>
<meta itemprop="image" content="/images/avatar.gif">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="plantegg">
</span>
<header class="post-header">
<h2 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2117/06/07/关于本博/" itemprop="url">关于本博</a></h2>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2117-06-07T18:30:03+08:00">
2117-06-07
</time>
</span>
<span class="post-category">
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/others/" itemprop="url" rel="index">
<span itemprop="name">others</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<h2 id="关于本博"><a href="#关于本博" class="headerlink" title="关于本博"></a>关于本博</h2><p>find me on twitter: <a href="https://twitter.com/plantegg" target="_blank" rel="noopener">@plantegg</a></p>
<p>Github: <a href="https://github.com/plantegg/programmer_case" target="_blank" rel="noopener">欢迎star</a> </p>
<p>知识星球：<a href="https://t.zsxq.com/0cSFEUh2J" target="_blank" rel="noopener">https://t.zsxq.com/0cSFEUh2J</a></p>
<p>关注基础知识，一次把问题搞清楚，从案例出发深挖相关知识。</p>
<p>以前觉得自己一看就懂，实际是一问就打鼓，一用就糊涂。所以现在开始记录并总结再联系案例，一般是先把零散知识记录下来（看到过），慢慢地相关知识积累更多，直到碰到实践案例或是有点领悟到于是发现这块知识可以整理成一篇系统些的文章（基本快懂了）。</p>
<p>“技术变化太快，容易过时”，我的看法是网络知识、操作系统、计算机原理等核心概念知识的寿命会比你的职业生涯还长。这些都是40岁之后还会还会很有用</p>
<p><a href="https://plantegg.github.io/2018/05/23/%E5%A6%82%E4%BD%95%E5%9C%A8%E5%B7%A5%E4%BD%9C%E4%B8%AD%E5%AD%A6%E4%B9%A0/">如何在工作中学习</a> 所有方法我都记录在这篇文章中了，希望对你能有所帮助。</p>
<p>所有新文章从<a href="https://plantegg.github.io/archives">这里可以看到</a>，即使再简单的一篇总结我可以持续总结三五年，有新的发现、感悟都是直接在原文上增减，不会发表新的文章。</p>
<p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20220421102225491.png" alt="image-20220421102225491"></p>
<p>为什么写博客而不是公众号，我见过20年前的互联网，深度依赖搜索引擎，所以还是喜欢博客。另外技术类文章更适合电脑阅读（随时摘录、实验）</p>
<h2 id="精华文章推荐（2021年前）"><a href="#精华文章推荐（2021年前）" class="headerlink" title="精华文章推荐（2021年前）"></a>精华文章推荐（2021年前）</h2><h4 id="在2010年前后MySQL、PG、Oracle数据库在使用NUMA的时候碰到了性能问题，流传最广的这篇-MySQL-–-The-MySQL-“swap-insanity”-problem-and-the-effects-of-the-NUMA-architecture-http-blog-jcole-us-2010-09-28-mysql-swap-insanity-and-the-numa-architecture-文章描述了性能问题的原因-文章中把原因找错了-以及解决方案：关闭NUMA。-实际这个原因是kernel实现的一个低级bug，这个Bug在2014年修复了https-github-com-torvalds-linux-commit-4f9b16a64753d0bb607454347036dc997fd03b82，但是修复这么多年后仍然以讹传讹，这篇文章希望正本清源、扭转错误的认识。"><a href="#在2010年前后MySQL、PG、Oracle数据库在使用NUMA的时候碰到了性能问题，流传最广的这篇-MySQL-–-The-MySQL-“swap-insanity”-problem-and-the-effects-of-the-NUMA-architecture-http-blog-jcole-us-2010-09-28-mysql-swap-insanity-and-the-numa-architecture-文章描述了性能问题的原因-文章中把原因找错了-以及解决方案：关闭NUMA。-实际这个原因是kernel实现的一个低级bug，这个Bug在2014年修复了https-github-com-torvalds-linux-commit-4f9b16a64753d0bb607454347036dc997fd03b82，但是修复这么多年后仍然以讹传讹，这篇文章希望正本清源、扭转错误的认识。" class="headerlink" title="在2010年前后MySQL、PG、Oracle数据库在使用NUMA的时候碰到了性能问题，流传最广的这篇 MySQL – The MySQL “swap insanity” problem and the effects of the NUMA architecture http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ 文章描述了性能问题的原因(文章中把原因找错了)以及解决方案：关闭NUMA。 实际这个原因是kernel实现的一个低级bug，这个Bug在2014年修复了https://github.com/torvalds/linux/commit/4f9b16a64753d0bb607454347036dc997fd03b82，但是修复这么多年后仍然以讹传讹，这篇文章希望正本清源、扭转错误的认识。"></a><a href="https://plantegg.github.io/2021/05/14/%E5%8D%81%E5%B9%B4%E5%90%8E%E6%95%B0%E6%8D%AE%E5%BA%93%E8%BF%98%E6%98%AF%E4%B8%8D%E6%95%A2%E6%8B%A5%E6%8A%B1NUMA/">在2010年前后MySQL、PG、Oracle数据库在使用NUMA的时候碰到了性能问题，流传最广的这篇 MySQL – The MySQL “swap insanity” problem and the effects of the NUMA architecture http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ 文章描述了性能问题的原因(文章中把原因找错了)以及解决方案：关闭NUMA。 实际这个原因是kernel实现的一个低级bug，这个Bug在2014年修复了https://github.com/torvalds/linux/commit/4f9b16a64753d0bb607454347036dc997fd03b82，但是修复这么多年后仍然以讹传讹，这篇文章希望正本清源、扭转错误的认识。</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20210517082233798.png" alt="image-20210517082233798"></p>
<h4 id="CPU的制造和概念-从最底层的沙子开始用8篇文章来回答关于CPU的各种疑问以及大量的实验对比案例和测试数据来展示了CPU的各种原理，比如多核、超线程、NUMA、睿频、功耗、GPU、大小核再到分支预测、cache-line失效、加锁代价、IPC等各种指标（都有对应的代码和测试数据）。"><a href="#CPU的制造和概念-从最底层的沙子开始用8篇文章来回答关于CPU的各种疑问以及大量的实验对比案例和测试数据来展示了CPU的各种原理，比如多核、超线程、NUMA、睿频、功耗、GPU、大小核再到分支预测、cache-line失效、加锁代价、IPC等各种指标（都有对应的代码和测试数据）。" class="headerlink" title="CPU的制造和概念 从最底层的沙子开始用8篇文章来回答关于CPU的各种疑问以及大量的实验对比案例和测试数据来展示了CPU的各种原理，比如多核、超线程、NUMA、睿频、功耗、GPU、大小核再到分支预测、cache_line失效、加锁代价、IPC等各种指标（都有对应的代码和测试数据）。"></a><a href="https://plantegg.github.io/2021/06/01/CPU%E7%9A%84%E5%88%B6%E9%80%A0%E5%92%8C%E6%A6%82%E5%BF%B5/">CPU的制造和概念</a> 从最底层的沙子开始用8篇文章来回答关于CPU的各种疑问以及大量的实验对比案例和测试数据来展示了CPU的各种原理，比如多核、超线程、NUMA、睿频、功耗、GPU、大小核再到分支预测、cache_line失效、加锁代价、IPC等各种指标（都有对应的代码和测试数据）。</h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20210802161410524-1011377.png" alt="image-20210802161410524"> </p>
<h4 id="《Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的》-从一个参数引起的rt抖动定位到OS锁等待再到CPU-Pause指令，以及不同CPU型号对Pause使用cycles不同的影响，最终反馈到应用层面的rt全过程。在MySQL内核开发的时候考虑了Pause，但是没有考虑不同的CPU型号，所以换了CPU型号后性能差异比较大"><a href="#《Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的》-从一个参数引起的rt抖动定位到OS锁等待再到CPU-Pause指令，以及不同CPU型号对Pause使用cycles不同的影响，最终反馈到应用层面的rt全过程。在MySQL内核开发的时候考虑了Pause，但是没有考虑不同的CPU型号，所以换了CPU型号后性能差异比较大" class="headerlink" title="《Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的》 从一个参数引起的rt抖动定位到OS锁等待再到CPU Pause指令，以及不同CPU型号对Pause使用cycles不同的影响，最终反馈到应用层面的rt全过程。在MySQL内核开发的时候考虑了Pause，但是没有考虑不同的CPU型号，所以换了CPU型号后性能差异比较大"></a><a href="https://plantegg.github.io/2019/12/16/Intel%20PAUSE%E6%8C%87%E4%BB%A4%E5%8F%98%E5%8C%96%E6%98%AF%E5%A6%82%E4%BD%95%E5%BD%B1%E5%93%8D%E8%87%AA%E6%97%8B%E9%94%81%E4%BB%A5%E5%8F%8AMySQL%E7%9A%84%E6%80%A7%E8%83%BD%E7%9A%84/">《Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的》 从一个参数引起的rt抖动定位到OS锁等待再到CPU Pause指令，以及不同CPU型号对Pause使用cycles不同的影响，最终反馈到应用层面的rt全过程。在MySQL内核开发的时候考虑了Pause，但是没有考虑不同的CPU型号，所以换了CPU型号后性能差异比较大</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/d567449fe52725a9d0b9d4ec9baa372c.png" alt="image.png"></p>
<h4 id="10倍性能提升全过程-在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，一次全栈性能优化过程的详细记录和分析。"><a href="#10倍性能提升全过程-在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，一次全栈性能优化过程的详细记录和分析。" class="headerlink" title="10倍性能提升全过程 在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，一次全栈性能优化过程的详细记录和分析。"></a><a href="https://plantegg.github.io/2018/01/23/10+%E5%80%8D%E6%80%A7%E8%83%BD%E6%8F%90%E5%8D%87%E5%85%A8%E8%BF%87%E7%A8%8B/">10倍性能提升全过程</a> 在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，一次全栈性能优化过程的详细记录和分析。</h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/05703c168e63e96821ea9f921d83712b.png" alt="image.png"></p>
<h4 id="就是要你懂TCP–半连接队列和全连接队列：偶发性的连接reset异常、重启服务后短时间的连接异常，通过一篇文章阐明TCP连接的半连接队列和全连接队大小是怎么影响连接创建的，以及用什么工具来观察队列有没有溢出、连接为什么会RESET"><a href="#就是要你懂TCP–半连接队列和全连接队列：偶发性的连接reset异常、重启服务后短时间的连接异常，通过一篇文章阐明TCP连接的半连接队列和全连接队大小是怎么影响连接创建的，以及用什么工具来观察队列有没有溢出、连接为什么会RESET" class="headerlink" title="就是要你懂TCP–半连接队列和全连接队列：偶发性的连接reset异常、重启服务后短时间的连接异常，通过一篇文章阐明TCP连接的半连接队列和全连接队大小是怎么影响连接创建的，以及用什么工具来观察队列有没有溢出、连接为什么会RESET"></a><a href="https://plantegg.github.io/2017/06/07/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E5%8D%8A%E8%BF%9E%E6%8E%A5%E9%98%9F%E5%88%97%E5%92%8C%E5%85%A8%E8%BF%9E%E6%8E%A5%E9%98%9F%E5%88%97/">就是要你懂TCP–半连接队列和全连接队列：偶发性的连接reset异常、重启服务后短时间的连接异常，通过一篇文章阐明TCP连接的半连接队列和全连接队大小是怎么影响连接创建的，以及用什么工具来观察队列有没有溢出、连接为什么会RESET</a></h4><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/1579241362064-807d8378-6c54-4a2c-a888-ff2337df817c.png" alt="image.png" style="zoom:80%;">
<h4 id="就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的"><a href="#就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的" class="headerlink" title="就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的"></a><a href="https://plantegg.github.io/2019/09/28/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E6%80%A7%E8%83%BD%E5%92%8C%E5%8F%91%E9%80%81%E6%8E%A5%E6%94%B6Buffer%E7%9A%84%E5%85%B3%E7%B3%BB/">就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/e177d59ecb886daef5905ed80a84dfd2.png"></p>
<h4 id="就是要你懂网络–一个网络包的旅程：教科书式地阐述书本中的路由、网关、子网、Mac地址、IP地址是如何一起协作让网络包最终传输到目标机器上。-同时可以跟讲这块的RFC1180比较一下，RFC1180-写的确实很好，清晰简洁，图文并茂，结构逻辑合理，但是对于90-的程序员没有什么卵用，看完几周后就忘得差不多，因为他不是从实践的角度来阐述问题，中间没有很多为什么，所以一般资质的程序员看完当时感觉很好，实际还是不会灵活运用"><a href="#就是要你懂网络–一个网络包的旅程：教科书式地阐述书本中的路由、网关、子网、Mac地址、IP地址是如何一起协作让网络包最终传输到目标机器上。-同时可以跟讲这块的RFC1180比较一下，RFC1180-写的确实很好，清晰简洁，图文并茂，结构逻辑合理，但是对于90-的程序员没有什么卵用，看完几周后就忘得差不多，因为他不是从实践的角度来阐述问题，中间没有很多为什么，所以一般资质的程序员看完当时感觉很好，实际还是不会灵活运用" class="headerlink" title="就是要你懂网络–一个网络包的旅程：教科书式地阐述书本中的路由、网关、子网、Mac地址、IP地址是如何一起协作让网络包最终传输到目标机器上。 同时可以跟讲这块的RFC1180比较一下，RFC1180 写的确实很好，清晰简洁，图文并茂，结构逻辑合理，但是对于90%的程序员没有什么卵用，看完几周后就忘得差不多，因为他不是从实践的角度来阐述问题，中间没有很多为什么，所以一般资质的程序员看完当时感觉很好，实际还是不会灵活运用"></a><a href="https://plantegg.github.io/2019/05/15/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E7%BD%91%E7%BB%9C--%E4%B8%80%E4%B8%AA%E7%BD%91%E7%BB%9C%E5%8C%85%E7%9A%84%E6%97%85%E7%A8%8B/">就是要你懂网络–一个网络包的旅程：教科书式地阐述书本中的路由、网关、子网、Mac地址、IP地址是如何一起协作让网络包最终传输到目标机器上。</a> 同时可以跟讲这块的<a href="https://tools.ietf.org/html/rfc1180" target="_blank" rel="noopener">RFC1180</a>比较一下，RFC1180 写的确实很好，清晰简洁，图文并茂，结构逻辑合理，但是对于90%的程序员没有什么卵用，看完几周后就忘得差不多，因为他不是从实践的角度来阐述问题，中间没有很多为什么，所以一般资质的程序员看完当时感觉很好，实际还是不会灵活运用</h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/8f5d8518c1d92ed68d23218028e3cd11.png"></p>
<h4 id="国产CPU和Intel、AMD性能PK-从Intel、AMD、海光、鲲鹏920、飞腾2500-等CPU在TPCC、sysbench下的性能对比来分析他们的性能差距，同时分析内存延迟对性能的影响"><a href="#国产CPU和Intel、AMD性能PK-从Intel、AMD、海光、鲲鹏920、飞腾2500-等CPU在TPCC、sysbench下的性能对比来分析他们的性能差距，同时分析内存延迟对性能的影响" class="headerlink" title="国产CPU和Intel、AMD性能PK 从Intel、AMD、海光、鲲鹏920、飞腾2500 等CPU在TPCC、sysbench下的性能对比来分析他们的性能差距，同时分析内存延迟对性能的影响"></a><a href="https://plantegg.github.io/2022/01/13/%E4%B8%8D%E5%90%8CCPU%E6%80%A7%E8%83%BD%E5%A4%A7PK/">国产CPU和Intel、AMD性能PK</a> 从Intel、AMD、海光、鲲鹏920、飞腾2500 等CPU在TPCC、sysbench下的性能对比来分析他们的性能差距，同时分析内存延迟对性能的影响</h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20220319115644219.png" alt="image-20220319115644219"></p>
<h4 id="从网络路由连通性的原理上来看负载均衡lvs的DR、NAT、FullNAT到底搞了些什么鬼，以及为什么要这么搞，和带来的优缺点：《就是要你懂负载均衡–lvs和转发模式》"><a href="#从网络路由连通性的原理上来看负载均衡lvs的DR、NAT、FullNAT到底搞了些什么鬼，以及为什么要这么搞，和带来的优缺点：《就是要你懂负载均衡–lvs和转发模式》" class="headerlink" title="从网络路由连通性的原理上来看负载均衡lvs的DR、NAT、FullNAT到底搞了些什么鬼，以及为什么要这么搞，和带来的优缺点：《就是要你懂负载均衡–lvs和转发模式》"></a><a href="/2019/06/20/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1--lvs%E5%92%8C%E8%BD%AC%E5%8F%91%E6%A8%A1%E5%BC%8F/">从网络路由连通性的原理上来看负载均衡lvs的DR、NAT、FullNAT到底搞了些什么鬼，以及为什么要这么搞，和带来的优缺点：《就是要你懂负载均衡–lvs和转发模式》</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/94d55b926b5bb1573c4cab8353428712.png"></p>
<h4 id="LVS-20倍的负载不均衡，原来是内核的这个Bug，这个内核bug现在还在，可以稳定重现，有兴趣的话去重现一下，然后对照源代码以及抓包分析一下就清楚了。"><a href="#LVS-20倍的负载不均衡，原来是内核的这个Bug，这个内核bug现在还在，可以稳定重现，有兴趣的话去重现一下，然后对照源代码以及抓包分析一下就清楚了。" class="headerlink" title="LVS 20倍的负载不均衡，原来是内核的这个Bug，这个内核bug现在还在，可以稳定重现，有兴趣的话去重现一下，然后对照源代码以及抓包分析一下就清楚了。"></a><a href="/2019/07/19/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1--%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1%E8%B0%83%E5%BA%A6%E7%AE%97%E6%B3%95%E5%92%8C%E4%B8%BA%E4%BB%80%E4%B9%88%E4%B8%8D%E5%9D%87%E8%A1%A1/">LVS 20倍的负载不均衡，原来是内核的这个Bug</a>，这个内核bug现在还在，可以稳定重现，有兴趣的话去重现一下，然后对照源代码以及抓包分析一下就清楚了。</h4><h4 id="就是要你懂TCP–握手和挥手，不是你想象中三次握手、四次挥手就理解了TCP，本文从握手的本质–握手都做了什么事情、连接的本质是什么等来阐述握手、挥手的原理"><a href="#就是要你懂TCP–握手和挥手，不是你想象中三次握手、四次挥手就理解了TCP，本文从握手的本质–握手都做了什么事情、连接的本质是什么等来阐述握手、挥手的原理" class="headerlink" title="就是要你懂TCP–握手和挥手，不是你想象中三次握手、四次挥手就理解了TCP，本文从握手的本质–握手都做了什么事情、连接的本质是什么等来阐述握手、挥手的原理"></a><a href="/2017/06/02/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E8%BF%9E%E6%8E%A5%E5%92%8C%E6%8F%A1%E6%89%8B/">就是要你懂TCP–握手和挥手，不是你想象中三次握手、四次挥手就理解了TCP，本文从握手的本质–握手都做了什么事情、连接的本质是什么等来阐述握手、挥手的原理</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/6d66dadecb72e11e3e5ab765c6c3ea2e.png"></p>
<h4 id="nslookup-OK-but-ping-fail–看看老司机是如何解决问题的，解决问题的方法肯定比知识点重要多了，同时透过一个问题怎么样通篇来理解一大块知识，让这块原理真正在你的只是提示中扎根下来"><a href="#nslookup-OK-but-ping-fail–看看老司机是如何解决问题的，解决问题的方法肯定比知识点重要多了，同时透过一个问题怎么样通篇来理解一大块知识，让这块原理真正在你的只是提示中扎根下来" class="headerlink" title="nslookup OK but ping fail–看看老司机是如何解决问题的，解决问题的方法肯定比知识点重要多了，同时透过一个问题怎么样通篇来理解一大块知识，让这块原理真正在你的只是提示中扎根下来"></a><a href="/2019/01/09/nslookup-OK-but-ping-fail/">nslookup OK but ping fail–看看老司机是如何解决问题的，解决问题的方法肯定比知识点重要多了，同时透过一个问题怎么样通篇来理解一大块知识，让这块原理真正在你的只是提示中扎根下来</a></h4><p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/oss/ca466bb6430f1149958ceb41b9ffe591.png"></p>
<h4 id="如何在工作中学习-一篇很土但是很务实可以复制的方法论文章。不要讲举一反三、触类旁通，谁都知道要举一反三、触类旁通，但是为什么我总是不能够举一反三、触类旁通？"><a href="#如何在工作中学习-一篇很土但是很务实可以复制的方法论文章。不要讲举一反三、触类旁通，谁都知道要举一反三、触类旁通，但是为什么我总是不能够举一反三、触类旁通？" class="headerlink" title="如何在工作中学习 一篇很土但是很务实可以复制的方法论文章。不要讲举一反三、触类旁通，谁都知道要举一反三、触类旁通，但是为什么我总是不能够举一反三、触类旁通？"></a><a href="/2018/05/23/%E5%A6%82%E4%BD%95%E5%9C%A8%E5%B7%A5%E4%BD%9C%E4%B8%AD%E5%AD%A6%E4%B9%A0/">如何在工作中学习</a> 一篇很土但是很务实可以复制的方法论文章。不要讲举一反三、触类旁通，谁都知道要举一反三、触类旁通，但是为什么我总是不能够举一反三、触类旁通？</h4><h4 id="举三反一–从理论知识到实际问题的推导-坚决不让思路跑偏，如何从一个理论知识点推断可能的问题"><a href="#举三反一–从理论知识到实际问题的推导-坚决不让思路跑偏，如何从一个理论知识点推断可能的问题" class="headerlink" title="举三反一–从理论知识到实际问题的推导 坚决不让思路跑偏，如何从一个理论知识点推断可能的问题"></a><a href="/2020/11/02/%E4%B8%BE%E4%B8%89%E5%8F%8D%E4%B8%80--%E4%BB%8E%E7%90%86%E8%AE%BA%E7%9F%A5%E8%AF%86%E5%88%B0%E5%AE%9E%E9%99%85%E9%97%AE%E9%A2%98%E7%9A%84%E6%8E%A8%E5%AF%BC/">举三反一–从理论知识到实际问题的推导</a> 坚决不让思路跑偏，如何从一个理论知识点推断可能的问题</h4><h2 id="性能相关（2015-2018年）"><a href="#性能相关（2015-2018年）" class="headerlink" title="性能相关（2015-2018年）"></a>性能相关（2015-2018年）</h2><h4 id="就是要你懂TCP–半连接队列和全连接队列-偶发性的连接reset异常、重启服务后短时间的连接异常"><a href="#就是要你懂TCP–半连接队列和全连接队列-偶发性的连接reset异常、重启服务后短时间的连接异常" class="headerlink" title="就是要你懂TCP–半连接队列和全连接队列 偶发性的连接reset异常、重启服务后短时间的连接异常"></a><a href="/2017/06/07/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E5%8D%8A%E8%BF%9E%E6%8E%A5%E9%98%9F%E5%88%97%E5%92%8C%E5%85%A8%E8%BF%9E%E6%8E%A5%E9%98%9F%E5%88%97/">就是要你懂TCP–半连接队列和全连接队列</a> 偶发性的连接reset异常、重启服务后短时间的连接异常</h4><h4 id="就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的-发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响"><a href="#就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的-发送窗口大小-Buffer-、接收窗口大小-Buffer-对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响" class="headerlink" title="就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的 发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响"></a><a href="/2019/09/28/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E6%80%A7%E8%83%BD%E5%92%8C%E5%8F%91%E9%80%81%E6%8E%A5%E6%94%B6Buffer%E7%9A%84%E5%85%B3%E7%B3%BB/">就是要你懂TCP–性能和发送接收Buffer的关系：发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响。BDP、RT、带宽对传输速度又是怎么影响的</a> 发送窗口大小(Buffer)、接收窗口大小(Buffer)对TCP传输速度的影响，以及怎么观察窗口对传输速度的影响</h4><h4 id="就是要你懂TCP–性能优化大全"><a href="#就是要你懂TCP–性能优化大全" class="headerlink" title="就是要你懂TCP–性能优化大全"></a><a href="/2019/06/21/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E6%80%A7%E8%83%BD%E4%BC%98%E5%8C%96%E5%A4%A7%E5%85%A8/">就是要你懂TCP–性能优化大全</a></h4><h4 id="就是要你懂TCP–TCP性能问题-Nagle算法和delay-ack"><a href="#就是要你懂TCP–TCP性能问题-Nagle算法和delay-ack" class="headerlink" title="就是要你懂TCP–TCP性能问题 Nagle算法和delay ack"></a><a href="/2018/06/14/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E6%9C%80%E7%BB%8F%E5%85%B8%E7%9A%84TCP%E6%80%A7%E8%83%BD%E9%97%AE%E9%A2%98/">就是要你懂TCP–TCP性能问题</a> Nagle算法和delay ack</h4><h4 id="10倍性能提升全过程-在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，看看一个性能全栈工程师如何在各种工具加持下发现各种问题的。"><a href="#10倍性能提升全过程-在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，看看一个性能全栈工程师如何在各种工具加持下发现各种问题的。" class="headerlink" title="10倍性能提升全过程 在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，看看一个性能全栈工程师如何在各种工具加持下发现各种问题的。"></a><a href="/2018/01/23/10+%E5%80%8D%E6%80%A7%E8%83%BD%E6%8F%90%E5%8D%87%E5%85%A8%E8%BF%87%E7%A8%8B/">10倍性能提升全过程</a> 在双11的紧张流程下，将系统tps从500优化到5500，从网络到snat、再到Spring和StackTrace，看看一个性能全栈工程师如何在各种工具加持下发现各种问题的。</h4><h2 id="CPU系列文章（2021年完成）"><a href="#CPU系列文章（2021年完成）" class="headerlink" title="CPU系列文章（2021年完成）"></a>CPU系列文章（2021年完成）</h2><h4 id="CPU的制造和概念"><a href="#CPU的制造和概念" class="headerlink" title="CPU的制造和概念"></a><a href="/2021/06/01/CPU%E7%9A%84%E5%88%B6%E9%80%A0%E5%92%8C%E6%A6%82%E5%BF%B5/">CPU的制造和概念</a></h4><h4 id="十年后数据库还是不敢拥抱NUMA？"><a href="#十年后数据库还是不敢拥抱NUMA？" class="headerlink" title="十年后数据库还是不敢拥抱NUMA？"></a><a href="/2021/05/14/%E5%8D%81%E5%B9%B4%E5%90%8E%E6%95%B0%E6%8D%AE%E5%BA%93%E8%BF%98%E6%98%AF%E4%B8%8D%E6%95%A2%E6%8B%A5%E6%8A%B1NUMA/">十年后数据库还是不敢拥抱NUMA？</a></h4><h4 id="Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的-x2F-2019-x2F-12-x2F-16-x2F-Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的-x2F"><a href="#Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的-x2F-2019-x2F-12-x2F-16-x2F-Intel-PAUSE指令变化是如何影响自旋锁以及MySQL的性能的-x2F" class="headerlink" title="[Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的](/2019/12/16/Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的/)"></a>[Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的](/2019/12/16/Intel PAUSE指令变化是如何影响自旋锁以及MySQL的性能的/)</h4><h4 id="Perf-IPC以及CPU性能-x2F-2021-x2F-05-x2F-16-x2F-Perf-IPC以及CPU利用率-x2F"><a href="#Perf-IPC以及CPU性能-x2F-2021-x2F-05-x2F-16-x2F-Perf-IPC以及CPU利用率-x2F" class="headerlink" title="[Perf IPC以及CPU性能](/2021/05/16/Perf IPC以及CPU利用率/)"></a>[Perf IPC以及CPU性能](/2021/05/16/Perf IPC以及CPU利用率/)</h4><h4 id="CPU性能和CACHE"><a href="#CPU性能和CACHE" class="headerlink" title="CPU性能和CACHE"></a><a href="https://plantegg.github.io/2021/07/19/CPU%E6%80%A7%E8%83%BD%E5%92%8CCACHE/">CPU性能和CACHE</a></h4><h4 id="CPU-性能和Cache-Line-x2F-2021-x2F-05-x2F-16-x2F-CPU-Cache-Line-和性能-x2F"><a href="#CPU-性能和Cache-Line-x2F-2021-x2F-05-x2F-16-x2F-CPU-Cache-Line-和性能-x2F" class="headerlink" title="[CPU 性能和Cache Line](/2021/05/16/CPU Cache Line 和性能/)"></a>[CPU 性能和Cache Line](/2021/05/16/CPU Cache Line 和性能/)</h4><h4 id="AMD-Zen-CPU-架构-以及-AMD、海光、Intel、鲲鹏的性能对比"><a href="#AMD-Zen-CPU-架构-以及-AMD、海光、Intel、鲲鹏的性能对比" class="headerlink" title="AMD Zen CPU 架构 以及 AMD、海光、Intel、鲲鹏的性能对比"></a><a href="/2021/08/13/AMD_Zen_CPU%E6%9E%B6%E6%9E%84/">AMD Zen CPU 架构 以及 AMD、海光、Intel、鲲鹏的性能对比</a></h4><h4 id="Intel、海光、鲲鹏920、飞腾2500-CPU性能对比"><a href="#Intel、海光、鲲鹏920、飞腾2500-CPU性能对比" class="headerlink" title="Intel、海光、鲲鹏920、飞腾2500 CPU性能对比"></a><a href="/2021/06/18/%E5%87%A0%E6%AC%BECPU%E6%80%A7%E8%83%BD%E5%AF%B9%E6%AF%94/">Intel、海光、鲲鹏920、飞腾2500 CPU性能对比</a></h4><h2 id="网络相关基础知识（2017年完成）"><a href="#网络相关基础知识（2017年完成）" class="headerlink" title="网络相关基础知识（2017年完成）"></a>网络相关基础知识（2017年完成）</h2><h4 id="就是要你懂网络–一个网络包的旅程"><a href="#就是要你懂网络–一个网络包的旅程" class="headerlink" title="就是要你懂网络–一个网络包的旅程"></a><a href="/2019/05/15/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E7%BD%91%E7%BB%9C--%E4%B8%80%E4%B8%AA%E7%BD%91%E7%BB%9C%E5%8C%85%E7%9A%84%E6%97%85%E7%A8%8B/">就是要你懂网络–一个网络包的旅程</a></h4><h4 id="通过案例来理解MSS、MTU等相关TCP概念"><a href="#通过案例来理解MSS、MTU等相关TCP概念" class="headerlink" title="通过案例来理解MSS、MTU等相关TCP概念"></a><a href="/2018/05/07/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E9%80%9A%E8%BF%87%E6%A1%88%E4%BE%8B%E6%9D%A5%E5%AD%A6%E4%B9%A0MSS%E3%80%81MTU/">通过案例来理解MSS、MTU等相关TCP概念</a></h4><h4 id="就是要你懂TCP–握手和挥手"><a href="#就是要你懂TCP–握手和挥手" class="headerlink" title="就是要你懂TCP–握手和挥手"></a><a href="/2017/06/02/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--%E8%BF%9E%E6%8E%A5%E5%92%8C%E6%8F%A1%E6%89%8B/">就是要你懂TCP–握手和挥手</a></h4><h4 id="wireshark-dup-ack-issue-and-keepalive"><a href="#wireshark-dup-ack-issue-and-keepalive" class="headerlink" title="wireshark-dup-ack-issue and keepalive"></a><a href="/2017/06/02/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82TCP--wireshark-dup-ack-issue/">wireshark-dup-ack-issue and keepalive</a></h4><h4 id="一个没有遵守tcp规则导致的问题"><a href="#一个没有遵守tcp规则导致的问题" class="headerlink" title="一个没有遵守tcp规则导致的问题"></a><a href="/2018/11/26/%E4%B8%80%E4%B8%AA%E6%B2%A1%E6%9C%89%E9%81%B5%E5%AE%88tcp%E8%A7%84%E5%88%99%E5%AF%BC%E8%87%B4%E7%9A%84%E9%97%AE%E9%A2%98/">一个没有遵守tcp规则导致的问题</a></h4><h4 id="kubernetes-service-和-kube-proxy详解-x2F-2020-x2F-09-x2F-22-x2F-kubernetes-service-和-kube-proxy详解-x2F"><a href="#kubernetes-service-和-kube-proxy详解-x2F-2020-x2F-09-x2F-22-x2F-kubernetes-service-和-kube-proxy详解-x2F" class="headerlink" title="[kubernetes service 和 kube-proxy详解](/2020/09/22/kubernetes service 和 kube-proxy详解/)"></a>[kubernetes service 和 kube-proxy详解](/2020/09/22/kubernetes service 和 kube-proxy详解/)</h4><h2 id="DNS相关"><a href="#DNS相关" class="headerlink" title="DNS相关"></a>DNS相关</h2><h4 id="就是要你懂DNS–一文搞懂域名解析相关问题"><a href="#就是要你懂DNS–一文搞懂域名解析相关问题" class="headerlink" title="就是要你懂DNS–一文搞懂域名解析相关问题"></a><a href="/2019/06/09/%E4%B8%80%E6%96%87%E6%90%9E%E6%87%82%E5%9F%9F%E5%90%8D%E8%A7%A3%E6%9E%90%E7%9B%B8%E5%85%B3%E9%97%AE%E9%A2%98/">就是要你懂DNS–一文搞懂域名解析相关问题</a></h4><h4 id="nslookup-OK-but-ping-fail"><a href="#nslookup-OK-but-ping-fail" class="headerlink" title="nslookup OK but ping fail"></a><a href="/2019/01/09/nslookup-OK-but-ping-fail/">nslookup OK but ping fail</a></h4><h4 id="Docker中的DNS解析过程"><a href="#Docker中的DNS解析过程" class="headerlink" title="Docker中的DNS解析过程"></a><a href="/2019/01/12/Docker%E4%B8%AD%E7%9A%84DNS%E8%A7%A3%E6%9E%90%E8%BF%87%E7%A8%8B/">Docker中的DNS解析过程</a></h4><h4 id="windows7的wifi总是报DNS域名异常无法上网"><a href="#windows7的wifi总是报DNS域名异常无法上网" class="headerlink" title="windows7的wifi总是报DNS域名异常无法上网"></a><a href="/2019/01/10/windows7%E7%9A%84wifi%E6%80%BB%E6%98%AF%E6%8A%A5DNS%E5%9F%9F%E5%90%8D%E5%BC%82%E5%B8%B8%E6%97%A0%E6%B3%95%E4%B8%8A%E7%BD%91/">windows7的wifi总是报DNS域名异常无法上网</a></h4><h2 id="LVS-负载均衡"><a href="#LVS-负载均衡" class="headerlink" title="LVS 负载均衡"></a>LVS 负载均衡</h2><h4 id="就是要你懂负载均衡–lvs和转发模式"><a href="#就是要你懂负载均衡–lvs和转发模式" class="headerlink" title="就是要你懂负载均衡–lvs和转发模式"></a><a href="/2019/06/20/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1--lvs%E5%92%8C%E8%BD%AC%E5%8F%91%E6%A8%A1%E5%BC%8F/">就是要你懂负载均衡–lvs和转发模式</a></h4><h4 id="就是要你懂负载均衡–负载均衡调度算法和为什么不均衡"><a href="#就是要你懂负载均衡–负载均衡调度算法和为什么不均衡" class="headerlink" title="就是要你懂负载均衡–负载均衡调度算法和为什么不均衡"></a><a href="/2019/07/19/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1--%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1%E8%B0%83%E5%BA%A6%E7%AE%97%E6%B3%95%E5%92%8C%E4%B8%BA%E4%BB%80%E4%B9%88%E4%B8%8D%E5%9D%87%E8%A1%A1/">就是要你懂负载均衡–负载均衡调度算法和为什么不均衡</a></h4><h2 id="网络工具"><a href="#网络工具" class="headerlink" title="网络工具"></a>网络工具</h2><h4 id="就是要你懂Unix-Socket-进行抓包解析"><a href="#就是要你懂Unix-Socket-进行抓包解析" class="headerlink" title="就是要你懂Unix Socket 进行抓包解析"></a><a href="/2018/01/01/%E9%80%9A%E8%BF%87tcpdump%E5%AF%B9Unix%20Socket%20%E8%BF%9B%E8%A1%8C%E6%8A%93%E5%8C%85%E8%A7%A3%E6%9E%90/">就是要你懂Unix Socket 进行抓包解析</a></h4><h4 id="就是要你懂网络监控–ss用法大全"><a href="#就是要你懂网络监控–ss用法大全" class="headerlink" title="就是要你懂网络监控–ss用法大全"></a><a href="/2016/10/12/ss%E7%94%A8%E6%B3%95%E5%A4%A7%E5%85%A8/">就是要你懂网络监控–ss用法大全</a></h4><h4 id="就是要你懂抓包–WireShark之命令行版tshark"><a href="#就是要你懂抓包–WireShark之命令行版tshark" class="headerlink" title="就是要你懂抓包–WireShark之命令行版tshark"></a><a href="/2019/06/21/%E5%B0%B1%E6%98%AF%E8%A6%81%E4%BD%A0%E6%87%82%E6%8A%93%E5%8C%85--WireShark%E4%B9%8B%E5%91%BD%E4%BB%A4%E8%A1%8C%E7%89%88tshark/">就是要你懂抓包–WireShark之命令行版tshark</a></h4><h4 id="netstat-timer-keepalive-explain"><a href="#netstat-timer-keepalive-explain" class="headerlink" title="netstat timer keepalive explain"></a><a href="/2017/08/28/netstat%20%E7%AD%89%E7%BD%91%E7%BB%9C%E5%B7%A5%E5%85%B7/">netstat timer keepalive explain</a></h4><h4 id="Git-HTTP-Proxy-and-SSH-Proxy"><a href="#Git-HTTP-Proxy-and-SSH-Proxy" class="headerlink" title="Git HTTP Proxy and SSH Proxy"></a><a href="/2018/03/14/%E5%A6%82%E4%BD%95%E8%AE%BE%E7%BD%AEgit%20Proxy/">Git HTTP Proxy and SSH Proxy</a></h4>
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="https://plantegg.github.io/2024/12/29/net.ipv4.tcp_tw_recycle/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="twitter @plantegg">
<meta itemprop="description" content>
<meta itemprop="image" content="/images/avatar.gif">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="plantegg">
</span>
<header class="post-header">
<h2 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2024/12/29/net.ipv4.tcp_tw_recycle/" itemprop="url">为什么你的 SYN 包被丢 net.ipv4.tcp_tw_recycle</a></h2>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2024-12-29T17:30:03+08:00">
2024-12-29
</time>
</span>
<span class="post-category">
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/tcp/" itemprop="url" rel="index">
<span itemprop="name">tcp</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<h1 id="为什么你的-SYN-包被丢-net-ipv4-tcp-tw-recycle"><a href="#为什么你的-SYN-包被丢-net-ipv4-tcp-tw-recycle" class="headerlink" title="为什么你的 SYN 包被丢 net.ipv4.tcp_tw_recycle"></a>为什么你的 SYN 包被丢 net.ipv4.tcp_tw_recycle</h1><p>本来这是我计划在知识星球里要写的<a href="https://articles.zsxq.com/id_1fdbevh4fzf0.html" target="_blank" rel="noopener">连续剧</a>，我打算好好多写几篇的(每篇都计划重现一个场景/坑点)，后来没看到任何一个同学参与，这样的话写了你们看完也没有体感，所以我直接公布答案吧，还能节省点你们的时间，记住干货就好：<strong>不要开 net.ipv4.tcp_tw_recycle</strong></p>
<p>作为全网最权威/最全面的 net.ipv4.tcp_tw_recycle 问题分析还是从知识星球分享出来，希望更多的人避免踩坑</p>
<h2 id="答案"><a href="#答案" class="headerlink" title="答案"></a>答案</h2><p>首先不通了是因为服务端开启了 net.ipv4.tcp_tw_recycle，需要判断握手包的时间得保持递增(T2 - T1 >1)</p>
<p>tcpping 一直是通的，因为服务端没有记录到 T1，T1 是每次 FIN 断开时记录，T2 是每个 SYN 包中携带。当 curl 然后断开时走了 FIN 服务端记录下 T1，下次 tcpping 就可以比较了，所以有一半概率不通，直到 1 分钟后 T1 一直没有跟新，超过 60 秒的 T1 失效，后面连接正常</p>
<h2 id="为什么要有-net-ipv4-tcp-tw-recycle？"><a href="#为什么要有-net-ipv4-tcp-tw-recycle？" class="headerlink" title="为什么要有 net.ipv4.tcp_tw_recycle？"></a>为什么要有 net.ipv4.tcp_tw_recycle？</h2><p>net.ipv4.tcp_tw_recycle 是一个 Linux 内核参数，用于控制 TCP 连接的 TIME_WAIT 状态的处理方式。这个参数的主要作用是加速 TIME_WAIT 套接字的回收。</p>
<p>参考：<a href="https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux" target="_blank" rel="noopener">Coping with the TCP TIME-WAIT state on busy Linux servers</a> </p>
<h3 id="PAWS-Protection-Against-Wrapped-Sequences"><a href="#PAWS-Protection-Against-Wrapped-Sequences" class="headerlink" title="PAWS(Protection Against Wrapped Sequences)"></a>PAWS(Protection Against Wrapped Sequences)</h3><p>TCP 包的 seq 是有限的(4字节 32bit)，会在达到最大值后回绕到零，这种情况称为”seq回绕”，seq 回绕后怎么判断这个 seq 是重复的(丢弃) 还是可以接受的？</p>
<p>引入 <a href="https://perthcharles.github.io/2015/08/27/timestamp-intro/" target="_blank" rel="noopener">PAWS</a> 的目的是确保即使seq 回绕发生，也能正确地处理序列号，除了 seq 外额外在 TCP options 里面增加了 timestamp 来作为维护数据包的seq 正确的判断。时间戳随每个数据包发送，并且单调增加，因此即使序列号回绕，接收方也可以使用时间戳来确定数据包的真实顺序，这就是 PAWS</p>
<p><a href="https://perthcharles.github.io/2015/08/27/timestamp-NAT/" target="_blank" rel="noopener">PAWS会检查syn 网络包的 timestamps</a> ，来判断这个syn包的发送时间是否早于上一次同 ip/stream（3.10 是 per ip/4.10 是 per stream） 的 fin包，如果早就扔掉，这也是导致syn 握手失败的一个高发原因，尤其是在NAT场景下。原本 PAWS 是每个连接的维度，但同时开启tcp_timestamp和tcp_tw_recycle之后，PAWS就变成per host粒度了</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">timestamp为TCP/IP协议栈提供了两个功能： </span><br><span class="line"> a. 更加准确的RTT测量数据，尤其是有丢包时 -- RTTM </span><br><span class="line"> b. 保证了在极端情况下，TCP的可靠性 -- PAWS</span><br></pre></td></tr></table></figure>
<p>不同 OS 内核版本因为 timestamp 生成不一样导致 PAWS 行为还不一样，通过参数来控制：net.ipv4.tcp_timestamps</p>
<h2 id="服务端如何通过判断时间戳来丢包？"><a href="#服务端如何通过判断时间戳来丢包？" class="headerlink" title="服务端如何通过判断时间戳来丢包？"></a>服务端如何通过判断时间戳来丢包？</h2><p>对同一个 src-ip 记录最后一次 FIN 包的时间戳为 T1，当这个 src-ip 有 SYN 包时取 SYN 包中的时间戳为 T2</p>
<p>如果 T2-T1 小于 1 就扔掉这个 SYN 包</p>
<p>一旦发生这种 SYN 包被丢弃，对应的监控指标(LINUX_MIB_PAWSPASSIVEREJECTED)：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">//第二个指标包含第一个，passive connections rejected 了也一定会是 SYN dropped</span><br><span class="line">#netstat -s |egrep "SYNs to LISTEN sockets dropped|passive connections rejected because"</span><br><span class="line"> 960055 passive connections rejected because of time stamp</span><br><span class="line"> 1049368 SYNs to LISTEN sockets dropped</span><br><span class="line"></span><br><span class="line">#netstat -s |egrep "SYNs to LISTEN sockets dropped|passive connections rejected because"</span><br><span class="line"> 960535 passive connections rejected because of time stamp</span><br><span class="line"> 1049848 SYNs to LISTEN sockets dropped</span><br><span class="line"></span><br><span class="line">#netstat -s |egrep "SYNs to LISTEN sockets dropped|passive connections rejected because"</span><br><span class="line"> 961015 passive connections rejected because of time stamp</span><br><span class="line"> 1050328 SYNs to LISTEN sockets dropped</span><br></pre></td></tr></table></figure>
<p>这个指标也很重要，我喜欢这种</p>
<h3 id="服务端丢包条件更多细节"><a href="#服务端丢包条件更多细节" class="headerlink" title="服务端丢包条件更多细节"></a>服务端丢包条件更多细节</h3><p>服务端设置 net.ipv4.tcp_tw_recycle 为 1 是必要条件，然后同时满足了这两个条件：</p>
<ol>
<li><code>(u32)get_seconds() - tm->tcpm_ts_stamp < TCP_PAWS_MSL（=60）</code>：容易满足，几乎总是满足。<strong>对比的是本地时间</strong>。收到syn的<strong>本地时间</strong>相比上次收包记录的<strong>本地时间</strong>，小于60s</li>
<li><code>(s32)(tm->tcpm_ts - req->ts_recent) > TCP_PAWS_WINDOW（=1）</code>：对比的是tcp时间戳，上次更新的tcp时间戳 - 这次syn的tcp时间戳，大于1（并且小于231）。也就是这次syn的tcp时间戳，如果<strong>小于</strong>上次记录到的时间戳（ms级），就会被丢掉。</li>
</ol>
<p>这里tm和req对应什么？一个四元组，还是ip地址，还是其他？3.10<strong>对应的是ip地址</strong>(不同内核版本不一样)</p>
<p>上次记录的时间戳是什么？注意这里对比的都是tm时间，是在连接关闭相关阶段，通过<code>tcp_remember_stamp</code>或<code>tcp_tw_remember_stamp</code>函数记录的，具体情况比较多。</p>
<h4 id="服务端将客户端的时间戳保存在哪里？"><a href="#服务端将客户端的时间戳保存在哪里？" class="headerlink" title="服务端将客户端的时间戳保存在哪里？"></a>服务端将客户端的时间戳保存在哪里？</h4><p>6u(2.6.32)代码：</p>
<p>由于inet_timewait_sock在连接进入tw状态会被释放掉，其中记录最近一次接收报文的timestamp信息会丢失；VJ 的思路，把此tcp stamp信息放入路由cache表的rtable中struct inet_peer中，rtable中只保srcIP，dstIP的PATH信息，没有端口号信息，也就是同src-dstIP(即使端口不同)的所有连接受同一个timestamp限制。</p>
<p>7u(3.10.0)代码：</p>
<p>3.5版本以后的内核版本不再使用rtable记录，tcp stamp信息改为存放在目标地址出接口net中存放的tcp_metrics_block，timestamp判断逻辑跟6u比增加了“如果之前有记录timestamp且在一个MSL内，而本次连接无timestamp时，请求被丢弃”的逻辑，这么修改的原因参见：</p>
<p><a href="https://patchwork.ozlabs.org/patch/380021/" target="_blank" rel="noopener">https://patchwork.ozlabs.org/patch/380021/</a></p>
<p><a href="https://patchwork.ozlabs.org/patch/379163/" target="_blank" rel="noopener">https://patchwork.ozlabs.org/patch/379163/</a></p>
<p>2017 年的这个讨论<a href="https://patchwork.ozlabs.org/project/netdev/patch/20170315203046.158791-1-soheil.kdev@gmail.com/" target="_blank" rel="noopener">https://patchwork.ozlabs.org/project/netdev/patch/20170315203046.158791-1-soheil.kdev@gmail.com/</a> 要去掉这个全局存放，改成可以按客户端 port 来记录</p>
<h2 id="客户端如何生成时间戳？"><a href="#客户端如何生成时间戳？" class="headerlink" title="客户端如何生成时间戳？"></a>客户端如何生成时间戳？</h2><ul>
<li>3.10 内核是按 <strong>客户端 ip</strong> 来生成 timestamp，也就是不管跟谁通信都是全局单调递增</li>
<li>4.19(4.12)是按 <strong>ip 对</strong>（per-destination timestamp<strong>）</strong>来生 timestamp ，也就是一对 ip 之间保证单调递增；</li>
<li>4.10之前是 per-client 生成递增 timestamp ，4.10 改成 per-connection 生成递增 timestamp(导致了兼容 net.ipv4.tcp_tw_recycle问题严重)，4.11 改成 per-destination-host 生成递增 timestamp(<strong>downgrade to per-host timestamp offsets</strong>)；4.12 去掉 net.ipv4.tcp_tw_recycle 参数永远解决问题</li>
</ul>
<h2 id="有哪些场景会触发-net-ipv4-tcp-tw-recycle-丢包"><a href="#有哪些场景会触发-net-ipv4-tcp-tw-recycle-丢包" class="headerlink" title="有哪些场景会触发 net.ipv4.tcp_tw_recycle 丢包"></a>有哪些场景会触发 net.ipv4.tcp_tw_recycle 丢包</h2><p>服务端的内核参数 net.ipv4.tcp_tw_recycle(<a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4396e46187ca5070219b81773c4e65088dac50cc" target="_blank" rel="noopener">4.12内核 </a> 中删除这个参数了) 和 net.ipv4.tcp_timestamps 的值都为 1时，服务器会检查每一个 SYN报文中的时间戳（Timestamp，跟同一ip下最近一次 FIN包时间对比），若 <a href="https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux" target="_blank" rel="noopener">Timestamp 不是递增的关系</a>，就扔掉这个SYN包（<strong>诊断</strong>：netstat -s | grep “ passive connections rejected because of time stamp”），常见触发时间戳非递增场景：</p>
<ol>
<li><a href="https://lwn.net/Articles/708021/" target="_blank" rel="noopener">4.10 内核</a>，一直必现大概率性丢包。<a href="https://github.com/torvalds/linux/commit/95a22caee396cef0bb2ca8fafdd82966a49367bb" target="_blank" rel="noopener">4.11 改成了</a> per-destination host的算法 //内核改来改去也是坑点</li>
<li>tcpping 这种时间戳按连接随机的，必现大概率持续丢包</li>
<li><strong>同一个客户端通过直连或者 NAT 后两条链路到同一个服务端</strong>，客户端生成时间戳是 by dst ip，导致大概率持续丢包</li>
<li>经过NAT/LVS 后多个客户端被当成一个客户端，小概率偶尔出现——通过 tc qdisc 可以来构造丢包重现该场景</li>
<li>网路链路复杂/链路长容易导致包乱序，进而出发丢包，取决于网络会小概率出现</li>
<li>客户端修改 net.ipv4.tcp_timestamps <ul>
<li>1->0，触发持续60秒大概率必现的丢包，60秒后恢复</li>
<li>0->1 持续大概率一直丢包60秒; 60秒过后如果网络延时略高且客户端并发大一直有上一次 FIN 时间戳大于后续SYN 会一直概率性丢包持续下去；如果停掉所有流量，重启客户端流量，恢复正常</li>
<li>2->1 丢包，情况同2</li>
<li>1->2 不触发丢包</li>
</ul>
</li>
</ol>
<p>其它 SYN 连不上的场景延伸阅读：<a href="https://plantegg.github.io/2020/05/24/%E7%A8%8B%E5%BA%8F%E5%91%98%E5%A6%82%E4%BD%95%E5%AD%A6%E4%B9%A0%E5%92%8C%E6%9E%84%E5%BB%BA%E7%BD%91%E7%BB%9C%E7%9F%A5%E8%AF%86%E4%BD%93%E7%B3%BB/">程序员如何学习和构建网络知识体系</a> </p>
<h3 id="一些特殊场景"><a href="#一些特殊场景" class="headerlink" title="一些特殊场景"></a>一些特殊场景</h3><p>这些特殊场景很可怕，不知不觉会产生 T2 不大于 T1 的情况，导致连接异常</p>
<h4 id="DNAT-x2F-ENAT"><a href="#DNAT-x2F-ENAT" class="headerlink" title="DNAT/ENAT"></a>DNAT/ENAT</h4><p>请求经过 DNAT 后 Server 端看到的 src-ip 是 client 的 IP，客户端同时通过直连(绿色)和走 LVS(黑色)两条链路就会大概率不通：</p>
<p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog//image-20240822161109563.png" alt="image-20240822161109563"></p>
<h4 id="没有挥手断开场景"><a href="#没有挥手断开场景" class="headerlink" title="没有挥手断开场景"></a>没有挥手断开场景</h4><p>有些 HA 探测都是握手/select 1/ RESET 连接，不走 FIN 四次挥手(比如 Jedis，见小作业应用断开连接的时候如何让 OS 走 RST 流程：<a href="https://articles.zsxq.com/id_v0mhaadx3cx5.html" target="_blank" rel="noopener">https://articles.zsxq.com/id_v0mhaadx3cx5.html</a> )，Server 端没有机会记录 T1，也就永远不会触发丢包，看着一切正常，直到某天来了个用户 curl 一下系统就崩了</p>
<p>比如 Jedis 就是直接 RST 断开连接，从不走 FIN 四次挥手</p>
<h2 id="延伸"><a href="#延伸" class="headerlink" title="延伸"></a>延伸</h2><p>如果服务端所用<a href="https://developer.aliyun.com/article/1262180" target="_blank" rel="noopener">端口是 time_wait 状态</a>，这时新连接 SYN 握手包刚好和 time_wait 的5元组重复，这个时候服务端不会回复 SYN+ACK 而是回复 time_wait 前的ack </p>
<h2 id="其它"><a href="#其它" class="headerlink" title="其它"></a>其它</h2><p>Server 在握手的第三阶段（TCP_NEW_SYN_RECV），等待对端进行握手的第三步回 ACK时候，如果收到RST 内核会对报文进行PAWS校验，如果 RST 带的 timestamp(TVal) 不递增就会因为通不过 PAWS 校验而被扔掉</p>
<p><a href="https://github.com/torvalds/linux/commit/7faee5c0d514162853a343d93e4a0b6bb8bfec21" target="_blank" rel="noopener">https://github.com/torvalds/linux/commit/7faee5c0d514162853a343d93e4a0b6bb8bfec21</a> 这个 commit 去掉了TCP_SKB_CB(skb)->when = tcp_time_stamp，导致 3.18 的内核版本linger close主动发送的 RST 中 ts_val为0，而<a href="https://github.com/torvalds/linux/commit/675ee231d960af2af3606b4480324e26797eb010" target="_blank" rel="noopener">修复的commit在 675ee231d960af2af3606b4480324e26797eb010</a>，直到 4.10 才合并进内核</p>
<h2 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h2><p>per-connection random offset：<a href="https://lwn.net/Articles/708021/" target="_blank" rel="noopener">https://lwn.net/Articles/708021/</a></p>
<h2 id="如果你觉得看完对你很有帮助可以通过如下方式找到我"><a href="#如果你觉得看完对你很有帮助可以通过如下方式找到我" class="headerlink" title="如果你觉得看完对你很有帮助可以通过如下方式找到我"></a>如果你觉得看完对你很有帮助可以通过如下方式找到我</h2><p>find me on twitter: <a href="https://twitter.com/plantegg" target="_blank" rel="noopener">@plantegg</a></p>
<p>知识星球：<a href="https://t.zsxq.com/0cSFEUh2J" target="_blank" rel="noopener">https://t.zsxq.com/0cSFEUh2J</a></p>
<p>开了一个星球，在里面讲解一些案例、知识、学习方法，肯定没法让大家称为顶尖程序员(我自己都不是)，只是希望用我的方法、知识、经验、案例作为你的垫脚石，帮助你快速、早日成为一个基本合格的程序员。</p>
<p>争取在星球内：</p>
<ul>
<li>养成基本动手能力</li>
<li>拥有起码的分析推理能力–按我接触的程序员，大多都是没有逻辑的</li>
<li>知识上教会你几个关键的知识点</li>
</ul>
<img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20240324161113874-5525702.png" alt="image-20240324161113874" style="zoom:50%;">
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="https://plantegg.github.io/2024/12/29/一次 Sysbench opening tables 卡慢的分析过程/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="twitter @plantegg">
<meta itemprop="description" content>
<meta itemprop="image" content="/images/avatar.gif">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="plantegg">
</span>
<header class="post-header">
<h2 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2024/12/29/一次 Sysbench opening tables 卡慢的分析过程/" itemprop="url">一次 Sysbench opening tables 卡慢的分析过程</a></h2>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2024-12-29T17:30:03+08:00">
2024-12-29
</time>
</span>
<span class="post-category">
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/MySQL/" itemprop="url" rel="index">
<span itemprop="name">MySQL</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<h1 id="一次-Sysbench-opening-tables-卡慢的分析过程"><a href="#一次-Sysbench-opening-tables-卡慢的分析过程" class="headerlink" title="一次 Sysbench opening tables 卡慢的分析过程"></a>一次 Sysbench opening tables 卡慢的分析过程</h1><h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>用 Sysbench 随便跑个压力，然后我用如下命令起压力，只达到了我预期的性能的 10%</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sysbench --mysql-user=root --mysql-password=123 --mysql-db=sbtest --mysql-host=e237 --mysql-port=3306 --tables=64 --threads=256 --table-size=2000000 --range-size=5 --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=all --time=1200 --report-interval=1 --histogram=off oltp_point_select run</span><br></pre></td></tr></table></figure>
<h2 id="分析"><a href="#分析" class="headerlink" title="分析"></a>分析</h2><p>看了下 MySQL 的进程状态，<strong>CPU 消耗很低</strong>，再看 processlist 都是 Opening tables，这问题我熟啊，table_open_cache 设置太小，直接干大 10 倍，悲催的是性能依然没有任何变化看了下 MySQL 的进程状态，CPU 消耗很低，再看 processlist 都是 Opening tables，这问题我熟啊，table_open_cache 设置太小，直接干大 10 倍，悲催的是性能依然没有任何变化</p>
<p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20241022175210162.png" alt="image-20241022175210162"></p>
<p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20241023104502834.png" alt="image-20241023104502834"></p>
<p>难道还有别的地方限制了？我去查了下 status 发现 Table_open_cache_overflows 一直是 0，从状态来看 table_open_cache 肯定够了：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line">#mysql -he237 -P3306 -uroot -p123 -e "show global status like '%open%' "</span><br><span class="line">mysql: [Warning] Using a password on the command line interface can be insecure.</span><br><span class="line">+----------------------------+---------+</span><br><span class="line">| Variable_name | Value |</span><br><span class="line">+----------------------------+---------+</span><br><span class="line">| Com_ha_open | 0 |</span><br><span class="line">| Com_show_open_tables | 0 |</span><br><span class="line">| Innodb_num_open_files | 48 |</span><br><span class="line">| Open_files | 14 |</span><br><span class="line">| Open_streams | 0 |</span><br><span class="line">| Open_table_definitions | 159 |</span><br><span class="line">| Open_tables | 1161 |</span><br><span class="line">| Opened_files | 173 |</span><br><span class="line">| Opened_table_definitions | 138 |</span><br><span class="line">| Opened_tables | 1168 |</span><br><span class="line">| Slave_open_temp_tables | 0 |</span><br><span class="line">| Table_open_cache_hits | 8125315 |</span><br><span class="line">| Table_open_cache_misses | 1168 |</span><br><span class="line">| Table_open_cache_overflows | 0 |</span><br><span class="line">+----------------------------+---------+</span><br><span class="line"></span><br><span class="line">#mysql -he237 -P3306 -uroot -p123 -e "show global status like '%Table_open%' "</span><br><span class="line">mysql: [Warning] Using a password on the command line interface can be insecure.</span><br><span class="line">+----------------------------+---------+</span><br><span class="line">| Variable_name | Value |</span><br><span class="line">+----------------------------+---------+</span><br><span class="line">| Table_open_cache_hits | 9039467 |</span><br><span class="line">| Table_open_cache_misses | 1170 |</span><br><span class="line">| Table_open_cache_overflows | 0 |</span><br><span class="line">+----------------------------+---------+</span><br><span class="line"></span><br><span class="line">#mysql -he237 -P3306 -uroot -p123 -e "show global variables like '%Table_open%' "</span><br><span class="line">mysql: [Warning] Using a password on the command line interface can be insecure.</span><br><span class="line">+----------------------------+-------+</span><br><span class="line">| Variable_name | Value |</span><br><span class="line">+----------------------------+-------+</span><br><span class="line">| table_open_cache | 8192 |</span><br><span class="line">| table_open_cache_instances | 16 |</span><br><span class="line">+----------------------------+-------+</span><br></pre></td></tr></table></figure>
<p>这些有点难绷，因为我用的别人的 sysbench, 于是自己编译了一个重压性能一下就正常了，于是我开始 dump 别人的 sysbench 完整参数，最后发现是我使用的时候配置错误将：–tables=32 设置成了 –tables=64 也就是我的 database 总共只有 32 张表，而我压测的时候写成了 64 张，还有 32 张表不存在导致。</p>
<p>而别人的 sysbench 默认添加了：–mysql-ignore-errors=all 也就是把报错信息都忽略了，导致控制台看不到异常信息</p>
<h3 id="碰到这种问题怎么办？"><a href="#碰到这种问题怎么办？" class="headerlink" title="碰到这种问题怎么办？"></a>碰到这种问题怎么办？</h3><p>我们经常碰到业务代码把报错信息吃掉了(类似设置了 –mysql-ignore-errors=all )，同时 SQL 里面拼错了表明或者写错了 Database 名也导致表不存在</p>
<p>所以这里的必杀技(银弹) 抓包(或者堆栈热点分析)：</p>
<p><img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog//image-20241023101835801.png" alt="image-20241023101835801"></p>
<p>上图中只要不是 1146 的都是表明正确的请求，可以看到 RT 是 0.1-0.2 毫秒之间；但是 response Error 1146 报错的 RT 就很大了，同时抓包里 1146 也给出了错误原因</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">+-------+------+---------------------------------------+</span><br><span class="line">| Level | Code | Message |</span><br><span class="line">+-------+------+---------------------------------------+</span><br><span class="line">| Error | 1146 | Table 'sbtest.sbtest42' doesn't exist |</span><br><span class="line">+-------+------+---------------------------------------+</span><br></pre></td></tr></table></figure>
<p>正常时 50 万 QPS 的 RT：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"> time port avg_rt svc_rt up_rt QPS drop rtt</span><br><span class="line">2024-10-23 10:14:57 P3306 227 228 0 532688 0 34</span><br><span class="line">2024-10-23 10:14:58 P3306 227 228 0 533439 0 34</span><br></pre></td></tr></table></figure>
<p>异常时 5 万 QPS 的 RT：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"> time port avg_rt svc_rt up_rt QPS drop rtt</span><br><span class="line">2024-10-23 10:13:56 P3306 2201 2201 0 58910 0 34</span><br><span class="line">2024-10-23 10:13:57 P3306 2195 2195 0 59141 0 34</span><br><span class="line">2024-10-23 10:13:58 P3306 2203 2203 0 58923 0 34</span><br><span class="line">2024-10-23 10:13:59 P3306 2190 2191 0 59266 0 34</span><br><span class="line">2024-10-23 10:14:00 P3306 2198 2198 0 59018 0 34</span><br><span class="line">2024-10-23 10:14:01 P3306 2242 2242 0 57926 0 34</span><br></pre></td></tr></table></figure>
<p>从 RT 确实可以看出来是 3306 端口返回/响应慢了，我在 MySQLD 的日志里也搜索了，应该是没有记录这种 1146 错误</p>
<p>如果多看几次 processlist 的话还会发现 Opening table 的 SQL 对应的表明都是大于 31 的，表名小的 SQL 就不会出现 Opening table </p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>这个问题我第一时间没有想到抓包，显示根据经验 Opening tables 就是打开表慢了，然后调大 cache 参数，还不好用就觉得超出我的理解有点慌！</p>
<p>然后想到去比较参数/版本的差异，运气好发现了参数的差异；如果运气不好我重新编译然后复制白屏的命令参数估计还是发现不了。</p>
<p>所以我在想有什么更好的办法能识别这种问题，最后的结论居然还是抓个包看看，并且真管用，正好和这篇方法论呼应一下：<a href="https://articles.zsxq.com/id_mnp5z56gl0wi.html" target="_blank" rel="noopener">https://articles.zsxq.com/id_mnp5z56gl0wi.html</a> </p>
<h2 id="延伸"><a href="#延伸" class="headerlink" title="延伸"></a>延伸</h2><p>很多时候开发很坑人，把业务异常堆栈吃了不输出，就拿这个例子来说也有业务写错表名，然后报错又不输出就会出现和问题一样的问题，导致分析问题的时候发现很奇怪好好的系统就是慢，这个时候除了抓包还可以通过 perf/jstack 去看看堆栈，抓下热点</p>
<p>推上也有一些讨论，可以参考下别人的思路：<a href="https://x.com/plantegg/status/1851066206163521712" target="_blank" rel="noopener">https://x.com/plantegg/status/1851066206163521712</a> </p>
<h2 id="如果你觉得看完对你很有帮助可以通过如下方式找到我"><a href="#如果你觉得看完对你很有帮助可以通过如下方式找到我" class="headerlink" title="如果你觉得看完对你很有帮助可以通过如下方式找到我"></a>如果你觉得看完对你很有帮助可以通过如下方式找到我</h2><p>find me on twitter: <a href="https://twitter.com/plantegg" target="_blank" rel="noopener">@plantegg</a></p>
<p>知识星球：<a href="https://t.zsxq.com/0cSFEUh2J" target="_blank" rel="noopener">https://t.zsxq.com/0cSFEUh2J</a></p>
<p>开了一个星球，在里面讲解一些案例、知识、学习方法，肯定没法让大家称为顶尖程序员(我自己都不是)，只是希望用我的方法、知识、经验、案例作为你的垫脚石，帮助你快速、早日成为一个基本合格的程序员。</p>
<p>争取在星球内：</p>
<ul>
<li>养成基本动手能力</li>
<li>拥有起码的分析推理能力–按我接触的程序员，大多都是没有逻辑的</li>
<li>知识上教会你几个关键的知识点</li>
</ul>
<img src="https://cdn.jsdelivr.net/gh/plantegg/plantegg.github.io/images/951413iMgBlog/image-20240324161113874-5525694.png" alt="image-20240324161113874" style="zoom:50%;">
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<footer class="post-footer">
<div class="post-eof"></div>
</footer>
</article>
<article class="post post-type-normal " itemscope itemtype="http://schema.org/Article">
<link itemprop="mainEntityOfPage" href="https://plantegg.github.io/2024/12/09/一次网络连接残留的分析/">
<span hidden itemprop="author" itemscope itemtype="http://schema.org/Person">
<meta itemprop="name" content="twitter @plantegg">
<meta itemprop="description" content>
<meta itemprop="image" content="/images/avatar.gif">
</span>
<span hidden itemprop="publisher" itemscope itemtype="http://schema.org/Organization">
<meta itemprop="name" content="plantegg">
</span>
<header class="post-header">
<h2 class="post-title" itemprop="name headline">
<a class="post-title-link" href="/2024/12/09/一次网络连接残留的分析/" itemprop="url">一次网络连接残留的分析</a></h2>
<div class="post-meta">
<span class="post-time">
<span class="post-meta-item-icon">
<i class="fa fa-calendar-o"></i>
</span>
<span class="post-meta-item-text">发表于</span>
<time title="创建于" itemprop="dateCreated datePublished" datetime="2024-12-09T17:30:03+08:00">
2024-12-09
</time>
</span>
<span class="post-category">
<span class="post-meta-divider">|</span>
<span class="post-meta-item-icon">
<i class="fa fa-folder-o"></i>
</span>
<span class="post-meta-item-text">分类于</span>
<span itemprop="about" itemscope itemtype="http://schema.org/Thing">
<a href="/categories/tcp/" itemprop="url" rel="index">
<span itemprop="name">tcp</span>
</a>
</span>
</span>
</div>
</header>
<div class="post-body" itemprop="articleBody">
<h1 id="一次网络连接残留的分析"><a href="#一次网络连接残留的分析" class="headerlink" title="一次网络连接残留的分析"></a>一次网络连接残留的分析</h1><p>本来放在知识星球的收费文章，也网络直播给星球成员讲解过这个问题以及这篇文章的内容，作删减和调整后也发博客吧</p>
<h2 id="问题描述"><a href="#问题描述" class="headerlink" title="问题描述"></a>问题描述</h2><p>LVS TCP 探活一般是 3 次握手(验证服务节点还在)后立即发送一个 RST packet 来断开连接(效率高，不需要走四次挥手)，但是在我们的LVS 后面的 RS 上发现有大量的探活连接残留，需要分析为什么？</p>
<p>一通分析下来发现是 RST 包 和第三次握手的 ack 到达对端乱序了，导致 RST 被drop 掉了。但是还需要进一步分析 drop 的时候和 RST 包里面带的 timestamp 有没有关系？</p>
<p>可以用 Scapy 来实验验证如下 4 个场景：</p>
<ol>
<li>正常三次握手，然后发送 RST 看看是否被 drop —— 期望 RST 不被 drop，连接正常释放，作为对比项</li>
<li>正常 2 次握手，然后立即发送 RST（正常带 timestamp），再发送 ack(制造乱序)，看看 RST 会不会被 drop，如果 RST drop 后连接还能正常握手成功并残留吗？</li>
<li>正常 2 次握手，然后立即发送 RST（不带 timestamp），再发送 ack(制造乱序)，看看 RST 会不会被 drop</li>
<li>正常 2 次握手，然后立即发送 RST（带 timestamp，但是 timestamp 为 0），再发送 ack(制造乱序)，看看 RST 会不会被 drop</li>
</ol>
<p>重现场景构造如下：通过客户端+服务端来尝试重现，客户端用 scapy 来构造任意网络包，服务端通过 python 起一个 WEB 服务</p>
<h3 id="客户端"><a href="#客户端" class="headerlink" title="客户端"></a>客户端</h3><p>因为最新的 scapy 需要 python3.7 ，可以搞一个内核版本较高的 Linux 来测试(星球统一 99 块的实验 ECS 就符合要求)，安装命令大概是这样：yum install python3-scapy</p>
<p>用 scapy 脚本构造如上 3 个场景的网络包，代码和使用帮助我放到这里了：<a href="https://github.com/plantegg/programmer_case/commit/e71ade38050c48170c7d6fb5922f78188a96435b#diff-3d18b8aa76586e6c59227e020ba22ef1ef8c5416764d0a923b198ad824996eda" target="_blank" rel="noopener">https://github.com/plantegg/programmer_case/commit/e71ade38050c48170c7d6fb5922f78188a96435b#diff-3d18b8aa76586e6c59227e020ba22ef1ef8c5416764d0a923b198ad824996eda</a></p>
<p>如果需要构造带 timestamp 的RST 用如下代码段，乱序通过调整 ack和 RST 的顺序来实现</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 构造 ACK 包</span></span><br><span class="line">ack = TCP(sport=source_port,</span><br><span class="line"> dport=target_port,</span><br><span class="line"> flags=<span class="string">'A'</span>,</span><br><span class="line"> seq=syn_ack.ack,</span><br><span class="line"> ack=syn_ack.seq + <span class="number">1</span>,</span><br><span class="line"> options=[(<span class="string">'NOP'</span>, <span class="literal">None</span>), (<span class="string">'NOP'</span>, <span class="literal">None</span>),</span><br><span class="line"> (<span class="string">'Timestamp'</span>, (int(time.time()), <span class="number">0</span>))]) //重点调整这里的时间戳，以及 rst 和 ack 包的顺序</span><br><span class="line"></span><br><span class="line"><span class="comment"># 发送 ACK</span></span><br><span class="line">send(ip/ack)</span><br></pre></td></tr></table></figure>
<p>在scapy 机器上drop 掉OS 自动发送的 RST(因为连接是 scapy 伪造的，OS 收到 syn+ack 后会 OS系统会发 RST(这个 RST不带 timestamp))</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">iptables -A OUTPUT -p tcp --dport 8000 --tcp-flags RST RST ! --tcp-option 8 -j DROP</span><br><span class="line"></span><br><span class="line">//清理</span><br><span class="line">iptables -D OUTPUT -p tcp --dport 8000 --tcp-flags RST RST ! --tcp-option 8 -j DROP</span><br></pre></td></tr></table></figure>
<p>scapy 构造的包流程，可以看到不走内核 tcp 协议栈，也不走 nf_hook（防火墙），不受上面的 iptables 规则限制，所以能发送到服务端：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">***************** c7d8ea00 ***************</span><br><span class="line">[100167.011693] [__dev_queue_xmit ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:0, flags:R</span><br><span class="line">[100167.011702] [dev_hard_start_xmit ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:0, flags:R *skb is successfully sent to the NIC driver*</span><br><span class="line">[100167.011714] [consume_skb ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:0, flags:R *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c700e300 ***************</span><br><span class="line">[100167.024811] [__dev_queue_xmit ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:2680597246, flags:A</span><br><span class="line">[100167.024821] [dev_hard_start_xmit ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:2680597246, flags:A *skb is successfully sent to the NIC driver*</span><br><span class="line">[100167.024891] [consume_skb ] TCP: 172.26.137.131:8146 -> 172.26.137.130:8000 seq:12346, ack:2680597246, flags:A *packet is freed (normally)*</span><br></pre></td></tr></table></figure>
<h3 id="Server-端"><a href="#Server-端" class="headerlink" title="Server 端"></a>Server 端</h3><p>先记住一个知识点，后面看内核调用堆栈会用得上确认是否被丢包</p>
<blockquote>
<p>一个网络包正常处理流程最后调 consume_skb 来释放，如果网络包需要 Drop 就调 <code>kfree_skb</code> 来丢包</p>
</blockquote>
<p>server端 安装 netstrace来监控包是否被drop，并通过 python 拉起一个端口：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python -m http.server 8000</span><br></pre></td></tr></table></figure>
<h4 id="tcpdump-确认-8000-端口收到的包"><a href="#tcpdump-确认-8000-端口收到的包" class="headerlink" title="tcpdump 确认 8000 端口收到的包"></a>tcpdump 确认 8000 端口收到的包</h4><p>在 8000端口机器上执行抓包验证收到的包顺序和所携带的 timestamp，包含 3 个场景的包：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">#tcpdump -i eth0 port 8000 -nn</span><br><span class="line">//场景 2：正常 2 次握手，然后立即发送 RST（带 timestamp）</span><br><span class="line">13:56:26.614701 IP 172.26.137.131.54321 > 172.26.137.130.8000: Flags [S], seq 2754757912, win 8192, options [mss 1460,nop,nop,TS val 1732514186 ecr 0], length 0</span><br><span class="line">13:56:26.614815 IP 172.26.137.130.8000 > 172.26.137.131.54321: Flags [S.], seq 1579697129, ack 2754757913, win 65160, options [mss 1460,nop,nop,TS val 2888180099 ecr 1732514186], length 0</span><br><span class="line">13:56:26.633997 IP 172.26.137.131.54321 > 172.26.137.130.8000: Flags [R], seq 2754757913, win 8192, options [mss 1460,nop,nop,TS val 1732514186 ecr 0], length 0 //留意端口号 54321 和 seq 2754757913 跟 nettrace 对应</span><br><span class="line">13:56:26.654954 IP 172.26.137.131.54321 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [nop,nop,TS val 1732514186 ecr 0], length 0</span><br><span class="line">13:56:26.655042 IP 172.26.137.130.8000 > 172.26.137.131.54321: Flags [R], seq 1579697130, win 0, length 0</span><br><span class="line"></span><br><span class="line">//场景 3：正常 2 次握手，然后立即发送 RST（不带 timestamp）, 注意这里的 tcp options 是 null</span><br><span class="line">13:56:28.993723 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [S], seq 54243194, win 8192, options [mss 1460,nop,nop,TS val 1732514188 ecr 0], length 0</span><br><span class="line">13:56:28.993809 IP 172.26.137.130.8000 > 172.26.137.131.12345: Flags [S.], seq 1983242893, ack 54243195, win 65160, options [mss 1460,nop,nop,TS val 2888182478 ecr 1732514188], length 0</span><br><span class="line">13:56:29.012982 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [R], seq 54243195, win 8192, length 0 //留意端口号 12345 和 seq 54243195 跟 nettrace 对应</span><br><span class="line">13:56:29.029886 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [nop,nop,TS val 1732514189 ecr 0], length 0</span><br><span class="line">13:56:29.029983 IP 172.26.137.130.8000 > 172.26.137.131.12345: Flags [R], seq 1983242894, win 0, length 0 //OS 触发</span><br><span class="line">13:56:29.050888 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [R], seq 54243195, win 8192, length 0</span><br><span class="line"></span><br><span class="line">//场景 1：正常握手，然后 RST</span><br><span class="line">13:56:30.399672 IP 172.26.137.131.22345 > 172.26.137.130.8000: Flags [S], seq 1038081714, win 8192, options [mss 1460,nop,nop,TS val 1732514190 ecr 0], length 0</span><br><span class="line">13:56:30.399770 IP 172.26.137.130.8000 > 172.26.137.131.22345: Flags [S.], seq 3263478059, ack 1038081715, win 65160, options [mss 1460,nop,nop,TS val 2888183884 ecr 1732514190], length 0</span><br><span class="line">13:56:30.426005 IP 172.26.137.131.22345 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [nop,nop,TS val 1732514190 ecr 0], length 0</span><br><span class="line">13:56:30.448876 IP 172.26.137.131.22345 > 172.26.137.130.8000: Flags [R], seq 1038081715, win 8192, length 0</span><br></pre></td></tr></table></figure>
<h4 id="场景-1：正常三次握手后再-RST，作为对比"><a href="#场景-1：正常三次握手后再-RST，作为对比" class="headerlink" title="场景 1：正常三次握手后再 RST，作为对比"></a>场景 1：正常三次握手后再 RST，作为对比</h4><p>netstrace 命令和结果</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><span class="line">#netstat -P 8000</span><br><span class="line">***************** c22c8c00,c22c8000 ***************</span><br><span class="line">[4912187.018483] [__ip_local_out ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018485] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *ipv4 in chain: OUTPUT*</span><br><span class="line">[4912187.018487] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912187.018489] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912187.018493] [ip_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018495] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *ipv4 in chain: POST_ROUTING*</span><br><span class="line">[4912187.018496] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *iptables table:, chain:POSTROU*</span><br><span class="line">[4912187.018499] [ip_finish_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018502] [ip_finish_output2 ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018506] [__dev_queue_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018510] [dev_hard_start_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *skb is successfully sent to the NIC driver*</span><br><span class="line">[4912187.018512] [skb_clone ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018516] [tpacket_rcv ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA</span><br><span class="line">[4912187.018519] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *packet is freed (normally)*</span><br><span class="line">[4912187.018533] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:22345 seq:3263478059, ack:1038081715, flags:SA *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8a00,c22c8f00 ***************</span><br><span class="line">[4912187.044742] [napi_gro_receive_entry] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044749] [dev_gro_receive ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044751] [__netif_receive_skb_core] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044753] [tpacket_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044758] [ip_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044760] [ip_rcv_core ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044762] [skb_clone ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044766] [nf_hook_slow ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912187.044769] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *iptables table:, chain:PREROUT*</span><br><span class="line">[4912187.044772] [ip_rcv_finish ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044776] [ip_route_input_slow ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044781] [fib_validate_source ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044785] [ip_local_deliver ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044786] [nf_hook_slow ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *ipv4 in chain: INPUT*</span><br><span class="line">[4912187.044787] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912187.044789] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912187.044791] [ip_local_deliver_finish] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044794] [tcp_v4_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044806] [tcp_child_process ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044810] [tcp_rcv_state_process] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *TCP socket state has changed*</span><br><span class="line">[4912187.044813] [tcp_ack ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044818] [__kfree_skb ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044825] [packet_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A</span><br><span class="line">[4912187.044827] [consume_skb ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:A *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8900,c22c8a00 ***************</span><br><span class="line">[4912187.067611] [napi_gro_receive_entry] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067617] [dev_gro_receive ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067622] [__netif_receive_skb_core] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067624] [tpacket_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067628] [ip_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067630] [ip_rcv_core ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067631] [skb_clone ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067634] [nf_hook_slow ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912187.067636] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *iptables table:, chain:PREROUT*</span><br><span class="line">[4912187.067639] [ip_rcv_finish ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067640] [ip_local_deliver ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067642] [nf_hook_slow ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *ipv4 in chain: INPUT*</span><br><span class="line">[4912187.067643] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912187.067644] [nft_do_chain ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912187.067646] [ip_local_deliver_finish] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067648] [tcp_v4_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067650] [tcp_filter ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067651] [tcp_v4_do_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067653] [tcp_rcv_established ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067659] [__kfree_skb ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067685] [packet_rcv ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R</span><br><span class="line">[4912187.067687] [consume_skb ] TCP: 172.26.137.131:22345 -> 172.26.137.130:8000 seq:1038081715, ack:3263478060, flags:R *packet is freed (normally)* //RST packet 被正常处理，没有发生 drop</span><br></pre></td></tr></table></figure>
<h4 id="场景-2：正常-2-次握手，然后立即发送-RST-带-timestamp"><a href="#场景-2：正常-2-次握手，然后立即发送-RST-带-timestamp" class="headerlink" title="场景 2：正常 2 次握手，然后立即发送 RST(带 timestamp)"></a>场景 2：正常 2 次握手，然后立即发送 RST(带 timestamp)</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br></pre></td><td class="code"><pre><span class="line">#netstat -P 8000</span><br><span class="line">//场景 2：正常 2 次握手，然后立即发送 RST（带 timestamp）—— RST 被 drop 了</span><br><span class="line">***************** c22c8900,c22c8300 *************** //8000 端口回复的 syn+ack</span><br><span class="line">[4912183.233533] [__ip_local_out ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233535] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *ipv4 in chain: OUTPUT*</span><br><span class="line">[4912183.233537] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912183.233538] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912183.233541] [ip_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233542] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *ipv4 in chain: POST_ROUTING*</span><br><span class="line">[4912183.233543] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *iptables table:, chain:POSTROU*</span><br><span class="line">[4912183.233546] [ip_finish_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233549] [ip_finish_output2 ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233552] [__dev_queue_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233555] [dev_hard_start_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *skb is successfully sent to the NIC driver*</span><br><span class="line">[4912183.233557] [skb_clone ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233561] [tpacket_rcv ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA</span><br><span class="line">[4912183.233565] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *packet is freed (normally)*</span><br><span class="line">[4912183.233581] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:54321 seq:1579697129, ack:2754757913, flags:SA *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8000,c22c8c00 ***************//客户端发送的 RST 比 ack 先到</span><br><span class="line">[4912183.252733] [napi_gro_receive_entry] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252741] [dev_gro_receive ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252743] [__netif_receive_skb_core] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252745] [tpacket_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252749] [ip_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252750] [ip_rcv_core ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252752] [skb_clone ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252757] [nf_hook_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912183.252759] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *iptables table:, chain:PREROUT*</span><br><span class="line">[4912183.252761] [ip_rcv_finish ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252765] [ip_route_input_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252771] [fib_validate_source ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252773] [ip_local_deliver ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252775] [nf_hook_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *ipv4 in chain: INPUT*</span><br><span class="line">[4912183.252777] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912183.252779] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912183.252782] [ip_local_deliver_finish] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252783] [tcp_v4_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252789] [kfree_skb ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *tcp_v4_rcv+0x65* *packet is dropped by kernel* //被 drop 了</span><br><span class="line">[4912183.252792] [packet_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R</span><br><span class="line">[4912183.252794] [consume_skb ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:R *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8900,c22c8200 ***************</span><br><span class="line">[4912183.273690] [napi_gro_receive_entry] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273697] [dev_gro_receive ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273700] [__netif_receive_skb_core] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273701] [tpacket_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273705] [ip_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273707] [ip_rcv_core ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273708] [skb_clone ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273711] [nf_hook_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912183.273714] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *iptables table:, chain:PREROUT*</span><br><span class="line">[4912183.273716] [ip_rcv_finish ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273719] [ip_route_input_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273724] [fib_validate_source ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273726] [ip_local_deliver ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273728] [nf_hook_slow ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *ipv4 in chain: INPUT*</span><br><span class="line">[4912183.273733] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912183.273735] [nft_do_chain ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912183.273737] [ip_local_deliver_finish] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273738] [tcp_v4_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273742] [__inet_lookup_listener] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273744] [tcp_filter ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273746] [tcp_v4_do_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273750] [tcp_rcv_state_process] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *TCP socket state has changed*</span><br><span class="line">[4912183.273754] [tcp_v4_send_reset ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273798] [kfree_skb ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *tcp_v4_do_rcv+0x6c* *packet is dropped by kernel*</span><br><span class="line">[4912183.273801] [packet_rcv ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A</span><br><span class="line">[4912183.273803] [consume_skb ] TCP: 172.26.137.131:54321 -> 172.26.137.130:8000 seq:2754757913, ack:1579697130, flags:A *packet is freed (normally)*</span><br></pre></td></tr></table></figure>
<h4 id="场景-3：正常-2-次握手，然后立即发送-RST（不带-timestamp）"><a href="#场景-3：正常-2-次握手，然后立即发送-RST（不带-timestamp）" class="headerlink" title="场景 3：正常 2 次握手，然后立即发送 RST（不带 timestamp）"></a>场景 3：正常 2 次握手，然后立即发送 RST（不带 timestamp）</h4><p>可以看到 RST 被 drop 然后 握手失败</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><span class="line">***************** c22c8900,c22c8f00 ***************</span><br><span class="line">[4912185.612533] [__ip_local_out ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612535] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *ipv4 in chain: OUTPUT*</span><br><span class="line">[4912185.612536] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912185.612538] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *iptables table:, chain:OUTPUT*</span><br><span class="line">[4912185.612539] [ip_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612541] [nf_hook_slow ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *ipv4 in chain: POST_ROUTING*</span><br><span class="line">[4912185.612542] [nft_do_chain ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *iptables table:, chain:POSTROU*</span><br><span class="line">[4912185.612544] [ip_finish_output ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612546] [ip_finish_output2 ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612547] [__dev_queue_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612550] [dev_hard_start_xmit ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *skb is successfully sent to the NIC driver*</span><br><span class="line">[4912185.612552] [skb_clone ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612555] [tpacket_rcv ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA</span><br><span class="line">[4912185.612558] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *packet is freed (normally)*</span><br><span class="line">[4912185.612573] [consume_skb ] TCP: 172.26.137.130:8000 -> 172.26.137.131:12345 seq:1983242893, ack:54243195, flags:SA *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8f00,c22c8800 ***************</span><br><span class="line">[4912185.631719] [napi_gro_receive_entry] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631726] [dev_gro_receive ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631728] [__netif_receive_skb_core] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631730] [tpacket_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631734] [ip_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631736] [ip_rcv_core ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631737] [skb_clone ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631744] [nf_hook_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912185.631746] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *iptables table:, chain:PREROUT*</span><br><span class="line">[4912185.631748] [ip_rcv_finish ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631754] [ip_route_input_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631759] [fib_validate_source ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631762] [ip_local_deliver ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631763] [nf_hook_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *ipv4 in chain: INPUT*</span><br><span class="line">[4912185.631765] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912185.631767] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *iptables table:, chain:INPUT*</span><br><span class="line">[4912185.631770] [ip_local_deliver_finish] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631772] [tcp_v4_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631777] [kfree_skb ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *tcp_v4_rcv+0x65* *packet is dropped by kernel*</span><br><span class="line">[4912185.631780] [packet_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R</span><br><span class="line">[4912185.631783] [consume_skb ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:R *packet is freed (normally)*</span><br><span class="line"></span><br><span class="line">***************** c22c8600,c22c8100 ***************</span><br><span class="line">[4912185.648623] [napi_gro_receive_entry] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648630] [dev_gro_receive ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648632] [__netif_receive_skb_core] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648633] [tpacket_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648637] [ip_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648639] [ip_rcv_core ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648640] [skb_clone ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648643] [nf_hook_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *ipv4 in chain: PRE_ROUTING*</span><br><span class="line">[4912185.648645] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *iptables table:, chain:PREROUT*</span><br><span class="line">[4912185.648647] [ip_rcv_finish ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648650] [ip_route_input_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648656] [fib_validate_source ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648659] [ip_local_deliver ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648660] [nf_hook_slow ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *ipv4 in chain: INPUT*</span><br><span class="line">[4912185.648662] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912185.648664] [nft_do_chain ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *iptables table:, chain:INPUT*</span><br><span class="line">[4912185.648667] [ip_local_deliver_finish] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648672] [tcp_v4_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648677] [__inet_lookup_listener] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648679] [tcp_filter ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648681] [tcp_v4_do_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648685] [tcp_rcv_state_process] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *TCP socket state has changed*</span><br><span class="line">[4912185.648689] [tcp_v4_send_reset ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648739] [kfree_skb ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *tcp_v4_do_rcv+0x6c* *packet is dropped by kernel*</span><br><span class="line">[4912185.648741] [packet_rcv ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A</span><br><span class="line">[4912185.648743] [consume_skb ] TCP: 172.26.137.131:12345 -> 172.26.137.130:8000 seq:54243195, ack:1983242894, flags:A *packet is freed (normally)*</span><br></pre></td></tr></table></figure>
<p>上面三个场景都没能重现问题，所以继续构造场景 4</p>
<h4 id="场景-4-timestamp-不递增"><a href="#场景-4-timestamp-不递增" class="headerlink" title="场景 4 timestamp 不递增"></a>场景 4 timestamp 不递增</h4><p>保证 tcp options 里面有 timestamp，且不递增，这时终于重现了连接残留：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line">//这表示有 tcp 连接残留在 8000 端口上，而实际上期望连接要因为有 RST 而被释放</span><br><span class="line">#netstat -ant |grep 8000</span><br><span class="line">tcp 4 0 0.0.0.0:8000 0.0.0.0:* LISTEN</span><br><span class="line">tcp 0 0 172.26.137.130:8000 172.26.137.131:19723 ESTABLISHED</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">//此时对应的抓包，注意这里 Server 端也没有回复 RST，前面 3 个场景 Server 端 8000 都会回 RST，从而不会残留</span><br><span class="line">//连接残留：ts 为 0，RST 被忽略，导致连接残留</span><br><span class="line">16:09:55.669693 IP 172.26.137.131.19723 > 172.26.137.130.8000: Flags [S], seq 12345, win 8192, options [TS val 1732608595 ecr 0,eol], length 0</span><br><span class="line">16:09:55.669708 IP 172.26.137.130.8000 > 172.26.137.131.19723: Flags [S.], seq 3736478060, ack 12346, win 65160, options [mss 1460,nop,nop,TS val 2982589154 ecr 1732608595], length 0</span><br><span class="line">16:09:55.687943 IP 172.26.137.131.19723 > 172.26.137.130.8000: Flags [R], seq 12346, win 8192, options [TS val 0 ecr 2982589154,eol], length 0</span><br><span class="line">16:09:55.703896 IP 172.26.137.131.19723 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [TS val 1732608595 ecr 0,eol], length 0</span><br><span class="line"></span><br><span class="line">//连接残留：ts 没递增，RST 被忽略，导致连接残留</span><br><span class="line">17:18:26.739344 IP 172.26.137.131.59541 > 172.26.137.130.8000: Flags [S], seq 12345, win 8192, options [TS val 1732612706 ecr 0,eol], length 0</span><br><span class="line">17:18:26.739358 IP 172.26.137.130.8000 > 172.26.137.131.59541: Flags [S.], seq 3510510105, ack 12346, win 65160, options [mss 1460,nop,nop,TS val 2986700224 ecr 1732612706], length 0</span><br><span class="line">17:18:26.756574 IP 172.26.137.131.59541 > 172.26.137.130.8000: Flags [R], seq 12346, win 8192, options [mss 1460,TS val 1732611916 ecr 0,eol], length 0</span><br><span class="line">17:18:26.870569 IP 172.26.137.131.59541 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [TS val 1732612706 ecr 0,eol], length 0</span><br></pre></td></tr></table></figure>
<p>不会导致连接残留的情况：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line">//连接不残留， ts 递增</span><br><span class="line">16:24:33.516952 IP 172.26.137.131.19544 > 172.26.137.130.8000: Flags [S], seq 12345, win 8192, options [TS val 1732609473 ecr 0,eol], length 0</span><br><span class="line">16:24:33.516967 IP 172.26.137.130.8000 > 172.26.137.131.19544: Flags [S.], seq 1834771950, ack 12346, win 65160, options [mss 1460,nop,nop,TS val 2983467001 ecr 1732609473], length 0</span><br><span class="line">16:24:33.539178 IP 172.26.137.131.19544 > 172.26.137.130.8000: Flags [R], seq 12346, win 8192, options [TS val 1732609473 ecr 0,eol], length 0</span><br><span class="line">16:24:33.556153 IP 172.26.137.131.19544 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [TS val 1732609473 ecr 0,eol], length 0</span><br><span class="line">16:24:33.556164 IP 172.26.137.130.8000 > 172.26.137.131.19544: Flags [R], seq 1834771951, win 0, length 0</span><br><span class="line"></span><br><span class="line">//连接不残留， 有 options 但是没有 ts</span><br><span class="line">17:05:16.217333 IP 172.26.137.131.22567 > 172.26.137.130.8000: Flags [S], seq 12345, win 8192, options [TS val 1732611916 ecr 0,eol], length 0</span><br><span class="line">17:05:16.217351 IP 172.26.137.130.8000 > 172.26.137.131.22567: Flags [S.], seq 3503286934, ack 12346, win 65160, options [mss 1460,nop,nop,TS val 2985909702 ecr 1732611916], length 0</span><br><span class="line">17:05:16.229589 IP 172.26.137.131.22567 > 172.26.137.130.8000: Flags [R], seq 12346, win 8192, options [mss 1460], length 0</span><br><span class="line">17:05:16.346564 IP 172.26.137.131.22567 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [TS val 1732611916 ecr 0,eol], length 0</span><br><span class="line">17:05:16.346578 IP 172.26.137.130.8000 > 172.26.137.131.22567: Flags [R], seq 3503286935, win 0, length 0</span><br><span class="line"></span><br><span class="line">//连接不残留，options 为 null</span><br><span class="line">16:29:38.618811 IP 172.26.137.131.33190 > 172.26.137.130.8000: Flags [S], seq 12345, win 8192, options [TS val 1732609778 ecr 0,eol], length 0</span><br><span class="line">16:29:38.618824 IP 172.26.137.130.8000 > 172.26.137.131.33190: Flags [S.], seq 1867663284, ack 12346, win 65160, options [mss 1460,nop,nop,TS val 2983772103 ecr 1732609778], length 0</span><br><span class="line">16:29:38.647039 IP 172.26.137.131.33190 > 172.26.137.130.8000: Flags [R], seq 12346, win 8192, length 0</span><br><span class="line">16:29:38.670061 IP 172.26.137.131.33190 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [TS val 1732609778 ecr 0,eol], length 0</span><br><span class="line">16:29:38.670073 IP 172.26.137.130.8000 > 172.26.137.131.33190: Flags [R], seq 1867663285, win 0, length 0</span><br><span class="line"></span><br><span class="line">//连接不残留， 有 options ，但 ts 为 nop</span><br><span class="line">17:37:37.476343 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [S], seq 2331525453, win 8192, options [mss 1460,nop,nop,TS val 1732613857 ecr 0], length 0</span><br><span class="line">17:37:37.476460 IP 172.26.137.130.8000 > 172.26.137.131.12345: Flags [S.], seq 230155727, ack 2331525454, win 65160, options [mss 1460,nop,nop,TS val 2987850961 ecr 1732613857], length 0</span><br><span class="line">17:37:37.494579 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [R], seq 2331525454, win 8192, options [nop,nop,eol], length 0</span><br><span class="line">17:37:37.511431 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [nop,nop,TS val 1732613857 ecr 0], length 0</span><br><span class="line">17:37:37.511546 IP 172.26.137.130.8000 > 172.26.137.131.12345: Flags [R], seq 230155728, win 0, length 0</span><br><span class="line">17:37:37.526369 IP 172.26.137.131.12345 > 172.26.137.130.8000: Flags [R], seq 2331525454, win 8192, length 0</span><br></pre></td></tr></table></figure>
<h3 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h3><p>最终重现的必要条件：<strong>内核在三次握手阶段（TCP_NEW_SYN_RECV），收到的RST 包里有 timestamp 且不递增</strong> 就会丢弃 RST</p>
<p>注意：</p>
<ul>
<li>如果 RST 的 seq 不递增也会导致连接残留，这属于 seq 回绕了 // /proc/net/netstat 中没找到 有哪个指标对应的监控</li>
<li>要区分 timestamp 没有和 timestamp 为 0 的情况，为 0 表示有，大概率回绕了//场景 1-3 都忽略了这个问题</li>
<li>options=[(‘NOP’, None), (‘NOP’, None)]) 表示没有 timestamp，也不能重现问题</li>
<li>以上案例 2/3/4 场景下 nettrace 看到的 RST 都被 drop 了，但是不妨碍连接的释放 //这个还需要分析为什么连接 RST 起作用了但是还是会 drop RST 包</li>
<li>如果出现连接残留，也会导致全连接队列增大直到溢出</li>
<li>三次握手成功后的通信阶段（established），此时只校验 RST 的 seq 有没有回绕，不校验 timestamp，这样连接能正确释放</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">//三次握手成功，如果 RST 带的 timestamp 不递增也会正确触发释放连接，也就是 ESTABLISHED 时只校验 RST 的 seq 有没有回绕，不校验 timestamp</span><br><span class="line">//如下抓包的连接被正确释放了，所以 LVS 会用这个逻辑来释放连接，但是一旦乱序就嗝屁了</span><br><span class="line">12:19:58.588218 IP 172.26.137.131.1406 > 172.26.137.130.8000: Flags [S], seq 2800159571, win 8192, options [mss 1460,nop,nop,TS val 1732681198 ecr 0], length 0</span><br><span class="line">12:19:58.588233 IP 172.26.137.130.8000 > 172.26.137.131.1406: Flags [S.], seq 3011503126, ack 2800159572, win 65160, options [mss 1460,nop,nop,TS val 3055192072 ecr 1732681198], length 0</span><br><span class="line">12:19:58.606594 IP 172.26.137.131.1406 > 172.26.137.130.8000: Flags [.], ack 1, win 8192, options [nop,nop,TS val 1732681198 ecr 0], length 0</span><br><span class="line">12:19:58.624392 IP 172.26.137.131.1406 > 172.26.137.130.8000: Flags [R], seq 2800159572, win 8192, options [nop,nop,TS val 0 ecr 0], length 0</span><br></pre></td></tr></table></figure>
<h4 id="对应的内核-commit"><a href="#对应的内核-commit" class="headerlink" title="对应的内核 commit"></a>对应的内核 commit</h4><p>Server 在握手的第三阶段（TCP_NEW_SYN_RECV），等待对端进行握手的第三步回 ACK时候，如果收到RST 内核会对报文进行PAWS校验，如果 RST 带的 timestamp(TVal) 不递增就会因为通不过 PAWS 校验而被扔掉</p>
<p>问题引入：<a href="https://github.com/torvalds/linux/commit/7faee5c0d514162853a343d93e4a0b6bb8bfec21" target="_blank" rel="noopener">https://github.com/torvalds/linux/commit/7faee5c0d514162853a343d93e4a0b6bb8bfec21</a> 这个 commit 去掉了TCP_SKB_CB(skb)->when = tcp_time_stamp，导致 3.18 的内核版本linger close主动发送的 RST 中 ts_val为0</p>
<p>问题修复：<a href="https://github.com/torvalds/linux/commit/675ee231d960af2af3606b4480324e26797eb010" target="_blank" rel="noopener">修复的commit在 675ee231d960af2af3606b4480324e26797eb010</a>，直到 4.10 才合并进内核</p>
<h4 id="监控"><a href="#监控" class="headerlink" title="监控"></a>监控</h4><p>对应这种握手阶段连接建立如何监控呢？</p>
<p>从内核代码 net/ipv4/tcp_minisocks.c/tcp_check_req 函数会对报文调用 tcp_paws_reject 函数进行 paws_reject 检测，tcp_paws_reject 如果返回值为true，则 tcp_check_req 返回NULL，并且记录 LINUX_MIB_PAWSESTABREJECTED 计数</p>
<p>可以观察 /proc/net/netstat 中的监控指标：PAWSEstab</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">//内核中的指标</span><br><span class="line">SNMP_MIB_ITEM("PAWSEstab", LINUX_MIB_PAWSESTABREJECTED)</span><br><span class="line"></span><br><span class="line">//尝试了 5 次 RST的 timestamp 不递增导致的残留，监控到这个值每次变化累加 1</span><br><span class="line">TcpExt:PAWSEstab 1 -> 1 -> 1 -> 1 -> 1</span><br></pre></td></tr></table></figure>
<p>虽然三次握手没有完成，但是在服务端连接已经是 ESTABLISHED，所以这里的统计指标还是 PAWSEstab，可以通过 netstat -s 来查看：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#netstat -s |grep -E -i "timestamp|paws"</span><br><span class="line"> 71 packets rejected in established connections because of timestamp //无论是三次握手阶段的 RST 还是握手成功后的请求只要 timestamp 不递增就会 drop</span><br></pre></td></tr></table></figure>
<p>这个指标对应在 netstat 源码(net-tools) 中的解释：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">{"PAWSEstab", N_("%llu packets rejected in established connections because of timestamp"), opt_number},</span><br><span class="line"> {"PAWSPassive", N_("%llu passive connections rejected because of time stamp"), opt_number},</span><br></pre></td></tr></table></figure>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>星球里之前也写过 scapy 的入门以及使用案例： <a href="https://articles.zsxq.com/id_6r1xkzwdb8zp.html" target="_blank" rel="noopener">scapy 重现网络问题真香</a></p>
<p>就像学英语的时候要精读，分析 case 也需要深挖，可以挖上一到两周，不要每天假学习(似乎啥都看了，当时啥都懂，过几个月啥都不懂)</p>
<p>掌握技能比掌握知识点和问题的原因更重要</p>
<p>nettrace 也真的很好用/很好玩，可以帮你学到很多内核知识</p>
<h2 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h2><p><a href="https://cloud.tencent.com/developer/article/2210423" target="_blank" rel="noopener">https://cloud.tencent.com/developer/article/2210423</a></p>
<p><a href="https://articles.zsxq.com/id_52ha2j6r5gow.html" target="_blank" rel="noopener">为什么你的 SYN 包被丢 net.ipv4.tcp_tw_recycle</a></p>
<p><a href="https://articles.zsxq.com/id_6r1xkzwdb8zp.html" target="_blank" rel="noopener">从一个fin 卡顿问题到 scapy 的使用</a></p>
<h2 id="如果你觉得看完对你很有帮助可以通过如下方式找到我"><a href="#如果你觉得看完对你很有帮助可以通过如下方式找到我" class="headerlink" title="如果你觉得看完对你很有帮助可以通过如下方式找到我"></a>如果你觉得看完对你很有帮助可以通过如下方式找到我</h2><p>find me on twitter: <a href="https://twitter.com/plantegg" target="_blank" rel="noopener">@plantegg</a></p>
<p>知识星球：<a href="https://t.zsxq.com/0cSFEUh2J" target="_blank" rel="noopener">https://t.zsxq.com/0cSFEUh2J</a></p>