Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memcached is not working with DPDK on Intel servers #25

Open
slivne opened this issue Feb 5, 2015 · 8 comments
Open

Memcached is not working with DPDK on Intel servers #25

slivne opened this issue Feb 5, 2015 · 8 comments
Labels
tcp/ip Native (in-Seastar) TCP/IP stack

Comments

@slivne
Copy link
Contributor

slivne commented Feb 5, 2015

DPDK 1.8.0
seastar : 89763c95c92b44d3482e7500ddd5c14b8c47b76en
also seastar: 29366cb
(same results)

  • boot seastar on dpdk1 with
    '''
    sudo build/release/apps/memcached/memcached --network-stack native --dhcp 0 --host-ipv4-addr 192.168.20.101 --netmask-ipv4-addr 255.255.255.0 --gw-ipv4-addr 192.168.20.185 --smp 1 --dpdk-pmd
    '''
  • run memaslap on dpdk2 with
    '''
    memaslap -s 192.168.20.101:11211 -t 20s
    '''
  • after memaslap ends on dpdk2 send pings
    '''
    ping 192.168.20.101
    '''

seastar httpd with dpdk does work on the servers

'''
sudo build/release/apps/httpd/httpd --network-stack nave --dhcp 0 --host-ipv4-addr 192.168.20.101 --netmask-ipv4-addr 255.255.255.0 --gw-ipv4-addr 192.168.20.185 --smp 1 --dpdk-pmd
'''
from packet 75 it seems that seastar has lost network connectivity

after memaslap ends send pings confirms this

I'll attach a wireshark capture
[shlomi@dpdk2 ~]$ sudo tshark -i enp129s0 -r /tmp/memached_capture.cap
Running as user "root" and group "root". This could be dangerous.
1 0.000000000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
2 0.000032000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
3 1.000040000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
4 1.000068000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
5 2.000068000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
6 2.000098000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
7 3.000085000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
8 3.000112000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
9 4.000116000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
10 4.000145000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
11 5.000132000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
12 5.000158000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
13 6.000160000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
14 6.000192000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
15 7.000172000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
16 7.000201000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
17 8.000211000 192.168.20.101 -> 239.192.74.66 collectd 1041 Host=dpdk1, 15 values for 4 plugins, 0 messages
18 8.000240000 192.168.20.101 -> 239.192.74.66 collectd 246 Host=dpdk1, 3 values for 1 plugin, 0 messages
19 8.279197000 192.168.20.185 -> 192.168.20.101 TCP 74 51693→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032647 TSecr=0 WS=128
20 8.279263000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51693 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
21 8.279337000 192.168.20.185 -> 192.168.20.101 TCP 54 51693→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
22 8.280717000 192.168.20.185 -> 192.168.20.101 TCP 74 51694→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032649 TSecr=0 WS=128
23 8.280774000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51694 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
24 8.280822000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
25 8.281946000 192.168.20.185 -> 192.168.20.101 TCP 74 51695→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032650 TSecr=0 WS=128
26 8.281992000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51695 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
27 8.282038000 192.168.20.185 -> 192.168.20.101 TCP 54 51695→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
28 8.283147000 192.168.20.185 -> 192.168.20.101 TCP 74 51696→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032651 TSecr=0 WS=128
29 8.283192000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51696 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
30 8.283233000 192.168.20.185 -> 192.168.20.101 TCP 54 51696→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
31 8.284336000 192.168.20.185 -> 192.168.20.101 TCP 74 51697→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032652 TSecr=0 WS=128
32 8.284380000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51697 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
33 8.284430000 192.168.20.185 -> 192.168.20.101 TCP 54 51697→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
34 8.285531000 192.168.20.185 -> 192.168.20.101 TCP 74 51698→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032654 TSecr=0 WS=128
35 8.285576000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51698 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
36 8.285618000 192.168.20.185 -> 192.168.20.101 TCP 54 51698→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
37 8.286725000 192.168.20.185 -> 192.168.20.101 TCP 74 51699→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032655 TSecr=0 WS=128
38 8.286761000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51699 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
39 8.286802000 192.168.20.185 -> 192.168.20.101 TCP 54 51699→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
40 8.287924000 192.168.20.185 -> 192.168.20.101 TCP 74 51700→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032656 TSecr=0 WS=128
41 8.287966000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51700 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
42 8.288007000 192.168.20.185 -> 192.168.20.101 TCP 54 51700→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
43 8.289107000 192.168.20.185 -> 192.168.20.101 TCP 74 51701→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032657 TSecr=0 WS=128
44 8.289152000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51701 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
45 8.289194000 192.168.20.185 -> 192.168.20.101 TCP 54 51701→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
46 8.290280000 192.168.20.185 -> 192.168.20.101 TCP 74 51702→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032658 TSecr=0 WS=128
47 8.290325000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51702 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
48 8.290366000 192.168.20.185 -> 192.168.20.101 TCP 54 51702→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
49 8.291402000 192.168.20.185 -> 192.168.20.101 TCP 74 51703→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032659 TSecr=0 WS=128
50 8.291442000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51703 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
51 8.291483000 192.168.20.185 -> 192.168.20.101 TCP 54 51703→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
52 8.292512000 192.168.20.185 -> 192.168.20.101 TCP 74 51704→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032661 TSecr=0 WS=128
53 8.292557000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51704 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
54 8.292598000 192.168.20.185 -> 192.168.20.101 TCP 54 51704→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
55 8.293634000 192.168.20.185 -> 192.168.20.101 TCP 74 51705→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032662 TSecr=0 WS=128
56 8.293679000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51705 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
57 8.293720000 192.168.20.185 -> 192.168.20.101 TCP 54 51705→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
58 8.294760000 192.168.20.185 -> 192.168.20.101 TCP 74 51706→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032663 TSecr=0 WS=128
59 8.294805000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51706 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
60 8.294846000 192.168.20.185 -> 192.168.20.101 TCP 54 51706→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
61 8.295890000 192.168.20.185 -> 192.168.20.101 TCP 74 51707→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032664 TSecr=0 WS=128
62 8.295926000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51707 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
63 8.295967000 192.168.20.185 -> 192.168.20.101 TCP 54 51707→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
64 8.297002000 192.168.20.185 -> 192.168.20.101 TCP 74 51708→11211 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=11032665 TSecr=0 WS=128
65 8.297047000 192.168.20.101 -> 192.168.20.185 TCP 62 11211→51708 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
66 8.297089000 192.168.20.185 -> 192.168.20.101 TCP 54 51708→11211 [ACK] Seq=1 Ack=1 Win=29312 Len=0
67 8.297145000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
68 8.297168000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
69 8.297184000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
70 8.297198000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
71 8.297215000 192.168.20.101 -> 192.168.20.185 MEMCACHE 62 STORED
72 8.297222000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
73 8.297224000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
74 8.297225000 192.168.20.101 -> 192.168.20.185 MEMCACHE 62 STORED
75 8.297235000 192.168.20.185 -> 192.168.20.101 MEMCACHE 1159 set \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf. 0 0 1024
76 8.297243000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [ACK] Seq=1106 Ack=9 Win=29312 Len=0
77 8.297457000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
78 8.297467000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
79 8.297475000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
80 8.297484000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
81 8.297493000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
82 8.297516000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
83 8.297529000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
84 8.297540000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
85 8.497421000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
86 8.497432000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
87 8.497437000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
88 8.497441000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
89 8.498412000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
90 8.498419000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
91 8.498424000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
92 8.498429000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Previous segment not captured] get \020p\024\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
93 8.498434000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
94 8.498440000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
95 8.498445000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
96 8.900453000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
97 8.900467000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
98 8.900472000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
99 8.900477000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
100 8.901460000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
101 8.901475000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
102 8.901480000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
103 8.901484000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\024\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
104 8.901489000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\020\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
105 8.901493000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
106 8.901498000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
107 9.707471000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
108 9.707485000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
109 9.707490000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
110 9.707494000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
111 9.707499000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
112 9.707503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
113 9.707508000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
114 9.707522000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
115 9.707527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
116 9.707533000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
117 9.707538000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
118 9.707543000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
119 9.707555000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
120 9.707560000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
121 11.319491000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
122 11.319506000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
123 11.319510000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
124 11.319515000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
125 11.319521000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
126 11.319525000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
127 11.319530000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
128 11.319545000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
129 11.319550000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
130 11.319557000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
131 11.319562000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
132 11.319567000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
133 11.319578000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
134 11.319583000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
135 11.535564000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020P\021\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
136 11.535583000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\360\021\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
137 11.535589000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\220\022\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
138 11.535594000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \0200\023\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
139 11.535599000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\320\023\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
140 11.535605000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020p\024\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
141 11.535610000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020\020\025\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
142 11.535621000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 get \020P\026\020\020\020\020\020StOIPA4ySHgoaLAOv1KiqE.MhgnpRkwtbIA.TEwvWah4vrTpsnVg-T1h
143 14.543471000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
144 14.543487000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
145 14.543492000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
146 14.543497000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
147 14.543503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
148 14.543507000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
149 14.543513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
150 14.543527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\260\025\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
151 14.543531000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
152 14.543536000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\026\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
153 14.543543000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\027\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
154 14.543547000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
155 14.543560000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\030\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
156 14.543565000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020p\031\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
157 20.991488000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
158 20.991503000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
159 20.991508000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
160 20.991513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
161 20.991518000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
162 20.991522000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
163 20.991527000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
164 29.301230000 192.168.20.185 -> 192.168.20.101 TCP 54 51693→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
165 29.301290000 192.168.20.185 -> 192.168.20.101 TCP 54 51694→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
166 29.301312000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51695→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
167 29.301327000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51696→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
168 29.301341000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51697→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
169 29.301345000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51698→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
170 29.301348000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51699→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
171 29.301386000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51700→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
172 29.301388000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51701→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
173 29.301389000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51702→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
174 29.301391000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51703→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
175 29.301395000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51704→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
176 29.301608000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51705→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
177 29.301609000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51706→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
178 29.301611000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51707→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
179 29.301611000 192.168.20.185 -> 192.168.20.101 TCP 54 [TCP ACKed unseen segment] 51708→11211 [FIN, ACK] Seq=1176 Ack=9 Win=29312 Len=0
180 33.871513000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\020\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
181 33.871530000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP Retransmission] get \020\260\020\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
182 33.871537000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020P\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
183 33.871545000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\360\021\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
184 33.871551000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\220\022\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
185 33.871558000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \0200\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
186 33.871565000 192.168.20.185 -> 192.168.20.101 MEMCACHE 124 [TCP ACKed unseen segment] [TCP Retransmission] get \020\320\023\020\020\020\020\020wQngU18jgs8nK-7r1WKvqcCar.jxZvRTJD7oEFVu5dfPcmFeHZX59jf.
187 45.301172000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=1/256, ttl=64
188 46.300493000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=2/512, ttl=64
189 47.300532000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=3/768, ttl=64
190 48.300682000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=4/1024, ttl=64
191 49.300541000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=5/1280, ttl=64
192 50.300491000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=6/1536, ttl=64
193 50.319452000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
194 51.300489000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=7/1792, ttl=64
195 51.321471000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
196 52.300480000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=8/2048, ttl=64
197 52.323450000 IntelCor_2d:3a:00 -> IntelCor_27:d3:ea ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
198 53.300465000 192.168.20.185 -> 192.168.20.101 ICMP 98 Echo (ping) request id=0x1f9d, seq=9/2304, ttl=64
199 54.300466000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
200 55.303461000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
201 56.305472000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
202 59.663499000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
203 60.665480000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
204 61.667452000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
@slivne
Copy link
Contributor Author

slivne commented Feb 5, 2015

https://drive.google.com/open?id=0B3W1iB9JfvzyejBxTlZ5XzgwaVE&authuser=0

@asias
Copy link
Contributor

asias commented Feb 10, 2015

I used tcp_server rxrx test + debug print. DPDK were dropping some packets. Only seq=2925 and seq=11401 of the data packets are received and passed from eth to ip/tcp. We even did not see the other data packets at layer 2.

[asias@dpdk1 seastar]$  /home/asias/src/seastar/build/release/tests/tcp_server --smp 1 --network-stack native --host-ipv4-addr 192.168.20.101 --gw-ipv4-addr 192.168.20.185 --collectd 0 --tso on --memory 4096M --ufo off --dpdk-pmd
Seastar TCP server listening on port 10000 ...         
dpdk_pool 2560
dpdk_pool 2560
============== got eth: p.len=56
============== got eth: p.len=74
--------------ip rx p.len=60
to l4
tcp_received p.len()=40
tcp_received src_port=39231
============== got eth: p.len=56
--------------ip rx p.len=42
1: ip_len=40
to l4
tcp_received p.len()=20
tcp_received src_port=39231
============== got eth: p.len=58
--------------ip rx p.len=44
to l4
tcp_received p.len()=24
tcp_received src_port=39231
Got data for us
============== got eth: p.len=1230
--------------ip rx p.len=1216
to l4
tcp_received p.len()=1196
tcp_received src_port=39231
ooo port=39231, nr=0, seq=2925, seq_abs=4089585917
============== got eth: p.len=946
--------------ip rx p.len=932
to l4
tcp_received p.len()=912
tcp_received src_port=39231
ooo port=39231, nr=1, seq=11401, seq_abs=4089594393
dpdk_pool 2561                                                                                                    
dpdk_pool 2561

[asias@dpdk2 ~]$ go run client.go -host 192.168.20.101 -test rxrx  -conn 1
========== rxrx ============
Server:  192.168.20.101:10000
Connections:  1

[asias@dpdk2 ~]$ sudo tshark -i enp129s0 -c 100
Running as user "root" and group "root". This could be dangerous.
Capturing on 'enp129s0'
1 0.000000 IntelCor_2d:3a:00 -> Broadcast ARP 42 Who has 192.168.20.101? Tell 192.168.20.185
2 0.000087 IntelCor_27:d3:ea -> IntelCor_2d:3a:00 ARP 60 192.168.20.101 is at 68:05:ca:27:d3:ea
3 0.000103 192.168.20.185 -> 192.168.20.101 TCP 74 39231→10000 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=411871771 TSecr=0 WS=128
4 0.000175 192.168.20.101 -> 192.168.20.185 TCP 62 10000→39231 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
5 0.000243 192.168.20.185 -> 192.168.20.101 TCP 54 39231→10000 [ACK] Seq=1 Ack=1 Win=29312 Len=0
6 0.000336 192.168.20.185 -> 192.168.20.101 TCP 58 39231→10000 [PSH, ACK] Seq=1 Ack=1 Win=29312 Len=4
7 0.000374 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
8 0.000378 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=1465 Ack=1 Win=29312 Len=1460
9 0.000380 192.168.20.185 -> 192.168.20.101 TCP 1230 39231→10000 [PSH, ACK] Seq=2925 Ack=1 Win=29312 Len=1176
10 0.000401 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=4101 Ack=1 Win=29312 Len=1460
11 0.000403 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=5561 Ack=1 Win=29312 Len=1460
12 0.000404 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=7021 Ack=1 Win=29312 Len=1460
13 0.000406 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=8481 Ack=1 Win=29312 Len=1460
14 0.000407 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=9941 Ack=1 Win=29312 Len=1460
15 0.000409 192.168.20.185 -> 192.168.20.101 TCP 946 39231→10000 [PSH, ACK] Seq=11401 Ack=1 Win=29312 Len=892
16 0.000454 192.168.20.101 -> 192.168.20.185 TCP 60 10000→39231 [ACK] Seq=1 Ack=5 Win=3737600 Len=0
17 0.000492 192.168.20.101 -> 192.168.20.185 TCP 60 [TCP Dup ACK 16#1] 10000→39231 [ACK] Seq=1 Ack=5 Win=3737600 Len=0
18 0.000512 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=12293 Ack=1 Win=29312 Len=1460
19 0.000577 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=13753 Ack=1 Win=29312 Len=1460
20 0.000585 192.168.20.185 -> 192.168.20.101 TCP 1514 39231→10000 [ACK] Seq=15213 Ack=1 Win=29312 Len=1460
21 0.200614 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
22 0.603570 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
23 1.408608 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
24 3.020611 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460
25 6.244586 192.168.20.185 -> 192.168.20.101 TCP 1514 [TCP Retransmission] 39231→10000 [ACK] Seq=5 Ack=1 Win=29312 Len=1460

@asias
Copy link
Contributor

asias commented Feb 10, 2015

Added more debug print in dpdk_qp::process_packets().

We did not see other data packets in dpdk_qp::process_packets() neither.

And we do did drop packet because of ip or tcp csum error.

            if (m->ol_flags & (PKT_RX_IP_CKSUM_BAD | PKT_RX_L4_CKSUM_BAD)) {
                // Packet with bad checksum, just drop it.
                print("dpdk_qp drop pkt\n");
                continue;
            }

EAL:   0000:03:00.1 not managed by UIO driver, skipping
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1584 rte_i40e_pmd
EAL:   0000:05:00.0 not managed by VFIO driver, skipping
EAL:   0000:05:00.0 not managed by UIO driver, skipping
EAL: PCI device 0000:81:00.0 on NUMA socket -1
EAL:   probe driver: 8086:1584 rte_i40e_pmd
EAL:   0000:81:00.0 not managed by VFIO driver, skipping
EAL:   PCI memory mapped at 0x7ffef13ff000
EAL:   PCI memory mapped at 0x7ffff7fea000
PMD: eth_i40e_dev_init(): FW 4.1 API 1.1 NVM 04.01.00 eetrack 80001121
PMD: eth_i40e_dev_init(): Failed to stop lldp
PMD: i40e_pf_parameter_init(): Max supported VSIs:130
PMD: i40e_pf_parameter_init(): PF queue pairs:64
PMD: i40e_pf_parameter_init(): Max VMDQ VSI num:63
PMD: i40e_pf_parameter_init(): VMDQ queue pairs:4
[New Thread 0x7ffef0fff700 (LWP 144203)]
ports number: 1
Port 0: max_rx_queues 316 max_tx_queues 316
Port 0: using 1 queue
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
Port 0 init ... done:
Creating mbuf pool 'dpdk_net_pktmbuf_pool0' [3072 mbufs] ...
PMD: i40e_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
PMD: i40e_dev_tx_queue_setup(): Using full-featured tx path

Checking link status
Created DPDK device
.done
Port 0 Link Up - speed 40000 Mbps - full-duplex
Seastar TCP server listening on port 10000 ...
^^^^^^^^^^^^^^^^ dpdk_qp p.len=74
dpdk_qp to l2
============== got eth: p.len=74
--------------ip rx p.len=60
to l4
tcp_received p.len()=40
tcp_received src_port=39235
^^^^^^^^^^^^^^^^ dpdk_qp p.len=56
dpdk_qp to l2
============== got eth: p.len=56
--------------ip rx p.len=42
1: ip_len=40
to l4
tcp_received p.len()=20
tcp_received src_port=39235
^^^^^^^^^^^^^^^^ dpdk_qp p.len=58
dpdk_qp to l2
============== got eth: p.len=58
--------------ip rx p.len=44
to l4
tcp_received p.len()=24
tcp_received src_port=39235
Got data for us
^^^^^^^^^^^^^^^^ dpdk_qp p.len=1230
dpdk_qp to l2
============== got eth: p.len=1230
--------------ip rx p.len=1216
to l4
tcp_received p.len()=1196
tcp_received src_port=39235
ooo port=39235, nr=0, seq=2925, seq_abs=2182892504
^^^^^^^^^^^^^^^^ dpdk_qp p.len=946
dpdk_qp to l2
============== got eth: p.len=946
--------------ip rx p.len=932
to l4
tcp_received p.len()=912
tcp_received src_port=39235
ooo port=39235, nr=1, seq=11401, seq_abs=2182900980
dpdk_pool 2561
^^^^^^^^^^^^^^^^ dpdk_qp p.len=56
dpdk_qp to l2
============== got eth: p.len=56
dpdk_pool 2561
dpdk_pool 2561

@gleb-cloudius
Copy link
Contributor

The problem is that memcached send packets with 9 fragments each, but this HW is only capable of sending packets with 8 frags ot less if TSO is not enabled and it is not enabled for DPDK yet. I haven't found how many frags are supported with TSO, but definitely greater than 8.

@nyh
Copy link
Contributor

nyh commented Feb 12, 2015

On Thu, Feb 12, 2015 at 11:17 AM, Gleb Natapov notifications@github.com
wrote:

The problem is that memcached send packets with 9 fragments each, but this
HW is only capable of sending packets with 8 frags ot less if TSO

Interesting. While we can - and probably should - fix memcached to send
fewer fragments, someone can come along and write a new application that
again tries to send 9 fragments. So definitely Seastar itself needs to
realize it was given too many fragments and merge some of them (by copying)
until we have fewer than the limit. Yesterday there was also a discussion
on how such merging for tiny fragments is also needed for performance.

Nadav Har'El
nyh@cloudius-systems.com

@gleb-cloudius
Copy link
Contributor

On Thu, Feb 12, 2015 at 01:33:37AM -0800, nyh wrote:

On Thu, Feb 12, 2015 at 11:17 AM, Gleb Natapov notifications@github.com
wrote:

The problem is that memcached send packets with 9 fragments each, but this
HW is only capable of sending packets with 8 frags ot less if TSO

Interesting. While we can - and probably should - fix memcached to send
fewer fragments, someone can come along and write a new application that
again tries to send 9 fragments. So definitely Seastar itself needs to
realize it was given too many fragments and merge some of them (by copying)
until we have fewer than the limit. Yesterday there was also a discussion
on how such merging for tiny fragments is also needed for performance.

The fix is to use TSO if it was not clear from my previous message :)
TSO is needed anyway for reasonable TCP performance.

        Gleb.

@nyh
Copy link
Contributor

nyh commented Feb 12, 2015

On Thu, Feb 12, 2015 at 11:36 AM, Gleb Natapov notifications@github.com
wrote:

The fix is to use TSO if it was not clear from my previous message :)
TSO is needed anyway for reasonable TCP performance.

It's not clear TSO would be a complete solution because:

  1. It's not clear to me that the 8-fragment limit of the NIC has a
    8-fragment limit is only when TSO is not enabled. (sounds reasonable, but
    it needs checking...).
  2. If there's such a thing of a limit on the number of fragments, Seastar
    needs to be aware of it. If it's not 8 and it's 64, Seastar still needs to
    do something different if someone tries to send 65 fragments...

Nadav Har'El
nyh@cloudius-systems.com

@gleb-cloudius
Copy link
Contributor

On Thu, Feb 12, 2015 at 01:41:19AM -0800, nyh wrote:

On Thu, Feb 12, 2015 at 11:36 AM, Gleb Natapov notifications@github.com
wrote:

The fix is to use TSO if it was not clear from my previous message :)
TSO is needed anyway for reasonable TCP performance.

It's not clear TSO would be a complete solution because:

  1. It's not clear to me that the 8-fragment limit of the NIC has a
    8-fragment limit is only when TSO is not enabled. (sounds reasonable, but
    it needs checking...).
    Spec says this specifically that's why I wrote what I wrote. This is not
    a guess work.
  2. If there's such a thing of a limit on the number of fragments, Seastar
    needs to be aware of it. If it's not 8 and it's 64, Seastar still needs to
    do something different if someone tries to send 65 fragments...

Of course, currently we linearize if there is more then 32 frags.

        Gleb.

@vladzcloudius vladzcloudius mentioned this issue Jul 29, 2015
Closed
@nlfox nlfox mentioned this issue Jan 13, 2018
Open
@frank8989 frank8989 mentioned this issue Feb 26, 2018
Open
@closertotheheart closertotheheart mentioned this issue Jan 30, 2019
Closed
@Howe2015 Howe2015 mentioned this issue Aug 31, 2020
Open
avikivity pushed a commit that referenced this issue Apr 6, 2021
@bhalevy @avikivity
test: closeable: allocate variables accessed by continuations using d…
4d69526 
…o_with

Fixes failures in debug mode:
```
$ build/debug/tests/unit/closeable_test -l all -t deferred_close_test
WARNING: debug mode. Not for benchmarking or production
random-seed=3064133628
Running 1 test case...
Entering test module "../../tests/unit/closeable_test.cc"
../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test"
../../src/testing/seastar_test.cc(43): info: check true has passed
==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
terminate called after throwing an instance of 'seastar::broken_promise'
  what():  broken promise
==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592)
False positive error reports may follow
For details see google/sanitizers#189
=================================================================
==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
    #0 0x7fbf45f39d0b  (/lib64/libasan.so.6+0xb3d0b)
    #1 0x7fbf45f57d4e  (/lib64/libasan.so.6+0xd1d4e)
    #2 0x7fbf45f3e724  (/lib64/libasan.so.6+0xb8724)
    #3 0x7fbf45eb3e5b  (/lib64/libasan.so.6+0x2de5b)
    #4 0x7fbf45eb51e8  (/lib64/libasan.so.6+0x2f1e8)
    #5 0x7fbf45eb7694  (/lib64/libasan.so.6+0x31694)
    #6 0x7fbf45f39398  (/lib64/libasan.so.6+0xb3398)
    #7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b)
    #8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116
    #9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121
    #10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810)
    #11 0x7fbf44bcd897  (/lib64/libgcc_s.so.1+0xf897)
    #12 0x7fbf44bcea5f  (/lib64/libgcc_s.so.1+0x10a5f)
    #13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8)
    #14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148
    #15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb)
    #16 0x7fbf45e10a21  (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21)
    #17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0)
    #18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094)
    #19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921)
    #20 0x7fbf45e5eae1  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1)
    #21 0x7fbf45e5ed31  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31)
    #22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547)
    #23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618)
    #24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77
    #25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65
    #26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20210406100911.12278-1-bhalevy@scylladb.com>
@nyh nyh added the tcp/ip Native (in-Seastar) TCP/IP stack label Feb 15, 2022
@nlescoua nlescoua mentioned this issue Mar 28, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tcp/ip Native (in-Seastar) TCP/IP stack
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@asias @nyh @slivne @gleb-cloudius