benchmark: 希望增加连接数较大的数据对比 #906
Comments
|
相同环境，对比了下我自己go框架版本的： package main
import (
"crypto/rand"
"fmt"
"net/http"
"os"
"os/signal"
"time"
"github.com/lesismal/nbio/nbhttp"
)
var (
data = make([]byte, 32)
)
type HttpHandler struct{}
func (h *HttpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
w.Header().Add("Date", time.Now().Format(time.RFC1123)) //"Mon, 02 Jan 2006 15:04:05 MST"
w.Header().Add("Content-Type", "text/plain; charset=UTF-8")
w.Write(data)
}
func main() {
rand.Read(data)
svr := nbhttp.NewServer(nbhttp.Config{
Network: "tcp",
Addrs: []string{"localhost:8888"},
NPoller: 4,
Handler: &HttpHandler{},
})
err := svr.Start()
if err != nil {
fmt.Printf("nbio.Start failed: %v\n", err)
return
}
defer svr.Stop()
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
<-interrupt
}200连接数ubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c200 --timeout 8 -t4 http://127.0.0.1:8888
Running 10s test @ http://127.0.0.1:8888
4 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 819.47us 1.36ms 25.33ms 93.47%
Req/Sec 85.61k 10.85k 112.94k 67.00%
Latency Distribution
50% 448.00us
75% 836.00us
90% 1.60ms
99% 7.55ms
3409325 requests in 10.01s, 484.46MB read
Requests/sec: 340599.12
Transfer/sec: 48.40MBubuntu@ubuntu:~/workflow/benchmark$ top -d 1 | grep _server
67665 ubuntu 20 0 2106524 54348 5660 S 303.0 0.7 0:26.98 go_server
67665 ubuntu 20 0 2106524 54348 5660 S 433.0 0.7 0:31.31 go_server
67665 ubuntu 20 0 2106524 53600 5660 S 427.7 0.7 0:35.63 go_server
67665 ubuntu 20 0 2106524 53600 5660 S 425.0 0.7 0:39.88 go_server
67665 ubuntu 20 0 2106524 55404 5660 S 418.8 0.7 0:44.11 go_server
67665 ubuntu 20 0 2106524 54792 5660 S 434.0 0.7 0:48.45 go_server
67665 ubuntu 20 0 2106524 54792 5660 R 427.7 0.7 0:52.77 go_server
67665 ubuntu 20 0 2106524 53168 5660 R 420.0 0.7 0:56.97 go_server
67665 ubuntu 20 0 2106524 54476 5660 R 422.8 0.7 1:01.24 go_server
67665 ubuntu 20 0 2106524 53080 5660 R 431.0 0.7 1:05.55 go_server
67665 ubuntu 20 0 2106524 53080 5660 S 106.9 0.7 1:06.63 go_server20k连接数ubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c20000 --timeout 8 -t4 http://127.0.0.1:8888
Running 10s test @ http://127.0.0.1:8888
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 80.88ms 35.35ms 358.53ms 68.47%
Req/Sec 49.74k 10.34k 91.54k 71.07%
Latency Distribution
50% 77.24ms
75% 103.26ms
90% 127.54ms
99% 175.82ms
1930526 requests in 10.07s, 274.32MB read
Requests/sec: 191790.50
Transfer/sec: 27.25MB 67665 ubuntu 20 0 2106524 54132 5660 R 44.6 0.7 1:07.08 go_server
67665 ubuntu 20 0 2107644 100180 5660 S 466.3 1.2 1:11.79 go_server
67665 ubuntu 20 0 2323080 111464 5724 S 476.0 1.4 1:16.55 go_server
67665 ubuntu 20 0 2323768 127076 5724 S 469.3 1.6 1:21.29 go_server
67665 ubuntu 20 0 2324408 130948 5724 S 469.0 1.6 1:25.98 go_server
67665 ubuntu 20 0 2324408 129652 5724 S 467.3 1.6 1:30.70 go_server
67665 ubuntu 20 0 2324472 154580 5724 S 443.6 1.9 1:35.18 go_server
67665 ubuntu 20 0 2324536 134668 5724 S 446.0 1.7 1:39.64 go_server
67665 ubuntu 20 0 2324536 129472 5724 S 436.6 1.6 1:44.05 go_server
67665 ubuntu 20 0 2324536 127012 5724 S 438.6 1.6 1:48.48 go_server
67665 ubuntu 20 0 2324536 121236 5724 S 347.5 1.5 1:51.99 go_server |
|
哈哈哈。感谢帮我做这个benchmark。 |
|
为什么你的server在20K的情况下，Latency不太好但qps却比较高？ |
|
我知道了，我们的server不调参数的情况下，默认2000并发。超过了，就关连接了。需要调节一下： WFServerParams params = HTTP_SERVER_PARAMS_DEFAULT;
params.max_connections = 30000;
WFHttpServer server(¶ms, process);
... |
这个白天再搞了，正准备睡了，你们也是这么晚不睡啊，哈哈哈 |
|
哈哈，我还没有那么早。感谢帮我们实验啊。 |
qps已经下降了很多了呀:joy: latancy不能完全反应出并发量，主要看单个请求的latancy是否跟server处理这单个请求时的阻塞程度，如果server阻塞了则cpu利用率就下降了，然后处理其他请求就也慢、并发量自然就下降了，我估计你们高连接数时下降就是这个原因导致的 我的是golang runtime在调度，协程数量充足，runtime抢占式效率还行，所以处理单个请求并不会因为单个请求阻塞而让cpu空闲太多导致利用率下降，只要cpu跑的欢，并发量就还能得以保持 |
多交流互相促进就好。 等我闲了也学习研究下你们的代码，主要想看下IO线程和逻辑线程、消息处理的同步异步之类的细节，很久没搞cpp了，而且我自己主要是用c++11之前的。 |
猜测可能是go的异步是语言级别的调度，所以即使latency上去了，go依然可以通过workstealing的方式把qps做得很好。 |
cpp肯定能做到比go快，主要看IO和逻辑线程池、同步异步的配合细节了。 |
我们这个cpp简单又护发的。 |
应该是我们max connections没调的原因。主要我看到我们20K那个测试很多Socket error，连接被我们关了。我也记得20K没下降得这么离谱😂 |
酱紫啊:joy:，需要怎么改下最大连接数的，我再跑跑看 |
WFServerParams params = HTTP_SERVER_PARAMS_DEFAULT; |
|
我改成30w了，正常了: 200ubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c200 --timeout 8 -t4 http://127.0.0.1:9000
Running 10s test @ http://127.0.0.1:9000
4 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 574.73us 1.05ms 19.58ms 96.79%
Req/Sec 99.76k 9.86k 123.00k 67.75%
Latency Distribution
50% 374.00us
75% 564.00us
90% 834.00us
99% 6.21ms
3972666 requests in 10.01s, 655.43MB read
Requests/sec: 396710.13
Transfer/sec: 65.45MB20kubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c20000 --timeout 8 -t4 http://127.0.0.1:9000
Running 10s test @ http://127.0.0.1:9000
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.96ms 1.87ms 44.18ms 90.63%
Req/Sec 98.96k 31.18k 195.89k 67.84%
Latency Distribution
50% 1.47ms
75% 2.43ms
90% 3.73ms
99% 9.52ms
3936124 requests in 10.09s, 649.40MB read
Socket errors: connect 18983, read 0, write 0, timeout 0
Requests/sec: 390086.10
Transfer/sec: 64.36MB100kubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c100000 --timeout 8 -t4 http://127.0.0.1:9000
Running 10s test @ http://127.0.0.1:9000
4 threads and 100000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.95ms 1.62ms 27.62ms 86.82%
Req/Sec 99.63k 56.37k 203.55k 56.12%
Latency Distribution
50% 1.59ms
75% 2.42ms
90% 3.58ms
99% 8.32ms
3738175 requests in 10.04s, 616.75MB read
Socket errors: connect 98983, read 0, write 0, timeout 0
Requests/sec: 372448.68
Transfer/sec: 61.45MB |
但是max connections改好了之后，没有太明显下降啊，应该还是比nginx强吧:joy:。。。 |
|
哈哈哈。这个数据很开心：） |
晚安了，这个close了先，workflow性能很赞！ |
可能那次也是我们忘了调😓 |
那这个得算我辅助上分了，哈哈哈 |
|
20k和100k的测试，我觉得数据有点问题…… |
|
我知道了，可能是你测试间隔太短，大量端口还没有被系统回收，连接建立不起来。或者你不小心把server进程max open files设置为1024了？ 2002K20K28K超过30K的话，单个wrk进程就无端口可用了，连接无法建立。 |
哦对，100k的是没有端口可用的，但是20k的应该是没问题的，难道我记错了？但是go的代码是同一个目录同一个终端、http_server关闭后同一个终端启动的go_server然后go的好像是正常的。。。
我应该都是 |
|
go_server的测试没有报 ubuntu@ubuntu:~/wrk$ ./wrk --latency -d30 -c200 --timeout 8 -t4 http://127.0.0.1:9000
Running 30s test @ http://127.0.0.1:9000
4 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 567.03us 1.07ms 22.27ms 96.69%
Req/Sec 101.60k 13.16k 148.35k 74.83%
Latency Distribution
50% 362.00us
75% 550.00us
90% 823.00us
99% 6.41ms
12134767 requests in 30.03s, 1.96GB read
Requests/sec: 404057.05
Transfer/sec: 66.66MB
ubuntu@ubuntu:~/wrk$
ubuntu@ubuntu:~/wrk$ ./wrk --latency -d30 -c20000 --timeout 8 -t4 http://127.0.0.1:9000
Running 30s test @ http://127.0.0.1:9000
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 60.19ms 30.54ms 290.17ms 70.95%
Req/Sec 59.12k 15.97k 100.87k 68.30%
Latency Distribution
50% 55.61ms
75% 74.30ms
90% 100.77ms
99% 159.24ms
6844898 requests in 30.10s, 1.10GB read
Requests/sec: 227374.27
Transfer/sec: 37.51MBubuntu@ubuntu:~$ netstat -ntp | grep _server | grep ESTAB | wc -l
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
20000 |
|
我又试了下，把 ubuntu@ubuntu:~/wrk$ ./wrk --latency -d10 -c20000 --timeout 8 -t4 http://127.0.0.1:9000
Running 10s test @ http://127.0.0.1:9000
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.94ms 1.80ms 26.97ms 89.68%
Req/Sec 97.42k 38.16k 199.68k 68.51%
Latency Distribution
50% 1.43ms
75% 2.39ms
90% 3.79ms
99% 9.27ms
3863010 requests in 10.09s, 637.34MB read
Socket errors: connect 18983, read 0, write 0, timeout 0
Requests/sec: 382937.20
Transfer/sec: 63.18MB再测30s，吞吐量也还是比较高 ubuntu@ubuntu:~/wrk$ ./wrk --latency -d30 -c20000 --timeout 8 -t4 http://127.0.0.1:9000
Running 30s test @ http://127.0.0.1:9000
4 threads and 20000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.01ms 1.87ms 39.86ms 89.25%
Req/Sec 93.31k 33.93k 209.72k 67.03%
Latency Distribution
50% 1.46ms
75% 2.51ms
90% 4.00ms
99% 9.62ms
11115514 requests in 30.07s, 1.79GB read
Socket errors: connect 18983, read 0, write 0, timeout 0
Requests/sec: 369597.43
Transfer/sec: 60.98MB昨天的高吞吐估计确实是我在修改max connections 30w后，不小心开了新的终端测试忘记了 昨天半夜只看了吞吐量、qps，没注意到 |
|
那我reopen了，咱继续调查:joy: |
|
刚才开始阅读了一点源码，发现这里似乎存在一些问题： struct poller_params params = {
.max_open_files = 65536,
.create_message = Communicator::create_message,
.partial_written = Communicator::partial_written,
.callback = Communicator::callback,
.context = this
};max_open_files写死了，并且后续的nodes_buf、poller accept后对fd的判断，都被这个值限制了。但max_open_files其实并不是最大65535。65535只是一对IP的套接口数量的限制，因为两个IP和server port都是固定，剩下的可变的只能是client port，short类型所以才受这个限制。但如果client是来自多个IP、每个client IP都能一组，就可以远大于这个值。 |
|
18983 connect errors的，应该是你吧http_server的ulimit -n设置为1024了吧？不是程序的问题。这个数字太规律了。 |
对，应该是我新开的窗口忘记ulimit改大了，前面这里用大的和1024的值测试确认了1024时候的现象与错误结果吻合： |
只要是c++，就没有护发的啊，今天看了几小时c++感觉比我平时看go几天都累:joy: |
#909 |
我本地跑了下 http_server，机器是4c8t，给server设置了4线程
env
200连接数时表现很棒
20k连接数时表现不及预期
如何优化
连接数较大时cpu利用率下降、相对的性能下降也比较明显。
使用是默认的代码、没做修改，是否能够通过修改配置或代码来进行优化？
The text was updated successfully, but these errors were encountered: