Include "connection reset by peer" errors in the summary

[Jongy]

Proposal

Vegeta should not stop on read tcp ... read: connection reset by peer errors, but continue and include them in the final summary.

Background

I'm trying to use Vegeta to test the downtime effect of an application restart. If I run

echo 'GET http://localhost:8080/' | ./vegeta attack -duration 5s -rate 100 | ./vegeta report

while the app is not listening, I'll get

...
Status Codes  [code:count]                      0:500
Error Set:
Get "http://localhost:8080/": dial tcp 0.0.0.0:0->127.0.0.1:8080: connect: connection refused

as expected. But if the app is running when Vegeta is started, and I quickly kill the app and restart it, Vegeta stops when its existing connections receive ECONNRESET, and prints a set of Get "http://localhost:8080/": read tcp ... read: connection reset by peer. If I reset the app nicely (not killing any already-open connections) then nothing bad happens (only a few "connection refused" errors during the downtime).

I'd expect it to "catch" the errors, include them in the summary and continue. I think it already includes those errors in the summary, but it still bails out and prints a bunch of stacktraces. Just like Vegeta counts "connection refused" errors and continues on, it should count "connection reset" errors and continue on.

Workarounds

Didn't think of any currently.

[tsenart]

I think Vegeta does continue as you expect, but sending 500 requests that return immediately because they fail to establish a connection takes very little time.

[Jongy]

The reason I think Vegeta does not continue is that the total count of requests does not match duration x rate. For example, in the -duration 5s -rate 100 case, if I kill the app - Vegeta stops without 500 total requests.

[Jongy]

Here's a result demonstrating this: echo 'GET http://localhost:8000/' | ./vegeta attack -duration 5s -rate 100

stacktraces...
...
...
Requests      [total, rate, throughput]         178, 100.56, 47.04
Duration      [total, attack, wait]             3.657s, 1.77s, 1.887s
Latencies     [min, mean, 50, 90, 95, 99, max]  485.427_s, 65.066ms, 640.125_s, 846.552_s, 1.181ms, 1.924s, 1.936s
Bytes In      [total, mean]                     197456, 1109.30
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           96.63%
Status Codes  [code:count]                      0:6  200:172  
Error Set:
Get "http://localhost:8000/": read tcp 127.0.0.1:55655->127.0.0.1:8000: read: connection reset by peer
Get "http://localhost:8000/": read tcp 127.0.0.1:50491->127.0.0.1:8000: read: connection reset by peer
Get "http://localhost:8000/": read tcp 127.0.0.1:34533->127.0.0.1:8000: read: connection reset by peer
Get "http://localhost:8000/": read tcp 127.0.0.1:52321->127.0.0.1:8000: read: connection reset by peer
Get "http://localhost:8000/": read tcp 127.0.0.1:39267->127.0.0.1:8000: read: connection reset by peer
Get "http://localhost:8000/": read tcp 127.0.0.1:32869->127.0.0.1:8000: read: connection reset by peer

[tsenart]

Interesting. Did you omit stack traces? Can you show those? Sent via Superhuman iOS ( https://sprh.mn/?vip=tsenart@gmail.com )

…

On Sun, Jul 12 2020 at 10:01 AM, Yonatan Goldschmidt < ***@***.*** > wrote: Here's a result demonstrating this: echo 'GET http://localhost:8000/' | ./vegeta attack -duration 5s -rate 100 stacktraces... ... ... Requests [total, rate, throughput] 178, 100.56, 47.04 Duration [total, attack, wait] 3.657s, 1.77s, 1.887s Latencies [min, mean, 50, 90, 95, 99, max] 485.427_s, 65.066ms, 640.125_s, 846.552_s, 1.181ms, 1.924s, 1.936s Bytes In [total, mean] 197456, 1109.30 Bytes Out [total, mean] 0, 0.00 Success [ratio] 96.63% Status Codes [code:count] 0:6 200:172 Error Set: Get "http://localhost:8000/": read tcp 127.0.0.1:55655->127.0.0.1:8000: read: connection reset by peer Get "http://localhost:8000/": read tcp 127.0.0.1:50491->127.0.0.1:8000: read: connection reset by peer Get "http://localhost:8000/": read tcp 127.0.0.1:34533->127.0.0.1:8000: read: connection reset by peer Get "http://localhost:8000/": read tcp 127.0.0.1:52321->127.0.0.1:8000: read: connection reset by peer Get "http://localhost:8000/": read tcp 127.0.0.1:39267->127.0.0.1:8000: read: connection reset by peer Get "http://localhost:8000/": read tcp 127.0.0.1:32869->127.0.0.1:8000: read: connection reset by peer — You are receiving this because you commented. Reply to this email directly, view it on GitHub ( #527 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAQPDZLYSVHHFE5P452CQLR3FUUPANCNFSM4OQ4FF3Q ).

[Jongy]

There are tons of goroutines -> tons of stack traces, and there are many that are actually different... I spent some time looking at some of them, also looked in the code and I see it should handle connection reset by peer errors, which will be accounted in all send / receive error cases in lib/attack.go:hit.

So... I reached the top of the very long error message printed by Go, to find this bit first:

runtime: mlock of signal stack failed: 12                                                                                                                                                                  
runtime: increase the mlock limit (ulimit -l) or                                                                                                                                                           
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed  

Sigh. Very unexpected that Go needs so much mlock, apparently it's about golang/go#35777.

My ulimit -l is 64 (Docker's default?) According to the stacktraces, the failed allocations are of Go's runtime, so Vegeta shouldn't do something about it, but a nicer error message describing what you should do to overcome it would be nice (at least so you could understand it's an external error which caused Vegeta to bail, and not something internal). Generally for all Go-runtime related errors I believe it's nicer user experience if Vegeta tells you that + attempts to give a fix. For example, vegeta report could notice the input file is truncated and tell you about it.

Thanks, and sorry for the misleading bug report. I'm closing this but still thinking it's a good idea to improve Go-runtime related errors :)