Reactor netty in spring boot webflux application generates metrics with CancelledServerWebExchangeException exception and UNKNOWN outcome where are no issues

[MBalciunas]

We have a bunch of spring boot webflux services in our project and almost all has this same issue. We use prometheus for metrics and track the success of the requests. However in those services from 1% to 20% http server requests metrics consists of outcome=UNKNOWN with exception=CancelledServerWebExchangeException while there are no other indications of any issues in server responses or indication that clients cancel that many requests. examples:
http_server_requests_seconds_count{exception="CancelledServerWebExchangeException",method="GET",outcome="UNKNOWN",platform="UNKNOWN",status="401",uri="UNKNOWN",} 87.0
http_server_requests_seconds_count{exception="CancelledServerWebExchangeException",method="GET",outcome="UNKNOWN",platform="UNKNOWN",status="200",uri="UNKNOWN",} 110.0

I successfully reproduced this locally with basic webflux application template and single controller bombarding with https://httpd.apache.org/docs/2.4/programs/ab.html : ab -n 15000 -c 50 http://localhost:8080/v1/hello.

I tried substituting tomcat for netty and there were no more of these metrics logs.

While it seems it doesn't cause direct issues on services running in production, it still interferes in the correctness of the metrics and alerts. We can ignore all the UKNOWN outcomes but we can't know if those UNKNOWN come from actual server/client cancels or just this netty issue.

Someone already had this issue in the past but it was never resolved: https://stackoverflow.com/questions/69913027/webflux-cancelledserverwebexchangeexception-appears-in-metrics-for-seemingly-no

Versions used: SpringBoot: 2.7.2 and SpringBoot: 2.6.2, Kotlin: 1.7.10, JVM: 17

[MBalciunas]

I created example repo to reproduce this issue: https://github.com/MBalciunas/spring-webflux-netty-metrics-issue

[bclozel]

This looks like a duplicate of #29599.
What would like the outcome of this issue to be?

[MBalciunas]

It's not the duplicate of #29599 as that issue is concerned with the status of UNKNOWN metric.
In this ticket I raise an issue that outcome=UNKNOWN with exception=CancelledServerWebExchangeException seems to be generated by metrics for possibly no apparent reason. As I provided in examples description, this only happens with Netty and not tomcat.
So the outcome I'd like would be no more UNKNOWN metrics when there is no issue from anywhere or an answer if this is a correct behaviour. And if it is, why it happens when it seems it shouldn't.

[bclozel]

CancelledServerWebExchangeException doesn't happen with Tomcat, because Servlet containers are not notified when a client closes the connection. With Servlet containers, you're only getting an error if the server is failing to write to the client. This explains why you're getting this metric with Reactor Netty and not Tomcat.

This metric is produced when a client closes the connection prematurely. From a Framework perspective, we have no way to know if the client went away because the response took too much time and cancelled the call or if a intermediary network device is at fault. This is why we're using the outcome=UNKNOWN tag for this. From this perspective, we're emitting data for an event you might be interested in. The alternative would be that we completely ignore those requests from the metrics, but some would consider this a bug if they're tracking down an actual problem and there is no data for those requests.

I'm happy to improve the situation. Any suggestion?

Note: if I remember correctly, apache bench opens and closes a connection for each requests (arguably, this is a strange approach for benchmarks). Benchmarking the TCP layer like this locally might explain why some connections might be closed before the exchange is complete, from a server perspective.

[gzeskas]

Just tried an example that @MBalciunas provided and for me is the same.
ab reports that all 15k requests very successful but in spring actuar end-point there is 454 failed requests with exception: CancelledServerWebExchangeException

ab -n 15000 -c 50 -H "Authorization:Basic cHJvbWV0aGV1czpwcm9tZXRoZXVz" http://localhost:8080/v1/hello
This is ApacheBench, Version 2.3 <$Revision: 1901567 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1500 requests
Completed 3000 requests
Completed 4500 requests
Completed 6000 requests
Completed 7500 requests
Completed 9000 requests
Completed 10500 requests
Completed 12000 requests
Completed 13500 requests
Completed 15000 requests
Finished 15000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /v1/hello
Document Length:        5 bytes

Concurrency Level:      50
Time taken for tests:   127.981 seconds
Complete requests:      15000
Failed requests:        0
Total transferred:      4440000 bytes
HTML transferred:       75000 bytes
Requests per second:    117.20 [#/sec] (mean)
Time per request:       426.604 [ms] (mean)
Time per request:       8.532 [ms] (mean, across all concurrent requests)
Transfer rate:          33.88 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.1      0      27
Processing:    73  425 112.1    430     880
Waiting:       73  425 112.1    430     880
Total:         73  426 112.1    430     880

Percentage of the requests served within a certain time (ms)
  50%    430
  66%    468
  75%    494
  80%    511
  90%    558
  95%    603
  98%    658
  99%    694
 100%    880 (longest request)

http_server_requests_seconds_count{exception="CancelledServerWebExchangeException",method="GET",outcome="UNKNOWN",status="200",uri="/v1/hello",} 454.0
http_server_requests_seconds_sum{exception="CancelledServerWebExchangeException",method="GET",outcome="UNKNOWN",status="200",uri="/v1/hello",} 197.189522582
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/v1/hello",} 14651.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/v1/hello",} 6204.099123688

[MBalciunas]

And also that seems weird that the sum of metrics is bigger that request count.

requests: 15000.
SUCESS + UNKNOWN outcomes sum: 454.0 + 14651.0 = 15105.

[bclozel]

I'd like to focus on the issue being discussed here: in the case of connections closed prematurely by clients, how is the current behavior incorrect? What would be the expected behavior?

Note that apache bench is not highly regarded as a benchmarking tool in general and in this case, it's not even using the keep-alive feature.