Spring webflux stream fails silently

[el-git]

We are using spring webflux in a real world banking application to stream back data from a mongo repository. The evaluation and initial development was carried out against spring-boot-starter-parent version 2.0.0.M6. Following an upgrade to 2.0.1.RELEASE we are now experiencing an issue with one of the streams where the response will stop returning data mid-stream, and without hanging up. There are no exceptions reported in the logs. The connection simply remains open until the client times out. It took a while to track down what might be causing it, but it seemed to always occur mid-way through serializing a data item that contains large lists of BigDecimal items. This is however not deterministic, evern when dealing with fixed and ordered data sets. It does however usually occur within a given range e.g. after between 1.2 and 1.3GB

Using these clues I was able to knock together a basic application which replicates the issue in a standalone manner (generated using Spring Initializr)

Here are the main classes in the app - source is attached
stream-error-demo.ZIP
:

pom.xml

<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>2.0.0.M6</version>
	<!--<version>2.0.1.RELEASE</version>-->
	<relativePath/>
</parent>

RouterConfig.java

@Configuration
public class RouterConfig {

    @Resource
    private TestHandler testHandler;

    @Bean
    RouterFunction<ServerResponse> getInputDataRouterFunction() {
        return route(GET("/justStreamBigDecimals"), testHandler::justStreamBigDecimals);
    }

}

TestHandler.java

@Component(value = "testHandler")
public class TestHandler {

    public Mono<ServerResponse> justStreamBigDecimals(ServerRequest serverRequest) {

        Flux<DecimalList> testClassFlux = Flux.create((FluxSink<DecimalList> sink) -> {
            sink.onRequest(n -> {
                for (int i = 0; i < n; i++) {
                    sink.next(new DecimalList());
                }
            });
        });

        return ServerResponse
                .ok()
                .contentType(MediaType.APPLICATION_STREAM_JSON)
                .body(testClassFlux, DecimalList.class);
    }

    private class DecimalList {
        private List<BigDecimal> bigDecimals;

        public DecimalList() {
            this.bigDecimals = new ArrayList<>();
            for (int i = 0; i < 10000; i++) {
                bigDecimals.add(new BigDecimal("1.123123123123123123123123123123123123123123123"));
            }
        }

        public List<BigDecimal> getBigDecimals() {
            return bigDecimals;
        }

        public void setBigDecimals(List<BigDecimal> bigDecimals) {
            this.bigDecimals = bigDecimals;
        }
    }

}

Fire up the spring boot application and invoke the handler using curl:

curl http://localhost:8080/justStreamBigDecimals >> test.json

The handler will continue emitting DecimalList items indefinitely.

With the spring-boot-starter-parent set to 2.0.1.RELEASE, the following behaviour is observed:

curl http://localhost:8080/justStreamBigDecimals >> test.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                      Dload  Upload   Total   Spent    Left  Speed
100 26.0M    0 26.0M    0     0  2690k      0 --:--:--  0:00:09 --:--:--     0

Where the speed soon drops to zero, but the stream does not terminate. There are also no errors reported in the logs.

Changing this to 2.0.0.M6, the stream continues indefinitely as expected e.g.

curl http://localhost:8080/justStreamBigDecimals >> test.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                     Dload  Upload   Total   Spent    Left  Speed
100 1129M    0 1129M    0     0  11.0M      0 --:--:--  0:01:41 --:--:--  9.8M

Please let me know if there is any further information that will help in reproducing this issue.

[bclozel]

I can't seem to reproduce the issue you're describing here.
It might be an OS-specific issue in reactor-netty (since native netty support can be at play depending on the platform). What is your operating system (can you provide an uname -a or the Windows version)?

Several issues were recently fixed in reactor-netty, so you might want to try with Spring Boot 2.0.2.BUILD-SNAPSHOT which depends on the latest right now. Here are the repositories declaration for your convenience:

	<repositories>
		<repository>
			<id>spring-snapshots</id>
			<name>Spring Snapshots</name>
			<url>https://repo.spring.io/snapshot</url>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
		<repository>
			<id>spring-milestones</id>
			<name>Spring Milestones</name>
			<url>https://repo.spring.io/milestone</url>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
	</repositories>

	<pluginRepositories>
		<pluginRepository>
			<id>spring-snapshots</id>
			<name>Spring Snapshots</name>
			<url>https://repo.spring.io/snapshot</url>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
		<pluginRepository>
			<id>spring-milestones</id>
			<name>Spring Milestones</name>
			<url>https://repo.spring.io/milestone</url>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>

Also, does it happen still if you change the Flux generation to be:

Flux<DecimalList> testClassFlux = Flux.generate(sink -> sink.next(new DecimalList()));

Thanks!

[el-git]

Thanks for looking into this. In terms of versions, this has been seen on both redhat linux and windows 7:

Linux xxxxxx 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017 x86_64 x86_64 x86_64 GNU/Linux (java version "1.8.0_144")
Windows 7 Enterprise Service Pack 1 (java version "1.8.0_161")

I've also included the java version for each.

I'll try your suggestions and report back.

[el-git]

I tried both suggestions above, using snapshot build and modifying the Flux generation, and still experiencing the same issue.

[bclozel]

Thanks for the quick answer - I've managed to reproduce that in a Docker image.
I suspect this to be a reactor-netty issue, and I'm currently trying to narrow things down.

In the meantime, feel free to add Tomcat, Undertow or Jetty to your project, I've confirmed this scenario works once you've added:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-tomcat</artifactId>
</dependency>

I'll ping back here once I've narrowed down the issue. Thanks!

[el-git]

Great, that is good news that you are able to reproduce the issue! I can also confirm that this scenario works after switching to Tomcat, both with this demo and in our application. This should keep us going for now. Let me know when a fix is available and I can try it out. Thanks.

[bclozel]

I can't reproduce this issue with the latest Reactor SNAPSHOTs. This must have been fixed in a recent commit.

Taking your sample project and adding the following property fixes the issue (I also tried with Spring Boot 2.0.2.BUILD-SNAPSHOT and 2.1.0.BUILD-SNAPSHOT and both are fine):

<properties>
  <reactor-bom.version>Bismuth-BUILD-SNAPSHOT</reactor-bom.version>
</properties>

Can you try and confirm this fixes the issue (make sure to run mvn clean package -U to get the latest snapshot build.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[el-git]

Apologies for the delay getting back to you.

I'm currently struggling to pull in the BUILD-SNAPSHOT you have listed above. We use a proxy nexus server and this is complaining when I point it to http://repo.spring.io/snapshot/. The administrator is telling me this is down to nexus expecting a maven-metadata.xml file in the root of this location which it can't find. Does this sound plausible?

I've also tried this in my home setup using both windows 7 and windows 10 but I'm unable to reproduce the original problem on these environments.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.