Allow ExchangeStrategies customizations in WebClient

[totof3110]

Seems caused by 89d053d / #23884.

After upgrading from Spring Boot 2.2.0 to 2.2.1, WebClient started throwing org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144 when calling a JSON REST API (that has a large response size).

From the linked documentation (https://github.com/spring-projects/spring-framework/blob/master/src/docs/asciidoc/web/webflux.adoc#limits), it's not clear what configuration should be changed to make our API calls work again.

Is this expected behavior?

Full stack trace:

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
	at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ Body from POST <redacted> [DefaultClientResponse]
Stack trace:
		at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101)
		at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:94)
		at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:59)
		at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:119)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
		at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:213)
		at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:346)
		at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:348)
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:572)
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478)
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227)
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
		at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:502)
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
		at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:834)

[bclozel]

If you are using Spring Boot and creating your WebClient using a WebClient.Builder configured by Spring Boot (you can get it injected), you can use the new configuration property spring.codec.max-in-memory-size, see spring-projects/spring-boot#18828.

Otherwise if you’re creating that client from scratch you need to configure the codecs like:

ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
                .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(1024 * 10)).build();
WebClient webClient = WebClient.builder().exchangeStrategies(exchangeStrategies).build();

[totof3110]

Thanks @bclozel . Since I use the autowired WebClient.Builder, I've tried setting spring.codec.max-in-memory-size to a large size (500MB) but am still getting that error.

We happen to be using Spring HATEOAS and I wonder if something there is overriding this configuration. In particular there's class org.springframework.hateoas.config.WebClientConfigurer which transforms the WebClients with:

webClient.mutate().exchangeStrategies(hypermediaExchangeStrategies()).build()

I've already noticed that this seems to mess up the ObjectMapper used in the WebClient and drop all the com.fasterxml.jackson.databind.Modules that were previously autoconfigured.

Could that be the reason?

[bclozel]

We need to improve the documentation for the client side of things. This should help non Spring Boot users and developers building their own WebClient instances.

Now for HATEOAS, we're probably missing an extension point to allow it to customize the codecs without replacing them - we should provide ClientCodecConfigurer with an additional method (like we did for ServerCodecConfigurer).

[totof3110]

Thanks @bclozel !