Add first class support for kotlinx.serialization

[neostage]

using Spring Boot 2.4.0)
spring-boot-starter-actuator has strong dependencies on jackson so that I can't use kotlinx.serialization as default json message converter.

[sdeleuze]

Hi, thanks for raising this since indeed I think there is a need for a proper integration for Kotlin serialization in Boot that goes further than what is inherited from Spring Framework.

Current behavior is documented here and has several limitations in Boot context that we would be likely to improve.

The configuration is kind of Spring Framework oriented, a dedicated Boot support similar to JSON-B and related documentation would be much better for consistency.

The second point that should be improved is what you raised in the issue. After discussing with @bclozel I tend to think it is currently not straightforward to support Kotlin serialization and actuators because the web configuration is shared between your app (that wants to use Kotlin serialization) and and actuator (that needs Jackson or similar general purpose JSON library to be configured in your web configuration. And Kotlin serialization is designed to serialize only Kotlin classes annotated with @Serializable or collection of those.

On Framework side, maybe I could refine the converter selection mechanism to have a better support for having both Kotlin Serialization and Jackson via spring-projects/spring-framework#26147 in order to provide a workaround with manual config of both.

On Boot side, after discussing with @snicoll and @bclozel , it sounds like a candidate for Spring Boot 2.5 that would potentially involve the resolution of #20291. As well as dependency management (I am discussing the creation of a Kotlin serialization BOM with the Kotlin team).

[neostage]

Thanks for the detailed response and I'll look forward to future releases!

[sdeleuze]

@snicoll After working on a draft commit for spring-projects/spring-framework#26147, I confirm that I plan to configure both Kotlin serialization and Jackson when they are on the classpath, I think that's will make configuration easier and will make actuator and error endpoints still working. Kotlin serialization is more narrow so that seems to work as expected.

So for Boot 2.5, what would be mainly needed is autoconfiguration and documentation à la JSONB, and dependency management with the upcoming BOM that I have asked to Kotlin team (similar to Coroutines one).

[snicoll]

Thanks for the follow-up Sébastien. I've repurposed this issue and triaged it for 2.5.x.

[sdeleuze]

@neostage I have fixed spring-projects/spring-framework#26147 and made sure your use case work fine with Boot 2.4, please test Spring Framework 5.3.2-SNAPSHOT with both Jackson and Kotlin serialization in the classpath it should work as expected.

[hantsy]

I found when jackson and Kotlin serialization in the classpath, the WebTestClient in tests also selects Kotlin serialization over Jackson, even though I have set spring.http.converters.preferred-json-mapper=jackson. BTW, I do not use the Kotlin serialization annotations on the data classes.

[wilkinsona]

In that case, the Spring Framework change described above by @bclozel is not working as expected. Please provide a minimal sample that reproduces the behavior you've described.

[hantsy]

@wilkinsona The spring.http.converters.preferred-json-mapper=jackson from Google result is my fault, I can not find this in the Spring Boot reference. It only includes such property for mvc stack(spring.mvc.converters.preferred-json-mapper=jackson is enabled by default). I hope there is another similar config for webflux.

I have prepared an example project to describe the issues, webflux-json-mapper.zip

I have added kotlinx-serialization-json into the dependencies, so there are two json mapper libs i the project: Jackson and Kotlinx.

Run the HelloControllerTest, it will throw an exception like this.

Decoding error: Unexpected JSON token at offset 0: Expected beginning of the string, but got [ at path: $
JSON input: ["RED","GREEN","BLUE"]
org.springframework.core.codec.DecodingException: Decoding error: Unexpected JSON token at offset 0: Expected beginning of the string, but got [ at path: $
JSON input: ["RED","GREEN","BLUE"]
	at org.springframework.http.codec.KotlinSerializationStringDecoder.processException(KotlinSerializationStringDecoder.java:139)
	Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Body from GET /hello [DefaultClientResponse]
Original Stack Trace:
		at org.springframework.http.codec.KotlinSerializationStringDecoder.processException(KotlinSerializationStringDecoder.java:139)
		at org.springframework.http.codec.KotlinSerializationStringDecoder.lambda$decode$0(KotlinSerializationStringDecoder.java:110)
		at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:179)
		at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
		at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
		at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
		at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onNext(FluxConcatArray.java:180)
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
		at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onSubscribe(FluxConcatArray.java:172)
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
		at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:238)
		at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:79)
		at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:68)
		at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
		at reactor.core.publisher.Mono.block(Mono.java:1806)
		at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.getListBodySpec(DefaultWebTestClient.java:460)
		at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.expectBodyList(DefaultWebTestClient.java:450)
		at com.example.demo.HelloControllerTest.test hello endpoint(HelloControllerTest.kt:20)
		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:146)
		at reactor.core.publisher.Mono.block(Mono.java:1807)
		at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.getListBodySpec(DefaultWebTestClient.java:460)
		at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.expectBodyList(DefaultWebTestClient.java:450)
		at com.example.demo.HelloControllerTest.test hello endpoint(HelloControllerTest.kt:20)
		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 0: Expected beginning of the string, but got [ at path: $
JSON input: ["RED","GREEN","BLUE"]
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:598)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:596)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenient(AbstractJsonLexer.kt:467)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.unexpectedToken(AbstractJsonLexer.kt:220)
	at kotlinx.serialization.json.internal.StringJsonLexer.consumeNextToken(StringJsonLexer.kt:74)
	at kotlinx.serialization.json.internal.StringJsonLexer.consumeKeyString(StringJsonLexer.kt:86)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:383)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeString(StreamingJsonDecoder.kt:339)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeEnum(StreamingJsonDecoder.kt:352)
	at kotlinx.serialization.internal.EnumSerializer.deserialize(Enums.kt:139)
	at kotlinx.serialization.internal.EnumSerializer.deserialize(Enums.kt:105)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.json.Json.decodeFromString(Json.kt:107)
	at org.springframework.http.codec.KotlinSerializationStringDecoder.lambda$decode$0(KotlinSerializationStringDecoder.java:107)
	at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:179)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onNext(FluxConcatArray.java:180)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onSubscribe(FluxConcatArray.java:172)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:238)
	at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:79)
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:68)
	at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.Mono.block(Mono.java:1806)
	at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.getListBodySpec(DefaultWebTestClient.java:460)
	at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.expectBodyList(DefaultWebTestClient.java:450)
	at com.example.demo.HelloControllerTest.test hello endpoint(HelloControllerTest.kt:20)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)


Unexpected JSON token at offset 0: Expected beginning of the string, but got [ at path: $
JSON input: ["RED","GREEN","BLUE"]
kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 0: Expected beginning of the string, but got [ at path: $
JSON input: ["RED","GREEN","BLUE"]
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
	at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail(AbstractJsonLexer.kt:598)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.fail$default(AbstractJsonLexer.kt:596)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeStringLenient(AbstractJsonLexer.kt:467)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.unexpectedToken(AbstractJsonLexer.kt:220)
	at kotlinx.serialization.json.internal.StringJsonLexer.consumeNextToken(StringJsonLexer.kt:74)
	at kotlinx.serialization.json.internal.StringJsonLexer.consumeKeyString(StringJsonLexer.kt:86)
	at kotlinx.serialization.json.internal.AbstractJsonLexer.consumeString(AbstractJsonLexer.kt:383)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeString(StreamingJsonDecoder.kt:339)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeEnum(StreamingJsonDecoder.kt:352)
	at kotlinx.serialization.internal.EnumSerializer.deserialize(Enums.kt:139)
	at kotlinx.serialization.internal.EnumSerializer.deserialize(Enums.kt:105)
	at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:69)
	at kotlinx.serialization.json.Json.decodeFromString(Json.kt:107)
	at org.springframework.http.codec.KotlinSerializationStringDecoder.lambda$decode$0(KotlinSerializationStringDecoder.java:107)
	at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:179)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onNext(FluxConcatArray.java:180)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onSubscribe(FluxConcatArray.java:172)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:238)
	at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:79)
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:68)
	at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.Mono.block(Mono.java:1806)
	at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.getListBodySpec(DefaultWebTestClient.java:460)
	at org.springframework.test.web.reactive.server.DefaultWebTestClient$DefaultResponseSpec.expectBodyList(DefaultWebTestClient.java:450)
	at com.example.demo.HelloControllerTest.test hello endpoint(HelloControllerTest.kt:20)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[hantsy]

I have added a custom CodecCustomizer to resolve the issue. https://github.com/hantsy/spring-puzzles/blob/master/webflux-json-mapper/src/main/kotlin/com/example/demo/DemoApplication.kt#L41

[hantsy]

@bclozel @wilkinsona I created another WebMvc example project, https://github.com/hantsy/spring-puzzles/tree/master/webmvc-json-mapper, just expose a enum values in the endpoint, no Kotlinx specific annotation.

• Add kotlinx-serializaiton-json into dependencies
• Add property spring.mvc.converters.preferred-json-mapper=jackson in the application.properties

In the Controller test,

• When using MockMvc, it works well.
• When using WebTestClient by @Autowired or MockMvcWebTestClient.bindTo(mockMvc), it failed with the same exception in the above comment, I have to add custom codecs to switch to use Jackson and make it work.

[hantsy]

Jackson, Gson and Jsonb have their own Autoconfiguraiton classes, and I would like an autoconfiguration for kotlinx.serialization.

Thus if Jackson and Kotlinx coexist in a spring WebFlux application and I want to use Jackson in all cases, it is easier to setup.

• Exclude the kotlinx serialization auto configuration
• Or there is a similar property like spring.webflux.converters.preferred-json-mapper=jackson to pick up Jackson always.

[bclozel]

I guess enum values don't need to be annotated with @kotlinx.serialization.Serializable to be considered by kotlinx.serialization. That's a limitation we could document here or in Spring Framework. As for introducing a brand new, dedicated auto-configuration for kotlinx, that's something we can do but I don't think it's high priority given that the only limitation so far is for applications 1) having the dependency on the classpath and 2) serializing enums.

[hantsy]

@bclozel In our project, the Kotlinx serialization JSON is a transitive dependency from other libs we used in the project, which we did not add explicitly. We can not control this case.

[wilkinsona]

It’s starting to sound like a way to disable kotlinx is required, perhaps through an annotation that Framework’s codecs and message converters look for or through some mechanism in Boot. It would appear that the classpath alone isn’t a sufficiently strong signal.

[y-marion]

We also have a similar issue where we have the dependency on kotest-assertions-json in testImplementation (which brings in kotlinx.serialization as transitive dependency).
This causes an error in a MockMVC test, where an Endpoint returning ResponseEntity<Unit> is called, if we return a ProblemDetail (or any response with a body in case of a failure), as kotlinx.serialization.internal.UnitSerializer.serialize is called either way to serialize the Response.

I've written a small reproduction here: https://github.com/y-marion/spring-kotlinx-unit-error-repro
The tests for the controller do not work if executed as is, if the testImplementation("io.kotest:kotest-assertions-json:5.9.1") is removed, they work.
The controller also works as intended if the application is started via bootRun.

[sdeleuze]

It’s starting to sound like a way to disable kotlinx is required, perhaps through an annotation that Framework’s codecs and message converters look for or through some mechanism in Boot. It would appear that the classpath alone isn’t a sufficiently strong signal.

@wilkinsona @bclozel Feel free to ping me if you want to brainstorm on what we could do and when.

[philwebb]

We're going to add auto-configuration and look at adding an entry to the spring.mvc.converters.preferred-json-mapper property.

[bclozel]

I did some work on this issue and came to the following conclusion: adding "first class" support for kotlinx.serialization would only be useful if developers want to contribute a custom kotlinx.serialization.json.Json bean and configure it to their liking. This is not what has been requested.

Quite the opposite, it seems that kotlinx serialization conflicts with Jackson when they're both on the classpath. While there was an initial attempt to make the kotlin variant more selective (by looking at annotations), this obviously does not work for Java enums. Switching the order won't improve things: if Kotlin is first, it will handle java enums and @kotlinx.serialization.Serializable-classes, if Jackson is first, it will handle everything.

The only possible improvement in Spring Boot would be remove the Kotlin converter if spring.mvc.converters.preferred-json-mapper=jackson. Still, the default Spring Framework experience is flawed because ordered converters are conflicting with each other. Spring Framework currenty considers Jackson/Gson/Jsonb in this order, configuring a single one to avoid conflicts. I think this should be the same for kotlinx serialization.

I'm closing this issue in favor of spring-projects/spring-framework#34410

[sdeleuze]

I would like to add more context based on my findings and discussions with the Kotlin team.

@hantsy Does not change the fact that spring-projects/spring-framework#34410 is a meaningful change, but FWIW your webflux-json-mapper repro is incorrect. In HelloControllerTest, you try get a Color body while it should be an Array<Color>, hence the error you see. You can do that by using the import org.springframework.test.web.reactive.server.expectBody extension (which will be more easily discoverable once https://youtrack.jetbrains.com/issue/KTIJ-33094 is fixed).

One source of confusion for Kotlin Serialization is that for some use cases, the plugin is not needed (enum), while it is for other ones (mostly build time processing of classes annotated @Serializable to generate a serializer in the project own source code). If the related class has already been processed and is shipped in a dependency, the plugin is not needed. I can see how people could be confused, so better to have an explicit signal to enable Kotlin Serialization in Spring Boot and avoid to have both Jackson and Kotlin Serialization. We did those refinements in Spring Framework 7 milestones, allowing also to fix an annoying limitation since Spring Framework 7 will support open polymorphism with Kotlin Serialization.

As discussed with @bclozel, I think Add kotlinx.serialization as preferred JSON mapper option #44241 is pretty important to allow a smooth migration and/or activation experience.