Error: read ECONNRESET for large request body

[davidgj23]

Currently, on my project, we decided to migrate from Akka to Pekko, but we're getting a strange error in one of our endpoints that doesn't happen when using Akka. The particularity of this endpoint is that its request has a field with a very long string which makes the request size 592432 bytes.

When calling the endpoint via CURL or Postman we got: Error: read ECONNRESET immediately, but the microservice processed the request successfully behind the scenes. Moreover, Pekko logs this error to the console: [service-pekko.actor.default-dispatcher-14] [org.apache.pekko.actor.ActorSystemImpl(service)] Request timeout encountered for request [POST /my-endpoint-url Default(592432 bytes)]. It is worth mentioning that it works fine when using a shorter string to make the request lighter.

We have tried different settings configurations like the max-content-length but it's been worthless. Also, as I mentioned before, this doesn't happen in Akka, and it happens in all Pekko versions.

Thanks in advance for your support.

[pjfanning]

Please provide a reproducible test case. Your description is vague about the request and providing more detail would really help.

[davidgj23]

@pjfanning I created this example: pekko-test
Use the request.json to test it.
Thank you 🙏

[jrudolph]

Thanks for sharing the reproducer, @davidgj23. I have a look.

[jrudolph]

There's a head stage somewhere in the json processing path which leads to the behavior.

[jrudolph]

Probably similar/same problem as this: mdedetrich/pekko-streams-circe#15

[jrudolph]

It's somewhat unlikely that this was introduced in Pekko, it's more likely a Akka 2.5 vs 2.6 issue.

[jrudolph]

This needs to be fixed in pekko-streams-circe. We can keep it open here for the time being.

[mdedetrich]

@jrudolph Thanks for looking into this, I tried having a look at the issue in mdedetrich/pekko-streams-circe#15 but I couldn't reproduce it and the original author of the bug report couldn't create a minimal reproduction either (he only saw it in prod).

Since the core code in pekko-streams-circe is ultra stable and hasn't changed, the suspicion is the regression occured due to underlying pekko/akka-streams

[jrudolph]

In any case, using head was never guaranteed to work with any version of akka-http. It might have accidentally worked for some reason but that is kind of its own bug because it a kind of a resource leak that was probably fixed at some point.

[jrudolph]

This is a similar issue to #460 which has also more information about the rationale.

I guess it would be possible to have a more nuanced view even for pekko-http and revise the decision to have request entity cancellation mean connection closure. After all, user code can cancel or close the request/response stream (in contrast to the entity stream) if it means to do so. On the other hand, pekko-http routing code has no access to the request/response streams directly, so I wonder if there's already code relying on the current behavior (accidentally or not).

[jrudolph]

Here's the code responsible for the behavior:

pekko-http/http-core/src/main/scala/org/apache/pekko/http/impl/engine/server/HttpServerBluePrint.scala

Lines 140 to 141 in 03d1e95

	// application layer has cancelled or only partially consumed response entity:
	// connection will be closed

The difference between big and small request entities is probably that for small requests the whole request is received in one big buffer so the whole processing might be playing out in one thread in one go just in the right order to still work.

[pjfanning]

@davidgj23 if this issue is causing you problems, one option is to use "com.github.pjfanning" %% "pekko-http-circe" % "2.7.0" instead of "org.mdedetrich" %% "pekko-http-circe" % "1.0.0".

The import in your routes class would become:
import com.github.pjfanning.pekkohttpcirce.FailFastCirceSupport._

The code in my repo doesn't stream like org.mdedetrich lib and I would generally recommend using the org.mdedetrich lib.

I have tested this change with your POC.

[mdedetrich]

@davidgj23 if this issue is causing you problems, one option is to use "com.github.pjfanning" %% "pekko-http-circe" % "2.7.0" instead of "org.mdedetrich" %% "pekko-http-circe" % "1.0.0".

The import in your routes class would become: import com.github.pjfanning.pekkohttpcirce.FailFastCirceSupport._

The code in my repo doesn't stream like org.mdedetrich lib and I would generally recommend using the org.mdedetrich lib.

I have tested this change with your POC.

I am making a fixed release of pekko-streams-circe as we speak. Should be out in half an hour

[mdedetrich]

Pekko Streams Circe 1.1.0 was just released with the fix to the critical bug, release notes are here https://github.com/mdedetrich/pekko-streams-circe/releases/tag/v1.1.0

@davidgj23 Please update the version of this library if you happen to be using it

[davidgj23]

@mdedetrich it worked after updating to 1.1.0!
Thank you guys for your support.