How to increase message timeout to 5 minutes? #1723
Answered
by
thjaeckle
dimabarbul
asked this question in
Q&A
-
|
Hello I need to make ditto send messages and wait for response up to 5 minutes. I took develop branch and made following changes:
diff --git a/deployment/docker/docker-compose.yml b/deployment/docker/docker-compose.yml
index 0842ec5081..54cda333f5 100755
--- a/deployment/docker/docker-compose.yml
+++ b/deployment/docker/docker-compose.yml
@@ -157,8 +157,9 @@ services:
- TZ=Europe/Berlin
- BIND_HOSTNAME=0.0.0.0
- ENABLE_PRE_AUTHENTICATION=true
+ - REQUEST_TIMEOUT=305s
# Set additional configuration options here appending JAVA_TOOL_OPTIONS: -Dditto.gateway.authentication.devops.password=foobar -Dditto.gateway...
- - JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class=
+ - JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=2 -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport -XX:+UseStringDeduplication -Xss512k -XX:MaxRAMPercentage=50 -XX:+UseG1GC -XX:MaxGCPauseMillis=150 -Dakka.coordinated-shutdown.exit-jvm=on -Dakka.cluster.shutdown-after-unsuccessful-join-seed-nodes=180s -Dakka.cluster.failure-detector.threshold=15.0 -Dakka.cluster.failure-detector.expected-response-after=10s -Dakka.cluster.failure-detector.acceptable-heartbeat-pause=20s -Dakka.cluster.downing-provider-class= -Dditto.gateway.message.max-timeout=305s -Dditto.gateway.command.max-timeout=305s
# in order to write logs into a file you can enable this by setting the following env variable
# the log file(s) can be found in /var/log/ditto directory on the host machine
# - DITTO_LOGGING_FILE_APPENDER=true
diff --git a/deployment/docker/nginx.conf b/deployment/docker/nginx.conf
index f5429c6066..0fac5461f8 100755
--- a/deployment/docker/nginx.conf
+++ b/deployment/docker/nginx.conf
@@ -10,10 +10,10 @@ http {
default_type application/json;
# timeouts are configured slightly higher than gateway read-timeout of 60 seconds
- proxy_connect_timeout 70; # seconds, default: 60
- proxy_send_timeout 70; # seconds, default: 60
- proxy_read_timeout 70; # seconds, default: 60
- send_timeout 70; # seconds, default: 60
+ proxy_connect_timeout 310; # seconds, default: 60
+ proxy_send_timeout 310; # seconds, default: 60
+ proxy_read_timeout 310; # seconds, default: 60
+ send_timeout 310; # seconds, default: 60
client_header_buffer_size 8k; # allow longer URIs + headers (default: 1k)
large_client_header_buffers 4 16k;Request that I send is following (in this example I use 3 minutes timeout): $ date; curl -i -X POST -u ditto:ditto -H 'Content-Type: application/octet-stream' -w '\n' --data 'data' http://localhost:8080/api/2/things/test:sensor01/inbox/messages/test-message?timeout=180; date
Tue Aug 22 17:06:24 EEST 2023
HTTP/1.1 503 Service Unavailable
Server: nginx/1.21.6
Date: Tue, 22 Aug 2023 14:08:24 GMT
Content-Type: application/json
Content-Length: 195
Connection: keep-alive
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
{"status":503,"error":"gateway:service.unavailable","message":"The service you wanted to reach is currently not available.","description":"Please contact the service team if this happens again."}
Tue Aug 22 17:08:24 EEST 2023But the request times out in 2 minutes and returns 503 instead of 408. Interesting part here is that in 3 minutes there is a log stating again that request timed out and that status is 408. Here are corresponding logs from gateway service: I tried digging a bit and found following code in AbstractRoute.java: private static final scala.concurrent.duration.Duration AKKA_HTTP_TIMEOUT =
scala.concurrent.duration.Duration.create(2, TimeUnit.MINUTES);
...
private Route increaseHttpRequestTimeout(final java.util.function.Function<Duration, Route> inner,
final scala.concurrent.duration.Duration requestTimeout) {
return withRequestTimeout(AKKA_HTTP_TIMEOUT,
() -> inner.apply(Duration.ofMillis(requestTimeout.toMillis())));
}I'm not java developer and not familiar with Akka, so it's just my guess that this code is relevant. I changed duration in code above to 5 minutes, built custom docker images and tried the same message with local images and it worked - it timed out after 3 minutes with 408 status code. I guess, I might be missing some Akka configuration that allows changing it without tweaking the code, but I cannot find one. I googled a bit and tried adding different configurations mentioned on Akka website like akka.http.server.idle-timeout, akka.http.client.idle-timeout, akka.http.host-connection-pool.client.idle-timeout. I tried adding them to gateway, connectivity, things and other services, but with no luck. So, I'm wondering how can I increase message timeout to 5 minutes without updating source code? Or is it by design capped by 2 minutes? |
Beta Was this translation helpful? Give feedback.
Answered by
thjaeckle
Aug 23, 2023
Replies: 1 comment
-
|
Hi @dimabarbul Good question. The other message related increased timeouts you already did set correctly. However yes .. that hardcoded The comment also indicates that this was added as a "premature optimization" - not necessarily needed: /**
* Timeout for Akka HTTP. Timeout is normally managed in HttpRequestActor and AcknowledgementAggregatorActor.
* The Akka HTTP timeout is only there to prevent resource leak.
*/
private static final scala.concurrent.duration.Duration AKKA_HTTP_TIMEOUT =
scala.concurrent.duration.Duration.create(2, TimeUnit.MINUTES); |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
dimabarbul
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @dimabarbul
Good question.
The
akka.http.server.idle-timeoutis configured in Ditto to be610sby default - can however be overwritten via environment variableIDLE_TIMEOUT.The other message related increased timeouts you already did set correctly.
However yes .. that hardcoded
AKKA_HTTP_TIMEOUTof 2 minutes seems really to break the possibility to increase the effective timeout.What probably was the intention (I can only speculate) of the
AKKA_HTTP_TIMEOUTis to configure the absolute configured maximum timeout (and maybe some seconds more) - but for that the code would have to read out the configuredakka.http.server.idle-timeoutfrom the configuration.The comment also indicates t…