Bug: 503 UC sometimes, upstream_reset_before_response_started #8639
Comments
zuercher
added
the
question
|
I don't know if this issue could be related, but I have something similar, where envoy close the TCP connexion (TCP FIN) without reasons. I have TCP dumps showing the issue where Envoy close the TCP connexion, which is re-opened when I restart the client application. |
|
I faced similar issues using Envoy as a proxy in front of Quarkus applications (see quarkusio/quarkus#4572). This started when Quarkus changed their HTTP layer to Vert.x. It seems to have something to do with socket timeout and keep-alive (connection pooling) between Envoy and Vert.x that does not match. I managed to solve this by setting max_requests_per_connection to 1 (=disabled keep alive see https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/cds.proto.html?highlight=keepalive) Maybe something you can try? |
|
@enbohm Thanks for your information. I also think it is a connection reuse problem. I will try your method. I am also trying to set idleTimeout to a small value, hope to solve this problem. |
|
Upgrade to Istio 1.3.4 where some bugs were corrected in istio's pilot. |
|
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions. |
stale
bot
added
the
stale
|
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions. |
This worked for me. It seems that the server my envoy was connecting to had HTTP Keep-Alive disabled. |
|
Update: I reverted disabling HTTP Keep-Alive. Instead I used upstream_connection_options with Now I am still not getting any 503 UC. |
hi @amalaruja could you please provide the details where we have to made changes with sample code. |
Bug Template
Title: 503 UC sometimes
Description:
I use Istio1.2.2, but sometimes I encountered some 503|UC error in access log and error log.
It seems like the issue #6578 and #2871.
Repro steps:
It's not easy to recurrent the bug, but I can see it about several times in one node.
Admin and Stats Output:
{
"version": "a975561b980463f08689d3debe33bb9eefc80c3d/1.11.0-dev/Clean/RELEASE/BoringSSL",
"state": "LIVE",
"command_line_options": {
"base_id": "0",
"concurrency": 32,
"config_path": "/var/lib/istio/envoy-rev0.json",
"config_yaml": "",
"allow_unknown_fields": true,
"admin_address_path": "",
"local_address_ip_version": "v4",
"log_level": "debug",
"component_log_level": "misc:error",
"log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v",
"log_path": "/export/Logs/envoy.log",
"hot_restart_version": false,
"service_cluster": "xxxx-ingress",
"service_node": "router
10.226.201.2181.xxxx-ingress.default~default.mesh.xxxx.com","service_zone": "",
"mode": "Serve",
"max_stats": "0",
"max_obj_name_len": "0",
"disable_hot_restart": false,
"enable_mutex_tracing": false,
"restart_epoch": 0,
"cpuset_threads": false,
"file_flush_interval": "10s",
"drain_time": "45s",
"parent_shutdown_time": "60s"
},
"uptime_current_epoch": "89062s",
"uptime_all_epochs": "89062s"
}
Config:
Logs:
Call Stack:
The text was updated successfully, but these errors were encountered: