[Bug]: backend api keeps getting http 502 after upgrading to 1.53

[wansadi]

Is there an existing issue for this?

• I have searched the existing issues

Description

Been keeping up with latest version for a year everything works good till I bump into v1.53 2 weeks ago. Page kept failling to load. I found a request api/v1/consolidated-api/view was stopping page from complete. From K8s I found corresponding error log:

Wed, Jan 8 2025 3:36:39 pm
editor stderr | {"level":"debug","ts":1736321799.9084947,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:8080","duration":0.000218331,"request":{"remote_ip":"127.0.0.6","remote_port":"42623","client_ip":"172.22.1.175","proto":"HTTP/1.1","method":"GET","host":"appsmith-test.truckerpath.com","uri":"/api/v1/consolidated-api/view","headers":{"X-B3-Sampled":["0"],"X-B3-Spanid":["f19ec3357abd1a94"],"X-Envoy-Decorator-Operation":["appsmith.product.svc.cluster.local:80/*"],"X-Forwarded-Host":["appsmith-test.truckerpath.com"],"Sec-Ch-Ua-Mobile":["?0"],"X-Real-Ip":["172.22.1.175"],"Sec-Ch-Ua-Platform":["\"macOS\""],"X-Envoy-Peer-Metadata":["ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHwoMSU5TVEFOQ0VfSVBTEg8aDTE3Mi4yMi4xOC4yNDIKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE4LjIKnAMKBkxBQkVMUxKRAyqOAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwoSCgdyZWxlYXNlEgcaBWlzdGlvCjkKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0CiIKF3NpZGVjYXIuaXN0aW8uaW8vaW5qZWN0EgcaBWZhbHNlChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAovCgROQU1FEicaJWlzdGlvLWluZ3Jlc3NnYXRld2F5LTU0YzU0ODc2OWMtNWg4czUKGwoJTkFNRVNQQUNFEg4aDGlzdGlvLXN5c3RlbQpdCgVPV05FUhJUGlJrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvaXN0aW8tc3lzdGVtL2RlcGxveW1lbnRzL2lzdGlvLWluZ3Jlc3NnYXRld2F5ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAonCg1XT1JLTE9BRF9OQU1FEhYaFGlzdGlvLWluZ3Jlc3NnYXRld2F5"],"X-Envoy-External-Address":["172.22.25.162"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""],"X-Forwarded-Proto":["http"],"X-Envoy-Attempt-Count":["1"],"X-Envoy-Peer-Metadata-Id":["router~172.22.18.242~istio-ingressgateway-54c548769c

-5h8s5.istio-system~istio-system.svc.cluster.local"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"],"X-Appsmith-Request-Id":["eae73513-8c31-4931-a26f-736a06f5629a"],"X-Request-Id":["3a216823-0a0c-4444-8955-54f23800a9d2"],"X-B3-Traceid":["7bb9b24413ad60eef19ec3357abd1a94"],"Accept":["*/*"],"X-Forwarded-For":["172.22.1.175, 127.0.0.6"]}},"error":"dial tcp 127.0.0.1:8080: connect: connection refused"}
2025-01-08T07:36:39.909384172Z editor stderr | {"level":"debug","ts":1736321799.9086525,"logger":"http.log.error.log0","msg":"dial tcp 127.0.0.1:8080: connect: connection refused","request":{"remote_ip":"127.0.0.6","remote_port":"42623","client_ip":"172.22.1.175","proto":"HTTP/1.1","method":"GET","host":"appsmith-test.truckerpath.com","uri":"/api/v1/consolidated-api/view","headers":{"X-Forwarded-Proto":["http"],"X-Request-Id":["3a216823-0a0c-4444-8955-54f23800a9d2"],"X-B3-Traceid":["7bb9b24413ad60eef19ec3357abd1a94"],"Accept":["*/*"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""],"Sec-Ch-Ua-Platform":["\"macOS\""],"X-Envoy-Attempt-Count":["1"],"X-B3-Spanid":["f19ec3357abd1a94"],"X-Real-Ip":["172.22.1.175"],"X-Forwarded-For":["172.22.1.175"],"X-B3-Sampled":["0"],"X-Envoy-Decorator-Operation":["appsmith.product.svc.cluster.local:80/*"],"X-Envoy-Peer-Metadata-Id":["router~172.22.18.242~istio-ingressgateway-54c548769c-5h8s5.istio-system~istio-system.svc.cluster.local"],"X-Envoy-Peer-Metadata":["ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHwoMSU5TVEFOQ0VfSVBTEg8aDTE3Mi4yMi4xOC4yNDIKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE4LjIKnAMKBkxBQkVMUxKRAyqOAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW

8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwoSCgdyZWxlYXNlEgcaBWlzdGlvCjkKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0CiIKF3NpZGVjYXIuaXN0aW8uaW8vaW5qZWN0EgcaBWZhbHNlChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAovCgROQU1FEicaJWlzdGlvLWluZ3Jlc3NnYXRld2F5LTU0YzU0ODc2OWMtNWg4czUKGwoJTkFNRVNQQUNFEg4aDGlzdGlvLXN5c3RlbQpdCgVPV05FUhJUGlJrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvaXN0aW8tc3lzdGVtL2RlcGxveW1lbnRzL2lzdGlvLWluZ3Jlc3NnYXRld2F5ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAonCg1XT1JLTE9BRF9OQU1FEhYaFGlzdGlvLWluZ3Jlc3NnYXRld2F5"],"Sec-Ch-Ua-Mobile":["?0"],"X-Envoy-External-Address":["172.22.25.162"]}},"duration":0.000618318,"status":502,"err_id":"q7nvz839n","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
2025-01-08T07:36:39.909809387Z editor stdout | {"level":"error","ts":1736321799.9086723,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"127.0.0.6","remote_port":"42623","client_ip":"172.22.1.175","proto":"HTTP/1.1","method":"GET","host":"appsmith-test.truckerpath.com","uri":"/api/v1/consolidated-api/view","headers":{"Accept":["*/*"],"X-Forwarded-Proto":["http"],"X-Request-Id":["3a216823-0a0c-4444-8955-54f23800a9d2"],"X-B3-Traceid":["7bb9b24413ad60eef19ec3357abd1a94"],"Sec-Ch-Ua-Platform":["\"macOS\""],"X-Envoy-Attempt-Count":["1"],"X-B3-Spanid":["f19ec3357abd1a94"],"X-Real-Ip":["172.22.1.175"],"X-Forwarded-For":["172.22.1.175"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""],"X-B3-Sampled":["0"],"X-Envoy-Decorator-Operation":["appsmith.product.svc.cluster.local:80/*"],"X-Envoy-Peer-Metadata-Id":["router~172.22.18.242~istio-ingressgateway-5

4c548769c-5h8s5.istio-system~istio-system.svc.cluster.local"],"Sec-Ch-Ua-Mobile":["?0"],"X-Envoy-External-Address":["172.22.25.162"],"X-Envoy-Peer-Metadata":["ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHwoMSU5TVEFOQ0VfSVBTEg8aDTE3Mi4yMi4xOC4yNDIKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE4LjIKnAMKBkxBQkVMUxKRAyqOAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwoSCgdyZWxlYXNlEgcaBWlzdGlvCjkKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0CiIKF3NpZGVjYXIuaXN0aW8uaW8vaW5qZWN0EgcaBWZhbHNlChoKB01FU0hfSUQSDxoNY2x1c3Rlci5sb2NhbAovCgROQU1FEicaJWlzdGlvLWluZ3Jlc3NnYXRld2F5LTU0YzU0ODc2OWMtNWg4czUKGwoJTkFNRVNQQUNFEg4aDGlzdGlvLXN5c3RlbQpdCgVPV05FUhJUGlJrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvaXN0aW8tc3lzdGVtL2RlcGxveW1lbnRzL2lzdGlvLWluZ3Jlc3NnYXRld2F5ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAonCg1XT1JLTE9BRF9OQU1FEhYaFGlzdGlvLWluZ3Jlc3NnYXRld2F5"]}},"bytes_read":0,"user_id":"","duration":0.000618318,"size":15,"status":502,"resp_headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Request-Id":["3a216823-0a0c-4444-8955-54f23800a9d2"]},"uuid":"eae73513-8c31-4931-a26f-736a06f5629a"}

I have just 1 pod for appsmith deployment. Every time I just delete the pod and appsmith will be upgraded to latest by K8s's auto-recover

Steps To Reproduce

Upgrade to v1.53+ with k8s deployment

Public Sample App

No response

Environment

Production

Severity

High (Blocker to building or releasing)

Issue video log

No response

Version

self hosted 1.53

[devnoname120]

Same error here.

Edit: it was probably caused by something else. Either way v1.58 works now on my side.