org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 502

[Oliverwqcwrw]

I want to know when 502 will appear

ERROR 2022-03-23 20:58:19:117 grpc-default-executor-67 TraceSegmentServiceClient : Send UpstreamSegment to collector fail with a grpc i
nternal exception.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 502
invalid content-type: text/html; charset=utf-8
headers: Metadata(:status=502,server=Tengine,date=Wed, 23 Mar 2022 12:58:19 GMT,content-type=text/html; charset=utf-8,content-length=66

DATA-----------------------------

<title>502 Bad Gateway</title>

502 Bad Gateway

Sorry for the inconvenience.
Please report this message and include the following information to us.
Thank you very much!

URL:	http://xxx:12000/skywalking.v3.TraceSegmentReportService/collect
Server:	is-push-pro-push-app-skywalking-oapserver-01
Date:	2022/03/23 20:58:19

---

Powered by Tengine

---

tengine

k

502 Bad Gateway

Sorry for the inconvenience.
Please report this message and include the following information to us.
Thank you very much!

URL:	http://skywalking-oap-server.mbg.local:12000/skywalking.v3.TraceSegmentReportService/collect
Server:	is-push-pro-push-app-skywalking-oapserver-01
Date:	2022/03/23 20:58:19

---

Powered by Tengine

---

tengine

    at org.apache.skywalking.apm.dependencies.io.grpc.Status.asRuntimeException(Status.java:533)                                   
    at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java

:478)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(Cl
ientCallImpl.java:742)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(C
lientCallImpl.java:721)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.skywalking.apm.dependencies.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[Oliverwqcwrw]

We have many services that use the same OAP

[wu-sheng]

502 maybe mean OAP crashed.

We have many services that use the same OAP

I don't know, because this may be nothing relative to OAP, it could be some LB crashed, or database crashed, or database is too slow to respond. Check OAP logs, pure client side logs don't help much.

[Oliverwqcwrw]

502 maybe mean OAP crashed.

We have many services that use the same OAP

I don't know, because this may be nothing relative to OAP, it could be some LB crashed, or database crashed, or database is too slow to respond. Check OAP logs, pure client side logs don't help much.

There is another question that I would like to consult. After 502 appeared in one service, the Agent automatically recovered after a few days, and the other service has not recovered after 502 appeared, so is there any mechanism for this service

[wu-sheng]

AFAIK, there isn't. Maybe they connect to different backend pods?

[Oliverwqcwrw]

Ok, thanks for your reply. I'll take another look