LoggingHandler hangs the thread #2796

sparhomenko · 2018-01-20T18:02:46Z

Our application is running on Google Kubernetes Engine using gcr.io/google-appengine/jetty image and used com.google.cloud.logging.LoggingHandler to publish logs on Stackdriver. We noticed some worker threads becoming unresponsive over time. When the pod is shutting down we can see the following exception for each:

java.lang.RuntimeException: java.lang.InterruptedException
	at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:545)
	at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:525)
	at com.google.cloud.logging.LoggingHandler.publish(LoggingHandler.java:273)
	at java.util.logging.Logger.log(Logger.java:738)
	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:582)
	at org.slf4j.impl.JDK14LoggerAdapter.error(JDK14LoggerAdapter.java:500)
        ...
Caused by: java.lang.InterruptedException
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:449)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
	at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:543)
	... 30 more

We'll try to extract a thread dump to see why the future never completes, but the issue seems dangerous by itself: LoggingImpl.java:543 uses the non-timeout version of Future.get() which can cause any logger call to block the current thread forever unless interrupted. Would it be possible to use the timeout version with a reasonably big timeout, e.g. 60 seconds?

The text was updated successfully, but these errors were encountered:

pongad · 2018-01-22T00:38:20Z

This seems related to #1795. The underlying client provides no way to initiate a flush.

@sparhomenko Do you have a sense of how long we're waiting? The above bug might cause us to wait a few seconds, but it shouldn't cause a prolonged hang. I agree that the wait with no timeout is a problem, but I want to understand the problem better before sending fixes.

sparhomenko · 2018-01-22T21:30:26Z

@pongad Longest one we observed so far was 2.5 hours until manually interrupted. We're not sure what caused the future to hang for so long. The issue seems to be intermittently reproducible, I'll try to get more diagnostic information next time we hit it.

pongad · 2018-01-25T06:19:29Z

@sparhomenko Yikes! 2.5 hours is WAY beyond reasonable. I now suspect some kind of propagation problem somewhere. Maybe we don't properly complete the future for something. I'll try to dig into this as well, but a repro would be greatly appreciated.

ghost · 2018-02-11T01:36:07Z

Hi @pongad, Im now about to use google cloud logging API for logging in a multiThreaded java app, it'll be having near to 5k threads running concurrently, so any updates plz?

pongad · 2018-02-12T02:52:48Z

@tamersaleh I unfortunately don't have concrete progress to share yet. We plan to simplify the batching logic. The simplification should let us smoke out many of these bugs.

sparhomenko · 2018-02-12T23:05:01Z

@pongad I'm afraid we are not able to reproduce the issue anymore. The only useful thing we found was that on the pods where this was reproducible LoggingHandler actually stopped providing any logs after this exception occurred:

java.lang.RuntimeException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAVAILABLE: 503:Service Unavailable
	at com.google.cloud.logging.LoggingImpl$7.onFailure(LoggingImpl.java:577)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:52)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.common.util.concurrent.Futures$AbstractChainingFuture.run(Futures.java:1405)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:794)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:96)
	at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:78)
	at com.google.api.gax.grpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:68)
	at com.google.api.gax.grpc.BatchExecutor$1.onFailure(BatchExecutor.java:96)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:52)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.api.gax.retrying.RetryingFutureImpl.executeAttempt(RetryingFutureImpl.java:142)
	at com.google.api.gax.retrying.RetryingFutureImpl.access$500(RetryingFutureImpl.java:58)
	at com.google.api.gax.retrying.RetryingFutureImpl$AttemptFutureCallback.onFailure(RetryingFutureImpl.java:176)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:52)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:96)
	at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:78)
	at com.google.api.gax.grpc.ExceptionTransformingCallable$ExceptionTransformingFuture.onFailure(ExceptionTransformingCallable.java:108)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:52)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:463)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
	at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAVAILABLE: 503:Service Unavailable
	at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$1.apply(GrpcLoggingRpc.java:141)
	at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$1.apply(GrpcLoggingRpc.java:135)
	at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:124)
	at com.google.common.util.concurrent.Futures$CatchingFuture.doFallback(Futures.java:842)
	at com.google.common.util.concurrent.Futures$CatchingFuture.doFallback(Futures.java:834)
	at com.google.common.util.concurrent.Futures$AbstractCatchingFuture.run(Futures.java:789)
	... 48 more
Caused by: com.google.api.gax.grpc.ApiException: io.grpc.StatusRuntimeException: UNAVAILABLE: 503:Service Unavailable
	... 23 more
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: 503:Service Unavailable
	at io.grpc.Status.asRuntimeException(Status.java:540)
	... 15 more

These might have been caused by us producing too many logs, which we are not doing anymore.

One other suspicious thing is that latest version of google-appengine Docker image we use is still using a pretty old version of google-cloud-logging and related dependencies. So the issue might have been triggered by a problem in that version, or by a classpath conflict between the provided dependencies and the newer ones we package in our WAR.

dansomething · 2018-02-12T23:34:41Z

FWIW, we have experienced similar issues using both JUL and Logback as our client for google-cloud-logging. The hangs seem to correlate with GRPC connection issues. We have sadly never been able to reliably reproduce the issue. We eventually had to give up on using the "Enhanced Stackdriver Logging" feature of the jetty-runtime.

Here some stracktraces from our logs if that helps.

java.lang.RuntimeException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 49999922530ns at com.google.cloud.logging.LoggingImpl$7.onFailure(LoggingImpl.java:577) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:112) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at
 com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at
 com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55) at
 com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:82) at
 com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:96) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:157) at
 com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:134) at
 com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:116) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at
 com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at
 com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.setException(GrpcExceptionCallable.java:117) at
 com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:101) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:458) at
 io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:433) at
 io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) at
 io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:339) at
 io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:443) at
 io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:525) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:446) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:557) at
 io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at
 io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:107) at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
 java.util.concurrent.FutureTask.run(FutureTask.java:266) at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at
 java.lang.Thread.run(Thread.java:748) Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 49999922530ns at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:155) at
 com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:149) at
 com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:153) at
 com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:205) at
 com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:193) at
 com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:132)

java.lang.RuntimeException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAVAILABLE: Authentication backend unavailable. at com.google.cloud.logging.LoggingImpl$7.onFailure(LoggingImpl.java:577) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:112) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at
 com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at
 com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55) at
 com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:82) at
 com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:96) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:157) at
 com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:134) at
 com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:116) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at
 com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at
 com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.setException(GrpcExceptionCallable.java:117) at
 com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:101) at
 com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at
 com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at
 com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at
 com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at
 com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at
 com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at
 io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:458) at
 io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:433) at
 io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41) at
 io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:339) at
 io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:443) at
 io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:525) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:446) at
 io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:557) at
 io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at
 io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:107) at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
 java.util.concurrent.FutureTask.run(FutureTask.java:266) at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at
 java.lang.Thread.run(Thread.java:748) Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAVAILABLE: Authentication backend unavailable. at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:155) at
 com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:149) at
 com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:153) at
 com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:205) at
 com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:193) at
 com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:132)

ghost · 2018-02-13T01:52:42Z

@pongad my experience with cloud logging is just that, when I'm trying to call logging.write(Collections.singleton(logEntry)), it doesn't seem like its called asynchronously, and it makes 90% of my threads in a BLOCKED state

pongad · 2018-02-13T22:02:45Z

The method taking 1min to timeout is curious. That's the deadline we set on our RPCs but Logging shouldn't take that long. This might be related to threads being in BLOCKED state; if threads are blocked, they can't service RPC.

Could you share with us the stack trace of the BLOCKED threads?

@garrettjonesgoogle FYI

ghost · 2018-02-14T01:14:01Z

some of them BLOCKED, and others are WAITING (parking)

sample of "WAITING (parking)":

"user_mini_table_76c0c70d45e05cc_616" #838 prio=10 os_prio=0 tid=0x00007f511c494000 nid=0x541b waiting on condition [0x00007eae3bddf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000003c0478268> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at com.google.api.gax.batching.ThresholdBatcher.add(ThresholdBatcher.java:167)
        at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:73)
        at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
        at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
        at com.google.cloud.logging.spi.v2.GrpcLoggingRpc.write(GrpcLoggingRpc.java:223)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:593)
        at com.google.cloud.logging.LoggingImpl.writeLogEntries(LoggingImpl.java:559)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:522)
        at com.dating.util.CloudLogger.log(CloudLogger.java:106)
        at com.dating.util.CloudLogger.logInfo(CloudLogger.java:83)
        at com.dating.util.Matching.getMyBestMatch(Matching.java:120)
        at com.harmonica.match.FindMyMaleMatchService.getMatches(FindMyMaleMatchService.java:61)
        at com.harmonica.match.FindMyMaleMatchService.findMalesForGirls(FindMyMaleMatchService.java:40)
        at com.harmonica.match.MatchGroupService.lambda$getMatchBatch$2(MatchGroupService.java:236)
        at com.harmonica.match.MatchGroupService$$Lambda$24/1081662793.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

sample of "BLOCKED":

"user_mini_table_76c0c70d45e05cc_615" #837 prio=10 os_prio=0 tid=0x00007f511c492000 nid=0x541a in Object.wait() [0x00007eae7bde0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at com.google.api.gax.batching.BlockingSemaphore.acquire(BlockingSemaphore.java:61)
        - locked <0x00000003c04781e0> (a com.google.api.gax.batching.BlockingSemaphore)
        at com.google.api.gax.batching.FlowController.reserve(FlowController.java:197)
        at com.google.api.gax.batching.BatchingFlowController.reserve(BatchingFlowController.java:58)
        at com.google.api.gax.batching.ThresholdBatcher.add(ThresholdBatcher.java:166)
        at com.google.api.gax.rpc.BatchingCallable.futureCall(BatchingCallable.java:73)
        at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
        at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
        at com.google.cloud.logging.spi.v2.GrpcLoggingRpc.write(GrpcLoggingRpc.java:223)
        at com.google.cloud.logging.LoggingImpl.writeAsync(LoggingImpl.java:593)
        at com.google.cloud.logging.LoggingImpl.writeLogEntries(LoggingImpl.java:559)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:522)
        at com.dating.util.CloudLogger.log(CloudLogger.java:106)
        at com.dating.util.CloudLogger.logInfo(CloudLogger.java:83)
        at com.dating.util.Matching.getMyBestMatch(Matching.java:120)
        at com.harmonica.match.FindMyMaleMatchService.getMatches(FindMyMaleMatchService.java:61)
        at com.harmonica.match.FindMyMaleMatchService.findMalesForGirls(FindMyMaleMatchService.java:40)
        at com.harmonica.match.MatchGroupService.lambda$getMatchBatch$2(MatchGroupService.java:236)
        at com.harmonica.match.MatchGroupService$$Lambda$24/1081662793.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

the line of (CloudLogger.java:106) is actually: logging.write(Collections.singleton(logEntry));

pongad · 2018-02-14T03:59:33Z

@tamersaleh Both of these are somewhat expected.

For WAITING, the thread seems to be waiting to lock a shared structure. I'd expect the number to be relatively small though, since the threads should update the structure then unlock quickly.

For the BLOCKED, that's caused by Logging throttling the application because there are too many logs waiting to be sent. This usually happens when the logs are generated faster than they can be sent to the server.

I assume you're not deadlocked, but progressing slowly? Do you have a sense of how many logs you're sending and how large they are? If the RPCs are too slow, there are some enhancements we can make, like opening multiple connections, etc.

ghost · 2018-02-14T04:07:40Z

@pongad
yes thats exactly what I'm facing now, logs are generated in a speed > 100,000 entries / sec

ghost · 2018-02-14T04:09:24Z

@pongad
right now I did some changes I let threads send all logs to a ConcurrentLinkedQueue then a single thread polling logs from there and posting them in chunks of 4k entries by calling logging.write(entries) and this single thread is scheduled to run every 2 sec but even thought this is not fast enough, Im thinking if eh RPC is fast enough I can enlarge this chunk may be to 50k or may be more

ghost · 2018-02-14T07:51:21Z

@pongad I've got an idea,
what about I call POST https://logging.googleapis.com/v2/entries:write directly ?
can I post concurrent calls and batches calls faster than the client lib ?

ghost · 2018-02-15T12:23:01Z

@pongad thanks alot, so when this new change can be in maven repository?

csainty · 2018-08-23T06:56:27Z

What is the status here?

Our application is taken down almost every night by the RuntimeException being thrown deep inside the async logging handler - https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-clients/google-cloud-logging/src/main/java/com/google/cloud/logging/LoggingImpl.java#L578

How are we supposed to catch and handle this exception since it isn't in our own callstack?

pongad · 2018-09-20T00:22:15Z

The immediate issue should be resolved now (by bumping gax dependency). There should probably be a way to configure logging clients to drop logs instead of blocking for some applications. Turning this into feature request.

michael-barker · 2018-10-05T00:43:56Z

I have a sample project that hangs every time log.error(...) is called. This happens when running locally and deployed to GKE. I also tried bumping gax and gax-rpc to 1.32.0 but that didn't seem to help. This is a WebFlux app but appears to happen regardless of whether I use WebFlux or WebMVC.

edit: Look like I don't need spring-cloud-gcp-starter-logging in order for logs to be sent to Stackdriver. Remove that fixed the issue for me. Maybe the sample project could still be useful though for reproducing the original issue.

sergei-ivanov · 2018-10-19T10:47:11Z

Hi,

Having enabled Stackdriver logging (via spring-cloud-gcp-starter-logging and gcloud logback appender) about a week ago, we are facing recurring issues there.

First of all, it appears that whenever LoggingImpl has to flush, it is a blocking synchronous operation from the caller's perspective. I initially had an assumption that writeSynchronicity (which is set to ASYNC by default for logback) would ensure that all underlying API calls are offloaded to some background processing thread. It does not seem to be the case.

It would be great if all asynchronous and blocking operations (and configuration options, when available/applicable) were properly documented. Otherwise it involves a lot of guesswork on the part of library users.

So in order to mitigate sporadic delays caused by gcloud logging appender, we decided to wrap it inside an AsyncAppender. Unfortunately, it made matters worse. At some point the cloud backend failed with UNAVAILABLE: Authentication backend unavailable, which completely locked up the application's processing thread. I've done a thread dump (at the bottom of this message), and it looks like we effectively have a deadlock there.

The application thread is waiting on async appender, which has its buffer full and is blocked
The working thread of async appender is waiting for the gcloud logger to flush
The gRPC thread failed with an exception and tried to log that exception, which also resulted in it being blocked on async appender call!

So the application is in a state where a recovery is impossible.

My next step in trying to mitigate this will be increasing the queue size for AsyncAppender and enabling the neverBlock option on it. Which means some messages will be lost when the gcloud backend chokes up, but at least the application won't be deadlocked by that:

<configuration>

    <springProperty name="APP_NAME" source="spring.application.name" default="spring"/>

    <appender name="CLOUD" class="com.google.cloud.logging.logback.LoggingAppender">
        <flushLevel>WARN</flushLevel>
        <log>${APP_NAME}</log>
    </appender>
    <appender name="ASYNC_CLOUD" class="ch.qos.logback.classic.AsyncAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <!-- May result in message loss when cloud backend is unavailable, but won't block the application -->
        <neverBlock>true</neverBlock>
        <queueSize>2048</queueSize>
        <appender-ref ref="CLOUD"/>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="ASYNC_CLOUD"/>
        <!-- Other appenders here... -->
    </root>

</configuration>

Again, maybe it's worth updating the documentation (both the project's readme file, and the stackdriver docs) to include the recommended AsyncAppender settings.

And here comes a thread dump:

"http-nio-8080-exec-1" #58 daemon prio=5 os_prio=0 tid=0x00007f388ff7e000 nid=0x754d waiting on condition [0x00007f3834
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000733d7e4b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:
        at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
        at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:181)
        at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:172)
        at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:161)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.warn(Logger.java:688)
        at <application code>

"AsyncAppender-Worker-ASYNC_CLOUD" #15 daemon prio=5 os_prio=0 tid=0x00007f388cd28000 nid=0x7512 waiting on condition [
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000079bba6b18> (a com.google.common.util.concurrent.CollectionFuture$ListFuture)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
        at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:544)
        at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:526)
        at com.google.cloud.logging.logback.LoggingAppender.append(LoggingAppender.java:201)
        at com.google.cloud.logging.logback.LoggingAppender.append(LoggingAppender.java:63)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:290)
   Locked ownable synchronizers:
        - None

"Gax-2" #16 daemon prio=5 os_prio=0 tid=0x00007f3830410000 nid=0x7522 waiting on condition [0x00007f3870f53000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000733d7e4b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:
        at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
        at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:181)
        at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:172)
        at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:161)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
        at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
        at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
        at java.util.logging.Logger.log(Logger.java:738)
        at java.util.logging.Logger.doLog(Logger.java:765)
        at java.util.logging.Logger.log(Logger.java:876)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:907)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:112)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677)
        at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677)
        at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
        at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
        at com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55)
        at com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:84)
        at com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:96)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1126)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677)
        at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95)
        at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.ja
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1126)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:493)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:29
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   Locked ownable synchronizers:
        - <0x00000007343591f8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

…to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time.

dave-r12 · 2018-12-26T18:50:00Z

Hey, I've also hit this issue. I was able to replicate it (mostly reliable) with this piece of code:

    var latch1 = new CountDownLatch(1);
    var thread1 = new Thread(() -> {
      try {
        latch1.await();
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
      log.error(Strings.repeat("a", 210000));
    });
    var thread2 = new Thread(() -> {
      latch1.countDown();
      try {
        Thread.sleep(5);
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
      log.info("Hello");
    });

    thread1.start();
    thread2.start();

    thread1.join();
    thread2.join();

I'm unfamiliar with the implementation but I was seeing this crop up when we attempted to log a very big message, causing the GCP logging impl to throw a RuntimeException (message > 110K). In the above code it happens in the log.error() call. This seems to cause an issue when a separate, concurrent thread is also logging something but just after the exception! The sequence is deliberate.

…to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time.

* fix #3880 #2796 disable automatic flush on write to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time. * fix formatting * add flush before listing

ajaaym · 2019-01-02T18:04:27Z

fixed in #4254

@garrettjonesgoogle

* Fix: writeAtLeastOnce should use a ReadWriteSession (#3882) * Fix search folder for BQDT synth output (#3878) * Bigtable: Add integration test to test nonexistent row handling (#3870) * Fix #3292 (#3875) * Regenerate video-intelligence client (#3865) * Release v0.69.0 (#3890) * Bump next snapshot (#3892) * Bigtable: fix admin tests to able to run concurrently (#3895) * Fix ApiFutures.addCallback() (#3886) ApiFutures.addCallback() now requires a third argument (a non-direct executor). Ref: https://github.com/googleapis/api-common-java/blob/master/src/main/java/com/google/api/core/ApiFutures.java * Regenerate logging client with test fixes (#3907) * Regenerate DLP Client (#3823) * Regenerate vision client: Adds product search (#3888) * Regenerate vision client * vision proto libraries need com.google.api:api-common compile dependency * Regenerate dataproc client (#3906) Unsets empty list of retry status codes * Regenerate monitoring client (#3908) Unsets empty list of no-retry status codes * Move BOM info to right after Quick Start (#3555) * Move BOM info to right after Quick Start * split version management * change to old version combinations * Update README.md * Spanner gapic migration (#3881) * Add GapicSpannerRpc stub (#3016) * spanner: implement partitionRead (#3022) * spanner: implement LRO methods using GAPIC stub (#3039) * Add implementation to:listInstances, listInstanceConfigs, listDatabases, getInstance, getInstanceConfigs, getDatabase, dropDatabase, deleteInstance, getDatabaseDdl (#3043) * spanner: move admin clients to GAPIC stub (#3067) * Spanner: migrate all unary call methods to gapic and inject headers (#3112) * Spanner: migrate streaming methods to gapic (#3139) * Spanner gapic migration lro methods (#3166) Migrate longrunning methods to GAPIC including: createDatabase, updateDatabaseDdl, createInstance, and updateInstance. * Make ChannelPool work (#3258) * Spanner gapic migration error augmentation with interceptor (#3304) * Add interceptors (#3346) * Clean up Spanner before merging to master (#3362) * Spanner Gapic Migration: fix updateDatabaseDdl (#3403) * Regenerate compute client (#3916) Adds interconnect features * Regenerate dlp client (#3917) Add excluded_fields to BigQueryOptions * Add a feature to list the buckets (#3915) Also, update the README to reflect recent update to pseudoDirs. * Pubsub: use gapic client (#3581) * bump checkstyle version to build on Java 9 The version of checkstyle we currently use uses tools.jar which is removed from Java 9 and above as part of Project Jigsaw. This commit uses a newer version of checkstyle that does not use tools.jar. Running `mvn checkstyle:checkstyle` succeeded.� * maybe this will be java 7 compat...? * pubsub: use GAPIC stub * fix merge * pr comment * Update CODEOWNERS fallback match should be last * Storage NIO: Stop leaving leftover buckets in IT (#3898) * Stop leaving leftover buckets in ITGcsNIO This integration test now deletes all the buckets it creates. The issue was that requester-pays buckets couldn't be deleted by the existing code. This PR updates RemoteStorageHelper so it can be given a userProject for requester-pays buckets. * Don't set userProject if empty, add test RemoteStorageHelper now doesn't set the userProject option if it's empty. Also, added testForceDeleteRetriesWithUserProject unit test. * Logging: Add a traceSampled field to the LogEntry class. (#3863) The new field was added to the Stackdriver LogEntry protocol buffer message in googleapis/googleapis@7b34e5a#diff-a51916a680df6453b6d86586cd7d63f9R149. * Bigtable: add sync methods (#3856) * New Client: Cloud Scheduler (#3913) * Add synth.py for generating cloud scheduler client * Add generated cloud scheduler code * Fix proto/grpc artifact names * Add versions and pom.xml files * Update list of stub packages in google-cloud-clients * Add README * Exists method added in BigtableInstanceAdminClient (#3891) * Firestore: Validate that at least one argument is provided for Cursor (#3900) * Release google-cloud-java v0.70.0 (#3921) * Bump next snapshot (#3922) * bigtable emulator (#3840) This allows customers to easily interact with the bigtable emulator. Currently the Bigtable emulator is implemented in go. It is a fairly complex bit of code that would be very difficult to replicate in java. This makes it very difficult for users to interact with it in a java project. This PR attempts to make things easier for the user by wrapping the golang emulator in a jar and providing a JUnit rule to simplify lifecycle management of the emulator. It's implemented as 2 artifacts: - google-cloud-gcloud-maven-plugin: a maven plugin that can download artifacts from the gcloud sdk repo and add them as resources to a jar. This avoids the need to check in binaries into google-cloud-java and makes sure that the binaries are up to date. - google-cloud-bigtable-emulator: This uses the maven plugin to bundle the bigtable emulator in a jar and adds a java wrapper around the binary. The wrapper was extracted from the existing bigtable client integration tests: https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/it/env/Emulator.java * Update README to suggest importing BOM (#3269) * Import BOM @garrettjonesgoogle * google-cloud-bom * Update README.md * Spanner: use WatchdogProvider (#3923) * change to watchdog provider * Bigtable: [Breaking change] listTables() now returns relative table names (#3696) * Bigtable: be consistent and don't expose absolute table names * fix merge * fix merge * Update container synth script for new artman config name (#3925) * New Client: BigQuery Storage (#3935) * New Client: Cloud Security Command Center (#3885) * Add synth.py script and generate files * Add versions for release management, add README.md, pom.xml * Add google-cloud-security center as module for google-cloud-clients * Add securitycenter to google-api-grpc modules * Add securitycenter artifacts to the bom * security center depends on iam protos * Regenerate Security Center models/stubs * Fix artifact versions * Fix artifact versions * Fix artifact versions * Add Kokoro nightly build configs (#3926) * Add namespace and schema to settings.xml (#3941) Updates the release setup guide. * Regenerate bigquerystorage client (#3939) Adds documentation strings * fix: google-cloud-clients/pom.xml to reduce vulnerabilities (#3943) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JAVA-COMMONSFILEUPLOAD-30401 * Regenerate speech client (#3933) Adds `result_end_time` to `StreamingRecognitionResult` * Update google-auth-library-java, apiary core libraries (#3945) * Update google-auth-library-java, apiary core libraries * Fix static import from shaded guava package * Upgrade gax, grpc and guava dependencies (#3980) * Bump gax version to 1.35.0 and guava to 26.0-android * Update grpc dependency to 1.16.1 * Add temporary, specific animal-sniffer-annotation version * Release v0.71.0 (#3978) * Bump next snapshot (#3981) * Fix BigQuery NullPointerException when estimatedFields is empty (#3984) * BigQuery client throws NullPointerException when estimatedFields is empty * fix formatting * Regenerate container client (#3979) * Cloud Asset fix README links (#3985) Updated Cloud Asset links to point to the Cloud Asset API page. * Add synth metadata for many clients (#4020) * Storage: Added feature to change host name of signed url (#3910) * #3140 feature added to change hostname of generated Url * add test case * removed overload method and add method withExtHostName in SignUrlOption * rename EX_HOST to SERVICE_ENDPOINT,rename withExtHostName to withHostName and used google-java-formatter * update as per feedback * fix formatting * update comments * update Signed URLs Comment * Regenerate websecurityscanner client (#4080) * Delay download of Bigtable emulator (#4102) * Add gcloud.download.skip option for google-cloud-gcloud-maven-plugin * Skip emulator download when doing the initial install * Rename `skip` to `shouldSkipDownload` * [Snyk] Fix for 1 vulnerable dependencies (kr.motd.maven:os-maven-plugin) (#4097) * fix: google-cloud-util/google-cloud-compat-checker/pom.xml to reduce vulnerabilities The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JAVA-ORGCODEHAUSPLEXUS-31521 * Add guava dependency in google-cloud-compat-checker Previously, the kr.motd.maven:os-maven-plugin brought in a dependency that brought in guava. This artifact explicitly uses guava collections so we need to add the dependency here. * fix: google-cloud-testing/google-cloud-gcloud-maven-plugin/pom.xml to reduce vulnerabilities (#4100) The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JAVA-ORGCODEHAUSPLEXUS-31521 * Trace: list response returns empty list instead of null (#4079) * Regenerate speech client: remove unused imports (#4077) * Update generated code: List responses return empty list instead of null (#4104) * Regenerate automl client * Regenerate bigquerydatatransfer client * Regenerate bigtable-admin client * Regenerate compute client * Regenerate containeranalysis client * Regenerate dataproc client * Regenerate dialogflow client * Regenerate dlp client * Regenerate errorreporting client * Regenerate firestore client * Regenerate kms client * Regenerate logging client * Regenerate redis client * Regenerate scheduler client * Regenerate securitycenter client * Regenerate spanner client * Regenerate tasks client * Regenerate language client: remove unused imports (#4071) * Datastore: Add `skippedResults` in `QueryResults` (#4093) * Fix #3279 google-cloud-datastore Java library exposes too little of the information in query results * fix typo in comment * BigQuery Storage: Add shim layer settings and configuration (#4022) * Create a shim layer for the BigQuery Storage API. This change adds a simple shim layer to the BigQuery Storage API client. It is modeled after the shim layer in the Cloud Bigtable client, but unlike that implementation, it does not create chained instances of unary and server-streaming callables to perform translation; instead, its parameters are simple pass-throughs to the underlying gRPC client stub. The shim currently does nothing but read default values from the base client layer and pass them back through. * Set the default gRPC inbound message size. This change sets the default value for the maximum inbound message size in the BigQuery Storage API client. A ReadRowsResponse object can contain serialized row block of up to 10 MB in size if the underlying row(s) are sufficiently large. * Update the default timeout value for ReadRows. This change modifies the default client-side timeout value for the ReadRows API. ReadRows calls can last for long periods in the case of large tables. The default timeout value is set to a full day, which is the duration for which a read session is valid. * Add a resumption strategy for ReadRows. This change modifies the configuration for the ReadRows API to add a resumption strategy, allowing the connection to be resumed transparently in the case of transient errors. * Spanner: fix code formatting (#4105) * Storage NIO: Add new retry options in CloudStorageConfiguration (#3869) * Put new retry options in CloudStorageConfiguration Add retriableHttpCodes and reopenableExceptions so the user can, if they choose, customize the conditions under which we retry an access or reopen a channel after an exception closed it. This can be handy if a cloud provider starts transiently throwing exceptions that would normally indicate a fatal condition, but which in practice will pass if retried. The backoff logic in retries / reopens in unmodified. * Clean up dead code line * retriable -> retryable (in NIO Cloud Storage code) Also add a few basic tests to make sure the retryable options do what we expect. * Update testListFilesInRootDirectory to match reality In the built-in UNIX filesystem, the newDirectoryStream method will return absolute paths if given absolute paths as input, and relative paths if given relative paths as input. We're now seeing this too for the NIO Cloud Storage provider (this is a good thing), so I updated this test to reflect this new reality. * remove unused import * Add boilerplate file header * Keep old ctor, just @deprecated * Add back the *other* ctor that needs to be deprecated * add @deprecated outside of comment * Regenerate IoT client (#4069) Adds new API endpoints for: * sendCommandToDevice * bindDeviceToGateway * unbindDeviceFromGateway * Moving surefire-junit4 out of the dependency lists to allow others to better run tests. (#4058) * Try adding maven-surefire-plugin as a plugin to pull the correct surefire-junit. * Let's move the surefire-junit4 out of util as well to be consistant. * Fix indentation in pom * Regenerate dataproc client - Adds workflow templating (#4110) * Add missing fields on MonitoredResourceUtil (#3887) * added missing fields #3559 * Fix indentation * Add helpers to convert core Date class with java.util.Date class (#3911) * #3157 add methods toDate, fromDate * resolved test case Date.getYear() * update as per feedback * Update javadoc string * Bigtable: publish emulator to maven central (#4106) * Bigtable: publish emulator to maven central * make sure nexus plugin is not disabled * Release v0.72.0 (#4113) * Adding FieldMask support to GetAll() (#4017) * Bump next snapshot (#4118) * Logging: Allow deferred formatting for trace and spanId (#3661) * Allow deferred formating for trace and spanId. * Add back setTrace(String) and setSpanId(String) for binary compatibility * Update HttpStorageRpc.java (#4109) * Make TestServiceFactory and TestServiceRpcFactory interfaces public (#4112) * Java 11 compatibility (#3821) * Add the javax.annotation-api as a compile scope to the google-api-grpc projects * Upgrade maven-compiler-plugin 3.5.1->3.7.0 and maven-jar-plugin 2.6->3.1.0 * Add annotation-api compile dependency to google-cloud-clients * Fix objensis version 3.1.0->2.6 * Fix service option test visibility * Temporarily using Java 7 build to test Java 11 * Restore the java 7 test to use the java 7 image * Add release profile to bigtable emulator (#4119) * Release v0.72.0 * Add release profile to bigtable emulator * Add release profile to bigtable emulator * Add lint test and format generated code (#4114) * Add lint Kokoro tests * synthtool will now run the java formatter after generating the files * BigQuery: Fix TableId behaviour for non default BigQueryClient project (#4115) * Fixes create, update and delete methods * remove comments * Fix formatting * Run google-java-format on all artifacts (#4121) * Run google-java-format on all artifats * Format some missed files * Add more missed files * Fix formatting issues from the merge * Pub/Sub: Update ApiFutures.addCallback() to take 3 arguments (#4120) * Update ApiFutures.addCallback() * use DirectExecutor * nit * use directExecutor * run formatter * Update maven-javadoc-plugin to 3.0.1 and disable doclint. (#4167) 3.0.0 introduced a setting to disable doclint via config instead of a necessitating a conditional profile based on the java version. * Remove the old clients compatibility chart. (#4163) Any of these clients in question would be using versions older than a year and should upgrade. * BigQuery: insertAll always retries operation (#4165) * fix #3321 BigQuery insertAll always retries operation * Update Spanner description for GA (#4174) * Storage: Close response body after request (#4116) * 4107: Disconnect response after request * 4107: Added integration test testWriteChannelWithConnectionPool * 4107: Renamed blobName. * Fix lint * Storage: Fixed link to StorageExamples page (#4173) * 4169: Fixed link to StorageExamples page * 4169: Fixed Line 39 An App Engine application that manages a virtual bookshelf. * 4169: Removed WS * Integration test deflake (#4176) * Bump integration test timeout to 15 minutes from 10 minutes * Call the maven test directly - no need for verify_single_it.sh * provide spanner.testenv.instance property from Kokoro config * Bump integration test timeout to 20 minutes * Fix syntax * Remove exclusions for guava-jdk5 and google-api-client (#4177) We've updated the apiary clients to versions that no longer need these exclusions. * Storage: Give users the ability to disable gzip content encoding to increase throughput (#4170) * 3822: Added BlobTargetOption for user to set disabledGzipContent true * 3822: Changed method comment * 3822: Changed method comment with more explanation * 3822: Fixed comments after review. Added unit test * 3822: Fixed formatting errors in Storage class * 3822: Fixed formatting errors in StorageImplTest class * Spanner: Handle AbortedExceptions that occur before commit (#4181) * spanner: Handle AbortedExceptions that occur before commit The current TransactionRunner code checks for aborted transactions by using the isAborted() function. This function returns the value of "aborted" that is only set when processing resultSets. This does not work for DML which only uses the resultSet stats and does not actually call ResultSet.next(). This patch adds an additional check for the aborted exception type and retries such transactions. * throw before executeUpdate, verify flag is false after test * Update google-cloud-clients/google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITDMLTest.java Co-Authored-By: nithinsujir <118742+nithinsujir@users.noreply.github.com> * Put the spanner test instance config back * Document purpose of test-jar and testlib dependencies (#4164) * google-cloud-bom should not need google-cloud-core test-jar (#4168) * google-cloud-bom should not need google-cloud-core test-jar * Need to declare google-cloud-core version still * testing modules need to specify the google-cloud-core test-jar version * Regenerate pubsub client: Fix empty list returns (#4159) * Regenerate pubsub client * Fix the default maxInboundMessageSize config via synth * BigQuery: Allow job to extract or load table across projects (#4183) * Fix #3924 ExtractJobConfiguration's setProjectId makes cross-project BQ extracts impossible * Bigtable: fix emulator publishing (#4189) * Bigtable: fix emulator publishing Currently the emulator can't be used because it's parent pom is not published. THis PR removes the parent and copies the relevant bits into the emulator's pom Fixes #4187 * Add a couple more bits from the parent pom * oops * Release google-cloud-java v0.73.0 (#4191) * Release v0.73.0 * Fix version replacements * Move Spanner to GA section * Fix doclint option * Fix pom issues for Bigtable emulator * Bump next snapshot (#4192) * Add Container synth.metadata (#4193) * Add Pub/Sub synth.metadata (#4194) * Update variable naming. (#4199) * Update api-common-java javadoc link. (#4200) * Regenerate compute client: adding beta config classes/options (#4201) * Regenerate dataproc client: Adding beta APIs for workflow templates (#4202) * NIO: Enabled blocked tests. Fixed testCantCreateWithoutUserProject test. (#4182) * 3448: Ublocked tests. Fixed testCantCreateWithoutUserProject test. * 3448: Fixed code format. * 3448: Fixed code format issue * Bigtable: readRow methods added with Query as argument (#4092) * Regenerate automl client - No substantive changes (#4209) * Regenerate compute client (#4210) This PR was generated using Autosynth. 🌈 Adding new resource types and methods from updated Discovery doc. * Fix setProjectId for insertAll (#4196) * Pubsub: Test for blocking push subscriptions for vpc-sc enabled projects (#4214) * Bump gax version to 1.35.1 (#4217) * Add IamCredentials client (#4216) * Add IamCredentials client * Fix pom versions and formatting * Remove classes that were in my cache directory from a bad previous generation * Remove some more files cached from a bad synth * Fix xml formatting * Fix xml formatting * Update README descriptions and links * Fix generated code lint * Synth script should format java code * Release v0.74.0 (#4222) * Bump next snapshot (#4224) * BigQuery : Fix setProjectId (#4227) * Fix Change in TableId behaviour for non default BigQueryClient project * Fix setProjectId for TableInfo * add test cases and fix writechannel * fix format code * Add BadRecords to LoadStatistics (#4230) * Add script to help bootstrap new client libraries (#4111) * Add new client python script * Add requirements.txt, convert to use attrs * Start new artifact versions at 0.0.0-alpha. Next release will bump version * Switch to pathlib * Move main execution into main() * Standardize quotes to double-quotes * Update docstrings * Fix Path vs. string types when reading/writing files * Update synth.py template to include the java formatter * Update javadoc link for api-common-java. (#4236) * IAMCredentials: Fix name/description in pom (#4238) * Regenerate google-cloud-compute (#4213) Generation of Compute client with latest gapic-generator changes. This also contains one set of manual changes in ITComputeTest, where we remove the hack needed to parse a ResourceName string. * Ignore testRotateFromCustomerEncryptionToKmsKeyWithCustomerEncryption while we work out backend change (#4266) * Fix NPE in BQ's StandardTableDefinition (#4247) * Bigtable: add a separate callable for point reads (#4264) * Spanner: closed watch-dog daemon threads spawned by Spanner client (#4263) * Fixed Issue - shutting-down WatchDog daemon threads spawned by Spanner client * Fixed Issue - shutdown watchDog daemon threads spawned by Spanner client * Minor code formatting * Fixed Kokoro code format issue * Fix Kokoro release jobs (#4225) * Fix Kokoro release jobs. Needs loopback pinentry-mode * Skip install on deploy * skip stagingProfileId * Capture built artifacts * Enable promote/drop jobs via environment variable * Fix directory path and use -B for drop/release * promote/drop need the settings file * Fix path to settings file * Update RELEASING documentation to use the Kokoro jobs * Pub/Sub: Add message abandonment (#4250) * add abandonment * add unit test for abandon * update forget() to stop extending deadline * BigQuery: Fix Maximum Bytes Billed property. (#4262) * Fix Maximum Bytes Billed * remove whitespace * Update UsePubSubEmulatorSnippet.java (#4256) * Logging: disable severity based flush on write by default (#4254) * fix #3880 #2796 disable automatic flush on write to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time. * fix formatting * add flush before listing * Docs: Fix maven site generation (#4232) The configuration for aggregate changed from an attribute to a report * Bigtable: deprecate the use of typesafe names (#4257) * Bigtable: deprecate the use of typesafe names Unfortunately typesafe names create more issues for bigtable than they solve. A lot of the time users need to use both bigtable data & admin apis. Unfortunately, those apis define the same names in different namespaces. This forces users to use fully qualified names which is ugly. Also, absolute names don't benefit the bigtable client because the client is anchored at the instance and all names end up being relative * fix docs * format code * Fixed cannot call public methods in via reflection (#4249) * modified package-private to public * update format for storage/Option.java * BigQuery : Fix numBytes, numRows in TableInfo/Table (#4271) * Fix NumBytes and NumRows * update NumBytes and NumRows * remove setter * update methods and resolved test case * Cloud Bigtable: helloWorld sample (#4274) * helloWorld sample * test class modified * format change in ITHelloWorld * Add .tmpl extension for README.md so releasetool is not confused (#4280) * Release v0.75.0 (#4282) * Bump next snapshot (#4286) * Fix #4284 NPE for getZone (#4289) * Regenerate securitycenter client (#4290) * Bigtable admin: deprecate typesafe names part 2 (#4258) * Bigtable admin: deprecate typesafe names This is a followup on #4257: Unfortunately typesafe names create more issues for bigtable than they solve. A lot of the time users need to use both bigtable data & admin apis. Unfortunately, those apis define the same names in different namespaces. This forces users to use fully qualified names which is ugly. Also, absolute names don't benefit the bigtable client because the client is anchored at the instance and all names end up being relative * format code * fix year * Regenerate iamcredentials client (#4277) * Revert "Pub/Sub: Add message abandonment (#4250)" (#4297) This reverts commit 6e2c2dd. * updated links (#4287) * updated links * updated link * added new line * Release v0.76.0 (#4298) * Regenerate dataproc client (#4301) * Bump next snapshot (#4300) * Cloud Bigtable: instanceAdmin sample (#4299) * instanceAdmin sample and tests * comments added * instanceAdminExample changes * renamed variables * changes * Regenerate securitycenter client (#4311) * 3540: Used transformer for shaded plugin (#4305) * Added Query#fromProto to convert protobuf ReadRowsRequest (#4291) * Adding Query.fromProto * Adding Query.fromProto * adding changes as per feedback * updated javadoc as per feedback * reformatted Query & QueryTest.java * Bigquery: corrected equality check on subclasses of StringEnumValue (#4283) * Bigquery: corrected equality check on subclasses of StringEnumValue * update format for FieldTest.java * Fix #4269 update metata url to FQDN (#4278) * 4152: Added checking PAGE_TOKEN from options (#4303) * 3918: Added checking of attributes size for 0. Added unit test (#4304) * Bigtable: Merge bigtable-admin into the bigtable artifact (#4307) * Bigtable: merge bigtable-admin into the bigtable artifact * split exclusion change for future PR * fix admin pathes * fix deprecation warning * fix synth script * fix generated file * temporarily re-add kokoro scripts (and point them to the merged artifact) until the jobs have been removed * Remove dep from examples * fix admin integration tests * revert stray fix (will be added to a separate PR) * fix int tests * don't double format the code * fix up docs * Regenerate PubSub: documentation updates (#4293) * Regenerate monitoring client (#4316) * Regenerate bigtable client (#4318) * 3822: Fixed setDisableGZipContent call (#4314) * 3822: Fixed setDisableGZipContent call * 3822: Changes after review * Regenerate speech client: enable multi-channel features (#4317) * Release v0.77.0 (#4324) * Created enum Region.java to retrieve Regions without doing request. (#4309) * Bump next snapshot (#4325) * Bigtable: fix handling of unset rpc timeouts (#4323) When the rpc timeout is zero, then it should be treated as disabled not actual 0 * Bigtable: Expose single row read settings (#4321) * exposing ReadRowSettings thru BigtableDataSettings * fixed typo error * Regenerate compute client (#4327) * Regenerate speech client (#4328) * Update README version to use bom version (#4322) * BigQuery: Fix NPE for null table schema fields (#4338) * Add failing test for empty table schema * Fix NPE if table schema returns null for fields * Bump gax, grpc & opencensus version (#4336) * Cloud Bigtable: HelloWorld sample updates (#4339) * comments added in HelloWorld and ITHelloWorld * removed typsafe name * separate properties for bigtable.project and bigtable.instance * separate properties for bigtable.project and bigtable.instance (#4346) * Pub/Sub: default values in batch settings comments (#4350) * Pub/Sub: default values in batch settings comments Ref: https://github.com/googleapis/gax-java/blob/master/gax/src/main/java/com/google/api/gax/batching/BatchingSettings.java * Verbose comment * Update maven-surefire-plugin to 3.0.0-M3 to fix Java 8 classloader (#4344) * Update maven-surefire-plugin to 3.0.0-M3 to fix Java 8 classloader * Update failsafe plugin to 3.0.0-M3 too * Specify maven-surefire-plugin version in bigtable-emulator * Bigtable: Fixing a typo for KeyOffSet#geyKey to getKey (#4342) * Regenerate spanner client (#4332) * Spanner: remove junit from compile dependencies (#4334) * Firestore: change timestampsInSnapshots default to true. (#4353) BREAKING CHANGE: The `areTimestampsInSnapshotsEnabled()` setting is now enabled by default so timestamp fields read from a `DocumentSnapshot` will be returned as `Timestamp` objects instead of `Date`. Any code expecting to receive a `Date` object must be updated. * Regenerate clients with updated copyright year (#4382) * Regenerate asset client * Regenerate automl client * Regenerate bigquerydatatransfer client * Regenerate bigquerystorage client * Regenerate bigtable client * Regenerate container client * Regenerate containeranalysis client * Regenerate dataproc client * Regenerate dialogflow client * Regenerate dlp client * Regenerate errorreporting client * Regenerate iamcredentials client * Regenerate iot client * Regenerate kms client * Regenerate language client * Regenerate logging client * Regenerate monitoring client * Regenerate os-login client * Regenerate pubsub client * Regenerate redis client * Regenerate securitycenter client * Regenerate speech client * Regenerate tasks client * Regenerate trace client * Regenerate video-intelligence client * Regenerate websecurityscanner client * Release google-cloud-java v0.78.0 (#4386) * Regenerate compute client (#4359) * Cloud Bigtable: tableAdmin sample (#4313) * tableAdmin sample and tests * comments added * files renamed and other edits * some changes in TableAdminExample and tests * fixed timestamp to base 16 * separate properties for bigtable.project and bigtable.instance * Regenerate scheduler client (#4294) * Regenerate spanner client (#4388) * Bump next snapshot (#4391) * Regenerate asset client (#4395) * Regenerate scheduler client (#4396) * Firestore: Include a trailing /documents on root resource paths (#4352) This is required for v1 and accepted in v1beta1. Port of googleapis/nodejs-firestore@52c7381 * Release v0.79.0 (#4402) * Removing some unused dependencies (#4385) * Removing some unused dependencies Also, reducing scope of auto-value to provided. * Restoring Firestore auto-value * Removing more instances of easymock. * Removing non-deprecated uses of joda time. (#4351) * Removing non-deprecated uses of joda time. This works towards #3482 * Update pom.xml * Ran `mvn com.coveo:fmt-maven-plugin:format` * Fixing a bad merge * Bump next snapshot (#4405) * fix shading (#4406) * Fixing some deprecation warnings (#4390) * fixing some deprecation warnings * updated comment * BigQuery : Fix Location configurable at BigQueryOptions (#4329) * Fix Location configurable from BigQueryOptions * modified code * modified code and add test case * removed unused location * Generate Firestore API v1 (#4410) * Bigtable: make row & cell ordering more consistent. (#4421) * Bigtable: make row & cell ordering more consistent. * RowCells should always be ordered in lexicographically by family, then qualifier and finally by reverse chronological order * Although rows will always be ordered lexicographically by row key, they should not implement Comparable to avoid confusion when compareTo() == 0 and equals() is false. Instead that ordering was moved to a separate comparator. * Add helpers to filter cells by family & qualifier * tweaks * code style * code style

pongad self-assigned this Jan 22, 2018

pongad added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: logging Issues related to the Cloud Logging API. labels Jan 22, 2018

dansomething mentioned this issue Jan 24, 2018

Can Logback be used for logging instead of JUL? GoogleCloudPlatform/jetty-runtime#217

Closed

pongad mentioned this issue Feb 1, 2018

Fix LoggingHandler flush #1795

Closed

pongad added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Feb 6, 2018

pongad mentioned this issue Feb 15, 2018

RFC: logging: experimental batching writer #2910

Closed

pongad removed their assignment Jul 16, 2018

JustinBeckwith added the 🚨 This issue needs some love. label Jul 19, 2018

pongad added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Sep 20, 2018

sduskis removed the 🚨 This issue needs some love. label Oct 9, 2018

sduskis removed the priority: p2 Moderately-important priority. Fix may not be included in next release. label Oct 9, 2018

meltsufin mentioned this issue Oct 29, 2018

Trace with Logging doesn't work on App Engine Flex spring-attic/spring-cloud-gcp#1118

Closed

volodymyrd mentioned this issue Dec 18, 2018

Threads leak with Logback Appender for Google Cloud Logging #3880

Closed

ajaaym added a commit to ajaaym/google-cloud-java that referenced this issue Dec 26, 2018

fix googleapis#3880 googleapis#2796 disable automatic flush on write …

2fc12c3

…to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time.

ajaaym mentioned this issue Dec 26, 2018

Logging: disable severity based flush on write by default #4254

Merged

ajaaym added a commit to ajaaym/google-cloud-java that referenced this issue Dec 28, 2018

fix googleapis#3880 googleapis#2796 disable automatic flush on write …

41d03ca

…to avoid hanging and leaking threads as grpc also uses this appender which goes in to deadlock over time.

ajaaym closed this as completed Jan 2, 2019

praveen4463 mentioned this issue Oct 4, 2019

LogbackAppender logging log doesnt show up in stackdriver #6437

Closed

JustinBeckwith assigned ajaaym Feb 1, 2021

montss mentioned this issue Oct 16, 2022

Flushing logs slows down the application intensively (Error severity logs are flushed by default) we need a way to disable flushing googleapis/java-logging#1143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LoggingHandler hangs the thread #2796

LoggingHandler hangs the thread #2796

sparhomenko commented Jan 20, 2018

pongad commented Jan 22, 2018

sparhomenko commented Jan 22, 2018

pongad commented Jan 25, 2018

ghost commented Feb 11, 2018

pongad commented Feb 12, 2018

sparhomenko commented Feb 12, 2018

dansomething commented Feb 12, 2018

ghost commented Feb 13, 2018 •

edited by ghost

Loading

pongad commented Feb 13, 2018 •

edited

Loading

ghost commented Feb 14, 2018 •

edited by ghost

Loading

pongad commented Feb 14, 2018

ghost commented Feb 14, 2018

ghost commented Feb 14, 2018 •

edited by ghost

Loading

ghost commented Feb 14, 2018

ghost commented Feb 15, 2018

csainty commented Aug 23, 2018

pongad commented Sep 20, 2018

michael-barker commented Oct 5, 2018 •

edited

Loading

sergei-ivanov commented Oct 19, 2018

dave-r12 commented Dec 26, 2018

ajaaym commented Jan 2, 2019

LoggingHandler hangs the thread #2796

LoggingHandler hangs the thread #2796

Comments

sparhomenko commented Jan 20, 2018

pongad commented Jan 22, 2018

sparhomenko commented Jan 22, 2018

pongad commented Jan 25, 2018

ghost commented Feb 11, 2018

pongad commented Feb 12, 2018

sparhomenko commented Feb 12, 2018

dansomething commented Feb 12, 2018

ghost commented Feb 13, 2018 • edited by ghost Loading

pongad commented Feb 13, 2018 • edited Loading

ghost commented Feb 14, 2018 • edited by ghost Loading

pongad commented Feb 14, 2018

ghost commented Feb 14, 2018

ghost commented Feb 14, 2018 • edited by ghost Loading

ghost commented Feb 14, 2018

ghost commented Feb 15, 2018

csainty commented Aug 23, 2018

pongad commented Sep 20, 2018

michael-barker commented Oct 5, 2018 • edited Loading

sergei-ivanov commented Oct 19, 2018

dave-r12 commented Dec 26, 2018

ajaaym commented Jan 2, 2019

ghost commented Feb 13, 2018 •

edited by ghost

Loading

pongad commented Feb 13, 2018 •

edited

Loading

ghost commented Feb 14, 2018 •

edited by ghost

Loading

ghost commented Feb 14, 2018 •

edited by ghost

Loading

michael-barker commented Oct 5, 2018 •

edited

Loading