DEADLINE_EXCEEDED when using Cloud Pub/Sub library from Compute Engine

[codemoped]

When using the Cloud Pub/Sub library in a Java application that has been deployed in a Google Compute Engine instance, a DEADLINE_EXCEEDED exception occurs. The application resides in a Docker container within a Container Engine-managed Compute Engine instance. The Pub/Sub topic and the Compute Engine instance are in the same project.

The Compute Engine service account has project editor permissions.

No problems with the old google-api-services-pubsub library.

The code to create the instance looks like this:

InputStream credentialsStream = IOUtils.toInputStream(SERVICE_ACCOUNT_JSON, StandardCharsets.UTF_8);
Credentials credentials = ServiceAccountCredentials.fromStream(credentialsStream);
PubSubOptions options = PubSubOptions.newBuilder().setCredentials(credentials).build();
PubSub pubsub = options.getService();

Then doing something like this (or alternatively, create a topic):

for (Subscription subscription : pubsub.listSubscriptions().getValues()) {
...
}

results in the following:

com.google.cloud.pubsub.PubSubException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED

Versions:

google-cloud-pubsub - v0.6.0
google-auth-library-oauth2-http - v0.6.0
google-auth-library-credentials - v0.6.0
google-* - v1.22.0

I also found this issue at Stackoverflow

[colintemple]

+1. I was the one who asked the question on Stack Overflow.

In my case, the following simple code caused DEADLINE_EXCEEDED errors when run on a Compute Engine instance within the same project as the Pub/Sub topic I wanted to publish to:

PubSub pubSub = PubSubOptions.getDefaultInstance().toBuilder()
        .build().getService();

String messageId = pubSub.publish(topic, message);

I could communicate with Pub/Sub using google-api-services-pubsub library, or via gcloud from the command line on the instance.

[garrettjonesgoogle]

I intend on investigating but it will take a couple days due to other commitments.

[timkettering]

+1. Encountering this issue with any type of call made w/ the PubSub client.

[jonatanpettersson]

+1. We had to revert back to the API while waiting for a fix to this.

[BodySplash]

+1 I'm trying to find why in the mean time, even if I have no clue.

[garrettjonesgoogle]

Investigating now.

[BodySplash]

So far, not good. I tought maybe I was polling pubsub a bit to aggressively, but It works like a charm on my machine. Maybe a network issue of some sort in the way k8n nodes are configured ? 

[qtxo]

I had the same DEADLINE_EXCEEDED problem (for any pubsub request) in my windows 7 laptop, in GCE the same application runs fine. I fixed my problem forcing the Netty version to the one expected by google-cloud 0.8.0 that is 4.1.3.Final.
Before the fix I was using the version 4.1.5.Final as a dependency of vertx-core-3.3.3.jar. (I hope vertx is not affected by this change, but at least I know what the problem is)

I can reproduce DEADLINE_EXCEEDED errors in google-cloud-examples forcing Netty version to 4.1.5.Final or 4.16.Final, look at the pom.xml

[BodySplash]

interesting, I have indeed a version conflict with netty. I'll try to fix my version to 4.1.3.

[BodySplash]

didn't seem to change anything on my side :(

[garrettjonesgoogle]

I was able to make a successful call to Pub/Sub from a docker image on GCE. Could someone who is still having a problem give their pom.xml (or build.gradle as the case may be)?

[BodySplash]

Here you are. I tried de strip this files to the bare minimum.

• build.gradle for the executable jar
• build.gradle for my common jar containing pub-sub

[garrettjonesgoogle]

@BodySplash , starting with your build.gradle for your common jar, I added a simple file to call pubsub plus a dependency on com.github.jengelman.gradle.plugins:shadow:1.2.4 to create a fat jar, and it worked in a docker image on GCE. Could you provide a minimal Dockerfile too? Maybe something is different there.

[BodySplash]

It is very simple for now, maybe the issue is in the way I use the new lib. Or maybe, since I cleaned up a little bit my build.gradles before posting them, it solved the issue.
`
FROM java:8-jre-alpine
maintainer Jean-Baptiste Dusseaut <>

ENV destination /app
COPY build/libs/gsoi-ufo-cards.jar ${destination}/app.jar
WORKDIR ${destination}
CMD ["java", "-jar", "app.jar"]
`

[timkettering]

@garrettjonesgoogle - i'm still encountering the DEADLINE blocker. could you share your test project so i can run it locally and see if its something w/ my setup, or a system-wide issue?

[garrettjonesgoogle]

Here is my test project:

https://github.com/garrettjonesgoogle/gcloud-java/tree/deadline-exceeded-issue/google-cloud-example-docker-gradle

Notes:

• You need to figure out auth yourself.
• You can build and run it locally this way:

./gradlew jar shadowJar
java -cp build/libs/google-cloud-example-docker-gradle-all.jar com.google.cloud.pubsub.spi.v1.PublisherSmokeTest --project_id YOUR_PROJECT_HERE

[BodySplash]

I looked at your sample project quickly. Is setting project-id mandatory or is it just for local testing ?

Anyway, I'll try to give it a try as soon as possible, thank you for sharing it.

[garrettjonesgoogle]

PubSub needs to get a project somehow - either explicitly or implicitly (I think there are environmental sources for project). My sample code does it explicitly.

[BodySplash]

Ok I was finally abled to make my service work. Here is the catch : I was using java:8-jre-alpine as my base image (I like my images to be small). Looking at your own dockerfile, I saw you were installing manually a java 8 jdk on an ubuntu, that was really the last significant difference with my own setup.

I can't really say where the problem come from though. Is it using a jre or using alpine ? And in any case why ? Alpine and K8S had their issues in the past.
I'll try to base my Docker image on java:8-jdk to see if it still works.

It would be a pitty not to be able to use a jre alpine as base image.

[jonatanpettersson]

We're also using jre-alpine, so guess there's a common denominator there.

[BodySplash]

I played a little bit with different base images, it's really a matter of alpine vs debian/ubuntu.
My service is running fine with 8-jre, not with any version of alpine.

[garrettjonesgoogle]

I tried alpine myself and duplicated the issue. I turned on FINE logging and this showed up, which is the root cause of the DEADLINE_EXCEEDED error:

FINE: Unable to load the library: netty-tcnative-linux-x86_64.
java.lang.UnsatisfiedLinkError: /tmp/libnetty-tcnative-linux-x86_646695900156633723588.so: Error loading shared library libcrypt.so.1: No such file or directory (needed by /tmp/libnetty-tcnative-linux-x86_646695900156633723588.so)

[garrettjonesgoogle]

Actually this is the more appropriate error:

Jan 06, 2017 10:07:36 PM io.netty.handler.ssl.OpenSsl <clinit>
FINE: Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless the application has already loaded the symbols by some other means. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
...
Jan 06, 2017 10:07:36 PM io.netty.handler.ssl.OpenSsl <clinit>
FINE: Failed to initialize netty-tcnative; OpenSslEngine will be unavailable. See http://netty.io/wiki/forked-tomcat-native.html for more information.
java.lang.UnsatisfiedLinkError: org.apache.tomcat.jni.Library.version(I)I

[garrettjonesgoogle]

I did some research, and it seems that there are missing C libraries in Alpine because Alpine attempts to be very minimal, so there are numerous cases of products that won't work on Alpine. The core issue I saw when trying it myself is that netty-tcnative needs glibc, but glibc is not available out of the box in Alpine.

I think this is outside the scope of google-cloud-java, and is up to each person who wants to use Alpine to figure out how to pull in all of the right dependencies to satisfy the linker issues.

[BodySplash]

Hum interesting question. I tend to agree with you, but in the other hand alpine is becoming the de-facto standard as base image in the docker world. I guess at very least it should be clearly documented somewhere.
Thanks anyway for your support and the new lib. It has by far a much better developper experience than the previous one.

[vendettacoder]

+1

Faced the same issue running the google cloud api java client on ubuntu.
Apparently even the latest version of the google vision api : 0.9.4-beta needs netty-all 4.1.3.Final for its java client to work. So just force use this version in your existing project. Should work fine!

[ricardoneves93]

I am having this problem when running locally in my project.
I'm using google-cloud version 0.8.0.
I've tried the previous suggestions and nothing solved my problem. I'm still getting this exception:

java.util.concurrent.ExecutionException: com.google.cloud.pubsub.PubSubException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED.

Does anyone have any similar problem?

[garrettjonesgoogle]

@ricardoneves93 could you file a new issue? You might have a different root cause, and it would be easier to track it separately.

[kilaka]

Happening to me as well with google-cloud-speech v0.8.1-alpha and v0.8.0 on tomcat v8.5.8 on Ubuntu 16.04 64bit.

[vtvttech]

While using in raspberry pi 3 model B with raspbian os grpc channel created and topic is also created but when trying to publish message DEADLINE_EXCEEDED Runtime exception occurs.
It is running fine in ubuntu 16.04 but its failed every time in raspberry pi.
Also google cloud speech api having same issue its running fine in ubuntu but failing in raspberry pi.

[olivierboucher]

Is there any update on this? With the popularity of Alpine, I have trouble believing support with simply be cut out

[garrettjonesgoogle]

Note added in README about non-support of Alpine in #1430 .

@olivierboucher could you file a separate issue requesting support for Alpine? This particular issue has become a little messy.

[ayushj158]

We are facing similar issues on version ( <spring-cloud-gcp.version>1.0.0.M3</spring-cloud-gcp.version>) . Is there a fix already available for this is any of the version ?