Broken pipe on API call with Minikube

[zenbones]

I have a pretty basic minikube setup running in hyperv. Just trying to make sure the basics work, so this is copied from the examples. This code running inside the image of a deployment...

ApiClient client = ClientBuilder.cluster().build();

Configuration.setDefaultApiClient(client);

OkHttpClient httpClient = client.getHttpClient().newBuilder().readTimeout(0, TimeUnit.SECONDS).writeTimeout(0, TimeUnit.SECONDS).build();
client.setHttpClient(httpClient);

CoreV1Api api = new CoreV1Api();

V1Pod pod =
  new V1PodBuilder()
    .withNewMetadata()
    .withName("apod")
    .endMetadata()
    .withNewSpec()
    .addNewContainer()
    .withName("www")
    .withImage("nginx")
    .endContainer()
    .endSpec()
    .build();

api.createNamespacedPod("default", pod, null, null, null);

...at the point of api.createNamespacedPod(), leads to...

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.smallmind.spark.singularity.boot.SingularityEntryPoint.main(SingularityEntryPoint.java:65)
Caused by: io.kubernetes.client.openapi.ApiException: java.net.SocketException: Broken pipe (Write failed)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:898)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerController.main(WorkerController.java:58)
... 5 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at okio.Okio$1.write(Okio.java:79)
at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:185)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:499)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:489)
at okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:315)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:304)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall.execute(RealCall.java:81)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:894)
... 8 more

What's the best way to determine why I'm getting that error? Anything obvious I'm missing?

[alek-sys]

The same problem exists with kind cluster (kind v0.8.1 go1.14.2 darwin/amd64), however it works on GKE.

[zenbones]

If it helps, my problem is a minikube env on hyperv , win 10/amd64, docker 19.0.3.8 using ubuntu 18.0.4 containers

[brendandburns]

I'm not sure how minikube sets up it's networking. I suspect that the kubernetes service isn't setup properly, but I will try to reproduce this.

[habuma]

I've recreated this on Kind (v0.8.1 go1.14.2 darwin/amd64, running on Docker version 19.03.8, build afacb8b), doing nothing more than making a call to api.getAPIResources() (which, if I'm not mistaken, makes a GET request to /api/v1). Same exception and stack trace.

So, in the same pod, I tried to bypass the Kubernetes client by using Spring's RestTemplate to make a GET request to /api/v1. I had to set it up to ignore SSL problems and to fetch the token from /var/run/secrets/kubernetes.io/serviceaccount/token and put it into the Authorization header. But with that setup, the request came back successful.

But knowing that the Kubernetes client uses OkHttp and not RestTemplate, I tried making the request using the OkHttp client, setting up the SSL context and token the same as I had for RestTemplate (and having to truncate the trailing new line from the token file), and it failed with the same stack trace as when going through the Kubernetes client.

A little deeper digging makes it seem that the exception is thrown while dealing with the SSL handshake, as per the following stack trace. It's deep in some low-level socket code where it happens, but if I were guessing, I'd say that OkHttp is fine and Kind/Minikube are both fine. But the combination isn't so great. I guess that at some point OkHttp sends or does something that the K8s API on Kind/Minikube handle by closing the connection...but OkHttp expecting to continue using that connection throws the Broken Pipe exception because the connection has been closed by the K8s API on those clusters. My guess continues to suppose that whatever OkHttp does to cause the K8s API to close, doesn't result in a closed connection on GKE (or other K8s platforms)...and whatever it is that OkHttp does isn't done by RestTemplate so that even on Kind/Minikube, the connection stays open. Again...all guesses in this paragraph.

[habuma]

I've found a way to make this work!

In short, I was reading that there are some TLS 1.3 changes that might impact this and that aren't available until more recent versions of Java. So...long story short...I realized that I was still targeting Java 1.8 in my project's build. I bumped it up to target Java 11 in the build (which also involved me switching to Java 11 in SDKMan...for "reasons" I still have 1.8 as the default in SDKMan), and suddenly it works! What's more, it works on both Kind and Minikube!

@zenbones Any chance you are building to target Java 1.8? If so, maybe try targeting Java 11? Would like to hear someone else try this and confirm that it works.

(Note, that I'm building using Spring Boot 2.3.1 and using its support for creating a container image. Java 11 works fine, but Java 12 and 13 not so much because at build time it can't find compatible dependencies for the Paketo BellSoft Liberica Buildpack 2.8.0 while building the image. It does seem to build fine with Java 14, though it fails at runtime with a new exception: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request.

[zenbones]

Wow, that's good detective work. Thanks. Our build does target Java 8, and we do pan to move to Java 11, but Oracle has given control of J2EE to the Eclipse Foundation, but not the javax.* namespace. So Eclipse is in the process of moving javax.* to jarakta.*, and we (and most everyone) is using pieces of the javax.* modules. We also use libraries with javax.* dependencies, so if we move it'll be wrong fro some our transient dependencies, and if we don't move we get caught by others that have shifted. This is all working itself out by the end of the year (I believe), so I was hoping to wait until then before making the jump to Java 11.

Sounds like this is not anyone's fault and there's probably little to be done and it's on us to make the Java11 move... but I'm wondering if forking the project and rebuilding it under Java 8 might also be a temporary solution... maybe going so far as to swap out the OKHttp client for something else (maybe the Apache client). I may be able to try that.

[habuma]

Swapping out OKHttp for Apache (or anything else) seems like quite a chore. The ApiClient class appears to be coupled to OkHtttp. Of course, it depends on how much of ApiClient you need. Ultimately, it just provides a convenience for making HTTP requests against the K8s API, so you could write your own convenient abstraction over the API that uses whatever HTTP client you want. I happen to know that Spring's RestTemplate with an Apache HttpClient client factory works fine in Java 8.

But yeah, I don't think this is a bug as much as a compatibility issue. As @alek-sys noted above, I also know it to work on GKE, regardless of the Java version in play (even Java 14 plays nice there). Just Kind and Minikube sorta require that your app that's using ApiClient be on Java 11.

[brendandburns]

fwiw, all of this code is generated, so swapping out the client library isn't a workable solution since we need the ability to re-generate the library.

For now, I would just use kind to create a Kubernetes 1.17 cluster instead of 1.18 (unless you really need 1.18 APIs) and I believe it should work w/ Java 8, kubernetes 1.18 made some TLS changes.

[zenbones]

Not sure what I can or can't do for now, but I can live with transient issues and find workarounds. I'm more concerned with ongoing problems. Are there any plans to make this compatible with Java 8, Minikube and kubernetes 1.18, or is the plan to let people upgrade out of Java 8 in order to find a solution?

[brendandburns]

This library is compatible w/ Java8 and Kubernetes 1.18. This library is tested and working with both JDK8 and JDK11 and both cloud (Azure, GKE) and on-premise (my home cluster) with Kubernetes 1.16, 1.17 and 1.18. So I'm confident that it works correctly.

There was a bug in JDK 11 (https://bugs.openjdk.java.net/browse/JDK-8236039) that has been fixed in recent versions that may have caused the problem cited w/ JDK 14 above (#1003 (comment)) as well as other certificate problems depending on JDK version.

I have not done recent testing w/ minikube so it's possible that there is some problem in there certificate generation. I will try to repro when I have time.

If you have problems w/ Java8 + this library and a complete install of Kubernetes (e.g. not minikube) let me know, in the meantime I think it's a problem specifically w/ minikube and we'll try to reproduce when we have cycles.

[zenbones]

And in the meantime I've sorted out a minimal Java 11 version of our code so I'll try this again and let you know.

[zenbones]

Whatever's happening is not just Java 11. I get the same broken pipe under Minikube with...

openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed mode, sharing)

I guess the next step is to try this in a non-Minikube environment. Back to the drawing board...

[zenbones]

And running in EKS on kubernetes 1.16 I get...

exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.smallmind.spark.singularity.boot.SingularityEntryPoint.main(SingularityEntryPoint.java:65)
Caused by: io.kubernetes.client.openapi.ApiException:
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerController.main(WorkerController.java:58)
... 5 more

This is copied from your example code and was about the simplest example I could find. Is this tested against anything more than GKE? I don't have an account but maybe I'll try to set one up there.

Oh, do you have that example you used to test on your home cluster? Maybe I could just drop that into a container and see if it works in Minikube or EKS?

[sarveshkaushal]

I tried to repro this on my machine (MacOS) without any success. Here are the steps I followed:

1. Installed Minikube(v1.11.0) with virtual box(6.1.10) on MacOS(10.15.5)
2. Created a Java(1.8.0_212 openjdk jdk alpine) Spring Boot(2.4.0-SNAPSHOT) application which has a controller mapped to /run URI.
3. Have the above example code (given in first comment) run when a request for /run resource is made.
4. Created a deployment to run my java spring boot application on single node K8 cluster created by Minikube.
5. Started port forwarding on port 8080. (Spring boot app runs on 8080 in my case)
6. Send a request to http://localhost:8080/run. I am able to create the pod (apod) as given in example.

[zenbones]

Odd, as the second report was for...

The same problem exists with kind cluster (kind v0.8.1 go1.14.2 darwin/amd64), however it works on GKE.

...but I'm game. I'll update minikube to the latest win10 version, and put the code into a Jersey resource, run the 1.8 JVM starting a Grizzly instance on Ubuntu, none of which should be a significant difference from your setup. Fingers crossed.