Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java kubernetes client error #8584

Closed
zenbones opened this issue Jun 27, 2020 · 8 comments
Closed

java kubernetes client error #8584

zenbones opened this issue Jun 27, 2020 · 8 comments

Comments

@zenbones
Copy link

I have a pretty basic minikube setup running in hyperv. Just trying to make sure the basics work, and this is copied from the examples. This code running inside the image of a deployment...

ApiClient client = ClientBuilder.cluster().build();

Configuration.setDefaultApiClient(client);

OkHttpClient httpClient = client.getHttpClient().newBuilder().readTimeout(0, TimeUnit.SECONDS).writeTimeout(0, TimeUnit.SECONDS).build();
client.setHttpClient(httpClient);

CoreV1Api api = new CoreV1Api();

V1Pod pod =
  new V1PodBuilder()
    .withNewMetadata()
    .withName("apod")
    .endMetadata()
    .withNewSpec()
    .addNewContainer()
    .withName("www")
    .withImage("nginx")
    .endContainer()
    .endSpec()
    .build();

api.createNamespacedPod("default", pod, null, null, null);

...at the point of api.createNamespacedPod(), leads to...

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.smallmind.spark.singularity.boot.SingularityEntryPoint.main(SingularityEntryPoint.java:65)
Caused by: io.kubernetes.client.openapi.ApiException: java.net.SocketException: Broken pipe (Write failed)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:898)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerController.main(WorkerController.java:58)
... 5 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at okio.Okio$1.write(Okio.java:79)
at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:185)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:499)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:489)
at okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:315)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:304)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall.execute(RealCall.java:81)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:894)
... 8 more

What's the best way to determine why I'm getting that error? Anything obvious I'm missing?

@RA489
Copy link

RA489 commented Jul 2, 2020

/triage support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Jul 2, 2020
@franck102
Copy link

I am seeing the exact same problem. It affects K8s operators that we have had running for over a year, and they all started failing recently withe the same error, for all operations (list, watch...).

Here is a sample exception:
We have several homemade operators that have happily been using the K8s java client to talk to the api server on minikube, and recently they all started to fail.
This affects any operations (list, watch...) and it seems to be related to SSL. The exception is always the same:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Namespace]  with name: [null]  in namespace: [dmp-system]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:155)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:608)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
	at com.fico.dmp.planning.client.KubernetesClientFactory.getKubernetesClient(KubernetesClientFactory.java:37)
	at com.fico.dmp.planning.KubernetesPlanRunner.exec(KubernetesPlanRunner.java:31)
	at com.fico.dmp.planning.openshift.ManageComponentInstanceInOpenshiftOperator.reconcile(ManageComponentInstanceInOpenshiftOperator.java:172)
	at com.fico.dmp.container.plan.PlansController.doStart(PlansController.java:62)
	at com.fico.dmp.container.plan.PlansController.startPlan(PlansController.java:48)
	at com.fico.dmp.container.services.ComponentController.update(ComponentController.java:315)
	... 68 common frames omitted
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at okio.Okio$1.write(Okio.java:79)
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
	at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:203)
	at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:514)
	at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:504)
	at okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:299)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:288)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at okhttp3.RealCall.execute(RealCall.java:93)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:412)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:151)
	... 76 common frames omitted

** minikube info

$ minikube version
minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

 $ minikube config view
- WantReportError: true
- coredns: false
- cpus: 6
- dashboard: true
- heapster: true
- memory: 16384
- metrics-server: true
- disk-size: 30G
- ingress: true
- kube-dns: true
- vm-driver: virtualbox

** Kubernetes client:

      <dependency>
        <groupId>io.kubernetes</groupId>
        <artifactId>client-java-extended</artifactId>
        <version>8.0.2</version>
      </dependency>

@franck102
Copy link

franck102 commented Jul 9, 2020

The issue is also discussed here:
kubernetes-client/java#1003

@franck102
Copy link

With SSL debugging I get the stack trace below. Some highlights, can someone confirm whether a client certificate is required to connect from a pod inside the cluster to the API server?

%% Initialized:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
...
upcoming handshake states: client certificate[11](optional)
...
Warning: no suitable certificate found - continuing without client authentication
pool-2-thread-1, WRITE: TLSv1.2 Handshake, length = 77
check handshake state: finished[20]
update handshake state: finished[20]
*** Finished
pool-2-thread-1, setSoTimeout(0) called
pool-2-thread-1, WRITE: TLSv1.2 Application Data, length = 64
pool-2-thread-1, WRITE: TLSv1.2 Application Data, length = 64
pool-2-thread-1, handling exception: java.net.SocketException: Broken pipe (Write failed)
%% Invalidated:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
pool-2-thread-1, SEND TLSv1.2 ALERT:  fatal, description = unexpected_message

Full stack trace:

%% Initialized:  [Session-1, SSL_NULL_WITH_NULL_NULL]
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
%% No cached client session
update handshake state: client_hello[1]
upcoming handshake states: server_hello[2]
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1594326316 bytes = { 105, 64, 29, 26, 148, 221, 156, 161, 36, 133, 157, 67, 29, 236, 211, 174, 176, 244, 116, 26, 18, 210, 20, 249, 170, 11, 209, 119 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, SSL_RSA_WITH_3DES_EDE_CBC_SHA]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
Extension extended_master_secret
Extension application_layer_protocol_negotiation, protocol names: [h2][http/1.1]
Extension renegotiation_info, renegotiated_connection: <empty>
***
pool-2-thread-1, WRITE: TLSv1.2 Handshake, length = 144
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 58
check handshake state: server_hello[2]
*** ServerHello, TLSv1.2
RandomCookie:  GMT: 946265005 bytes = { 57, 119, 91, 192, 35, 210, 148, 98, 109, 234, 173, 121, 121, 3, 210, 101, 198, 79, 19, 100, 68, 79, 87, 78, 71, 82, 68, 1 }
Session ID:  {}
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
Extension application_layer_protocol_negotiation, protocol names: [h2]
***
%% Initialized:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
** TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
update handshake state: server_hello[2]
upcoming handshake states: server certificate[11]
upcoming handshake states: server_key_exchange[12](optional)
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 967
check handshake state: certificate[11]
update handshake state: certificate[11]
upcoming handshake states: server_key_exchange[12](optional)
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: CN=minikube, O=system:masters
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 26146449242114331395340209000766494637241403891413319963805392667549868704233365564641954907021426571991142620933182367152709971403600217712523152164815925606214221792265885189021221099205504585889594996682245459830592704862417760951670124987677903359542388154476371935097265336136751048026399554435358530316854111257152880593339985639285317522335672124388309347626256600118424890863267692190434143693571626859302637746785764759497727211362720440133446366173719286574374701944205811070260877140341414896652392511079930778693735810589186938465022237503808363005779759265717521153755692048544914444458710006073037037639
  public exponent: 65537
  Validity: [From: Wed Jul 08 08:41:47 UTC 2020,
               To: Fri Jul 09 08:41:47 UTC 2021]
  Issuer: CN=minikubeCA
  SerialNumber: [    02]

Certificate Extensions: 4
[1]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[2]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

[3]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

[4]: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: minikubeCA
  DNSName: control-plane.minikube.internal
  DNSName: kubernetes.default.svc.cluster.local
  DNSName: kubernetes.default.svc
  DNSName: kubernetes.default
  DNSName: kubernetes
  DNSName: localhost
  IPAddress: 192.168.99.100
  IPAddress: 10.96.0.1
  IPAddress: 127.0.0.1
  IPAddress: 10.0.0.1
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: 05 77 12 6E B3 30 3D 57   59 C4 9A 3C 4B E6 6A 47  .w.n.0=WY..<K.jG
0010: AD 07 86 0D 62 77 97 17   89 5B C4 DF 5A 15 65 38  ....bw...[..Z.e8
0020: F8 18 51 77 F2 1B E2 9F   3D 34 6D 1E 9C 68 BA 4C  ..Qw....=4m..h.L
0030: 81 35 B6 D3 3D 75 94 0F   A2 5D 66 48 3A 3A B1 4B  .5..=u...]fH::.K
0040: C1 19 A6 DB CA AE 58 A6   2B 92 E2 B9 BB CA C9 84  ......X.+.......
0050: A8 E5 68 A7 5C 7D EE 1A   16 09 3F 4C E8 C5 8A A9  ..h.\.....?L....
0060: 59 B2 6E 1A 36 5C 43 F8   FD 3D BA 72 B4 D2 48 25  Y.n.6\C..=.r..H%
0070: 92 88 15 1F 2C CE 2F B2   8E 59 8B 0A F2 24 BD 81  ....,./..Y...$..
0080: 0C DF EF 61 F5 D7 CA A0   35 E0 AE E2 CC 2D B1 DA  ...a....5....-..
0090: 37 F1 A1 2D 88 B0 BC B6   7B 0E 1D F4 61 23 1E A7  7..-........a#..
00A0: 87 66 54 61 BF 0E F5 D0   D2 16 80 70 C3 FB 0F 46  .fTa.......p...F
00B0: 63 B5 C5 8E 1A FE 5F D2   FF 8C BE 7E CD 3D 4E 6E  c....._......=Nn
00C0: 8E 1F 2E F6 B9 00 81 79   0B 44 A6 89 B9 95 D5 A6  .......y.D......
00D0: 43 F4 BF 9E 40 7D 8A F5   43 0D 4A 4F DD F0 21 BD  C...@...C.JO..!.
00E0: 76 FA D9 2A 35 99 E0 93   06 85 B6 4A 92 1E BE 38  v..*5......J...8
00F0: 90 4F 50 99 A7 4D EB 5C   77 6F 09 D1 97 E0 16 89  .OP..M.\wo......

]
***
Found trusted certificate:
[
[
  Version: V3
  Subject: CN=minikubeCA
  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits
  params: null
  modulus: 31293793297420462513971980375749477673823969029720400756809351475051341952614102627770458975690930290146126398807112313496265726423583228533931268586878270978465541219881065428228958881271087765417532271157341190475387336788101695229730789913571654773560846965616319774823635451095831522921911130273150613480144775193748500026074846104355996481328333136561237534857006129695230596318279324220637374344983958712384574606773292812272412784898978342387810869403292715240858796686310128148562467528117468778821441988972038250942196636424115565634714782004732740149059735939151847781281422573916327689175141976610680918437
  public exponent: 65537
  Validity: [From: Thu Nov 22 13:33:49 UTC 2018,
               To: Mon Nov 20 13:33:49 UTC 2028]
  Issuer: CN=minikubeCA
  SerialNumber: [    01]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

[2]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]

[3]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
  Key_CertSign
]

]
  Algorithm: [SHA256withRSA]
  Signature:
0000: D5 42 45 DF FF AD F0 EE   27 CE 50 5A 60 92 E7 E5  .BE.....'.PZ`...
0010: EE 69 7A 00 A0 EE 9B 76   2A 45 00 7E 91 34 B0 5B  .iz....v*E...4.[
0020: 73 78 BD DC E6 0B 5D 34   3F 75 34 E3 AC FC AB 37  sx....]4?u4....7
0030: E4 46 8F 9F 60 7C 7D 42   2D 05 58 73 3F D8 C1 F9  .F..`..B-.Xs?...
0040: 2B 61 56 BE 20 E6 5F 2E   8C AC E9 2C BF 39 9F 2F  +aV. ._....,.9./
0050: 6A 5C DB D8 22 DD 40 EF   58 AD 9F 42 EB 8D D0 0C  j\..".@.X..B....
0060: BD BB 9C 74 0E B1 1D E4   5F 2D 1A 22 CF 0C F7 E4  ...t...._-."....
0070: 3C F8 AD D4 9B C1 B9 D2   71 7E A1 51 C5 EE 22 93  <.......q..Q..".
0080: 79 09 12 3D C0 98 B2 4D   24 DC 4C A7 AD F8 4B 34  y..=...M$.L...K4
0090: 66 97 DE 03 29 9D 23 3B   32 45 84 BA 28 E0 39 9F  f...).#;2E..(.9.
00A0: 9F DA 97 06 90 7F 7C 3C   D9 54 6A DC 6E 02 95 01  .......<.Tj.n...
00B0: CB D7 D1 02 13 30 F4 DD   0C 7C 23 73 83 CC 67 17  .....0....#s..g.
00C0: 5E 84 93 DA 0A F4 D7 70   6C F6 A2 44 A7 54 2C ED  ^......pl..D.T,.
00D0: C5 70 21 AA 4E 4D 40 18   25 2E 0C C1 86 99 8F 5A  .p!.NM@.%......Z
00E0: 5B F3 C6 DD 70 E4 0F 54   21 B9 D3 7A 53 24 92 11  [...p..T!..zS$..
00F0: 1F 61 5B E0 B1 62 D7 23   70 43 CA E9 DF 97 71 C6  .a[..b.#pC....q.

]
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 333
check handshake state: server_key_exchange[12]
update handshake state: server_key_exchange[12]
upcoming handshake states: certificate_request[13](optional)
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** ECDH ServerKeyExchange
Signature Algorithm SHA512withRSA
Server key: Sun EC public key, 256 bits
  public x coord: 58449391755953405122348774537058054757220068840967545951716180272862379002301
  public y coord: 114986651139100385519354884145129784763025031585640739967843909174229168067018
  parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 83
check handshake state: unknown[13]
*** CertificateRequest
Cert Types: RSA, ECDSA
Supported Signature Algorithms: SHA256withRSA, SHA256withECDSA, Unknown (hash:0x8, signature:0x7), SHA384withRSA, SHA512withRSA, SHA384withECDSA, SHA512withECDSA, SHA1withRSA, SHA1withECDSA
Cert Authorities:
<CN=minikubeCA>
<CN=front-proxy-ca>
update handshake state: unknown[13]
upcoming handshake states: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 4
check handshake state: server_hello_done[14]
update handshake state: server_hello_done[14]
upcoming handshake states: client certificate[11](optional)
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** ServerHelloDone
Warning: no suitable certificate found - continuing without client authentication
*** Certificate chain
<Empty>
***
update handshake state: certificate[11]
upcoming handshake states: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
*** ECDHClientKeyExchange
ECDH Public value:  { 4, 163, 120, 247, 36, 224, 29, 37, 28, 193, 231, 253, 65, 255, 165, 255, 153, 182, 71, 151, 108, 5, 224, 18, 29, 182, 47, 243, 255, 165, 43, 159, 231, 53, 143, 93, 2, 6, 80, 147, 153, 160, 87, 6, 141, 108, 55, 59, 222, 205, 234, 172, 50, 160, 237, 32, 236, 198, 73, 68, 236, 39, 87, 239, 202 }
update handshake state: client_key_exchange[16]
upcoming handshake states: certificate_verify[15](optional)
upcoming handshake states: client change_cipher_spec[-1]
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
pool-2-thread-1, WRITE: TLSv1.2 Handshake, length = 77
SESSION KEYGEN:
PreMaster Secret:
0000: 25 7A 59 C0 F3 D9 AC DA   AE 04 F7 28 42 55 8A 2B  %zY........(BU.+
0010: 6F 78 EE 4C 89 73 D6 D5   45 EA E7 F4 83 5E A3 A9  ox.L.s..E....^..
CONNECTION KEYGEN:
Client Nonce:
0000: 5F 07 7D 2C 69 40 1D 1A   94 DD 9C A1 24 85 9D 43  _..,i@......$..C
0010: 1D EC D3 AE B0 F4 74 1A   12 D2 14 F9 AA 0B D1 77  ......t........w
Server Nonce:
0000: 38 67 DC AD 39 77 5B C0   23 D2 94 62 6D EA AD 79  8g..9w[.#..bm..y
0010: 79 03 D2 65 C6 4F 13 64   44 4F 57 4E 47 52 44 01  y..e.O.dDOWNGRD.
Master Secret:
0000: C3 54 25 ED 4A 4E A8 A4   CF 6B F4 DA F5 39 C8 BA  .T%.JN...k...9..
0010: 4A 7D 9B EF DC 30 FD 4C   19 87 D9 03 D3 57 75 7A  J....0.L.....Wuz
0020: 78 98 85 3A 6D 9A 4A ED   0D C5 63 53 D0 2D C7 A6  x..:m.J...cS.-..
Client MAC write Secret:
0000: 06 67 23 B9 BD 9B D5 CE   8C 52 B6 44 95 82 43 4F  .g#......R.D..CO
0010: D7 D8 90 5F                                        ..._
Server MAC write Secret:
0000: 48 52 2D 4C 74 6A 1B 88   EB 00 C7 9D 8E B7 2A CD  HR-Ltj........*.
0010: 4D 20 86 31                                        M .1
Client write key:
0000: 23 52 B5 86 9E 5A 8D 8C   E9 FE 8D 1D AF 49 9F 05  #R...Z.......I..
0010: 0E CE 20 E6 7F 92 17 E9   CF C6 C1 DD 90 FB 56 81  .. ...........V.
Server write key:
0000: 70 A2 C7 A4 B6 4B 72 85   07 FB 9C 6C 4A 86 17 D1  p....Kr....lJ...
0010: F1 E9 78 E4 43 B8 D1 97   45 06 C6 C3 43 AE 0A F6  ..x.C...E...C...
... no IV derived for this protocol
update handshake state: change_cipher_spec
upcoming handshake states: client finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
pool-2-thread-1, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data:  { 23, 237, 40, 123, 54, 247, 160, 213, 206, 174, 174, 46 }
***
update handshake state: finished[20]
upcoming handshake states: server change_cipher_spec[-1]
upcoming handshake states: server finished[20]
pool-2-thread-1, WRITE: TLSv1.2 Handshake, length = 64
pool-2-thread-1, READ: TLSv1.2 Change Cipher Spec, length = 1
update handshake state: change_cipher_spec
upcoming handshake states: server finished[20]
pool-2-thread-1, READ: TLSv1.2 Handshake, length = 64
check handshake state: finished[20]
update handshake state: finished[20]
*** Finished
verify_data:  { 132, 7, 141, 122, 118, 156, 241, 14, 52, 91, 105, 0 }
***
%% Didn't cache non-resumable client session: [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
pool-2-thread-1, setSoTimeout(0) called
pool-2-thread-1, WRITE: TLSv1.2 Application Data, length = 64
pool-2-thread-1, WRITE: TLSv1.2 Application Data, length = 64
pool-2-thread-1, handling exception: java.net.SocketException: Broken pipe (Write failed)
%% Invalidated:  [Session-2, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
pool-2-thread-1, SEND TLSv1.2 ALERT:  fatal, description = unexpected_message
pool-2-thread-1, WRITE: TLSv1.2 Alert, length = 48
pool-2-thread-1, Exception sending alert: java.net.SocketException: Broken pipe (Write failed)
pool-2-thread-1, called closeSocket()
pool-2-thread-1, called close()
pool-2-thread-1, called closeInternal(true)
METER 20:25:19.428 [pool-2-thread-1] [] WARN  c.f.d.m.operator.MeteringOperator - Watch request failed
io.kubernetes.client.openapi.ApiException: java.net.SocketException: Broken pipe (Write failed)
	at io.kubernetes.client.util.Watch.createWatch(Watch.java:109)
	at com.fico.dmp.metering.operator.rules.RulesManager.watch(RulesManager.java:92)
	at com.fico.dmp.metering.operator.rules.RulesManager.run(RulesManager.java:73)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at okio.Okio$1.write(Okio.java:79)
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
	at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:185)

@priyawadhwa
Copy link

Hey @zenbones thanks for opening this issue. Honestly, I'm not super familiar with this so I'm not sure exactly what the issue is. Some suggestions I can think of:

  1. Since this recently started failing, I'd check that the kubernetes version minikube is the same as the cluster where this worked
  2. Increase the CPU/memory allocated to minikube via the --memory and --cpus flags to minikube start, maybe we're trying to create a pod but the resources aren't there

@franck102 you said:

I am seeing the exact same problem. It affects K8s operators that we have had running for over a year, and they all started failing recently withe the same error, for all operations (list, watch...).

Have you been running these operators in minikube? and did they only start failing in minikube?

@fabiokung
Copy link

@medyagh
Copy link
Member

medyagh commented Sep 16, 2020

@zenbones
@fabiokung
I see this
Prohibited TLS 1.2 Cipher Suite: c014

I wonder if the java client for kubernetes is using the different Cert format.

@medyagh medyagh changed the title Minikube kubernetes service closes pipe on java kuberntes client java client for kuberentes error Sep 16, 2020
@medyagh medyagh added triage/long-term-support and removed kind/support Categorizes issue or PR as a support question. labels Sep 16, 2020
@medyagh medyagh changed the title java client for kuberentes error java kubernetes client error Sep 16, 2020
@medyagh
Copy link
Member

medyagh commented Sep 16, 2020

I recommend following this thread for better support #8584

@medyagh medyagh closed this as completed Sep 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants