Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException #64

Open
adrianovieira opened this issue Mar 17, 2021 · 3 comments
Open

org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException #64

adrianovieira opened this issue Mar 17, 2021 · 3 comments

Comments

@adrianovieira
Copy link

adrianovieira commented Mar 17, 2021 •
edited
Loading

This error occurs after importing some data to elasticsearch

  • environment

    OS: Linux Fedora-33
elasticsearch: 7.11.2
ssl: yes; self-signed certificate
java: 
  openjdk version "1.8.0_265"
  OpenJDK Runtime Environment (Zulu 8.48.0.53-CA-linux64) (build 1.8.0_265-b11)
  OpenJDK 64-Bit Server VM (Zulu 8.48.0.53-CA-linux64) (build 25.265-b11, mixed mode)

  • Embulk and plugins

    Started Embulk v0.9.23
Loaded plugin embulk-input-postgresql (0.11.1)
Loaded plugin embulk-output-elasticsearch (0.4.7)
Loaded plugin embulk-filter-column (0.7.1)

  • logging: 

     2021-03-17 16:08:24.241 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
 2021-03-17 16:08:37.086 -0300 [INFO] (0025:task-0000): > 12,85 seconds
 2021-03-17 16:08:37.088 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
 2021-03-17 16:08:37.089 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
 2021-03-17 16:08:37.090 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
 2021-03-17 16:08:37.093 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
 2021-03-17 16:08:37.096 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
 2021-03-17 16:08:37.098 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
 2021-03-17 16:08:37.191 -0300 [INFO] (0025:task-0000): > 0,09 seconds
 2021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
 2021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
 2021-03-17 16:08:37.193 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
 2021-03-17 16:08:37.195 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
 2021-03-17 16:08:37.197 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
 2021-03-17 16:08:37.199 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
 2021-03-17 16:08:37.255 -0300 [INFO] (0025:task-0000): > 0,06 seconds
 2021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
 2021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
 2021-03-17 16:08:37.257 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
 2021-03-17 16:08:37.259 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
 2021-03-17 16:08:37.260 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
 2021-03-17 16:08:37.261 -0300 [INFO] (0025:task-0000): > 0,00 seconds
 2021-03-17 16:08:37.263 -0300 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
 org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
 	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
 	at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
 	at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
 	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
 	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
 	at org.embulk.spi.Exec.doWith(Exec.java:22)
 	at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
 	at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
 	at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
 	at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
 	at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
 	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
 	at org.embulk.cli.Main.main(Main.java:64)
 Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
 	at com.google.common.base.Throwables.propagate(Throwables.java:160)
 	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:164)
 	at org.embulk.output.elasticsearch.ElasticsearchHttpClient.sendRequest(ElasticsearchHttpClient.java:322)
 	at org.embulk.output.elasticsearch.ElasticsearchHttpClient.push(ElasticsearchHttpClient.java:84)
 	at org.embulk.output.elasticsearch.ElasticsearchRecordBuffer.bufferRecord(ElasticsearchRecordBuffer.java:68)
 	at org.embulk.base.restclient.RestClientPageOutput.add(RestClientPageOutput.java:44)
 	at org.embulk.spi.PageBuilder.doFlush(PageBuilder.java:214)
 	at org.embulk.spi.PageBuilder.flush(PageBuilder.java:219)
 	at org.embulk.spi.PageBuilder.addRecord(PageBuilder.java:198)
 	at org.embulk.filter.column.ColumnFilterPlugin$1.add(ColumnFilterPlugin.java:237)
 	at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:351)
 	at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:291)
 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 	at java.lang.Thread.run(Thread.java:748)
 	Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
 		... 16 more
 	Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
 		at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
 		at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
 		at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
 		at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
 		at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
 		at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
 		... 14 more
 	Caused by: org.eclipse.jetty.io.EofException
 		at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
 		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
 		at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
 		at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
 		at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
 		... 1 more
 	Caused by: java.io.IOException: Pipe quebrado
 		at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
 		at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
 		at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
 		at sun.nio.ch.IOUtil.write(IOUtil.java:65)
 		at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
 		at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
 		... 7 more
 	Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
 		... 16 more
 	Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
 		at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
 		at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
 		at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
 		at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
 		at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
 		at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
 		... 14 more
 	Caused by: org.eclipse.jetty.io.EofException
 		at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
 		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
 		at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
 		at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
 		at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
 		... 1 more
 	Caused by: java.io.IOException: Pipe quebrado
 		at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
 		at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
 		at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
 		at sun.nio.ch.IOUtil.write(IOUtil.java:65)
 		at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
 		at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
 		... 7 more
 Caused by: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
 	at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
 	at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
 	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
 	at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
 	at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
 	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
 	... 14 more
 Caused by: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
 	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:277)
 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1309)
 	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:182)
 	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:129)
 	at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
 	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
 	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
 	... 1 more

 Error: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
@hiroyuki-sato
Copy link
Member

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

  • Did you import your certificate into the JVM environment? (i.e use keytool.)
  • Have you ever tried import without SSL
    I would like to know whether this issue relates to SSL or not.

Best regards.

@adrianovieira
Copy link
Author

adrianovieira commented May 12, 2021 •
edited
Loading

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

  • Did you import your certificate into the JVM environment? (i.e use keytool.)

yes, I did it (if I hadn't done, the issue would be a certificate related).

  • Have you ever tried import without SSL

yes, on a "single-node" lab cluster! but it doesn't work for the production cluster which has more than one node and is over SSL.

I would like to know whether this issue relates to SSL or not.

Best regards.

@hiroyuki-sato
Copy link
Member

Hello, @adrianovieira Thank you for your reply.

  1. I'll try to create the same environment. I'll appreciate it if you send me the reproduce configuration of the elasticsearch.
  2. Multiple clusters without SSL worked fine. Correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hiroyuki-sato @adrianovieira