Elasticsearch version 6.8.6 oss unable to start #51060

nnbrandon · 2020-01-15T17:49:57Z

Elasticsearch version (bin/elasticsearch --version): Version: 6.8.6, Build: oss/rpm/3d9f765/2019-12-13T17:11:52.013738Z, JVM: 1.8.0_181

Plugins installed: []

JVM version (java -version):
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

OS version (uname -a if on a Unix-like system): Linux

Description of the problem including expected versus actual behavior:
Hi all,

I am trying to do a fresh install of elasticsearch-oss-6.8.6 with OpenJDK version 1.8.0_181. An elasticsearch-oss-6.5 was able to successfully install before but I am receiving errors related to the keystore and trust manager. There were no issues related to the trust manager and keystore before. It seems like for some reason, elasticsearch is trying to do SSL configurations but we never specified it. There is no SSL options or configurations in the elasticsearch.yml file.

Steps to reproduce:

Download the https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.8.6.rpm
I had to rebuild the rpm file with "Requires: java-1.8.0-openjdk"
Build and install elasticsearch

Provide logs (if relevant):

[2020-01-13T19:11:29,780][ERROR][o.e.b.Bootstrap          ] [xSBU6-2] Exception
org.elasticsearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
	at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:70) ~[?:?]
	at org.elasticsearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:109) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:145) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:115) ~[?:?]
	at org.elasticsearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:88) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$11(Node.java:472) ~[elasticsearch-6.8.6.jar:6.8.6]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_181]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_181]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_181]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_181]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_181]
	at org.elasticsearch.node.Node.<init>(Node.java:475) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.6.jar:6.8.6]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.6.jar:6.8.6]
Caused by: java.security.KeyStoreException: problem accessing trust storejava.io.IOException: Invalid keystore format
	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:74) ~[?:?]
	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_181]
	at org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:151) ~[?:?]
	at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68) ~[?:?]
	... 24 more

Log File:
elasticsearch.log

The text was updated successfully, but these errors were encountered:

nnbrandon · 2020-01-15T23:07:38Z

Based off of this call stack,

Version 6.8: https://github.com/elastic/elasticsearch/blob/6.8/server/src/main/java/org/elasticsearch/node/Node.java#L472, elasticsearch is trying to create a component for ReindexPlugin.

Version 6.8: https://github.com/elastic/elasticsearch/blob/6.8/modules/reindex/src/main/java/org/elasticsearch/index/reindex/ReindexPlugin.java#L88, it calls a function ReindexSslConfig() and I am unsure as to why it is trying to config it with SSL.

Version 6.5, https://github.com/elastic/elasticsearch/blob/6.5/modules/reindex/src/main/java/org/elasticsearch/index/reindex/ReindexPlugin.java, there is no mentions of ReindexSslConfig() and this is mainly due to version 6.5 not supporting the library ssl-config.

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

nnbrandon · 2020-01-16T01:20:56Z

@tvernum, do you possibly know what is going on here? I saw that you created this PR wanted to get your input if possible. We did not introduce any SSL settings to reindex from remote.

tvernum · 2020-01-16T02:46:11Z

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

Open Source Elasticsearch has always (well, for a very long time) supported reindexing over https. In 6.7 we added support for configuring the SSL parameters for that connection.

So it makes sense that you see a differne between 6.8 and 6.5

Can you give an exact OS version and/or JDK package name?
OS vendors repackage openjdk and CA Certs is one of the areas that they sometimes fiddle with, so if I'm going to reproduce this I'll need some more info on the environment.

elasticmachine · 2020-01-16T02:46:31Z

Pinging @elastic/es-distributed (:Distributed/Reindex)

elasticmachine · 2020-01-16T02:46:32Z

Pinging @elastic/es-security (:Security/Network)

tvernum · 2020-01-16T03:06:07Z

In your log file I see

 -Djava.security.properties==/ci/hubble/elasticsearch-java.security

My strong suspicion is that your custom security properties are the root cause of this issue.
Are you able to provide a copy of that file?

nnbrandon · 2020-01-16T07:37:05Z

In your log file I see
 -Djava.security.properties==/ci/hubble/elasticsearch-java.security
My strong suspicion is that your custom security properties are the root cause of this issue.
Are you able to provide a copy of that file?

ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=PKIX

I believe that these two lines in there may be the reason why. I'll give it an attempt without it and will ping the results.

nnbrandon · 2020-01-16T17:00:59Z

Why is elasticsearch trying to configure out ReindexPlugin with SSL even though this is an OSS version?

Open Source Elasticsearch has always (well, for a very long time) supported reindexing over https. In 6.7 we added support for configuring the SSL parameters for that connection.

So it makes sense that you see a differne between 6.8 and 6.5

Can you give an exact OS version and/or JDK package name?
OS vendors repackage openjdk and CA Certs is one of the areas that they sometimes fiddle with, so if I'm going to reproduce this I'll need some more info on the environment.

OS Version:

cat /etc/os-release
NAME="MgmtOS"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="MgmtOS 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

JDK:

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Thank you @tvernum

nnbrandon · 2020-01-16T17:46:27Z

@tvernum I didn't have much luck removing the lines I indicated. Here are the contents of the security properties file. Thank you ahead of time for taking a look.

security-properties.txt

tvernum · 2020-01-17T03:02:58Z

I can't reproduce this using the security properties you posted, so it looks like it will be something more subtle.
I'll try and work out a reproduction but I don't have any promises on timeframe.

nnbrandon · 2020-01-17T17:34:11Z

@tvernum, I was able to find out the problem and it was from the security properties file that you had suspicions on. It was mainly due to a keystore mismatch with Java's keystore. I'll go ahead and close this issue. Thank you for taking a look at it.

tvernum added :Distributed Indexing/Reindex Issues relating to reindex that are not caused by issues further down :Security/TLS SSL/TLS, Certificates labels Jan 16, 2020

tvernum self-assigned this Jan 16, 2020

nnbrandon closed this as completed Jan 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch version 6.8.6 oss unable to start #51060

Elasticsearch version 6.8.6 oss unable to start #51060

nnbrandon commented Jan 15, 2020 •

edited

Loading

nnbrandon commented Jan 15, 2020 •

edited

Loading

nnbrandon commented Jan 16, 2020

tvernum commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

tvernum commented Jan 16, 2020

nnbrandon commented Jan 16, 2020

nnbrandon commented Jan 16, 2020 •

edited

Loading

nnbrandon commented Jan 16, 2020

tvernum commented Jan 17, 2020 •

edited

Loading

nnbrandon commented Jan 17, 2020

Elasticsearch version 6.8.6 oss unable to start #51060

Elasticsearch version 6.8.6 oss unable to start #51060

Comments

nnbrandon commented Jan 15, 2020 • edited Loading

nnbrandon commented Jan 15, 2020 • edited Loading

nnbrandon commented Jan 16, 2020

tvernum commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

elasticmachine commented Jan 16, 2020

tvernum commented Jan 16, 2020

nnbrandon commented Jan 16, 2020

nnbrandon commented Jan 16, 2020 • edited Loading

nnbrandon commented Jan 16, 2020

tvernum commented Jan 17, 2020 • edited Loading

nnbrandon commented Jan 17, 2020

nnbrandon commented Jan 15, 2020 •

edited

Loading

nnbrandon commented Jan 15, 2020 •

edited

Loading

nnbrandon commented Jan 16, 2020 •

edited

Loading

tvernum commented Jan 17, 2020 •

edited

Loading