Fresh install of Elasticsearch v8.16.1 with security disabled causes crash

[Aadniz]

Elasticsearch Version

Version: 8.16.1

Installed Plugins

No response

Java Version

bundled

OS Version

Linux kusari-ab2289 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linux

Problem Description

After the installation was done, it automatically added this piece of configuration to /etc/elasticsearch/elasticsearch.yml:

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 10-12-2024 12:52:13
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["kusari-ab2289"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Upon commenting out this whole section, and changing http.host: 0.0.0.0 to http.host: 127.0.0.1, it will crash with the following error:

[2024-12-10T13:21:23,683][ERROR][o.e.b.Elasticsearch      ] [kusari-ab2289] fatal exception while booting Elasticsearch
org.elasticsearch.ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl - [xpack.security.transport.ssl.enabled] is not set, but the following settings have been configured in elasticsearch.yml : [xpack.security.transport.ssl.keystore.secure_password,xpack.security.transport.ssl.truststore.secure_password]
	at org.elasticsearch.xpack.core.ssl.SSLService.validateServerConfiguration(SSLService.java:653) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:627) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:160) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:496) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:325) ~[?:?]
	at org.elasticsearch.node.NodeConstruction.lambda$construct$16(NodeConstruction.java:884) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:254) ~[elasticsearch-8.16.1.jar:?]
	at java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:289) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215) ~[?:?]
	at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:636) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:656) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:662) ~[?:?]
	at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:667) ~[?:?]
	at org.elasticsearch.node.NodeConstruction.construct(NodeConstruction.java:906) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:288) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.node.Node.<init>(Node.java:200) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240) ~[elasticsearch-8.16.1.jar:?]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.16.1.jar:?]

I would expect it to not crash, and to now be able to get a successful 200 response doing curl http://127.0.0.1:9200/

Steps to Reproduce

1. Fresh installation of Debian 12 Bookworm
2. Follow the instructions at https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html only following the first 2 steps "Import the Elasticsearch PGP Key" and "Installing from the APT repository"
3. Comment out the auto-generated security configuration in /etc/elasticsearch/elasticsearch.yml
4. Start systemd service

Logs (if relevant)

elasticsearch.log

[Aadniz]

Instead of commenting out everything, rather change all the enabled: true options to enabled: false.

I would call this solution a workaround, and not addressing the main problem here. This configuration is autogenerated, appended, after having been set up. By so, it is expected to be able to comment this out without encountering any problems.

How the settings are set up after installation is misleading. I suggest you either merge these settings without being autogenerated, or fix this crash.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)