[BUG] DemoConfig gets read after upgrade

[dhoffend]

What is the bug?
After upgrading opensearch software on our server, the can't start because the democonfig installer is re-adding the security demo configuration in out working production config.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Install oder opensearch version
2. configuration plugin.security in an hierarchic style and certificates
3. run opensearch
4. apt upgrade
5. opensearch won't start because of missing certificates and config issues

What is the expected behavior?
It should matter if the yml configuration is writting in an hierarchic or flat style. If the configuration is valid, opensearch democonfig installer should respect set

What is your host/environment?

• OS: Debian 11
• Version 2.16

Snippet of opensearch.yml

plugins:
    security:
        allow_unsafe_democertificates:           false
        allow_default_init_securityindex:        false
        enable_snapshot_restore_privilege:       true
        check_snapshot_restore_write_privileges: true
        ssl:
            transport:
                pemcert_filepath:              /etc/opensearch/opensearch.fullchain.crt
                pemkey_filepath:               /etc/opensearch/opensearch.key
                pemtrustedcas_filepath:        /etc/opensearch/internal-ca.pem
                enforce_hostname_verification: true
            http:
                enabled:                true
                pemcert_filepath:       /etc/opensearch/opensearch.fullchain.crt
                pemkey_filepath:        /etc/opensearch/opensearch.key
                pemtrustedcas_filepath: /etc/opensearch/internal-ca.pem

Do you have any additional context?

After upgrading opensearch the configuration looks like

plugins:
    security:
        allow_unsafe_democertificates:           false
        allow_default_init_securityindex:        false
        enable_snapshot_restore_privilege:       true
        check_snapshot_restore_write_privileges: true

        ssl:
            transport:
                pemcert_filepath:              /etc/opensearch/opensearch.fullchain.crt
                pemkey_filepath:               /etc/opensearch/opensearch.key
                pemtrustedcas_filepath:        /etc/opensearch/internal-ca.pem
                enforce_hostname_verification: true

            http:
                enabled:                true
                pemcert_filepath:       /etc/opensearch/opensearch.fullchain.crt
                pemkey_filepath:        /etc/opensearch/opensearch.key
                pemtrustedcas_filepath: /etc/opensearch/internal-ca.pem
[...]

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=kirk,OU=client,O=client,L=test,C=de']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

Here is the part where democonfig looks for the existence of "plugin.security"
https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/tools/democonfig/SecuritySettingsConfigurer.java#L106-L128

[cwperks]

[Triage] Thank you for filing this issue @dhoffend. This looks like an actual bug. Thank you for the report.

[derek-ho]

Fixed with #4793