Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modify the security configuration steps to avoid securityadmin use #1968

Closed
5 tasks
okynos opened this issue Dec 7, 2022 · 5 comments
Closed
5 tasks

Modify the security configuration steps to avoid securityadmin use #1968

okynos opened this issue Dec 7, 2022 · 5 comments
Assignees
@davidcr01
Labels
level/task Subtask issue qa_known Issues that are already known by the QA team type/change Change requested type/feature New feature request wazuh-indexer

Comments

@okynos
Copy link
Contributor

okynos commented Dec 7, 2022

Description

Hello!

We have detected in this issue #1582 that there is a configuration option to set up automatically the security configuration without running security admin tool which is deprecated.

The parameter is plugins.security.allow_default_init_securityindex: true

Tasks

  • Include the parameter as a default in the package installation.
  • Build and deploy the modified package.
  • Test in all recommended systems.

Validation
@okynos okynos added type/feature New feature request wazuh-indexer labels Dec 7, 2022
@okynos okynos mentioned this issue Dec 7, 2022
Closed
@alberpilot alberpilot assigned okynos and unassigned okynos Dec 7, 2022
@DFolchA DFolchA mentioned this issue Feb 14, 2023
Closed
@davidcr01 davidcr01 self-assigned this Feb 21, 2023
@davidcr01
Copy link
Contributor

davidcr01 commented Feb 21, 2023 •
edited
Loading

Update Report

Testing

I will test the behavior of the indexer in multiple scenarios: the installation of a single-node indexer and a distributed installation.

In these tests, the /etc/wazuh-indexer/opensearch.yml is modified by adding the plugins.security.allow_default_init_securityindex: true option.

Single node

Installation and stating service

Show log 
root@ubuntu-focal:/home/vagrant# systemctl daemon-reload
root@ubuntu-focal:/home/vagrant# systemctl enable wazuh-indexer
Synchronizing state of wazuh-indexer.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable wazuh-indexer
Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service → /lib/systemd/system/wazuh-indexer.service.
root@ubuntu-focal:/home/vagrant# systemctl start wazuh-indexer
root@ubuntu-focal:/home/vagrant# systemctl sttus wazuh-indexer
Unknown operation sttus.
root@ubuntu-focal:/home/vagrant# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-02-21 10:28:33 UTC; 8s ago
       Docs: https://documentation.wazuh.com
   Main PID: 3403 (java)
      Tasks: 60 (limit: 3545)
     Memory: 1.2G
     CGroup: /system.slice/wazuh-indexer.service
             └─3403 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.ne>

Feb 21 10:28:13 ubuntu-focal systemd[1]: Starting Wazuh-indexer...
Feb 21 10:28:20 ubuntu-focal systemd-entrypoint[3403]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 21 10:28:20 ubuntu-focal systemd-entrypoint[3403]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (f>
Feb 21 10:28:20 ubuntu-focal systemd-entrypoint[3403]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSe>
Feb 21 10:28:20 ubuntu-focal systemd-entrypoint[3403]: WARNING: System::setSecurityManager will be removed in a future release
Feb 21 10:28:21 ubuntu-focal systemd-entrypoint[3403]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 21 10:28:21 ubuntu-focal systemd-entrypoint[3403]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (fil>
Feb 21 10:28:21 ubuntu-focal systemd-entrypoint[3403]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Securi>

wazuh-cluster.log

Show log 
[2023-02-21T10:28:33,682][INFO ][o.o.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]]]).
[2023-02-21T10:28:33,708][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:33,711][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Index .opendistro_security created?: true
[2023-02-21T10:28:33,712][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node started, try to initialize it. Wait for at least yellow cluster state....
[2023-02-21T10:28:33,717][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'config' with /etc/wazuh-indexer/opensearch-security/config.yml and populate it with empty doc if file missing and populateEmptyI>
[2023-02-21T10:28:33,839][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] create_mapping
[2023-02-21T10:28:33,859][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:33,949][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'config' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:33,949][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'roles' with /etc/wazuh-indexer/opensearch-security/roles.yml and populate it with empty doc if file missing and populateEmptyIfF>
[2023-02-21T10:28:33,978][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:33,996][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,026][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'roles' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,027][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml and populate it with empty doc if file missing and p>
[2023-02-21T10:28:34,048][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,068][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,092][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'rolesmapping' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,092][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml and populate it with empty doc if file missing and>
[2023-02-21T10:28:34,112][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,137][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,160][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'internalusers' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,161][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml and populate it with empty doc if file missing and p>
[2023-02-21T10:28:34,179][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,199][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,215][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'actiongroups' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,216][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml and populate it with empty doc if file missing and populateEmpt>
[2023-02-21T10:28:34,237][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,253][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,273][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'tenants' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,273][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml and populate it with empty doc if file missing and populateEmp>
[2023-02-21T10:28:34,296][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,319][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,340][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'nodesdn' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,341][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml and populate it with empty doc if file missing and populate>
[2023-02-21T10:28:34,361][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,381][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,404][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'whitelist' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,404][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml and populate it with empty doc if file missing and populate>
[2023-02-21T10:28:34,433][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,453][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,472][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'allowlist' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,472][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Will update 'audit' with /etc/wazuh-indexer/opensearch-security/audit.yml and populate it with empty doc if file missing and populateEmptyIfF>
[2023-02-21T10:28:34,520][INFO ][o.o.c.m.MetadataMappingService] [node-1] [.opendistro_security/nvVCSqIiSvyTnZE6SQMPkQ] update_mapping [_doc]
[2023-02-21T10:28:34,540][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2023-02-21T10:28:34,557][INFO ][o.o.s.s.ConfigHelper     ] [node-1] Doc with id 'audit' and version 2 is updated in .opendistro_security index.
[2023-02-21T10:28:34,720][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-1] Auditing on REST API is enabled.

Found errors

  • The first message is not an error itself.
  • The second is a general ERROR message during the node startup.

✔️ The number of errors are reduced.

Show log 
root@ubuntu-focal:/home/vagrant# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error"
[2023-02-21T10:28:21,795][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3285845650814918236, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2023-02-21T10:28:29,931][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

Multi node (two nodes)

Installation and stating service
Same for both nodes.

Show log 
[root@ip-172-31-19-254 ec2-user]# systemctl daemon-reload
[root@ip-172-31-19-254 ec2-user]# systemctl enable wazuh-indexer
Created symlink from /etc/systemd/system/multi-user.target.wants/wazuh-indexer.service to /usr/lib/systemd/system/wazuh-indexer.service.
[root@ip-172-31-19-254 ec2-user]# systemctl start wazuh-indexer
[root@ip-172-31-19-254 ec2-user]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-02-21 12:08:49 UTC; 6s ago
     Docs: https://documentation.wazuh.com
 Main PID: 3680 (java)
   CGroup: /system.slice/wazuh-indexer.service
           └─3680 /usr/share/wazuh-indexer/jdk/bin/java -Xshare...

Feb 21 12:07:54 ip-172-31-19-254.ec2.internal systemd[1]: Start...
Feb 21 12:08:15 ip-172-31-19-254.ec2.internal systemd-entrypoint[3680]: ...
Feb 21 12:08:15 ip-172-31-19-254.ec2.internal systemd-entrypoint[3680]: ...
Feb 21 12:08:15 ip-172-31-19-254.ec2.internal systemd-entrypoint[3680]: ...
Feb 21 12:08:15 ip-172-31-19-254.ec2.internal systemd-entrypoint[3680]: ...
Feb 21 12:08:15 ip-172-31-19-254.ec2.internal systemd-entrypoint[3680]: ...
Feb 21 12:08:49 ip-172-31-19-254.ec2.internal systemd[1]: Start...
Hint: Some lines were ellipsized, use -l to show in full.

wazuh-cluster.log

Show log node-1 
[2023-02-21T12:08:29,851][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:08:39,854][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:08:49,856][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:08:49,864][WARN ][o.o.n.Node               ] [node-1] timed out while waiting for initial discovery state - timeout: 30s
[2023-02-21T12:08:49,875][INFO ][o.o.h.AbstractHttpServerTransport] [node-1] publish_address {172.31.19.254:9200}, bound_addresses {172.31.19.254:9200}
[2023-02-21T12:08:49,877][INFO ][o.o.n.Node               ] [node-1] started
[2023-02-21T12:08:49,881][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started
[2023-02-21T12:08:49,885][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will attempt to create index .opendistro_security and default configs if they are absent
[2023-02-21T12:08:49,889][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: []
[2023-02-21T12:08:49,890][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: true
[2023-02-21T12:08:59,857][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:09:09,859][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:09:19,860][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [172.31.29.181:9300] from hosts providers and [{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2023-02-21T12:09:19,911][ERROR][o.o.s.c.ConfigurationRepository] [node-1] Cannot apply default config (this is maybe not an error!)
Show log node-2 
[2023-02-21T12:10:16,000][WARN ][o.o.s.a.r.AuditMessageRouter] [node-2] No default storage available, audit log may not work properly. Please check configuration.
[2023-02-21T12:10:16,000][INFO ][o.o.s.a.i.AuditLogImpl   ] [node-2] Message routing enabled: false
[2023-02-21T12:10:16,090][INFO ][o.o.s.f.SecurityFilter   ] [node-2] <NONE> indices are made immutable.
[2023-02-21T12:10:16,798][INFO ][o.o.a.b.ADCircuitBreakerService] [node-2] Registered memory breaker.
[2023-02-21T12:10:18,099][INFO ][o.o.t.NettyAllocator     ] [node-2] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2023-02-21T12:10:18,312][INFO ][o.o.d.DiscoveryModule    ] [node-2] using discovery type [zen] and seed hosts providers [settings]
[2023-02-21T12:10:19,639][WARN ][o.o.g.DanglingIndicesState] [node-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-02-21T12:10:20,353][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-2] PerformanceAnalyzer Enabled: false
[2023-02-21T12:10:20,436][INFO ][o.o.n.Node               ] [node-2] initialized
[2023-02-21T12:10:20,438][INFO ][o.o.n.Node               ] [node-2] starting ...
[2023-02-21T12:10:20,600][INFO ][o.o.t.TransportService   ] [node-2] publish_address {172.31.29.181:9300}, bound_addresses {172.31.29.181:9300}
[2023-02-21T12:10:20,896][INFO ][o.o.b.BootstrapChecks    ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-02-21T12:10:22,661][INFO ][o.o.c.c.Coordinator	  ] [node-2] setting initial configuration to VotingConfiguration{olWtkAazTw220jKAk4cCwQ,GNmLg8q0TeOLzu-SjlpsEA}
[2023-02-21T12:10:22,882][INFO ][o.o.c.c.JoinHelper	  ] [node-2] failed to join {node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={node-2}{olWtkAazTw220jKAk4cCwQ}{VR1huDZTT7OCpJCkb2j1WQ}{172.31.29.181}{172.31.29.181:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumT$
org.opensearch.transport.RemoteTransportException: [node-1][172.31.19.254:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.cluster.coordination.CoordinationStateRejectedException: incoming term 1 does not match current term 2
        at org.opensearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:254) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1175) ~[opensearch-1.2.4.jar:1.2.4]
        at java.util.Optional.ifPresent(Optional.java:176) ~[?:?]
        at org.opensearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:644) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:607) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:136) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.TransportService.connectToNode(TransportService.java:430) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.TransportService.connectToNode(TransportService.java:414) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:592) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:172) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:139) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193) ~[?:?]
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:336) ~[?:?]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:647) ~[?:?]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:64) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:373) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) ~[opensearch-1.2.4.jar:1.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2023-02-21T12:10:22,937][INFO ][o.o.c.s.MasterService    ] [node-2] elected-as-master ([2] nodes joined)[{node-2}{olWtkAazTw220jKAk4cCwQ}{VR1huDZTT7OCpJCkb2j1WQ}{172.31.29.181}{172.31.29.181:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, {node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=true} e$
[2023-02-21T12:10:23,030][INFO ][o.o.c.c.CoordinationState] [node-2] cluster UUID set to [ThryhQl3R6CggOo7UdQKhQ]
[2023-02-21T12:10:23,106][INFO ][o.o.c.s.ClusterApplierService] [node-2] master node changed {previous [], current [{node-2}{olWtkAazTw220jKAk4cCwQ}{VR1huDZTT7OCpJCkb2j1WQ}{172.31.29.181}{172.31.29.181:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{node-1}{GNmLg8q0TeOLzu-SjlpsEA}{lqTlU18-Q8mzfjeCrnLkLQ}{172.31.19.254}{172.31.19.254:9300}{dimr}{shard_indexing_pressure_enabled=t$
[2023-02-21T12:10:23,162][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-2] Config override setting update called with empty string. Ignoring.
[2023-02-21T12:10:23,163][INFO ][o.o.a.c.ADClusterEventListener] [node-2] Cluster is not recovered yet.
[2023-02-21T12:10:23,185][INFO ][o.o.c.r.a.DiskThresholdMonitor] [node-2] skipping monitor as a check is already in progress
[2023-02-21T12:10:23,214][INFO ][o.o.h.AbstractHttpServerTransport] [node-2] publish_address {172.31.29.181:9200}, bound_addresses {172.31.29.181:9200}
[2023-02-21T12:10:23,215][INFO ][o.o.n.Node               ] [node-2] started
[2023-02-21T12:10:23,235][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-2] Node started

Found errors

  • In the node-1, the errors are related to the security-admin is not initialized unless the other nodes are initialized.
  • In the node-2, there are no errors related to this, the same as in the first test.

❌ The number of errors is not reduced.

Show errors of node-1 
[2023-02-21T12:08:14,997][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2023-02-21T12:09:19,911][ERROR][o.o.s.c.ConfigurationRepository] [node-1] Cannot apply default config (this is maybe not an error!)
[2023-02-21T12:09:19,937][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:09:27,940][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:09:35,942][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:09:43,945][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:09:51,948][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:09:59,951][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:10:07,955][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
[2023-02-21T12:10:15,965][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
Show errors of node-2 
[root@ip-172-31-29-181 ec2-user]# cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -e "Error"
[2023-02-21T12:10:01,481][INFO ][o.o.n.Node               ] [node-2] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8214277236289611877, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2023-02-21T12:10:15,997][ERROR][o.o.s.a.s.SinkProvider   ] [node-2] Default endpoint could not be created, auditlog will not work properly.
[root@ip-172-31-29-181 ec2-user]#

Next steps

It is necessary to discuss with the team if it is worth performing this change. If it is, the next steps would be:

  • Generate a new indexer package with the change.
  • Edit the Installation Assistant by removing the security-admin call.
  • Test the change:
    • AIO installation using the Installation Assistant.
      • In DEB systems.
      • In RPM systems.
    • AIO installation following the Step-by-step installation.
      • In DEB systems.
      • In RPM systems.
    • Distributed installation using the Installation Assistant.
      • In DEB systems.
      • In RPM systems.
    • Distributed installation following the Step-by-step installation.
      • In DEB systems.
      • In RPM systems.
  • Request the documentation team to remove the "Run the security admin script" of the Step-by-step installation page.

@davidcr01
Copy link
Contributor

davidcr01 commented Feb 22, 2023 •
edited
Loading

Update Report

Development

The indexer packages were generated by running the generate_package.sh of the stack/indexer/deb and stack/indexer/rpm folders.

If we install these new packages, we can see that the new option is added in the opensearch.yml configuration file:

root@ubuntu-focal:/home/vagrant# cat /etc/wazuh-indexer/opensearch.yml | grep default_init_securityindex

plugins.security.allow_default_init_securityindex: true

⚠️ This issue goes to "In review" until its approval by the team.

@davidcr01
Copy link
Contributor

davidcr01 commented Feb 23, 2023 •
edited
Loading

Update Report

Test

A test has been done to check the behavior of this option. I tried to:

  • Install the Wazuh indexer without the new parameter.
  • Change the passwords
  • Make a backup of the internal_users.yml file.
  • Reinstall the Wazuh indexer with the new parameter.
  • Remove the internal_users.yml default file and copy the internal_users.yml modified file.

This test checks if the Wazuh indexer with the new parameter (run securityadmin by default) loads the modified internal_users.yml.

🔴 The result is that the Wazuh indexer reads this configuration but it does not use it. Instead, it uses the default configuration. If we read the description of the parameter (https://opensearch.org/docs/latest/security/configuration/yaml/):

When set to true, the security plugin uses default security settings if an attempt to create the security index fails when OpenSearch launches

In conclusion, if we want to change the passwords of the internal_users.yml file, the securityadmin plugin has to be executed to load the modified security index, and that is actually what we want to avoid. This parameter seems not to solve the proposed problem.

Maybe it would be recommended to wait for OpenSearch to release the alternative of the securityadmin plugin to think about a solution.

@davidcr01
Copy link
Contributor

This issue goes to "On hold" until OpenSearch releases the securityadmin alternative. opensearch-project/security#1755

@rauldpm rauldpm mentioned this issue Mar 20, 2023
Closed
2 tasks
@teddytpc1
Copy link
Member

Blocked by: opensearch-project/security#1755

@teddytpc1 teddytpc1 added level/task Subtask issue type/change Change requested labels Apr 20, 2023
@javimed javimed mentioned this issue Jul 12, 2023
Closed
@rauldpm rauldpm added the qa_known Issues that are already known by the QA team label Oct 9, 2023
This was referenced Oct 9, 2023
Closed
Closed
@davidcr01 davidcr01 mentioned this issue Nov 30, 2023
Closed
8 tasks
@teddytpc1 teddytpc1 mentioned this issue Dec 26, 2023
Closed
8 tasks
@Rebits Rebits mentioned this issue Jul 11, 2024
Closed
@rauldpm rauldpm mentioned this issue Jul 23, 2024
Closed
1 task
@hossam1522 hossam1522 mentioned this issue Aug 22, 2024
Closed
1 task
@juliamagan juliamagan closed this as not planned Won't fix, can't repro, duplicate, stale Aug 28, 2024
@hossam1522 hossam1522 mentioned this issue Sep 2, 2024
Closed
1 task
@pro-akim pro-akim mentioned this issue Sep 24, 2024
Closed
1 task
@joaquinsgi joaquinsgi mentioned this issue Sep 30, 2024
Closed
1 task
@pro-akim pro-akim mentioned this issue Oct 7, 2024
Closed
1 task
@rauldpm rauldpm mentioned this issue Oct 16, 2024
Closed
1 task
@hossam1522 hossam1522 mentioned this issue Oct 21, 2024
Closed
1 task
@dgamboa1605 dgamboa1605 mentioned this issue Nov 13, 2024
Closed
1 task
@DiegoRatto DiegoRatto mentioned this issue Dec 9, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidcr01 davidcr01

Labels
level/task Subtask issue qa_known Issues that are already known by the QA team type/change Change requested type/feature New feature request wazuh-indexer
Projects
None yet
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

1968-avoid-securityadmin-4.5 wazuh/wazuh-packages
5 participants
@okynos @rauldpm @teddytpc1 @davidcr01 @juliamagan