Error defining attributes in Wazuh indexer log files

[rauldpm]

Description

During the OVA v4.4.0 Alpha 1 analysis (wazuh/wazuh#15513 (comment)), the following error log was found in the Wazuh indexer journalctl output:

[root@wazuh-server wazuh-user]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal" 
Dec 07 16:01:17 wazuh-server systemd-entrypoint[997]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Dec 07 16:01:17 wazuh-server systemd-entrypoint[997]: 2022-12-07 16:01:17,690 main ERROR Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster.log" got access denied ("java.lang.RuntimePermission" "accessUserInformation") java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Dec 07 16:01:17 wazuh-server systemd-entrypoint[997]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Dec 07 16:01:17 wazuh-server systemd-entrypoint[997]: 2022-12-07 16:01:17,684 main ERROR Could not define attribute view on path "/var/log/wazuh-indexer/wazuh-cluster_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation") java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")

When checking these files, they exist with the following permissions:

-rw-r--r--. 1 wazuh-indexer wazuh-indexer  32103 Dec  7 17:16 wazuh-cluster.log
-rw-r--r--. 1 wazuh-indexer wazuh-indexer  66759 Dec  7 17:16 wazuh-cluster_server.json

Apparently, these error logs do not always appear, so it is necessary to identify when these logs appear, and if it is a problem with the SPECS files.

Tasks

• Investigate the origin and cause of these logs (DEB and RPM)
• Modify what is necessary to avoid the appearance of these logs.
• Validate the changes (DEB and RPM)
• Analyze and check if these logs are also found in version 4.4.0-2.4.0 of the Wazuh indexer.

Validation

• Wazuh indexer does not show error logs with the mentioned files.

[c-bordon]

Research Summary

I was carrying out various tests and I was not able to replicate these errors in the logs, outside of the OVA, creating VMs from the OVA always gave me an error, but when I install the service directly by packages or using Wazuh installation assistant these logs did not appear.

I generated a new OVA in the 4.4 branch, and with this new OVA the errors do not appear in the logs:

[root@wazuh-server ~]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal" 
dic 13 10:00:08 wazuh-server systemd-entrypoint[735]: WARNING: System::setSecurityManager will be removed in a future release
dic 13 10:00:08 wazuh-server systemd-entrypoint[735]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
dic 13 10:00:08 wazuh-server systemd-entrypoint[735]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
dic 13 10:00:08 wazuh-server systemd-entrypoint[735]: WARNING: A terminally deprecated method in java.lang.System has been called
dic 13 10:00:07 wazuh-server systemd-entrypoint[735]: WARNING: System::setSecurityManager will be removed in a future release
dic 13 10:00:07 wazuh-server systemd-entrypoint[735]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
dic 13 10:00:07 wazuh-server systemd-entrypoint[735]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.3.0.jar)
dic 13 10:00:07 wazuh-server systemd-entrypoint[735]: WARNING: A terminally deprecated method in java.lang.System has been called

Conclusion

Apparently in the process of creating the OVA we had some problem that generated the errors in the logs, we will have to wait for the creation of the OVA alpha 2 or RC1 to validate that the errors do not appear

[c-bordon]

I could not find similar errors in Opensearch, I leave the links of the searches:
https://www.google.com/search?q=opensearch-cluster.log+access+denied&sxsrf=ALiCzsa18MAC21rZ9zSydhCI9Jv2cCX_0w%3A1670940723348&ei=M4iYY9jqFL3R1sQP4L28cA&ved=0ahUKEwiY0IWi4_b7AhW9qJUCHeAeDw4Q4dUDCBA&uact=5&oq=opensearch-cluster.log+access+denied&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIFCAAQogQ6BAgjECc6BAgAEEM6BQgAEJECOgUIABCABDoOCC4QgAQQxwEQ0QMQywE6CAgAEIAEEMsBOgcIIxCxAhAnOgcIABCABBAKOg0ILhCABBDHARDRAxAKOgQIABAeOgcIABCABBANOgYIABAeEA06BwgAEB4Q8QQ6BQghEKABOgcIIRCgARAKSgQIQRgASgQIRhgAUABYmUBgy0VoA3AAeACAAZ4BiAH3HJIBBTI3LjEymAEAoAEBwAEB&sclient=gws-wiz-serp
https://www.google.com/search?q=org.opensearch.cli.Command.mainWithoutErrorHandling%28Command.java%3A138%29&sxsrf=ALiCzsYwk3EGVsKyFh_-T1q8fgmeyA3ZvA%3A1670940674324&ei=AoiYY-ulE9_K1sQPzaa7qAo&ved=0ahUKEwjrrtWK4_b7AhVfpZUCHU3TDqUQ4dUDCBA&uact=5&oq=org.opensearch.cli.Command.mainWithoutErrorHandling%28Command.java%3A138%29&gs_lcp=Cgxnd3Mtd2l6LXNlcnAQA0oECEEYAEoECEYYAFAAWABgzwFoAHABeACAAT2IAT2SAQExmAEAoAECoAEBwAEB&sclient=gws-wiz-serp
https://www.google.com/search?q=ERROR+Could+not+define+attribute+view+on+path&oq=ERROR+Could+not+define+attribute+view+on+path&aqs=chrome..69i57j0i22i30.382j0j7&sourceid=chrome&ie=UTF-8

[okynos]

On hold by Alpha2 tests.

[c-bordon]

New tests are carried out with the OVA created for Alpha 2 and the error is detected again in the logs:

A new OVA is re-made locally using the generate_ova.sh script and the error does not exist:

[root@wazuh-server ~]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal" 
dic 26 12:54:47 wazuh-server systemd-entrypoint[734]: WARNING: System::setSecurityManager will be removed in a future release
dic 26 12:54:47 wazuh-server systemd-entrypoint[734]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
dic 26 12:54:47 wazuh-server systemd-entrypoint[734]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
dic 26 12:54:47 wazuh-server systemd-entrypoint[734]: WARNING: A terminally deprecated method in java.lang.System has been called
dic 26 12:54:46 wazuh-server systemd-entrypoint[734]: WARNING: System::setSecurityManager will be removed in a future release
dic 26 12:54:46 wazuh-server systemd-entrypoint[734]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
dic 26 12:54:46 wazuh-server systemd-entrypoint[734]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
dic 26 12:54:46 wazuh-server systemd-entrypoint[734]: WARNING: A terminally deprecated method in java.lang.System has been called

The unattended script is also tested in a VM with Centos 7 and it works correctly. An additional test was carried out on a Centos 7 VM using the provision.sh script and the installation was also detected without problems:

[root@centos72 ova]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal" 
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: System::setSecurityManager will be removed in a future release
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: A terminally deprecated method in java.lang.System has been called
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: System::setSecurityManager will be removed in a future release
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
dic 26 17:52:50 wazuh-server systemd-entrypoint[15643]: WARNING: A terminally deprecated method in java.lang.System has been called

A new OVA is created again to confirm this while I analyze where this error could be generated: https://ci.wazuh.info/view/Packages/job/Packages_Builder_OVA/185/console

I'm also trying to replicate the process with the AMI used to create the OVA on EC2, although I still have no conclusions about what can make the error

[c-bordon]

Update Report

Tasks

• I was doing more tests but for now I can't reproduce the error, the new tests included creating an instance with the ami in AWS and using the provision.sh script but the error does not appear, it does not appear after deleting the logs either, the current tests that I am doing correspond to exporting that same VM to validate if it is something that can happen after the export

Next steps:

• Carry out tests with the exported ova: https://s3.amazonaws.com/warehouse.wazuh.com/trash/vmexport-i-0a33766df47cb970a.ova

[c-bordon]

Update Report

Tasks

• I was carrying out several tests that I mention later
• Test with the vm exported from AWS, create an instance with the ami used in the jenkins process (ami-0ad8beb5c406ea8e4), clone the git repository using the 4.4 branch, and use the provision.sh script. After this export the VM with aws CLI: aws ec2 create-instance-export-task --instance-id i-0e516e1f038891fda --target-environment vmware --export-to-s3-task "ContainerFormat=ova, DiskImageFormat=VMDK , S3Bucket=warehouse.wazuh.com, S3Prefix=trash/vm" After this I used the script setOVADefault.sh and after launching the OVA on my machine the log continues to appear
• I tried OVAs 4.3.0 and 4.3.9 and neither of them shows the error in the logs

[root@wazuh-server ~]# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.3.0"
WAZUH_REVISION="40310"
WAZUH_TYPE="server"
[root@wazuh-server ~]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal"
dic 28 12:27:27 wazuh-server systemd-entrypoint[901]: WARNING: All illegal access operations will be denied in a future release
dic 28 12:27:27 wazuh-server systemd-entrypoint[901]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
dic 28 12:27:27 wazuh-server systemd-entrypoint[901]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
dic 28 12:27:27 wazuh-server systemd-entrypoint[901]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
dic 28 12:27:27 wazuh-server systemd-entrypoint[901]: WARNING: An illegal reflective access operation has occurred

• Validate that when exporting EC2 from AWS and the services are turned off, (indexer, dashboard)

Conlusion

• At the moment I was able to verify that the problem occurs with OpenSearch 2.3.0 and 2.4.0 and it occurs at the time of exporting the AWS instance, at some point a permissions change may occur but I cannot detect where this can be done since these steps do not make permission changes declaratively

[c-bordon]

It goes to on Hold due to priority over this issue

[c-bordon]

In progress again: Testing OVAs with Opensearch packages

[c-bordon]

Update Report

Tasks

• I did a test extracting an OVA with the Opensearch installation, but when I download the OVA the errors do not appear:

[root@wazuh-server ~]# systemctl start opensearch
[root@wazuh-server ~]# systemctl start opensearch-dashboards
[root@wazuh-server ~]# journalctl -r -u opensearch.service | grep -i -E "error|critical|warning|fatal" 
ene 03 14:41:15 wazuh-server systemd-entrypoint[429]: WARNING: System::setSecurityManager will be removed in a future release
ene 03 14:41:15 wazuh-server systemd-entrypoint[429]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
ene 03 14:41:15 wazuh-server systemd-entrypoint[429]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.4.1.jar)
ene 03 14:41:15 wazuh-server systemd-entrypoint[429]: WARNING: A terminally deprecated method in java.lang.System has been called
ene 03 14:41:14 wazuh-server systemd-entrypoint[429]: WARNING: System::setSecurityManager will be removed in a future release
ene 03 14:41:14 wazuh-server systemd-entrypoint[429]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
ene 03 14:41:14 wazuh-server systemd-entrypoint[429]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.4.1.jar)
ene 03 14:41:14 wazuh-server systemd-entrypoint[429]: WARNING: A terminally deprecated method in java.lang.System has been called

• I am going to test again with the provision script since I saw some differences with the state of the services when importing

[rauldpm]

From 3 points to 5 points.

[c-bordon]

The error was found again in the OVA of RC1wazuh/wazuh#16148 (comment)

[DFolchA]

UPDATE

Apply possible fix by deleting logs before exporting OVA.
57acac3

[DFolchA]

Update

Test new OVA error does not appear on Wazuh indexer.

We found a new error in the dashboard:

[DFolchA]

Update

Fixed error from the last comment in 7f6d88a.

Generate new OVA locally and check that the mentioned error does not appear and that everything is running correctly.

[root@wazuh-server ~]# journalctl -r -u wazuh-indexer.service | grep -i -E "error|critical|warning|fatal"
Feb 24 13:12:11 wazuh-server systemd-entrypoint[1229]: WARNING: System::setSecurityManager will be removed in a future release
Feb 24 13:12:11 wazuh-server systemd-entrypoint[1229]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 24 13:12:11 wazuh-server systemd-entrypoint[1229]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 24 13:12:11 wazuh-server systemd-entrypoint[1229]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 24 13:12:09 wazuh-server systemd-entrypoint[1229]: WARNING: System::setSecurityManager will be removed in a future release
Feb 24 13:12:09 wazuh-server systemd-entrypoint[1229]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 24 13:12:09 wazuh-server systemd-entrypoint[1229]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 24 13:12:09 wazuh-server systemd-entrypoint[1229]: WARNING: A terminally deprecated method in java.lang.System has been called