Release 4.9.0 - Beta 1 - Footprint Metrics - SYSCHECK (2.5d)

[wazuhci]

Footprint metrics information

Main release stage issue #	#25025
Main footprint metrics issue #	#25005
Version	4.9.0
Release stage #	Beta 1
Tag	https://github.com/wazuh/wazuh/tree/v4.9.0-beta1

Stress test documentation

Packages used

• Repository: packages-dev.wazuh.com

• Package path: pre-release

• Package revision: 1

• Jenkins build: https://ci.wazuh.info/job/Test_stress/5471/

---

Manager

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5471_manager_2024-08-05.zip

• CSV

  monitor-manager-Test_stress_B5471_manager-pre-release.csv
  Test_stress_B5471_manager_analysisd_events.csv
  Test_stress_B5471_manager_analysisd_state.csv
  Test_stress_B5471_manager_remoted_state.csv

Centos agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5471_centos_2024-08-05.zip

• CSV

  monitor-agent-Test_stress_B5471_centos-pre-release.csv
  Test_stress_B5471_centos_agentd_state.csv

Ubuntu agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5471_ubuntu_2024-08-05.zip

• CSV

  monitor-agent-Test_stress_B5471_ubuntu-pre-release.csv
  Test_stress_B5471_ubuntu_agentd_state.csv

Windows agent

• Plots

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

• Logs and configuration

  ossec_Test_stress_B5471_windows_2024-08-05.zip

• CSV

  monitor-winagent-Test_stress_B5471_windows-pre-release.csv
  Test_stress_B5471_windows_agentd_state.csv

macOS agent

• Plots
• Logs and configuration
• CSV

Solaris agent

• Plots
• Logs and configuration
• CSV

Conclusion 🟡

Plots compared to #24467

Graphs 🟢

Logs 🟡

Known issues:

• Vulnerability Detector module is enabled at the start of the stress test #22565
• https://github.com/wazuh/wazuh-jenkins/issues/4867
• https://github.com/wazuh/wazuh-jenkins/issues/5287
• Real time Windows callback process error in Syscheck stress test #12074

[hossam1522]

Analysis Report

Compared to:

• Release 4.8.1 - RC 2 - Footprint Metrics - SYSCHECK (2.5d) #24467

Graph Report 🟢

Manager

• Dropped_events: increase of syscheck to 30 in the second part of the test

Centos

• VMS: decrease of modulesd.

Ubuntu

• VMS: decrease of modulesd.

Windows

• No abnormalities were found.

Log Report 🟡

Manager

• Expected in Stress Test

2024/08/02 20:40:27 wazuh-modulesd:content-updater: WARNING: The offsets download has been interrupted.
2024/08/03 02:42:06 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.
2024/08/03 06:20:39 wazuh-remoted: WARNING: Message queue is full (10). Events may be lost.
2024/08/04 01:03:33 wazuh-analysisd: WARNING: Syscheck decoder queue is full.

• Reported in: Vulnerability Detector module is enabled at the start of the stress test #22565

2024/08/02 20:38:22 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2024/08/02 20:38:22 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-172-31-0-193.ec2.internal', retrying until the connection is successful.

Centos

• Expected in Stress Test

2024/08/03 02:41:21 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.
2024/08/03 02:41:59 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.

• Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867

2024/08/02 20:40:27 wazuh-agentd: ERROR: (1137): Lost connection with manager. Setting lock.
2024/08/02 20:40:27 wazuh-agentd: ERROR: (1216): Unable to connect to '[172.31.0.193]:1514/tcp': 'Connection refused'.

Ubuntu

• Expected in Stress Test

2024/08/03 02:41:22 wazuh-agentd: WARNING: Agent buffer is full: Events may be lost.
2024/08/03 02:42:22 wazuh-syscheckd: WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data.

• Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867

2024/08/02 20:40:35 wazuh-agentd: ERROR: (1216): Unable to connect to '[172.31.0.193]:1514/tcp': 'Connection refused'.

Windows

• Expected in Stress Test

2024/08/02 21:06:28 wazuh-agent: WARNING: Agent buffer at 90 %.
2024/08/03 02:41:23 wazuh-agent: WARNING: (6906): Real time process: no data. Probably buffer overflow.
2024/08/03 02:41:42 wazuh-agent: WARNING: Agent buffer is full: Events may be lost.

• Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287

2024/08/03 02:43:52 wazuh-agent: ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.18282'. Error code: 2
2024/08/03 02:43:52 wazuh-agent: WARNING: At get_user(c:\tmp\syscheck_test\files\fimstress.18282): CreateFile(): The system cannot find the file specified. (2)
2024/08/03 03:10:03 wazuh-agent: WARNING: (6922): Cannot open 'c:\tmp\syscheck_test\directories\dir2368': No such file or directory
2024/08/03 03:56:34 wazuh-agent: WARNING: (6922): Cannot open 'c:\tmp\syscheck_test\directories': No such file or directory
2024/08/03 12:19:45 wazuh-agent: ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.3975698'. Error code: 5
[2024-08-03_15:18:49] [ERROR] (create_delete): files\fimStress.5205000 file cannot be deleted.

• Reported in Real time Windows callback process error in Syscheck stress test #12074

2024/08/04 20:41:51 wazuh-agent: ERROR: (6613): Real time Windows callback process: 'Access is denied.' (5).

[pro-akim]

Review Notes

Plots:

The following observations could be done

Manager

Dropped_events: increase of syscheck to 30 in the second part of the test

CentOs and Ubuntu

VMS: decrease of modulesd

Add those observations to the plots description please

Logs:

LGTM

[pro-akim]

Review Notes

LGTM