Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 4.9.2 - RC 1 - Vulnerability Detection E2E tests #26616

Closed
hossam1522 opened this issue Oct 29, 2024 · 5 comments
Closed

Release 4.9.2 - RC 1 - Vulnerability Detection E2E tests #26616

hossam1522 opened this issue Oct 29, 2024 · 5 comments

Comments

@hossam1522
Copy link
Member

hossam1522 commented Oct 29, 2024

Tests information

Main release stage issue #26599
Main Vulnerability Detection tests issue #26615
Version 4.9.2
Release stage RC 1
Tag https://github.com/wazuh/wazuh/tree/v4.9.2-rc1
Previous E2E Vulnerability tests issue #26165

Description

The following issue aims to run all Vulnerability Detector E2E tests for the current release candidate, report the results, and open new issues for any encountered errors.

Instruction

Run the vulnerability detector tests using the Test_e2e_system pipeline

Evidences

Build: https://ci.wazuh.info/job/Test_e2e_system/399/
Report: Test_e2e_system_399_test_vulnerability_detector.zip

Analysis

#26616 (comment)

Conclusion 🔴

@hossam1522
Copy link
Member Author

Build failed due to:

13:17:02  hudson.remoting.ProxyException: org.wazuh.JenkinsException: Deployment error[Test_e2e_system_396_20241029130837_macos_1400_7_0:hudson.AbortException: script returned exit code 255]
13:17:02  	at org.wazuh.QA.WazuhQAEnvironment.WazuhQAEnvironment.deploy(WazuhQAEnvironment.groovy:1205)
13:17:02  	at WorkflowScript.deployInstaces(WorkflowScript:268)
13:17:02  	at WorkflowScript.run(WorkflowScript:444)
13:17:02  	at ___cps.transform___(Native Method)
13:17:02  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
13:17:02  	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
13:17:02  	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
13:17:02  	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
13:17:02  	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
13:17:02  	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
13:17:02  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
13:17:02  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
13:17:02  	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.constructorCall(DefaultInvoker.java:25)
13:17:02  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:97)
13:17:02  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
13:17:02  	at jdk.internal.reflect.GeneratedMethodAccessor1465.invoke(Unknown Source)
13:17:02  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:17:02  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
13:17:02  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
13:17:02  	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
13:17:02  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
13:17:02  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
13:17:02  	at jdk.internal.reflect.GeneratedMethodAccessor1464.invoke(Unknown Source)
13:17:02  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:17:02  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
13:17:02  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
13:17:02  	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
13:17:02  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
13:17:02  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
13:17:02  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
13:17:02  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
13:17:02  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
13:17:02  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:403)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:97)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:315)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:279)
13:17:02  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
13:17:02  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
13:17:02  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
13:17:02  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
13:17:02  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
13:17:02  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
13:17:02  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
13:17:02  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
13:17:02  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
13:17:02  	at java.base/java.lang.Thread.run(Thread.java:829)
13:17:02  Finished: FAILURE

Same error as #26475 (comment)

@pro-akim
Copy link
Member

pro-akim commented Oct 29, 2024

Update

✔️ Checking availability of macOS Server (confirmed by DevOps Team)
Build: https://ci.wazuh.info/job/Test_e2e_system/397 🔴

The build has failed for the same reason.

After some research, we could find that the mac Server was not working properly,
After restart

Re-buid: https://ci.wazuh.info/job/Test_e2e_system/399/

@pro-akim pro-akim self-assigned this Oct 29, 2024
@joaquinsgi joaquinsgi self-assigned this Oct 30, 2024
@joaquinsgi
Copy link
Member

This issue is back to in progress, due to the build has finished

@joaquinsgi
Copy link
Member

joaquinsgi commented Oct 30, 2024

Analysis Report

test_install_vulnerable_package_when_agent_down[install_package]

Error:

E       AssertionError: 
E         Test test_install_vulnerable_package_when_agent_down[install_package] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check no_duplicated_vulnerabilities succeeded
E         -----
E         
E       assert False
E        +  where False = <bound method TestResult.get_test_result of <wazuh_testing.end_to_end.TestResult object at 0x7fb6e219afd0>>()
E        +    where <bound method TestResult.get_test_result of <wazuh_testing.end_to_end.TestResult object at 0x7fb6e219afd0>> = <wazuh_testing.end_to_end.TestResult object at 0x7fb6e219afd0>.get_test_result

In missing_vulnerabilities:

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.0.1",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities:

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "arm64"
        ]
    ]
}
test_vulnerability_detector_scans_cases[upgrade_package_maintain_vulnerability]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_maintain_vulnerability] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report.
E         Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report.
E         Check setup_operation_results failed. Evidences (['setup_failed_agents']) can be found in the report.
E         Check no_duplicated_vulnerabilities succeeded

In missing_vulnerabilities, missing_affected_alerts, missing_mitigated_alerts

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.0.1",
            "x86_64"
        ]
    ]
}
{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.1.0",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.6-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.6",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.6-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.6",
            "arm64"
        ]
    ]
}

In setup_failed_agents

[
    "agent1",
    "agent4",
    "agent3",
    "agent2",
    "agent5"
]
test_vulnerability_detector_scans_cases[upgrade_package_add_vulnerability]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_add_vulnerability] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report.
E         Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report.
E         Check setup_operation_results succeeded
E         Check no_duplicated_vulnerabilities succeeded

In missing_vulnerabilities, missing_affected_alerts, missing_mitigated_alerts

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.1.0",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.1.1-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.1.1",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.1.1-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.1.1",
            "arm64"
        ]
    ],
    "agent2": [
        [
            "CVE-2024-27982",
            "Node.js",
            "18.0.0",
            "x86_64"
        ],
        [
            "CVE-2024-27983",
            "Node.js",
            "18.0.0",
            "x86_64"
        ]
    ]
}
test_vulnerability_detector_scans_cases[upgrade_package_maintain_add_vulnerability]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_maintain_add_vulnerability] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index succeeded
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert succeeded
E         Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report.
E         Check setup_operation_results succeeded
E         Check no_duplicated_vulnerabilities succeeded

In missing_mitigated_alerts:

{
    "agent2": [
        [
            "CVE-2022-32223",
            "Node.js",
            "18.0.0",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities:

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.2.0-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.2.0",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.2.0-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.2.0",
            "arm64"
        ]
    ],
    "agent2": [
        [
            "CVE-2024-27982",
            "Node.js",
            "18.1.0",
            "x86_64"
        ],
        [
            "CVE-2024-27983",
            "Node.js",
            "18.1.0",
            "x86_64"
        ]
    ]
}
test_vulnerability_detector_scans_cases[upgrade_package_remove_vulnerability]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_remove_vulnerability] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index succeeded
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert succeeded
E         Check expected_vulnerability_mitigated_alert succeeded
E         Check setup_operation_results succeeded
E         Check no_duplicated_vulnerabilities succeeded

In unexpected_vulnerabilities:

{
    "agent2": [
        [
            "CVE-2024-27982",
            "Node.js",
            "18.20.0",
            "x86_64"
        ],
        [
            "CVE-2024-27983",
            "Node.js",
            "18.20.0",
            "x86_64"
        ]
    ]
}
test_vulnerability_detector_scans_cases[remove_package]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[remove_package] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index succeeded
E         Check no_unexpected_vulnerabilities_found_in_index succeeded
E         Check expected_vulnerability_affected_alert succeeded
E         Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report.
E         Check setup_operation_results succeeded
E         Check no_duplicated_vulnerabilities succeeded

In missing_mitigated_alerts:

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.0.1",
            "x86_64"
        ]
    ]
}
test_vulnerability_detector_scans_cases[upgrade_package_nonvulnerable_to_vulnerable]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[upgrade_package_nonvulnerable_to_vulnerable] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index succeeded
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert succeeded
E         Check expected_vulnerability_mitigated_alert succeeded
E         Check setup_operation_results failed. Evidences (['setup_failed_agents']) can be found in the report.
E         Check no_duplicated_vulnerabilities succeeded

In unexpected_vulnerabilities:

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.4.1-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.4.1",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.4.1-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "9.4.1",
            "arm64"
        ]
    ],
    "agent2": [
        [
            "CVE-2024-27982",
            "Node.js",
            "20.5.1",
            "x86_64"
        ],
        [
            "CVE-2024-27983",
            "Node.js",
            "20.5.1",
            "x86_64"
        ]
    ]
}
[
    "agent5",
    "agent4",
    "agent3",
    "agent1"
]
test_vulnerability_detector_scans_cases[install_package]

Error:

E       AssertionError: 
E         Test test_vulnerability_detector_scans_cases[install_package] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report.
E         Check expected_vulnerability_mitigated_alert succeeded
E         Check setup_operation_results succeeded
E         Check no_duplicated_vulnerabilities succeeded

In missing_vulnerabilities and missing_affected_alerts:

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.0.1",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities:

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "arm64"
        ]
    ]
}
test_change_agent_manager[install_package]

Error:

E       AssertionError: 
E         Test test_change_agent_manager[install_package] failed
E         
E         Check no_errors succeeded
E         Check operation_successfull_for_all_agents succeeded
E         Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report.
E         Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report.
E         Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report.
E         Check expected_vulnerability_mitigated_alert succeeded
E         Check no_duplicated_vulnerabilities succeeded

In missing_vulnerabilities and missing_affected_alerts:

{
    "agent2": [
        [
            "CVE-2022-0778",
            "Node.js",
            "17.0.1",
            "x86_64"
        ]
    ]
}

In unexpected_vulnerabilities:

{
    "agent1": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "x86_64"
        ]
    ],
    "agent3": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "x86_64"
        ]
    ],
    "agent4": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5-1",
            "arm64"
        ]
    ],
    "agent5": [
        [
            "CVE-2023-4822",
            "grafana",
            "8.5.5",
            "arm64"
        ]
    ]
}

Known issues

Not Known issues

New issues

@pro-akim
Copy link
Member

Review Notes

GJ
LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants