Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overcome monkeypatch for testing #2715

Open
fgibertoni opened this issue Feb 6, 2025 · 1 comment
Open

Overcome monkeypatch for testing #2715

fgibertoni opened this issue Feb 6, 2025 · 1 comment
Labels
backend_simple Simple Backend issues github_actions Pull requests that update Github_actions code maintenance Issues that requires updating dependencies and overall modernity of the project refactor tests

Comments

@fgibertoni
Copy link
Contributor

I think we should start to think about how to overcome the actual testing methodology based on monkeypatches.

The motivation I found for this are:

  • overall testing capabilities and stability: we would be able to test more extensively each analyzer thus resulting in a more stable and reliable project
  • we would be able to test the actual implementation of more complex analyzers in depth
  • a better experience for first-time contributors that already know how to use unittest and don't have to learn a new testing methodology

Of course, since most of the analyzers are now based on monkeypatches we can proceed in three ways:

  1. We can remove monkeypatch entirely and rewrite all tests for each plugin using unittest; this would for sure require quite a lot of work but the result would be much cleaner
  2. We can stop using monkeypatch in new plugins and keep the existing as they are as of now.; each new plugin should then have its test written with unittest
  3. We can keep using monkeypatch in really simple plugins that only rely on external APIs and don't have much computation (we already have some of them) and test only more complex ones

Also the pipeline inside .github/ may be affected by this change, so we should consider to update it if necessary.

I just wrote down some ideas, feel free to partecipate and add other ideas or suggestions 😄

@fgibertoni fgibertoni added backend_simple Simple Backend issues github_actions Pull requests that update Github_actions code maintenance Issues that requires updating dependencies and overall modernity of the project refactor tests labels Feb 6, 2025
@mlodic
Copy link
Member

mlodic commented Feb 6, 2025

I agree that the tests framework actually implemented contains some limitations.

At this time, it is already possible to run custom tests for each analyzer if someone wants. One of the few examples can be found here: https://github.com/intelowlproject/IntelOwl/blob/master/tests/api_app/analyzers_manager/observable_analyzers/test_nvd_cve.py

However atm this requires duplication of the mocking that would need to appear both in the main module and in the test module to properly work.

Ideally, it would make sense to create a custom python file for each analyzer containing its own patch and execution and to remove the old monkeypatches from the code.

One of the advantages of the monkeypatch system was not only the reduction of the code to write but thanks to the test_subclasses we executed jobs for each type of analyzable containing each analyzer. By removing the monkeypatch from the main modules, this would not be possible anymore.

Following the unittest principles, this would make sense because first we should test each part of the project indipendently from each other and the rest. At the same time, such "more integrated" tests would still be useful to understand that the main pieces works together. So it could make sense to rewrite these test_subclasses for ObservableAnalyzerTestCase and FileAnalyzerTestCase: the solution could be to test the Plugin start and related methods more appropriately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend_simple Simple Backend issues github_actions Pull requests that update Github_actions code maintenance Issues that requires updating dependencies and overall modernity of the project refactor tests
Projects
None yet
Development

No branches or pull requests

2 participants