Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize file processing #71

Closed
kissgyorgy opened this issue Nov 30, 2021 · 2 comments
Closed

Parallelize file processing #71

kissgyorgy opened this issue Nov 30, 2021 · 2 comments
Assignees
Labels
enhancement New feature or request performance performance improvements tasks

Comments

@kissgyorgy
Copy link
Contributor

We want to run the processing in a multiprocessing.pool.Pool.

In order to simplify the implementation, and for measuring things, we would do this in 2 steps:

  1. We can run unblob in for every new process_file call.
  2. We can look into yara.match callbacks and start the handling in the pool right away. (For the current code, this would need more complex changes.)
@kissgyorgy
Copy link
Contributor Author

Only after #100. We don't know if this would worth the cost of running on more memory or more CPU if we don't measure.

@qkaiser qkaiser added enhancement New feature or request performance performance improvements tasks labels Jan 13, 2022
@kissgyorgy
Copy link
Contributor Author

It has been implemented in #181

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance performance improvements tasks
Projects
None yet
Development

No branches or pull requests

2 participants