Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory slowly leaking with high amount of tasks #235

Open
iasjennen opened this issue Dec 3, 2024 · 3 comments
Open

memory slowly leaking with high amount of tasks #235

iasjennen opened this issue Dec 3, 2024 · 3 comments

Comments

@iasjennen
Copy link

hello,
I have the following use case:

a Spatie\Async\Pool instance runs permanently and processes a big amount of tasks.

This runs flawlessly, however over the course of days / weeks and processing >100000 tasks the main process requires more and more memory (>2GB).

I suspect the indefinite keeping of the history of finished processes and results causes the "memory leak", see src/Pool.php

...
$this->finished[$process->getPid()] = $process;
...
$this->results[] = $process->triggerSuccess();
...

I suggest to add explicit functions to do the house keeping, eg:

	public function clearFinished() {
		$thid->finished = [];
	}

	public function clearResults() {
		$thid->results = [];
	}

what do you think?

@iasjennen
Copy link
Author

after a couple of days of testing i can confirm:
This was the source of the memory leak and regularly wiping the finished / results prevents the issue from happening.

Tested inside application code by wiping every 1000 jobs:


				if((intval($id) % 1000) == 0){
					$this->pool->clearResults();
					$this->pool->clearFinished();
					$this->cliOutput->printLine('wiped task worker results');
				}

@iasjennen
Copy link
Author

@Nielsvanpach please review, when you have time

@Nielsvanpach
Copy link
Member

We're open to accept a PR for this!

iasjennen added a commit to iasjennen/async that referenced this issue Dec 9, 2024
Since the member variables of finished/results keep growing indefinitely over time this is a source of a memory leak.
This memory leak is resolvable by clearing the result caches with the newly introduced methods.
iasjennen added a commit to iasjennen/async that referenced this issue Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants