Memory leaks in PHPUnit

[brendt]

• Laravel Version: v6.6.0
• PHP Version: 7.4.0 and 7.3
• Database Driver & Version: sqlite 3.24.0 and MySQL 5.7.25

Description:

Recently we noticed our testsuite throwing out of memory errors, a problem that hadn't happened before. Doing a little debugging we found that each test adds around 2 MB of memory usage. Our test suite has around 1000 tests, and to have it fully run we need around 600 MB of memory allocated.

Around 80% of our test cases are "unit" tests, though they do interact with the database, 20% are integration tests that will simulate requests to controllers, using the built-in Laravel functionality.

We're using both the CreatesApplication and RefreshDatabase traits as provided by Laravel, and use an in-memory sqlite database, however for this issue I also ran our test suite on a MySQL database, where the same leak seems to happen.

I have looked at several older issues and stackoverflow questions, though none of them seem to provide a solution that works in this case.

Steps To Reproduce:

Add this teardown function in the base testcase, and observe.

    protected function tearDown(): void
    {
        parent::tearDown();

        echo memory_get_usage() . PHP_EOL;
    }

Output:

.57536280
.59370792
.63121592
.65270952
…
.83115760
.84632896
.85978784
.82666928
…

[gocanto]

Easy way for you to handle this would extending directly from PHPUnit and letting the framework aside. Hence, you will indeed remove these issues since DB is not in place anymore.

Now, if you wanna have DB interaction just for the peace of mind, I would suggest moving them as feature. Once they are there, create two test suits. One for Unit and One for feature and start moving tests from feature to unit.

This is not an answer for your issue, but I would start from there since I had to do the same in the past once you hit a big number of tests.

[mfn]

I'm still on 5.8 using 7.3 but I was curious and ran a private test suite with >6k tests (absolutely mixed: unit, integration, lots of DB) with your tearDown hack and all I saw is just numbers going up.

Actually the test suite is split into two pipelines 5k vs. 1.5k and the memory goes up from 104_318_304 to 458_096_144 (over 5k tests) and from 67_790_784 to 209_527_416 (1k5 tests).

I'm using a 3rd party service (Codeship) and the machine probably has enough RAM so I never noticed this.

My point: "probably has been that way" already, before you discovered it? Or can you confirm it's not happening with 5.8 for you?

Btw. this is with phpunit 8.3.x

[brendt]

Or can you confirm it's not happening with 5.8 for you

Also happens on 5.8

[deleugpn]

• Laravel Version: v6.6.0
• PHP Version: 7.2.19
• Database Driver & Version: MySQL 5.6.45

I don't know if it's relevant. I ran my test suite in a project with 650 tests, pretty much all of them touching the database, but I'm using actual MySQL and not SQLite.
The first test output was: 39'420'296
The last test output was: 90'611'232

I noticed that it doesn't constantly increase, e.g.

.48095376
.48720792
.49078208
.42856792
.43272440
...
.50419648
.50762032
.51165152
.45723920
.46062656

[brendt]

I've also noticed that sometimes memory usage is decreased, though overall it increases throughout the whole run.

[mfn]

I've also noticed that sometimes memory usage is decreased, though overall it increases throughout the whole run.

Same for me. Btw I'm using Postgres for the DB tests.

[driesvints]

Can we maybe create a simple test case which can reproduce this?

[mfn]

Can we maybe create a simple test case which can reproduce this?

Here we go: https://github.com/mfn/laravel-tests-memleak

Upon checkout:

• composer install
• php artisan generateHorde 50
• vendor/bin/phpunit
  Observe ever increasing memory

A bare bones installation with only adjustments for creating senseless tests. No database involved.

I just remembered some other issue/PR which talked about memory leaks where the culprit was some, I think, closure/reference related leak (but AFAIK this had to do with the Query builder or Eloquent, which doesn't apply in this case).

Or maybe it's such kind of issue with how the application is booted for the tests.

[staudenmeir]

This is a PHPUnit issue. The memory also increases if you remove Laravel from tests/TestCase.php:

<?php

namespace Tests;

abstract class TestCase extends \PHPUnit\Framework\TestCase
{
    //use CreatesApplication;

    protected function tearDown(): void
    {
        parent::tearDown();

        echo memory_get_usage() . PHP_EOL;
    }
}

It looks like the issue was introduced in PHPUnit 8.4. The memory also increases with 8.3, but not as much as with 8.4.

Probably related: sebastianbergmann/phpunit#3915

[dwightwatson]

I noticed that my memory usage increased after upgrading to PHP 7.4.0 - suddenly started hitting the default 128 MB limit. I did the same thing with memory_get_usage() but put it down to being an issue on my end, so glad to see it's not just me.

For reference: Laravel 6.6.1, PHP 7.4.0, Postgres 12.1.

I just tested against PHPUnit 7 as well to compare - similar memory creep but to a lesser extent.

PHPUnit 7.5.17: 137.00 MB
PHPUnit 8.4.3: 165.00 MB

[brendt]

@sebastianbergmann could it be that this is a PHPUnit issue and not a Laravel one?

[driesvints]

I'm gonna leave this open for a bit to follow up but it seems it's best to focus the discussion on the PHPUnit issue linked above by @staudenmeir.