OPCache wasted memory increases on each request

[icedevelopment]

Bug Report

Q	A
BC Break	no
Version	2.12.3

Summary

The OPCache wasted memory increases on each request if a paginator is used.

Current behavior

Whenever you make a request on a controller with a Doctrine\ORM\Query that uses Doctrine\ORM\Tools\Pagination\Paginator#getIterator(), the DQL to SQL parser cache is redone each time.
Seems that this is bad a side effect introduced in commit 023e946 which was merged in 2.6.5 so every version between this and 2.12.3 should be impacted.

How to reproduce

1. Install a Symfony app with Doctrine with the following standard cache configuration:

# config/packages/prod/doctrine.yaml
doctrine:
    orm:
        auto_generate_proxy_classes: false
        metadata_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        query_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        result_cache_driver:
            type: pool
            pool: doctrine.result_cache_pool

framework:
    cache:
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system

2. Create a controller that uses a Doctrine\ORM\Query with a paginator (i.e. knplabs/knp-paginator-bundle).

3. Set the environment to prod in .env (APP_ENV=prod and APP_DEBUG=0).

4. Make requests on the controller and watch opcache_get_status()['memory_usage']['wasted_memory'] increase each time until OPCache is full and resets itself.

The expireQueryCache flag is always set to true here:

orm/lib/Doctrine/ORM/Tools/Pagination/Paginator.php

Line 163 in c05e170

$whereInQuery->expireQueryCache();

This makes Doctrine reparse the DQL each time:

orm/lib/Doctrine/ORM/Query.php

Lines 261 to 272 in c05e170

	if (! $this->expireQueryCache && $cacheItem->isHit()) {
	$cached = $cacheItem->get();
	if ($cached instanceof ParserResult) {
	// Cache hit.
	$this->parserResult = $cached;
	return $this->parserResult;
	}
	}
	// Cache miss.
	$parser = new Parser($this);

Then Symfony will recreate the exact same file in its cache in var/cache/prod/pools/, invalidate and recompile it into OPCache:
https://github.com/symfony/cache/blob/7d8415956df68c8dcbc9468e119945e39bacead1/Adapter/PhpFilesAdapter.php#L254-L259

Expected behavior

The OPCache wasted_memory doesn't increase on each request.

[josephLaurent]

I got exactly the same issue, for me removing the following lines in the symfony config fix the issue:

doctrine:
    orm:
        auto_generate_proxy_classes: false
        metadata_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        query_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        result_cache_driver:
            type: pool
            pool: doctrine.result_cache_pool

Also, put this in the php.ini file prevent using the opcache and so fix the issue as well:

opcache.file_cache = NULL
opcache.file_cache_only = 0

But this are only quick fix that have impact on performances.
Is there any update on a possible fix on doctrine ?

[greg0ire]

I would not expect the metadata cache or the query cache to grow indefinitely. I would try removing only the line about the result cache. How do you use that cache inside your app, by the way. Regarding the news, we did not discuss this issue in private or in other threads AFAIK.

[josephLaurent]

I also use doctrine 2.12.3 with symfony. The case where the cache doesn't stop growing is when I use doctrine to query the database with a paginator in production environment. If fact, I'm facing the same issue than @icedevelopment in exactly the same case scenario.

[greg0ire]

Well let's say your result set is millions of records. Why would you expect the cache not to grow? The result cache is not meant to be used on each and every query you make through Doctrine ORM, and isn't by default (you have to call useResultCache explicitly IIRC. The cache system is not going to magically hold all the data in a table for free.

[icedevelopment]

@greg0ire The issue here is with the query cache (DQL to SQL transformation), not with the result cache.
For every SQL query used in an HTTP request, Doctrine will always do the DQL to SQL parsing and create a new query cache file with the same contents but a new unique filename everytime. This will make your var/cache/prod/pools/ grow on every HTTP request and wil increase your OPCache wasted memory all the way until it is forced to automatically restart.

[icedevelopment]

@greg0ire have you been able to have a look at this issue? This looks like a pretty bad bug performance-wise.

[greg0ire]

I was on holidays and I don't use the ORM myself, so no. After looking into this a bit, I think a solution could be to use a special implementation of the query cache driver just for the WHERE IN query, instead of calling expireQueryCache. Such an implementation is provided by symfony/cache, but not by psr/cache, sadly.

Can you try calling $whereInQuery->setQueryCache(new Symfony\Component\Cache\Adapter\NullAdapter()) and let me know if that mitigates the issue? If it does, we will need to figure out what to do (build our own null implementation of CacheItemPoolInterface or reuse one).

[icedevelopment]

Hey @greg0ire thank you for your help.

I've just tried calling $whereInQuery->setQueryCache(new \Symfony\Component\Cache\Adapter\NullAdapter()) before $whereInQuery->expireQueryCache(); and I can confirm that it does indeed mitigate the issue.

[greg0ire]

Thanks, that's great news! Would you be willing to work on a PR to use this trick instead of the expireQueryCache one?

[icedevelopment]

Thanks, that's great news! Would you be willing to work on a PR to use this trick instead of the expireQueryCache one?

I just opened #10095.
I created a new NullCacheItemPool based on Symfony\Component\Cache\Adapter\NullAdapter.
Let me know what you think.

[mpdude]

A suggested fix is in #10434.