Temp folder full

[josephLaurent]

Hello,

Lately, we came across the following issue using bref and lambda.

cache.WARNING: Failed to save key "0c97290e3f217e7b24eed5e136876318" of type Doctrine\ORM\Query\ParserResult: fwrite(): Write of 5359 bytes failed with errno=28 No space left on device

At first, we though that it came from the code using the /tmp folder and making it full.
But after some investigation, I found out that a folder is created (/tmp/cac0c3b3d1db88ccd79feeec7173b6f1) and grow bigger during the execution until that, sometime the cache become full.
I found out that doctrine (and maybe some other library) are writting in this folder but it only happens with this layer: ${bref:layer.php-81}.
With the layer: ${bref:layer.php-81-fpm} I don't have any issue.
We are using a lot of lambda function using both layer and all the event-driven functions (using ${bref:layer.php-81}) have the same issue, all the other don't.
The cache doesn't become full on every lambda but this folder (/tmp/cac0c3b3d1db88ccd79feeec7173b6f1) is always created.
I found out that the folder name change if we change the layer version, else it always stay the same.
Also, the tmp folder look like this just before it fails:

/tmp (514M)
 |-- /vendor (156M) -> Because we are using 'separateVendor' option in this example but it also happens without this option.
 |-- /cache (106M)
 |-- /cac0c3b3d1db88ccd79feeec7173b6f1 (253M) -> The undesired folder
      |-- /tmp (235M)
           |-- /vendor (45M)
           |-- /cache (190M)

A part of our serverless.yml:

functions:
    import:
        handler: import.php
        description: 'Import'
        reservedConcurrency: ${env:RESERVED_CONCURRENCY_IMPORT, 5}
        memorySize: 2048
        timeout: 900
        layers:
            - ${bref:layer.php-81}
        events:
            - sqs:
                arn:
                    Fn::GetAtt:
                        - import
                        - Arn
                batchSize: 1


    ...

resources:
    import:
        Type: "AWS::SQS::Queue"
        Properties:
            QueueName: mobireport-import-${opt:stage, self:provider.stage}
            VisibilityTimeout: 900
            MessageRetentionPeriod: 900

Also, increasing the ephemeral storage of the lambda resolve the issue (at least for some lambda execution) but I hope there is a better solution.
I don't know if this is a bug or if I miss something but in any case, help is welcome :)
Thanks in advance

[mnapoli]

Thanks for the great report 👍

This might be the opcache directory:

bref/runtime/layers/function/php.ini

Lines 11 to 20 in 2f78b22

	opcache.enable=1
	opcache.enable_cli=1
	; Store the opcodes into a file cache instead of memory
	; Since PHP runs on lambdas with a new process each time the memory cache is lost
	; TODO store in a subdirectory (but the problem is that the subdirectory doesn't exist when PHP starts...)
	opcache.file_cache="/tmp"
	; Disable the memory cache since it's useless
	; In my tests it allows to gain 30% of response time in a classic API controller
	opcache.file_cache_only=1

With FPM, opcache stores things in memory. In the "function" layer, we store on disk because the PHP process is restarted between invocations.

I also wonder if APCu might not be the culprit here 🤔 Because I find it curious that opcache's cache would grow infinitely. (or maybe you use a Doctrine "file" cache which caches data as PHP code in order to use opcache when reading the file, which leads to the evergrowing opcache cache).

Maybe Doctrine uses APCu, and APCu stores stuff in the same "space" as opcache.

What can be done? If you use BREF_LOOP_MAX, then your process might not be restarting that often, which means you could use in-memory cache (switch opcache.file_cache_only=0 in php.ini in your app). Maybe that would avoid the evergrowing file cache. BUT it would mean that any time the PHP process restarts you have an empty cache I guess?

Also as you mentioned the /tmp directory size can be increased.

Finally maybe it's worth understand which part specifically causes the issue: is this opcache directly? (in which case fine-tuning its config may help https://tideways.com/profiler/blog/fine-tune-your-opcache-configuration-to-avoid-caching-suprises)
Or is this APCu?

[josephLaurent]

Thanks a lot for your help.

opcache.file_cache_only=0 look like a good option for us because keeping cache between execution does't really matter for asyncronous call.
Note that we also need to set this to fix our issue: opcache.file_cache = NULL

[afu-dev]

Is it possible that Doctrine is storing query results in its cache folder (speaking of /tmp/cache), but when it clears that cache (just a few files), the opcache isn't cleared, resulting in an ever-growing cache?

This can explain why /tmp/cac0c3b3d1db88ccd79feeec7173b6f1/tmp/cache is bigger than /tmp/cache? 🤔

Edit: if so, opcache.blacklist_filename=/tmp/cache could help?
Edit2: /tmp/cac0c3b3d1db88ccd79feeec7173b6f1/tmp/cache instead of /tmp/cache/cac0c3b3d1db88ccd79feeec7173b6f1/tmp/cache.

[josephLaurent]

The folder that grow bigger and bigger is /tmp/cac0c3b3d1db88ccd79feeec7173b6f1. But the name change with every new layer so hard to put this: opcache.blacklist_filename=/tmp/cac0c3b3d1db88ccd79feeec7173b6f1 and not sure it is a good idea to put all the tmp directory in the blacklist ?
Also, opcache.blacklist_filename should not be used on source file instead of cache file ?
Don't hesitate if I said something wrong or misunderstood because I'm really not a opcache expert.

[afu-dev]

Yes, /tmp/cac0c3b3d1db88ccd79feeec7173b6f1 is the folder used by opcache to store the opcode. This is why I proposed to put opcache.blacklist_filename=/tmp/cache, or in other words, the folder used by Doctrine to cache the query results.

If /tmp/cac0c3b3d1db88ccd79feeec7173b6f1 stops growing with this directive, it will be a good lead to understand the origin of the bug.

[josephLaurent]

I tried to opcache.blacklist_filename=/tmp/cache but I still got the issue.
I found the exact folder that is growing /tmp/cac0c3b3d1db88ccd79feeec7173b6f1/tmp/cache/prod/pools.
As we use symfony with doctrine, removing the following lines from the doctrine.yaml config seems to solve the issue.

doctrine:
    orm:
        auto_generate_proxy_classes: false
        metadata_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        query_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        result_cache_driver:
            type: pool
            pool: doctrine.result_cache_pool

An issue has been open for the same issue here: doctrine/orm#9917
I think I can close this issue
Thank you for your help :)