[Bug]: The OPcache interned strings buffer is nearly full #31223
Comments
ghost
added
0. Needs triage
|
cc @MichaIng |
|
I did a sudo service apache2 restart and it went away but then comes back after a while |
|
@andyxheli On Debian with PHP8.0, I suggest to raise it with an own override config module, doubling it to 16 MiB: # Create an own PHP module with highest priority
cat << '_EOF_' > /etc/php/8.0/mods-available/local-opcache.ini
; Raise OPcache interned strings buffer for Nextcloud
; priority=99
opcache.interned_strings_buffer=16
_EOF_
# Enable the module
phpenmod local-opcache
# Restart Apache2 to apply the new module
systemctl restart apache2
This makes sense since the OPcache as well as its interned strings buffer are filled on demand by default, i.e. each script is loaded once requested the first time. When you restart Apache2, the OPcache instance is destroyed and it the buffer is filled while you using the Nextcloud web interface and the individual apps. Since this seems to be not as rare as I expected, it makes sense that we add this explicitly to the docs: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache Not sure whether it breaks the scope of the docs, but I could also add the steps like above to add a config override, the IMHO preferred way since no package files are touched, so that changes can be reverted easily and package manager upgrades can update the default configuration files without interactive confirmation and loosing customisations. In case, does anyone know whether |
|
@MichaIng Sorry I searched everywhere for something regarding this and nothing came up for me I even check the docs and I didn't see anything about a recommended opcache.interned_strings_buffer=16. so I though to submit a bug. I should change my settings from 8 to 16 ? |
|
Okay, no problem, I asked just in case it showed something wrong, e.g. you raised the value already or verified that in fact the buffer is not so full.
And this makes sense as it depends on the enabled apps and in case other PHP applications running on the same PHP/OPcache instance. In most cases, the default 8 MiB are sufficient, in rare cases, as far we could see, 16 MiB are required to keep everything stored in cache. But it wouldn't make sense to recommend this to everyone when the additional 8 MiB reserved in total OPcache was just empty in most cases 😉. So the warnings do now show up based on actual current cache usage. But as I said, it makes sense to mention it in the docs and link that docs section from within the admin panel warnings. I'll raise a PR the next days.
This I would recommend, using e.g. the commands I provided above. Note that this is not mandatory, it just assures that all strings present in Nextcloud's PHP scripts stay cached and hence speeds up access performance a bit. The interned strings buffer is quite a clever implementation which stores repeating strings across all cached scripts just a single time in shared memory and additionally persistent for future requests (which doesn't seem to be the default according to some readings). |
|
All my NC installations were working fine for years, but right after upgrading to 23.0.2 I get these errors on every single installation. |
|
I have the same problem, wenn i raise it to 16 it sayy raise it higher, wenn i raise it to 32 it says raise it higher |
Exactly the same. No matter how high I set new limits, error re-appears again and again. |
And it still does. Note that the admin panel warnings/recommendations are nothing mandatory, just a hint what may be improved according to how far Nextcloud is able to check it. But warnings may be wrong or not apply in a particular case, e.g. I keep ignoring warnings about missing Back to topic: I cannot imagine that one Nextcloud instance is able to use an interned strings buffer of more than 16 MiB, so @Copiis @toxpal are you running only a single Nextcloud instance on that webserver/PHP pool or are there other web applications (which do use PHP)? If it is really just one Nextcloud, can you show your |
My little Nextcloud has the same issue, too. I used opcache before, but immediately after upgrade from 22 to 23 a couple of days ago I got this opcache warning, too. I tried to get rid of it by doubling |
|
Same as for the others, if Nextcloud is really the only website/application running on that webserver and PHP instance, please post the output of
The admin panel does exactly this. It looks how much of the available interned strings buffer is in fact used, and when >90% of available size, it prints the recommendation. If Nextcloud is really the only website, then obviously it requires that much interned strings buffer in your particular case, at least when you want to assure that all strings are kept in cache. |
So this check is a new NC 23 feature? |
|
NC v23.0.1 to be precise. |
On the webserver is one Nextcloud an one Friendica instalation
|
Okay, in this case it is not so unexpected that the default is insufficient. Not sure how large Friendica is. You can btw monitor OPcache usage via opcache-gui which allows to browse and search through all cached scripts, see their size in cache etc. Sadly the interned strings buffer cannot be listed like that, and I'm not sure about any way to further analyse its content. E.g. I have a server with phpBB, Wordpress, Matomo and some own pages and it uses ~150 MiB interned strings buffer (I set it to 256 MiB, giving 192 MiB effective space, always 75%) and up to 400 MiB overall OPcache size. My private Nextcloud instance uses 3.5 MiB/32 MiB, so the ratio between interned strings buffer and overall OPcache usage (the letter includes the first) is quite different depending on which application is used, and in Nextcloud obviously depending on used apps. In case of phpBB and Wordpress they both create large (up to 1 MiB) temporary PHP scripts as some own cache layer, which probably plays a role, especially when those are not evicted on removal (which would be basically a bug), but I haven't analysed it further for now. |
|
you also also use CLI https://github.com/gordalina/cachetool |
|
Ah nice, I didn't know that one. Also good for the docs as alternative. |
|
Here a website that lists all the recommended monitoring options |
|
|
|
To not only see the tool's own PHP/OPcache instance, AFAIK such needs to be passed: --fcgi=/run/php/php8.0-fpm.socketBut this is for PHP-FPM. In case Apache2 with mod_php is used, instead this may work: --web-path=/path/to/your/document/root --web-url=https://my.domain.org |
|
The value of 8 also seems to be the default in docker images. And im getting the same warning running of nextcloud:apache |
|
8 MiB is the PHP default: https://www.php.net/manual/opcache.configuration.php |
|
Also have this warning after added redis cache to last version of server: memcache.distributed' => '\OC\Memcache\Redis', |
|
Redis and transactional file locking is completely unrelated to OPcache and the interned strings buffer. Simply follow the recommendation if you want to max performance. |
|
I have this error as well. Unfortunately, I'm very new to Docker (partly jumping in as a learning exercise), and so I don't know much yet about where the settings for associated services are or where (host, various containers) I'm supposed to enter commands, and I'm finding it difficult to find out. I have a stack with Nextcloud (23.0.1, from new), MariaDB and Nginx Proxy Manager. I fixed a few other errors, and it works, but please could someone give me an idea where to look to change the opcache settings. Tried to get a list of installed apps too, as others have posted, but I don't seem to be able to use the occ command anywhere. I have spent a fair bit of time trouble-shooting Nextcloud and other containers, which is part of the learning experience, but though I'm not an IT or CLI noob, I'm finding the level of assumed knowledge to be above me, so some small pointers to help me on my way would be really appreciated. |
|
There is no way (I'm aware of) to configure PHP settings with the Docker container. It needs to be changed by the maintainer here (which still follows the old static recommendation): https://github.com/nextcloud/docker/blob/master/Dockerfile-alpine.template#L81-L86 The all-in-one appliance switched to Ah, here it is already: nextcloud/docker#1702 |
I'm using nextcloud Linux server docker for unraid, |
Im on Z 25.0.3 |
|
Is there a way to enable a log to see what's taking so much space on OPcache usage? so we can see why this warning comes up ? |
=>
That is strange, I tested this up and down, and the logic is trivial. But what caught my eye even more is that you see a message which was never inside official Nextcloud code: vs The wording in your version is also wrong, those strings are called "interned strings". Are you using an official Nextcloud version? If so, does it show integrity check errors (in case recheck via CLI), and which locale/language are you using? |
This is a translation from german |
Ah okay, I was already wondering 😄. Can you show how you added the values to your # php -r 'if ((false) || ((true) && (2048 < 4096 / 4))) { echo("true\n"); } else { echo("false\n"); }'
falseAh, or there is really |
|
Let's get started, my php.ini entry for the Nexcloud domain looks like this. Do I understand this correctly, the |
|
this is what mine looks like not sure if it good ? zend_extension=opcache.so |
All right, after setting the |
Not necessarily. But if it is a quarter or larger, it the second warning shouldn't show up. It was already a half so the warning shouldn't have shown up before, but there was this little loophole that it still did show up if there was really zero free space in the interned strings buffer.
This was after applying the patch, right? Leave it like that for a while, as the strings buffer is unlikely to full directly, but over time. @andyxheli |
|
That is now the status quo. After I applied your patch > #36278 all OPcache warnings disappeared |
opcache.memory_consumption = 64
opcache.interned_strings_buffer = 32With 64/16 it should work as well. However, the |
No
No, there is also a Wordpress installation and a Friendica installation running in parallel.
I can't tell you that, I'm not that deep into the matter.
These are my active apps
|
|
I still could not find any permanent solution for this. Had |
I encourage you to find my posted solution above. It continues to work in 10 Nextcloud production environments. In a few of them, I needed to update php version for some apps that required it. The recommendation came back, and I simply entered the configuration again, and went right away. There's sadly clutter on this page which detracts people from being able to solve this. Best, oemb1905. |
Did you try this #36278 |
|
Tested with the patch up and down, and indeed was running also into the case where the interned strings buffer was exhausted up to the last byte, in which case the recommendation was kept being shown, with the patch not anymore. @Copiis echo '<?php echo '\''<pre>'\''; print_r(opcache_get_status(false));' > /path/to/webroot/opcache.phpand point your browser to |
|
I'd personally use: opcache.memory_consumption = 64
opcache.interned_strings_buffer = 16which gives more buffer to the overall OPcache, while the interned strings buffer warnings still won't show up anymore, as already a quarter of overall size (or more). If you run more than Nextcloud on the same PHP server, in which case you may need to increase |
PS |
($status['memory_usage']['used_memory'] / $status['memory_usage']['free_memory'] > 9)59835240 / 5189720 = 11.53 > 9 hence the recommendation should now show up. |
|
The changes look like this |
|
Yes, the message about the overall OPcache buffer shows up, which is correct as I calculated above. What does not show up is the message about the interned strings buffer, which has only 16 bytes left free space. It didn't show up in this circumstance since a longer time, when being defined as quarter of overall OPcache or larger (I'm repeating myself, is it really that difficult to understand? 😅). But it did still show up if the interned strings buffer had exactly 0 bytes left free space, which is fixed with the patch. Btw, your output shows that the interned strings buffer is set to 8 MiB (6 MiB usable) only, in which case, also the interned strings buffer message should show up, since |
A simple ctrl-f would do the job, so long as you expanded all historical posts with "Load more" - here ya go as a courtesy: #31223 (comment) ... the settings/values are for my use-case, the important part is the location imho. If it does not help, well best luck, but I've yet to have an instance of mine or others I help continue to complain since. Lmk either way, would love to know. |
|
I'm running nextcloud inside lxc container and had to limit the available memory to resolve this issue (server passed 64 GB memory to the container, now I limit on 8GB). Calculating based on 90% maybe not always best aproach. |
|
How is the system memory allocated to the container related to the size and usage of the PHP instance's OPcache and interned strings buffer? The OPcache size is 128 MiB by default, and I never saw a (single) Nextcloud instance using more than that. And the Nextcloud admin panel warns only when the interned string buffer size is less than 25% than that (and above 90% filled), hence less than 32 MiB in this case. This is all some orders of magnitudes below the system/container memory sizes you are mentioning. |
|
Hi @MichaIng, gave me the idea it may be worth a try to modify |
|
This is data from the PHP OPcache, not the system memory. And reducing RAM size of course reduces free memory, which would increase the chance for the panel recommendation to show up, instead of reducing it 😉. Moreover the particular line you are quoting is not about the interned strings buffer (topic here), but about the overall OPcache memory. This is completely unrelated to host or container system memory. The reason why you do not see the admin panel recommendation currently, is because it takes a while before it reappears, as the interned strings buffer is filled slowly. Since it is known that in some circumstances for unknown reasons, the interned strings buffer is filled slowly unlimited, we adjusted the setup check to not warn anymore when interned strings buffer is 25% of overall OPcache size already. Hence, if your OPcache size is 128 MiB (default), then set the interned strings buffer size to 32 MiB, and all is good. This has been mentioned allover above. Always good to reach through issues and review the attached pull requests, before drawing false conclusions 😉. |
and others
Bug description
The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 8.
Steps to reproduce
Expected behavior
NA
Installation method
Manual installation
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.0
Web server
Apache (supported)
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Updated to a major version (ex. 22.2.3 to 23.0.1)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
{ "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "1.1.1.14", "cloud.next.com" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "23.0.2.1", "overwrite.cli.url": "https:\/\/cloud.next.com", "htaccess.RewriteBase": "\/", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379 }, "simpleSignUpLink.shown": false, "trashbin_retention_obligation": "auto,30", "versions_retention_obligation": "auto,1", "skeletondirectory": "", "default_phone_region": "US", "allow_local_remote_servers": true, "mail_smtpmode": "smtp", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_sendmailmode": "smtp", "mail_smtpport": "587", "mail_smtpauthtype": "LOGIN", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "twofactor_enforced": "true", "twofactor_enforced_groups": [], "twofactor_enforced_excluded_groups": [], "maintenance": false, "updater.release.channel": "stable", "theme": "", "loglevel": 2, "updater.secret": "***REMOVED SENSITIVE VALUE***" } }List of activated Apps
Nextcloud Signing status
No response
Nextcloud Logs
No response
Additional info
.
The text was updated successfully, but these errors were encountered: