adjust opcache settings

[happyreacer]

After several hours (about 10-12 hours) I have the following warning in the settings overview:

`The PHP OPcache module is not properly configured:

The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 8.`

When I stop and restart the containers, the warning is gone.

[szaimen]

cc @MichaIng is this how it is supposed to work?

[szaimen]

nextcloud/vm@8740349

[MichaIng]

Jep works as intended when the interned strings buffer usage raises over 90% of configured size. I'm actually wondering which app (?) is causing this since I'm running my instance with max 3.5 MiB interned string buffer used 🤔.

[szaimen]

@happyreacer which apps do you have installed?

Please post the output of sudo docker exec -it nextcloud-aio-nextcloud php occ app:list
Thanks!

[szaimen]

@MichaIng is there maybe a way to find out which files get cached in opcache? In this way we could probably better investigate which app is doing that...

[MichaIng]

Generally OPcache stats can be obtained via:

<?php echo '<pre>'; print_r(opcache_get_status(false)); echo '</pre>';

This needs to be called from the same PHP parent process which runs Nextcloud to see the same cache (which is shared across all FPM children at least), so in CLI it is useless 😉. Change false to true to additionally get a full list of all cached scripts.

Neat is also the opcache-gui, simply download the index.php somewhere into your webroot and call it. Shows nice stats and gauges and searchable cached script lists.

For the OPcache itself it is trivial, all PHP files are cached, you can list them, evict them etc. For the interned strings buffer it is however more complicated and I'm not away of a way to actually list the content of this buffer. Until today I failed to understand which kind of strings are actually stored, asked on PHP mailing lists already, and other unexpected behaviour remains unclear: https://stackoverflow.com/q/67853338/16145737 (which indicates that we should recommend the next power of two instead "higher than ".)
Basically it is very badly documented. But the general concept behind it is pretty clever, storing recurring strings only a single time and request-persistent in a hash table in SHM to reduce (peak) memory usage and to avoid re-allocating strings for each request. Best description I found: https://www.npopov.com/2021/10/13/How-opcache-works.html#interned-strings

However, literal strings, symbols, internal class and function names alone are not sufficient to explain the actual interned strings buffer usage, as long as a PHP application does not create and remove large temporary PHP scripts without evicting them from cache when not used anymore, or when eviction does not or cannot include the cached interned strings.

[szaimen]

Thanks for the deep insights! Do you think that adding the same change that was made in the NcVM will solve the issue here, too?

[MichaIng]

It should.

The other fixed OPcache settings which were recommended before by Nextcloud, could be btw removed (when currently set). They are all PHP defaults anyway, some make things worse than better.

Together with the changed admin panel recommendations, I updated the Nextcloud docs according to the new recommendations and, aside of the usage based recommended cache size increments, other settings which can increase performance: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache
Tweaking revalidation could be actually a reasonable step for containers where config.php is not expected to change? Or does it make use of occ to enable/disable maintenance mode in cases?

[szaimen]

It should.

nice!

The other fixed OPcache settings which were recommended before by Nextcloud, could be btw removed (when currently set). They are all PHP defaults anyway, some make things worse than better.

You mean those?

all-in-one/Containers/nextcloud/Dockerfile

Lines 89 to 94 in 3041e51

	echo 'opcache.enable=1'; \
	echo 'opcache.interned_strings_buffer=8'; \
	echo 'opcache.max_accelerated_files=10000'; \
	echo 'opcache.memory_consumption=128'; \
	echo 'opcache.save_comments=1'; \
	echo 'opcache.revalidate_freq=1'; \

Together with the changed admin panel recommendations, I updated the Nextcloud docs according to the new recommendations and, aside of the usage based recommended cache size increments, other settings which can increase performance: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#enable-php-opcache

Okay, cool! Will have a look at them later...

Tweaking revalidation could be actually a reasonable step for containers where config.php is not expected to change? Or does it make use of occ to enable/disable maintenance mode in cases?

Not that I am aware of...
But config.php changes sometimes... See https://github.com/nextcloud/all-in-one/blob/main/Containers/nextcloud/entrypoint.sh

[MichaIng]

You mean those?

Yes exactly. These are all PHP defaults, some not even reasonable (10000 accelerated files is an invalid value which is increased automatically to 16229). Only opcache.revalidate_freq defaults to 2 seconds, but rechecking PHP files timestamps every second is IMHO total overkill as long as you are not currently developing the application (where disabling OPcache all together would be more reasonable). In case of Nextcloud, only manual or occ CLI edits of config.php are cases where a revalidation makes sense, and it is unlikely that such a change is done and then the Nextcloud website reloaded within a second, expecting the change to take effect 😉. Nextcloud core and app upgrades as well as internal config.php changes trigger related OPcache invalidation automatically.

[szaimen]

Honestly I didn't know that we set them until I've looked it up 😅
(those settings were copied from the "official" docker container...)

[MichaIng]

It was until recently part of the server tuning docs to apply them like that, so this has been adopted by most Nextcloud appliances, I guess 🙂.

[happyreacer]

@szaimen

@happyreacer which apps do you have installed?

Please post the output of sudo docker exec -it nextcloud-aio-nextcloud php occ app:list Thanks!

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - admin_audit: 1.13.0
  - calendar: 3.0.5
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contacts: 4.0.7
  - contactsinteraction: 1.4.0
  - cookbook: 0.9.9
  - dashboard: 7.3.0
  - dav: 1.21.0
  - event_update_notification: 1.4.0
  - extract: 1.3.3
  - federatedfilesharing: 1.13.0
  - federation: 1.13.0
  - files: 1.18.0
  - files_downloadactivity: 1.12.0
  - files_external: 1.15.0
  - files_pdfviewer: 2.4.0
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - impersonate: 1.10.0
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - mail: 1.11.6
  - nextcloud-aio: 0.1.0
  - nextcloud_announcements: 1.12.0
  - notes: 4.3.0
  - notifications: 2.11.1
  - notify_push: 0.3.0
  - oauth2: 1.11.0
  - password_policy: 1.13.0
  - passwords: 2022.1.20
  - photos: 1.5.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - quota_warning: 1.13.0
  - recommendations: 1.2.0
  - registration: 1.4.0
  - richdocuments: 5.0.1
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - sociallogin: 4.11.0
  - spreed: 13.0.2
  - support: 1.6.0
  - survey_client: 1.11.0
  - systemtags: 1.13.0
  - tasks: 0.14.2
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_email: 2.1.1
  - twofactor_nextcloud_notification: 3.3.1
  - user_status: 1.3.1
  - viewer: 1.7.0
  - weather_status: 1.3.0
  - workflowengine: 2.5.0
Disabled:
  - circles: 23.0.0
  - encryption
  - mediadc: 0.1.8
  - onlyoffice: 7.3.0
  - rainloop: 7.1.3
  - talk_matterbridge: 1.23.2
  - user_ldap
```
`

[szaimen]

Thanks!

[MichaIng]

My list is much smaller:

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - apporder: 0.14.0
  - calendar: 3.0.5
  - cloud_federation_api: 1.6.0
  - contacts: 4.0.7
  - dashboard: 7.3.0
  - dav: 1.21.0
  - federatedfilesharing: 1.13.0
  - files: 1.18.0
  - files_rightclick: 1.2.0
  - files_trashbin: 1.13.0
  - files_versions: 1.16.0
  - impersonate: 1.10.0
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - notes: 4.3.0
  - notifications: 2.11.1
  - notify_push: 0.3.0
  - oauth2: 1.11.0
  - photos: 1.5.0
  - provisioning_api: 1.13.0
  - ransomware_protection: 1.12.0
  - settings: 1.5.0
  - spreed: 13.0.2
  - survey_client: 1.11.0
  - tasks: 0.14.2
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - updatenotification: 1.13.0
  - viewer: 1.7.0
  - workflowengine: 2.5.0
Disabled:
  - admin_audit
  - circles: 22.0.0-alpha.9
  - comments: 1.2.0
  - contactsinteraction: 1.1.0
  - encryption
  - federation: 1.2.0
  - files_external
  - files_pdfviewer: 1.2.1
  - files_sharing: 1.15.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - nextcloud_announcements: 1.10.0
  - password_policy: 1.2.2
  - privacy: 1.0.0
  - recommendations: 0.4.0
  - serverinfo: 1.5.0
  - sharebymail: 1.2.0
  - support: 1.0.0
  - systemtags: 1.2.0
  - text: 1.1.0
  - user_ldap
  - user_status: 1.0.0
  - weather_status: 1.0.0

I once tested to max out OPcache usage by installing, enabling and accessing literally all apps available on the store, skipping only those which are incompatible with each other or incompatible with the NC version, after overriding the compatibility warning. And I reached only around 64 MiB. But to be true, I haven't monitored the interned strings buffer that time and I'm also not sure whether app usage (feeding with content) has any effect on it. In theory it shouldn't as it is related to PHP scripts only, not database query content or the resulting HTML documents served, but not 100% sure.

[szaimen]

Oh I see you disabled files_sharing 😅

Yes I mean Nextcloud AIO is meant to include all in one so it is logically that much more apps are used and enabled here ;)

Although I must say: that is really a loong list!

[MichaIng]

Jep, fully legit, and files_sharing indeed is usually enabled as well here, not sure why I disabled it, probably because I didn't need to share something externally from my private instance since a long time 😄.

I actually added the new checks to be able to reduce OPcache sizes, especially the overall one which for NC alone never needs to be above 64 MiB, thinking that it may reduce admin panel warnings, especially for users which do not have access to PHP settings, on shared hostings or such. Interesting that there are cases where the interned strings buffer is too low with 8 MiB. Causes questions/support/changes like this, probably annoying for the one or the other, on the other hand applying the change then may have the little performance benefit which those recommendation were intended for in the first place 🙂.

[snrkl]

to clarify - if someone is getting this warning, should we be proceeding with the tuning guide referenced, or are there other changes planned.. I have read through the ticket, but as I am no conversant in PHP, I am not confident of the path forward...

[MichaIng]

Simply follow the recommendation given by the admin panel. What we may do is fine tuning the used vs available ratio (or using the absolute free/unused value instead of a ratio) at which the warning is shown, but that doesn't affect the generally reasonable recommendation to have at least 10% of the limits left as buffer.

What we may also change is to not just say "higher than X" but recommend a concrete value, e.g. the next power of two for overall OPcache size and interned strings buffer, and the next valid prime number for max keys.

[pyyhttu]

I actually added the new checks to be able to reduce OPcache sizes, especially the overall one which for NC alone never needs to be above 64 MiB, thinking that it may reduce admin panel warnings, especially for users which do not have access to PHP settings, on shared hostings or such.

Read through the tuning guide as well as the other tickets associated to this.

I've got Nextcloud 23.0.2 running on a shared hosting, where my server admins pointed out I'm able to raise the opcache.interned_strings_buffer memory myself with .user.ini. So the tuning guide I raised from default 8 to 16, then to 32, but I'm still getting this error:

8MB: https://ibb.co/smNC6xL
32MB: https://ibb.co/BNcbfNf

phpinfo.php also sees opcache.interned_strings_buffer as 32MB: https://ibb.co/68LN7PR

Requesting to clarify this further - what is the recommendation to act on this on shared hosting?

[MichaIng]

Is it only a single Nextcloud instance served by this PHP pool? I assume that it's a dedicate PHP pool/instance per shared hosting user, otherwise this would be a security vulnerability. You can check it via custom PHP script or opcache-gui: #192 (comment)

... although on your phpinfo output I see 4096 MiB opcache.memory_consumption (128 is the default) and 200,000 cached keys (10,000 is the default), which might be intended for more than one user? Can you see the other user's PHP scripts in the OPcache with the above methods? 🤔

Also note that opcache.interned_strings_buffer is a PHP_INI_SYSTEM setting, so it always applies PHP pool/instance wide and cannot really be set per-dir via .user.ini: https://www.php.net/manual/de/configuration.changes.modes.php

You can see this as well in your phpinfo since local value and master value both have changed. In case of a shared hosting with shared PHP pool/instance, this would mean that you change the value for all users of that shared hosting until server/PHP restart or until another user changes the value the same way, showing how bad that idea would be 😄. If this was really the case, you should ask your hosting provider to split PHP instance/pool per user.