php-fpm: free(): invalid pointer when used with jemalloc

[makhomed]

Description

The following code:

0. install Rocky Linux 9.1
1. enable EPEL repo and install package jemalloc-5.2.1-2.el9.x86_64 from EPEL
2. install php82-php-fpm package from remi-safe repo, details here: https://rpms.remirepo.net/wizard/
3. enable jemalloc library for using byall all processes via /etc/ld.so.preload configuration file
4. systemctl start php82-php-fpm.service
5. obtain error php-fpm[1280]: free(): invalid pointer

Resulted in this output:

systemd[1]: Starting The PHP FastCGI Process Manager...
php-fpm[1717]: free(): invalid pointer
systemd[1]: Started Process Core Dump (PID 1718/UID 0).
systemd-coredump[1719]: Resource limits disable core dumping for process 1717 (php-fpm).
systemd-coredump[1719]: Process 1717 (php-fpm) of user 0 dumped core.
systemd[1]: systemd-coredump@1-1718-0.service: Deactivated successfully.
systemd[1]: php82-php-fpm.service: Main process exited, code=dumped, status=6/ABRT
systemd[1]: php82-php-fpm.service: Failed with result 'core-dump'.
systemd[1]: Failed to start The PHP FastCGI Process Manager.

But I expected this output instead:

Starting The PHP FastCGI Process Manager...
Started The PHP FastCGI Process Manager.

PHP Version

PHP 8.2.5

Operating System

Rocky Linux 9.1

[KapitanOczywisty]

Possible duplicate of #10670

[nielsdos]

Yes this is indeed the same LD_PRELOAD problem, see #10670 (comment)
And the workaround jemalloc uses has been deprecated in glibc: #10670 (comment) and therefore doesn't work anymore

[makhomed]

Yes this is indeed the same LD_PRELOAD problem, see #10670 (comment)
And the workaround jemalloc uses has been deprecated in glibc: #10670 (comment) and therefore doesn't work anymore

Ok, this is means what it is not possible to use jemalloc with php-fpm at all? May be in this case, php-fpm can detect jemalloc substitution of glibc malloc, and write graceful error message to user? Because error message "free(): invalid pointer" is not user friendly and looks like php internal bug loke double free or something like this.

Or this is not php bug at all, but this is jemalloc bug, and this bug should be fixed in jemalloc code?

If this is jemalloc bug - issue about it should be open it the jemalloc bug tracker ?

https://github.com/jemalloc/jemalloc

[nielsdos]

Right now it is indeed not possible to use the LD_PRELOAD hack to override the allocator. As a workaround you could try to get rid of the RTLD_DEEPBIND option in your build of PHP, but this will most likely cause issues with 3rd party libraries that define the same global symbols.
Alternatively, you can link your PHP extensions statically instead of dynamically which will also work around the problem.

May be in this case, php-fpm can detect jemalloc substitution of glibc malloc, and write graceful error message to user?

I'm not sure if that's possible to do in a reliable way. There are usages of LD_PRELOAD that do not involve allocators that work.

Or this is not php bug at all, but this is jemalloc bug, and this bug should be fixed in jemalloc code?

It's more of a limitation in how these alternative allocators are injected. Any application that uses RTLD_DEEPBIND to load its libraries can break. In the past jemalloc used a workaround that allowed it to be used with RTLD_DEEPBIND, but glibc removed support for that workaround recently.

It would probably be possible to add a configuration option to PHP to allow overriding the allocator at configure+compile time, but even then using LD_PRELOAD to override the allocator will not work.

[bukka]

Why do you try to use jemalloc with FPM in the first place? From what I remember it claims to be more scallable so it probably make sense in multithreaded apps but there are no threads in FPM so what would be the actual advantage?

[makhomed]

Why do you try to use jemalloc with FPM in the first place? From what I remember it claims to be more scallable so it probably make sense in multithreaded apps but there are no threads in FPM so what would be the actual advantage?

Yes, you are right, jemalloc is very good for Percona Server for MySQL (compiled statically) or for Redis (also compiled statically).

It was my mistake - I think what if jemalloc is good for Percona Server for MySQL and Redis - this is means what jemalloc will be good for every other software, so I enable jemalloc via /etc/ld.so.preload for all programs, and got crashes for php-fpm.

Now I realized my mistake and stop trying to use jemalloc for each program in the system.

[bukka]

So maybe it's not just better for multithreaded apps as Redis is single threaded and mentions better fragmentation here but it's mem intensive app so it's more likely it needs better allocator. PHP uses it's own allocator for program run that is optimized for PHP execution so it is less likely there will be significant improvement. FPM uses libc allocator for non request related usage (e.g. master process) so it might get some improvements but not sure if there will be any visible impact as the amount of new allocations is much smaller compare to databases.

[makhomed]

So maybe it's not just better for multithreaded apps as Redis is single threaded and mentions better fragmentation here but it's mem intensive app so it's more likely it needs better allocator.

As I can see from htop output - /usr/bin/redis-server is multithreaded app.
At least, /usr/bin/redis-server from redis-6.2.7-1.el9.x86_64 package from Rocky Linux 9.1 / RHEL 9.1

  [Main] [I/O]
    PID USER       PRI  NI NLWP  VIRT   RES   SHR S  CPU%▽MEM%   TIME+  Command
   1285 redis       20   0    5  275M 10724  7756 S   0.0  0.0  1:33.81 /usr/bin/redis-server 127.0.0.1:6379
   1296 redis       20   0    5  275M 10724  7756 S   0.0  0.0  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
   1297 redis       20   0    5  275M 10724  7756 S   0.0  0.0  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
   1298 redis       20   0    5  275M 10724  7756 S   0.0  0.0  0:00.00 /usr/bin/redis-server 127.0.0.1:6379
   1299 redis       20   0    5  275M 10724  7756 S   0.0  0.0  0:00.00 /usr/bin/redis-server 127.0.0.1:6379

PHP uses it's own allocator for program run that is optimized for PHP execution so it is less likely there will be significant improvement. FPM uses libc allocator for non request related usage (e.g. master process) so it might get some improvements but not sure if there will be any visible impact as the amount of new allocations is much smaller compare to databases.

Ok, Jakub thank you for detailed explanation, now I completely understand my mistake and stop trying to use jemalloc with php-fpm.

BTW, not only php-fpm, but nginx also has use its own, very effective memory allocator, which gives it a high speed of work.

nginx memory allocator source code

nginx memory management api

description how nginx memory allocator works