JIT segmentation fault in PHP 8.1

[cappadaan]

Description

PHP 8.1.0 + 8.1.1 produces segfault, randomly. Downgrading to 8.0 solves the issue.

--core dump---

BFD: Warning: coredump-php-fpm.30267 is truncated: expected core file size >= 5413076992, found: 35983360.
[New LWP 30267]
[New LWP 1887]
[New LWP 1886]
[New LWP 1888]
Cannot access memory at address 0x7f277dbb3128
Cannot access memory at address 0x7f277dbb3120
Failed to read a valid object file image from memory.
Core was generated by `php-fpm: pool xxxxxx '.

Program terminated with signal 11, Segmentation fault.
#0 0x000055bbf04c0f25 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER () at /usr/src/debug/php-8.1.1/Zend/zend_vm_execute.h:10137
10137 ce = CACHED_PTR(opline->op2.num);
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7ffdc940e3a8:
(gdb) bt
#0 0x000055bbf04c0f25 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER () at /usr/src/debug/php-8.1.1/Zend/zend_vm_execute.h:10137
Cannot access memory at address 0x7ffdc940e3a8
(gdb) frame 0
#0 0x000055bbf04c0f25 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER () at /usr/src/debug/php-8.1.1/Zend/zend_vm_execute.h:10137
10137 ce = CACHED_PTR(opline->op2.num);
(gdb) info frame
Stack level 0, frame at 0x7ffdc940e3b0:
rip = 0x55bbf04c0f25 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER (/usr/src/debug/php-8.1.1/Zend/zend_vm_execute.h:10137); saved rip Cannot access memory at address 0x7ffdc940e3a8

this is the only available info in the core dump.

PHP Version

PHP 8.1.0 + 8.1.1

Operating System

CentOS 7

[cmb69]

I'm afraid this is not actionable without further information, since you didn't provide a reproduce script, and the stack backtrace only shows 1 frame. Since you claim that happens randomly, providing a reproduce script may not be possible, but at the very least we need more info, such as whether OPcache is enabled, whether OPcache JIT is enabled (tracing or function), and in which environment (SAPI) this happens, and also whether that happens only occasionally (and recovers afterwards or not), or happens frequently. Also, try to run with the latest PHP-8.1 development version, where some PHP 8.1.1 bugs have been fixed.

[cappadaan]

• it happens in php-fpm
• randomly and frequent, cannot reproduce
• there is nothing more in the backtrace, so its hard to find what function causes this
• opcache settings:
  opcache.enable = on;
  opcache.enable_cli=1;
  opcache.jit_buffer_size=1G;
  opcache.validate_root = on;
  opcache.enable_cli = on;
  opcache.file_update_protection = 0;
  opcache.revalidate_freq = 0;
  opcache.validate_timestamps = 0;
  opcache.interned_strings_buffer = 64;
  opcache.max_accelerated_files = 16229;
  opcache.memory_consumption = 4096;
  opcache.fast_shutdown = 1;
  opcache.log_verbosity_level = 2;
  opcache.blacklist_filename = /etc/opcache-blacklist.txt;
  opcache.force_restart_timeout = 1800;

[cmb69]

Thanks for the further info! Does it also happen when JIT is disabled (i.e. opcache.jit_buffer_size=0)? If not, what happens if you use a smaller jit_buffer_size (1GB seems very much, maybe try 16M or 64M).

[cappadaan]

I disabled JIT for now, will update in a few days.

[cmb69]

Okay, I'll change back to feeback status (the ticket will be kept open for 2 weeks).

[cappadaan]

Disabling JIT seemed to solve the problem. So JIT is the cause of the segfault.

I have now enabled it and set the cache to 64M, will update later.

[cappadaan]

Again another crash with SIGSEGV after setting JIT to 64M.
So definitely JIT causes this crash.

For now we leave it off there is some sort of fix (or newer php version).

If you need more info just let me know.

[w3yyb]

It's similar to the problem ：https://bugs.php.net/bug.php?id=81664

[cmb69]

Okay, so we now there is a randomly but frequently segfault if (tracing) JIT is enabled under FPM, which happens at:

#0 0x000055bbf04c0f25 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER () at /usr/src/debug/php-8.1.1/Zend/zend_vm_execute.h:10137
10137 ce = CACHED_PTR(opline->op2.num);

I'm afraid that is insufficient information to be actionable, and generally it's hard to fix a bug which is not reproducible. :(

[meinemitternacht]

We are also encountering this bug (or a very similar one) at my company. We use Yocto to build target images for X86_64, and PHP 8.1 is now producing segfaults with JIT enabled. What debugging options would be helpful here?

[meinemitternacht]

@cmb69 After some troubleshooting, it seems that our issue was due to the first flag, CPU optimizations. When enabled, CPUs with the avx instruction set, but not avx2 or avx512..., exhibited segfaults when running JIT. It may be worthwhile to see why this is occurring, but it would be more appropriate for us to open a separate issue at some point in the future.

[cmb69]

@meinemitternacht, so you have segfaults with opcache.jit=1254, but not with opcache.jit=0254 on these machines?

[meinemitternacht]

@cmb69 That is correct.

After testing, this was determined to be incorrect.

[jorismak]

Because if plesk recommend it , that means it's supposed to be working. A lot a lot a lot of people are using it fine and enjoying the huge performance increases it brings. Maybe not enough JIT memory for the use case, or too much? Some other config change ?

…

On Thu, Nov 16, 2023, 16:10 alex-enginity ***@***.***> wrote: An entire Plesk server went down at my company just because I enabled JIT with 1254 as suggested by Plesk itself. This is nuts. The server I was replacing was online for the past 5 years, not a single downtime and 4 hours after enabling JIT, my whole server went down. This should be permanently disabled on PHP. I never had a single timeout or crash due to tweaking PHP options, this is the first time in 10 years that something like this happened to me. I'm curious, why is this code still there and why would Plesk be recommending people to enable it. Have the world gone mad? — Reply to this email directly, view it on GitHub <#7817 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMD2DVG6GVFT2EF7TQM4TTYEYUIHAVCNFSM5KUJMA22U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBRGQ3DINBWGU4Q> . You are receiving this because you commented.Message ID: ***@***.***>

[reza305z]

this is still happening to me on php8.2.8-fpm. the only temporary solution I found is to use function instead of tracing.

Same here with php8.2.12-fpm. I think maybe there is some php extension that causes this problem.

[fidoboy]

Same here. Can't enable JIT with PHP-FPM 8.1.25 because it crashes. Centos server with Apache 2.4.58

[fidoboy]

In my particular case I've discovered after some time of trial and error that the issue was caused by a shit called "Ioncube Loader" that was loading from PHP ini. I've removed it and now JIT is working as expected.

[alex-enginity]

Nice catch!. The server where I have this thing crash and take down most of my server, it crashed on PHP 8.1 and that particular version had Ioncube loader active, as I am using WHMCS on it.
The other versions don't have ioncube on them. So this could explain why all hell broke loose.

[fidoboy]

Nice catch!. The server where I have this thing crash and take down most of my server, it crashed on PHP 8.1 and that particular version had Ioncube loader active, as I am using WHMCS on it. The other versions don't have ioncube on them. So this could explain why all hell broke loose.

You'll see the change as soon as you remove that Ioncube Loader from your PHP setup. I hope that this useless thing disappears from the open source realm forever. In my case it was almost inmediate. Since I removed that useless module, all is working as expected and there is no seg faults at all.

[DavidAnderson684]

I've got many servers which segfault where php-fpm segfaults when I turn JIT on (PHP 8.0, 8.1, 8.2), but don't otherwise, and none of them are running ionCube. So unfortunately not all the problems are inside ionCube,

[fidoboy]

ou'll see the change as soon as you remove that Ioncube Loader from your PHP setup. I hope that this useless thing disappears from the open source realm forever. In my case it was almost inmediate. Since I removed that useless module, all is working as expected and there is no seg faults at all.

There are other issues involved obviously. Because in depth, the issue is that the design of JIT is already very young. But you can have for sure that Ioncube Loader is causing crashes with JIT.

[fidoboy]

In my checks, as soon as I enable the Ioncube thing there is a failure. And it works fine when it's disabled.

[alex-enginity]

It must be quick on your case because maybe the extension is interacting in a way that creates more issues faster than without it. But in my particular case, even PHP8.2 was affected after some more time. On PHP8.1 with ioncube loader it went nuts in just 2 hours but I also lost access to Plesk, which was using it's on webserver. This means that whatever JIT is doing, it is jeopardizing the entire system.

Even if they fix the issues, I don't see me enabling this feature in any forseable future. It's way too risky. Specially if other people's business are on the line. Stability means everything to me.

This featured should be labelled EXPERIMENTAL with a warning message. Not used widely like some of them are doing such as Plesk in their new Performance Booster.

[DavidAnderson684]

What would be useful would an option for the JIT cache for a pool to completely empty itself if any child process in the pool segfaults. I am using pools with pm = ondemand . What I notice in the logs is that once one process in that pool segfaults, all the others launched in that pool segfault, and that continues until the requests stop coming in and allow the number of workers to drop again to zero, and then things are OK. So, by inference, something in the JIT cache is corrupt, and that means that any future worker using it will segfault; if the cache could then be automatically emptied by the master process clearing it (whether directly or by killing all workers in the pool), then things would go a lot better.

[miermontoto]

PHP 8.3 could help fix this issue:

i haven't had the chance to test this thoroughly yet.

[DavidAnderson684]

Interestingly, I've had only a total of 5 segfaults in total of php-fpm since December 21st, which coincides with upgrading to 8.1.26 and 8.2.13 The week before that there were hundreds. The changelogs of both those versions include a fix of an issue causing a segfault in php-fpm.

[DavidAnderson684]

Well, scratch that theory: 83 today thus far already.

[marcing]

I have a doctrine repository fork which is segfaulting on Windows when running unit tests on PHP CLI (8.0, 8.1, 8.2, 8.3). I cannot isolate a smaller sample, but I noticed also linux + FPM crashes on other projects when JIT is enabled. It does not happen every time like on Windows though.
For Windows reproducible segfault check: https://github.com/ovos/doctrine1 + php tests/run.php = segfault after Doctrine_Ticket_1113_TestCase, every single time on a Ryzen 5900X CPU, but not on GitHub Actions (probably a different CPU?)

[adic3x]

I have SIGSEGV error on 8.3.1

So environment is dev on win11 x86-64, Docker with nginx and php-fpm 8.3.1 / wsl. php souce files mapped as volumes: - ./:/var/www/myproject/

php.ini:

zend_extension=opcache.so
opcache.enable=1
opcache.enable_cli=1
opcache.jit_buffer_size=134217728
opcache.jit=1235 ; I don't remember why not 1255 or 1205

I can't say how to reproduce the problem, but I have two ideas.

First. Is editing source files works correctly with JIT at all? In dev mode I edit php files and changes applied immediately. Must I disable JIT on dev environment when I use Docker?

And second, maybe this code reproduces SIGSEGV error:

trait foo
{
    protected static function load(MyClass $thisStatic, bool $condition): mixed
    {
        $query = Comment::where('use', '=', $thisStatic->id);

        if ($condition) {
            $query->where(function (mixed $query) use ($thisStatic): void {
                $query->where('timestamp', '>=', $thisStatic->start)
                      ->orWhere('created_at', '>=', date('Y-m-d H:i:s', $thisStatic->start));
            });
        } else {
            $query->where(function (mixed $query) use ($thisStatic, $divider): void {
                $query->whereBetween('timestamp', [$thisStatic->start, $thisStatic->end])
                      ->orWhere('timestamp', '>', $divider)
                      ->orWhereBetween('created_at', [date('Y-m-d H:i:s', $thisStatic->start), date('Y-m-d H:i:s', $thisStatic->end)])
                      ->orWhere('created_at', '>', date('Y-m-d H:i:s', $divider));
            });
        }

        return $query;
    }
}

It's Laravel Eloquent query builder. I could guess JIT engine may not correctly process Closure with arguments (argument is object passed to function).

However I'm not sure. It's hard to fix a non-reproducible problem.

I will try to rewrite query without Closure and with whereRaw instead. And use 1205 flags. I will write here if I find any information.

[dstogov]

First. Is editing source files works correctly with JIT at all? In dev mode I edit php files and changes applied immediately. Must I disable JIT on dev environment when I use Docker?

JIT should work transparently, but of course "editing" may trigger some bugs that are not visible in usual production environments and that are even harder to reproduce.

In case you could provide a docker image with reproducible scenario, we would be able to understand and fix the problem.

[xpader]

Still happend with php 8.1.27 in offical php docker container.

There's nothing different, except add some extensions like this:

FROM php:8.1.27-fpm-alpine3.19

RUN echo "Start building.." \
    set -ex \
    && apk update \
    # install system base packages
    && apk add tzdata tini git openssh-server openssh-client rsync \
    # config timezone
    && ln -sf /usr/share/zoneinfo/${TIMEZONE} /etc/localtime \
    && echo "${TIMEZONE}" > /etc/timezone \
    # install php extensions
    && apk add pcre-dev $PHPIZE_DEPS gmp-dev zlib-dev libpng-dev \
    && docker-php-ext-install bcmath gmp calendar exif gd pdo_mysql sockets opcache \
    && pecl install -n redis && docker-php-ext-enable redis.so \
    # show php version and extensions
    && php -v \
    && php -m

CMD ["tini", "--", "entrypoint.sh"]

Running by tini, use exec to instead sh process.

entrypoint.sh:

#!/bin/sh -e
/usr/sbin/sshd -e -E /var/log/sshd.log &

cp /opt/crontab /etc/crontabs/www && chown root:root /etc/crontabs/www && chmod 600 /etc/crontabs/www &
crond -L /var/log/crond.log &

exec /usr/local/sbin/php-fpm

[palgalik]

I ran into segmentation fault on 8.3.6 with my Symfony 7.x application with this config:

opcache.memory_consumption = 128
opcache.interned_strings_buffer = 8
opcache.max_accelerated_files = 16000
opcache.revalidate_freq = 60
opcache.fast_shutdown = 1
opcache.enable=1
opcache.enable_cli=1
opcache.jit=1235
opcache.jit_buffer_size=256M

[marcing]

https://github.com/ovos/doctrine1 + php tests/run.php still segfaulting as of PHP 8.3.6.

[jorismak]

As an experiment, did any of you compile PHP yourself? Or can you compile PHP yourself and try if it still segfaults? Not to use as a solution, but to test if this helps. Maybe it's the packaged PHP that is compiled to use architecture things that are not available on your local system (I'm just guessing here).

…

On Wed, 17 Apr 2024 at 16:06, Marcin G ***@***.***> wrote: https://github.com/ovos/doctrine1 + php tests/run.php still segfaulting as of PHP 8.3.6. — Reply to this email directly, view it on GitHub <#7817 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALMD2DTW3RJKDJL7VZI5L5LY5Z6VZAVCNFSM5KUJMA22U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBWGEZTINZTGIZA> . You are receiving this because you commented.Message ID: ***@***.***>

[marcing]

Dear @jorismak I tried few times in the past, but never managed to compile PHP on Windows. I'm using the official package from
https://www.php.net/downloads.php
However we have also segfaults on servers (Linux), so we do not use JIT in production right now, because we have to restart FPM every time it happens.

[klaussilveira]

On PHP 8.3.9 and Symfony 7.1.2, and with opcache.jit=1235, also segfault. Zend Engine v4.3.9, Zend OPcache v8.3.9.

[cmb69]

@jorismak, @marcing, it is unlikely that the official Windows builds would use SIMD instructions which are not available, since they still use only SSE2 (on x64) and SSE (on x86). If you still want to do your own builds, see https://wiki.php.net/internals/windows/stepbystepbuild_sdk_2. About a minimal build environment, see https://gist.github.com/cmb69/47b8c7fb392f5d79b245c74ac496632c.

As for the actual issue: these may actually be quite different issues, and it's almost impossible to fix; see #7817 (comment) on what might help.

To mitigate the crashes: try with tracing JIT disabled. If that doesn't work disable JIT altogether. If that still doesn't work, disable OPcache. Oh, and of course, if there are other "invasive" extensions (often zend_extensions) active, disable these, and see if it fixes (or at least improves) the situation. Info about what worked for you, and what not, wouldn't help us to fix the issue, but at least you would have stable system up and running.

[klaussilveira]

opcache.jit=1205 has been stable for me so far.

[mreho]

Hello, seems to be the same case here with a Wordpress installation that runs on PHP 8.3 (was the same on 8.1 and 8.2 using jit set at 1255)

[10-Aug-2024 13:39:51] WARNING: [pool www.site.tld] child 4001726 exited on signal 11 (SIGSEGV - core dumped) after 0.209034 seconds from start

Switching to jit 1205 seems to fix this issue

JIT seems experimental at this point..

[cmb69]

Info about what worked for you, and what not, wouldn't help us to fix the issue, […]

[daniellockyer]

This issue is really plaguing us. We were on 8.1.17, experiencing a really strange segfault within some Stripe-PHP code on one of our apps. I updated to 8.1.20, and that problem is fixed, but then I start getting segfaults on a totally different PHP on the same server. Had to revert that one to 7.3 😖 Changing JIT options doesn't seem to help with much

Unfortunately, I can't provide a repro but I have the following logs:

[33624850.446814] php-fpm8.1[5390]: segfault at 7fd70a5f1a38 ip 00007fed0cc18a6c sp 00007ffd2f24a3e0 error 4 in opcache.so[7fed0cc00000+df000] likely on CPU 7 (core 7, socket 0)

[cmb69]

Yesterday there was an accident in my household, so I called the ambulance. They asked: "Where do you live?" I replied: "In my house, of course!" For some reason, they still didn't show up. ;)

What I want to say: please don't reply to this thread if your issue isn't even related to JIT (I already expect that those who reported JIT related segfaults may actually face quite different issues). Instead open a new ticket, and try to provide as much useful information as you can.