SIGSEGV in libpcre2 on PHP 8.1 (ARM)

[Zenexer]

Frequently asked questions

• I have read Frequently Asked Questions

Describe the bug

I've noticed an unusual number of segfaults since upgrading from PHP 8.0 to PHP 8.1. They're rare enough that I haven't had any luck making a reliable repro, but they're frequent enough to cause problems.

Coredumps indicate that the issue is probably related to PCRE JIT:

(gdb) bt
#0  vld1q_u8 (__a=0xffff9e000000 <error: Cannot access memory at address 0xffff9e000000>) at /usr/lib/gcc/aarch64-linux-gnu/10/include/arm_neon.h:17550
#1  ffcps_1_utf (str_end=0xffff9dfffffa "", str_ptr=0xffff9e000000 <error: Cannot access memory at address 0xffff9e000000>, offs1=8, offs2=<optimized out>,
    chars=<optimized out>) at src/pcre2_jit_neon_inc.h:190
#2  0x0000ffffa6662118 in ?? ()
#3  0x0000ffff9dfffff8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

To Reproduce

Sorry, I don't have a reliable repro--it's been difficult to narrow down the cause. It's happening seemingly with any request handled by PHP-FPM, but very infrequently--maybe every thousand requests or so. There isn't any obvious correlation between the requests that segfault, other than that they probably all use preg_* functions at one point or another.

This is something that I can likely debug and even patch on my own, but I'm having trouble since it's difficult to reproduce. If anyone has any ideas that could lead to a more reliable repro, or even better debugging methods than simply throwing a coredump at gdb, that would be greatly appreciated.

Expected behavior

No segfaults

Distribution (please complete the following information):

• OS: Debian
• Architecture: aarch64
• Repository: packages.sury.org

Package(s) (please complete the following information):

# apt-cache policy php libpcre2-8-0
php:
  Installed: (none)
  Candidate: 2:8.1+92+0~20220117.43+debian11~1.gbpe0d14e
  Version table:
     2:8.1+92+0~20220117.43+debian11~1.gbpe0d14e 500
        500 https://packages.sury.org/php bullseye/main arm64 Packages
     2:7.4+76 500
        500 http://deb.debian.org/debian bullseye/main arm64 Packages
libpcre2-8-0:
  Installed: 10.39-2+0~20211122.14+debian11~1.gbp0d570b
  Candidate: 10.39-2+0~20211122.14+debian11~1.gbp0d570b
  Version table:
 *** 10.39-2+0~20211122.14+debian11~1.gbp0d570b 500
        500 https://packages.sury.org/php bullseye/main arm64 Packages
        100 /var/lib/dpkg/status
     10.36-2 500
        500 http://deb.debian.org/debian bullseye/main arm64 Packages

Additional context

• Segfaults seem to happen toward the end of request handling, not the beginning.
• There's an existing PHP issue indicating that libpcre2 needs to be compiled without --enable-jit-sealloc, but I don't see any indication that's been enabled. https://bugs.php.net/bug.php?id=79261
• My logs are showing that the issue either didn't exist in PHP 8.0.x or occurred significantly less often.
• I haven't tested whether this also occurs on x86_64.
• I'm running everything in Docker.

[oerdnj]

If you can get a coredump from the pcre2, Philip (pcre author) is usually very responsive and helpful.

[oerdnj]

The pcre2 has been updated to 10.40, could you perhaps try now?

[Zenexer]

I have another stability issue I need to sort out first, but as soon as I'm sure everything is stable with JIT disabled, I'll give it another shot.

[Zenexer]

Same issue. PHP 8.1.8 (cli) (built: Jul 11 2022 08:55:24) (NTS)

[svenauhagen]

Hi,

I am running into the same problem on our server with php8.1 and libpcre2.
I compiled libpcre2 without optimizations to get a better output.
It happens every now and then so it is not really consistent.

Here is the latest crash report, I will try to find the problem but help is appreciated :)

(gdb) bt full
#0 0x0000ffffbac124d0 in vld1q_u8 (__a=0xffff98600000 <error: Cannot access memory at address 0xffff98600000>) at /usr/lib/gcc/aarch64-linux-gnu/10/include/arm_neon.h:17551
No locals.
#1 ffcps_1 (str_end=0xffff985fffff "", str_ptr=0xffff98600000 <error: Cannot access memory at address 0xffff98600000>, offs1=11, offs2=10, chars=1164258605)
at src/pcre2_jit_neon_inc.h:190
qw = {mem = "\000\000\000\377", '\000' <repeats 11 times>, dw = {4278190080, 0}}
ic = {x = 1164258605, c = {c1 = 45 '-', c2 = 45 '-', c3 = 101 'e', c4 = 69 'E'}}
compare1_type = compare_match1
compare2_type = compare_match1i
cmp1a = {45 <repeats 16 times>}
cmp1b = {0 <repeats 16 times>}
cmp2a = {101 <repeats 16 times>}
cmp2b = {32 <repeats 16 times>}
diff = 1
char1a = 45 '-'
char2a = 101 'e'
char1b = 45 '-'
char2b = 69 'E'
p1 = 0xffff98600009 <error: Cannot access memory at address 0xffff98600009>
align_offset = 10
data = {0, 0, 0, 255, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0, 0}
prev_data = {112, 108, 117, 103, 105, 110, 115, 47, 119, 111, 111, 99, 111, 109, 109, 101}
data2 = {255, 0, 0, 255, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
eq = {0, 0, 0, 255, 0 <repeats 12 times>}
#2 0x0000ffff9d512090 in ?? ()
No symbol table info available.
#3 0x00ff0000ff000000 in ?? ()
No symbol table info available.

It does not happen all the time but every few thousand requests. Our site is used enough though that it happens every hour or so. It is also always the same piece of code:

const PATH_PATTERN_SEARCH_JSON = '#(DOMAIN_PLACEHOLDER)([a-z]+)([_A-Z])(-[-_a-z0-9]+).json$#i';
$domain_replace = 'default';
$search_pattern = str_replace( 'DOMAIN_PLACEHOLDER', $domain_replace, PATH_PATTERN_SEARCH_JSON );
preg_replace("#(woocommerce-)([a-z]+)([_A-Z])(-[-_a-z0-9]+).json$#i", $search_pattern, "/var/www/wordpress/wp-content/languages/plugins/woocommerce-en_US-d5ba42
31878fa2e7d350f73c3da91138.json");

Is that something that can help you?
It looks more to me like a one off in either the PHP memory allocation or in a loop and pcre2 gets a pointer to a char that is outside of the string end position.

Best
Sven

[drekinov]

Hi,
We have similar issue with php 8.2.11 and libpcre2 10.40-1.
I was able to collect core dump which has same backtrace

#0  vld1q_u8 (__a=0xffff88600000 <error: Cannot access memory at address 0xffff88600000>) at /usr/lib/gcc/aarch64-linux-gnu/11/include/arm_neon.h:16131
No locals.
#1  ffcps_default (str_end=0xffff885ffffe "", str_ptr=0xffff88600000 <error: Cannot access memory at address 0xffff88600000>, offs1=2, offs2=<optimized out>, chars=<optimized out>)
    at src/pcre2_jit_neon_inc.h:190
        qw = {mem = "\240o\300ݪ\252\000\000 LA\215\377\377\000", dw = {187650841538464, 281473051610144}}
        ic = <optimized out>

it is referencing same file and line src/pcre2_jit_neon_inc.h:190.

That is fixed in libpcre2 10.42 (Changelog: 5. Merged @carenas patch #175 which fixes #86 - segfault on aarch64 (ARM),)
PCRE2Project/pcre2@3bcfbdd - commit description of fix.
We are running Symfony application on EC2 ARM instances with Ubuntu 22.04 LTS.
Segfault appear after running some time for exactly one preg_replace_callback and only for some requests.
Can we expect libpcre2 update soon ?

[Zenexer]

Debugging this was a PITA, so I'd love to see it shipped :) In the meantime, you can disable PCRE JIT as a workaround. I haven't found it to affect performance much in most PHP use cases.

[carenas]

so I'd love to see it shipped :)

FWIW it shipped with 10.42, and it is small enough that it would make sense to backport, if upgrading the whole PCRE library is not an option.

[drekinov]

I know it is kind of redudant/obvious but want to mention that same lib version is utilized for php 8.3.x so issue continue :)

[carenas]

issue continue

ONLY if you use a version of PHP that uses the embedded PCRE2 and that hasn't been updated, unless you are talking of a different issue that "might" look similar.

there is at least another crasher in the ARM code that was ALSO fixed in the recent 10.43RC1 and that EVERYONE should be running instead (either as part of PHP or as an independent library)

FWIW if you need to keep using older PCRE for whatever reason, backporting the known crashers is the least that should be expected from a packaged PHP IMHO, let me know if I can help.

[drekinov]

there is new ondrej provided:
libpcre2-8-0:amd64 (10.42-3+ubuntu22.04.1+deb.sury.org+1)
Probably that should fix the issue.
Will make some tests on production :)