Performance degradation when updating from 10.17.0.7 to 10.19.0.9

[adri]

Description

When updating from 10.17.0.7 to 10.19.0.9 we saw an average response increase of ~10ms, from ~37ms to ~47ms. After reverting the response time decreased.

Steps to Reproduce

I don't know

Expected Behavior

No performance impact

Your Environment

Symfony v7.0
PHP 8.3.2
Debian 12

[newrelic-php-agent-bot]

Hi @adri! Thank you for bringing this to the team’s attention. Could you contact New Relic support by following these steps. This way we will be able to gather additional details about the issue at hand. If your subscription level does not include technical support support plan, the other option available to customers with free plans is to use New Relic’s community support channel: Explorers Hub. For all available options on how to get support, please see these resources.

[ruudk]

@newrelic-php-agent-bot What does that performance degradation has to do with our support plan. We are paying customers. We report that your latest SDK degrades performance. And the previous did not. Clearly something is off.

[zsistla]

@ruudk thanks for the question, and we'll try to clear up the intent. The performance degradation has nothing to do with the support plan and is an issue we are investigating. Sometimes (as in this case), in order to investigate further, we need certain details from customers that cannot be shared in a public repo. Our method for collecting this data in a manner to provide privacy and security to the customer is via our support channel which can be reached in various ways (as detailed in the bot message).

[slt]

We noticed this when our systems were upgraded from v10.17.0.7 to v10.18.0.8, our endpoints were performing dramatically worse, even basic nothing endpoints like health checks.

We build our base docker containers with the latest versions nightly so within a few days of the v10.18.0.8 release it was very clear something had gone wrong.

This is a health check endpoint that does nothing but return a string

This is a more involved endpoint which is usually consistently <60ms

Given the change log, we thought perhaps it was due to JIT finally being available.

New features

PHP's Just-In-Time Compilation is no longer disabled when the PHP Agent is enabled. PHP has this on-by-default starting with PHP 8.0.

We have since tested with many different configurations of JIT, including JIT off and it seems to us at least this performance regression is due v10.18.0.8 itself and not JIT. It seems the same issue is also present in v10.19.0.9

We've rolled back and pinned to v10.17.0.7 and are watching this issue & release announcements hoping for a fix, just commenting here to give it weight.

If it helps, our newrelic.ini has long had the transaction_tracer disabled to minimise performance impacts

newrelic.transaction_tracer.enabled = false
newrelic.transaction_tracer.detail = 0

[ebeltramo96]

We noticed the same as soon as we merged the new version, we had to roll it back to v10.17.0.7

[zsistla]

@slt @ebeltramo96
Would you be please be able to provide more details about your system:
OS
PHP version(s)
frameworks in use

If you follow the bot instructions, our support team could work with you to collect additional information such as debug logs and other sensitive system info in a secure manner.

[ACenterA]

10.20.0.10 has been released:

10.20.0.10

Anyone knows if this bug still occurs with that version?

[ppp0]

Clearly to be seen after upgrading from 10.17.0.7 to 10.22.0.12, albeit not so dramatic as reported above (~5%)

[adri]

Unfortunately upgrading to 10.22.0.12 still results in +12-13ms response times for us.

[ruudk]

Would you be please be able to provide more details about your system: OS PHP version(s) frameworks in use

If you follow the bot instructions, our support team could work with you to collect additional information such as debug logs and other sensitive system info in a secure manner.

OS: Debian 12 bookworm
PHP: PHP 8.3.11
Framework: Symfony 7.1

@zsistla I just created a Case #00224017

[workato-integration]

https://new-relic.atlassian.net/browse/NR-322938

[lavarou]

As @slt pointed out, release 10.18.0.8 added support for PHP’s Just-In-Time Compilation. This support required changes to how New Relic PHP Agent hooks into Zend Engine. I.e. for PHPs 8.0+ it no longer uses the method of overriding zend_execute hook but it uses observer API instead. This change came with an increased overhead. New Relic PHP Engineering team is looking into options to reduce the overhead when observer API is used.

[flokain]

thx for looking into this. This is blocking us from using distributed tracing, so we hope this is fixed soon.

[ebeltramo96]

is there any news on this topic?

[Mat-Moo]

Are you seeing the same in 11 agent? (I suspect so)

[flokain]

yes it is we are a paying customer and have filed this with our account manager anna tauriello. and she said she will escalate this internally so please get in touch with her, for more details and logs about our tests.
what i can tell we used newrelic php agent version 11.3.0.16z and then activated newrelic.distributed_tracing_enabled = trueannewrelic.infinite_tracing.trace_observer.host="******.aws-eu-central-1.tracing.edge.eu.nr-data.net. The effect you can see in the below graph average(duration) from transaction` increasing from 4ms 30ms