Way lower requests per second with HHVM 3.10.1 than with PHP 5.6.4 for most PHP frameworks in a "Hello World" benchmark - why?

[motin]

Compare the following benchmark results.

Is this to say that HHVM should mostly be used when using a lot of computation-intensive code in hack/php, but for "ordinary" web development, ordinary php may be the better choice?

Or, is this a configuration issue? Some explanation would be great, thanks. :)

PHP-FPM 5.6.4 with opcode cache

## HHVM 3.10.1

(Note: Neither phalcon nor ice works with HHVM, so their results are zeroed out in the HHVM graph)

[Orvid]

This looks a lot like a configuration issue, possibly the result of too little RAM and trying to swap. HHVM uses significantly more memory than PHP.

[motin]

Swap does not seem to be the case. I re-ran the benchmarks with vmstat running in the background, and there is no swapping going on:

root@2d1bea9e7307:/public/php-framework-benchmark# vmstat 2
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1162892  90312 1266528    0    0    53    49 1474 2203  3 11 86  0  0
 3  0      0 1160456  90320 1266680    0    0     0    12 3495 10535  7 31 61  0  0
 2  0      0 1156736  90320 1267200    0    0     0     0 9586 29300 10 89  2  0  0
 2  0      0 1155712  90320 1267332    0    0     0     0 15566 21879  4 93  4  0  0
 2  0      0 1154224  90328 1267732    0    0     0     8 14773 22760  8 88  3  0  0
 2  0      0 1150380  90328 1268644    0    0     0     0 10012 28610 12 86  3  0  0
 2  0      0 1146792  90336 1269548    0    0     0    10 9684 28316 16 81  3  0  0
 2  0      0 1144064  90336 1270264    0    0     0     0 11180 26548 14 84  3  0  0
 2  0      0 1143568  90336 1270320    0    0     0     0 16919 19289  4 92  4  0  0
 2  0      0 1142328  90344 1270384    0    0     0    10 17611 19393  4 92  4  0  0
 2  0      0 1140684  90344 1270416    0    0     0     0 17982 19404  1 95  4  0  0
 2  0      0 1140096  90352 1270552    0    0     0    20 16000 20317  4 93  3  0  0
 2  0      0 1140708  90356 1270580    0    0     0  1326 16706 19454  2 95  3  0  0
 2  0      0 1141936  90356 1270620    0    0     0     0 17701 18415  2 95  4  0  0
 2  0      0 1144168  90364 1270684    0    0     0    22 13020 17208 20 76  4  0  0
 2  0      0 1145240  90364 1270768    0    0     0     0 14851 17752 13 83  4  0  0
 2  0      0 1145364  90372 1270792    0    0     0    20 18403 18909  4 92  5  0  0
 2  0      0 1145228  90372 1270824    0    0     0     0 18076 18923  1 95  4  0  0
 2  0      0 1145600  90372 1270848    0    0     0     0 18049 18855  3 93  4  0  0
 2  0      0 1146964  90380 1270872    0    0     0    20 17627 17766  1 95  4  0  0
 2  0      0 1148264  90380 1270900    0    0     0   738 17936 19349  2 93  5  0  0
 1  0      0 1149120  90388 1270908    0    0     0    14 18436 19645  1 95  4  0  0
 2  0      0 1150372  90388 1270932    0    0     0     0 18666 20602  1 94  5  0  0
 2  0      0 1150640  90388 1270956    0    0     0     0 18993 20459  1 95  4  0  0
 3  0      0 1151364  90396 1270968    0    0     0     8 20360 19840  1 95  4  0  0
 2  0      0 1152120  90396 1270976    0    0     0     0 19307 20644  1 94  5  0  0
 2  0      0 1152488  90404 1270984    0    0     0    10 18414 19797  2 94  4  0  0
 2  0      0 1152996  90404 1271012    0    0     0     0 18936 19642  2 93  5  0  0
 2  0      0 1153028  90404 1271032    0    0     0     0 19085 20032  1 94  5  0  0
 2  0      0 1153276  90412 1271040    0    0     0     8 18452 18602  3 92  5  0  0
 0  0      0 1155412  90412 1271052    0    0     0   252 12257 13597  2 59 40  0  0
 0  0      0 1156148  90412 1271060    0    0     0     0  279  431  0  0 100  0  0
 1  0      0 1156828  90420 1271060    0    0     0    18  270  413  0  0 100  0  0
 0  0      0 1157308  90420 1271060    0    0     0     0  268  407  0  0 100  0  0

[motin]

This looks a lot like a configuration issue

Do you see any obvious issues with the configuration issued for the benchmark? https://github.com/motin/php-framework-benchmark/tree/multiple-stacks/docker/stack/php/hhvm

[fredemmott]

Based on the README in the project you forked from, this is expected:

"This project attempts to measure minimum overhead (minimum bootstrap cost) of PHP frameworks in real world"

We are usually slower than PHP7 at tiny 'hello-world' pages (even with the minimal overhead of modern frameworks, they still count as 'tiny'), and we don't put effort into optimizing them as they aren't representative of most real workloads so it would just be getting better numbers without actually helping anyone. Sticking to the default framework configurations also possibly makes them misleading, as checking that the underlying framework is configured appropriately is generally a safer/easier path to improved performance than replacing the engine, which is usually done at a later step.

While we consider these kinds of benchmarks irrelevant, to improve it without changing the general idea I suggest looking through hhvm.com/blog/9293/lockdown-results-and-hhvm-performance and https://github.com/hhvm/user-documentation for general hints. Specific points include:

• warmup. Like all JITs, HHVM is much slower for the first few requests. We run the benchmark twice (with the same HHVM process) and throw away the first set of results. If a real-world user has a low enough number of requests that the warmup period is a significant portion, the QPS is low enough that the relative performance of different engines is rarely of importance compared to eg the backing IO
• hhbbc (repo authoritative mode). Not compatible with all PHP code, but should be used where available
• hhvm.server.thread_count. This generally needs tuning per workload - it can vary from 2x cores (if using async IO heavily) to 10x in different situations
• the configuration tests we do: https://github.com/hhvm/oss-performance/blob/master/base/hhvm_config_check.php https://github.com/hhvm/oss-performance/blob/master/base/php-src_config_check.php https://github.com/hhvm/oss-performance/blob/master/base/SystemChecks.php - and it looks like you're not touching mysql, but just in case, https://github.com/hhvm/oss-performance/blob/master/base/DatabaseInstaller.php#L119-L127

[fredemmott]

Is this to say that HHVM should mostly be used when using a lot of computation-intensive code in hack/php, but for "ordinary" web development, ordinary php may be the better choice?

It shows that HHVM should not be used for 'hello world' pages where performance is of top concern. Lightweight pages are not reflective of real-world performance in more significant apps.

[motin]

@fredemmott Thanks for a great explanation!