Error building phar with compression enabled #80

localgod · 2014-04-24T09:24:07Z

I get the following error:

proc_open(): unable to create pipe Too many open files

when I enable the following option in the box.json file:

"compression": "BZ2",

I'm trying to build a phar app with the Cilex framework as the foundation.
I'm running PHP 5.4.27 on OSX 10.9.2

The text was updated successfully, but these errors were encountered:

ghost · 2014-04-24T18:25:28Z

Does this error message happen without the "compression" setting?

ghost · 2014-05-17T03:24:12Z

http://stackoverflow.com/questions/18744991/symfony2-process-component-unable-to-create-pipe-and-launch-a-new-process

Seems like the gc is still full of bugs (after 10 years). Typical php behavior, don't be afraid... ;-)

ghost · 2014-09-05T23:22:39Z

@localgod Can you use dev-master to see if you still see this issue?

localgod · 2014-09-06T09:46:56Z

I'm normally using the phar version from http://box-project.org/
When I try to install dev-master with composer I get this message:

Skipped installation of bin bin/box for package kherge/box: file not found in package

So I can not really test it.

ghost · 2014-09-06T16:26:19Z

Try:

composer require kherge/box=2.0.x-dev

kingcrunch · 2015-02-05T10:22:54Z

It seems, that this is still an issue with the current phar. Anything I can do?

PHP Fatal error:  Uncaught exception 'ErrorException' with message 'proc_open(): unable to create pipe Too many open files' in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php:974
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 974, Array)
#1 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(974): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(784): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(745): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(675): Symfony\Component\Console\Application->getTe in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php on line 974

kherge · 2015-02-05T21:11:51Z

I'm afraid I don't know what else I can do on my end.

I believe that this is a problem with the phar extension itself, which may not be closing open file handles.

ghost · 2015-02-05T21:17:15Z

@kherge You should reduce the problem to 1-10 lines of code, write a short example and send a bug report (https://bugs.php.net/) or find an existing one with a workaround. I thought the workaround I linked would solve the problem. :S Maybe different bugs can cause the same error message. Good luck!

lserni · 2015-03-01T21:09:08Z

I have the same problem. The cause is indeed in the phar extension; the "compressFiles" function compresses each file to a temporary one, and when it has finished, it reads them back together again, and then closes them all. Doing so requires N files to be open at once. (You can increase the limit using ulimit() under Unix/Linux, though).

kherge · 2015-03-01T21:17:34Z

@lserni Thanks for the additional info!

I'll see if I can write a small code example to trigger this issue.

lserni · 2015-03-04T12:12:32Z

The problem is actually in the "phar.c" file. This is a heavily snipped excerpt. Note that there is one cycle opening as many tmpfiles() as needed, and then another that makes use of them. There seems to be a check that avoids using those handles altogether, though.
An alternative would be to not store entry->cfp, but rather a pair ( offset, length ) with which to seek into a single huge temporary file with fixed cfp. Then all rewind(entry->cfp)'s would be replaced by fseek(hugefp, entry->offset, SEEK_SET), etc.:

newfile = php_stream_fopen_tmpfile();

for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   ...
   entry->cfp = php_stream_fopen_tmpfile();

}

/* now copy the actual file data to the new phar */
offset = php_stream_tell(newfile);
for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   if (entry->cfp) {
        file = entry->cfp;
        php_stream_rewind(file);
   }

   ...
   if (entry->cfp) {
        php_stream_close(entry->cfp);
        entry->cfp = NULL;
   }
}

/* finally, close the temp file, rename the original phar,
   move the temp to the old phar, unlink the old phar, and reload it into memory
*/
if (phar->fp && free_fp) {
    php_stream_close(phar->fp);
}

kherge · 2015-03-04T16:03:35Z

Found an open bug on PHP's bug tracker which pretty much describes what we're experiencing.

Here's a snippet that will reproduce the issue:

<?php

ini_set('memory_limit', -1);

$max_handles = (int) file_get_contents('/proc/sys/fs/file-max');

if (!is_dir('src')) {
    echo "Creating source files...\n";

    mkdir('src');

    $total_dirs = ceil($max_handles / 1000);

    for ($current_dir = 0; $current_dir < $total_dirs; $current_dir++) {
        mkdir("src/$current_dir");

        for ($current_file = 0; $current_file < 1000; $current_file++) {
            touch("src/$current_dir/$current_dir-$current_file.php");
        }
    }
} else {
    echo "Source files already exist, skipping.\n";
}

if (file_exists('bug.phar')) {
    echo "Removing previous phar...\n";

    unlink('bug.phar');
}

echo "Building new phar...\n";

$phar = new Phar('bug.phar');
$phar->buildFromIterator(
    new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator(
            'src',
            FilesystemIterator::SKIP_DOTS
        )
    ),
    'src'
);

echo "Compression files in phar...\n";

$phar = new Phar('bug.phar');
$phar->compressFiles(Phar::GZ);

EDIT: I also updated PHP's bug tracker.

ghost · 2015-03-04T16:18:23Z

PHP devs do not seem to care about PHP extensions since 2010. Lot of bugs are not resolved since then in the soap and phar extensions (but I guess you can find more extensions easily which suffered the same faith). I don't know whether you can do anything about this. You need some c skills, and fork, fix, pull request, but I guess these options are not available by PHP extensions (I may be wrong).

kherge · 2015-03-04T16:21:17Z

Unfortunately I'm not experienced enough in C, and even then it could be a while before we see anything become of the fix.

From what I recall, the PHP extension used to be a PHP native class. I'm thinking that until this issue gets resolved properly, I could re-create the extension in PHP again. It'll be much slower, but it'll be easier and quicker to fix.

ghost · 2015-03-04T18:26:53Z

I don't know, I am not experienced in this topic.

For me the phar was important because of the digital signature. With that you can write a deploy tool, which copies phar files into a folder via FTP or HTTP. If somebody steals your password they can do nothing, because they cannot sign their own package without the private key.

Maybe others use this technology for different purposes, I think before you write your own phar implementation in PHP you should think about what you want to solve with it. So it should not be a simple copy-paste of the current solution. E.g. some people want to use compressed, password encrypted files, which is not supported by the current extension. Some people use webphar (I guess that was the name), which is only 1.5-2x slower than the native php. By them your slower solution won't be good enough I think. (Note: I don't use PHP for a year or so, I dev now in nodejs.)

lserni · 2015-03-05T08:02:45Z

I feel quite confident in altering phar.c - I shall do so as soon as my new laptop arrives, the old and faithful one having given up the ghost some days ago - but I think the problem lies rather in compiling (and distributing) the resulting phar.so module, which must match the existing PHP installation.

For me on OpenSuSE for example the easiest option seems to be to download the php5-phar SRPM and use that to rebuild the RPM.

But a quick fix would be to simply increase the open file limit before making the PHAR file:

ulimit -Sn 4096

I've tried it, and (where available) it works.

ghost · 2015-03-05T15:33:28Z

@lserni If you want to modify the c file, I think the best option to talk with the php guys. They already have the tools necessary to distribute the changes. I cannot find any contact about how to contribute to the project, and the phar extension is built in since php 5.3, but has a pecl registration as well: http://pecl.php.net/package/phar According to the pecl page: Marcus Börger helly@php.net who you should ask about this. (If that mail address still exists. Nobody touched the extension since 2009-07-29. Maybe you should ask too, why these extensions aren't maintained properly.)

jameshalsall · 2015-07-24T11:02:44Z

@lserni thanks for that, works a charm 👍

vinkla · 2015-12-21T08:06:45Z

I'm having the same issue. When I run box build after adding compression to the JSON config I get this error thrown. I'm building the phar with PHP 7.0 and it works fine without the compression.

$ box build
Building...
PHP Fatal error:  Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)
#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()
#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)
#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()
#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Below is the content of the box.json file.

{
    "chmod": "0755",
    "directories": [
        "src"
    ],
    "files": [
        "LICENSE",
        "./vendor/guzzle/guzzle/src/Guzzle/Http/Resources/cacert.pem"
    ],
    "finder": [
        {
            "name": "*.php",
            "exclude": ["Tests", "tests"],
            "in": "vendor"
        }
    ],
    "git-version": "package_version",
    "main": "bin/climb",
    "output": "climb.phar",
    "compression": "GZ",
    "stub": true
}

lserni · 2015-12-21T16:14:23Z

Did you try with my "dirty fix", below? (If you did and it did not work,
you may need to use
sudo).

(Also: if you get the same error but some time later, then it means that
even 4096 files are
not enough. You need to increase again the value).

I did write the patch to avoid using temp files while building the phar,
but never got around
to test it to any serious extent.

> But a quick fix would be to simply increase the open file limit before
making the PHAR file:

ulimit -Sn 4096

On Mon, Dec 21, 2015 at 9:06 AM, Vincent Klaiber notifications@github.com
wrote:

I'm having the same issue. When I run box build after adding compression
to the JSON config I get this error thrown. I'm building the phar with PHP
7.0 and it works fine without the compression.

$ box build
Building...
PHP Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:#0 [internal function]: KevinGH\Box\Application->KevinGH\Box{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Fatal error: Uncaught ErrorException: proc_open(): unable to create pipe Too many open files in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php:978
Stack trace:#0 [internal function]: KevinGH\Box\Application->KevinGH\Box{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 978, Array)#1 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(978): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)#2 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(788): Symfony\Component\Console\Application->getSttyColumns()#3 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(749): Symfony\Component\Console\Application->getTerminalDimensions()#4 phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php(679): Symfony\Component\Console\Application->getTerminalWidth()#5 phar in phar:///usr/local/Cellar/box/2.5.3/libexec/box-2.5.3.phar/src/vendors/symfony/console/Application.php on line 978

Below is the content of the box.json file.

{
"chmod": "0755",
"directories": [
"src"
],
"files": [
"LICENSE",
"./vendor/guzzle/guzzle/src/Guzzle/Http/Resources/cacert.pem"
],
"finder": [
{
"name": "*.php",
"exclude": ["Tests", "tests"],
"in": "vendor"
}
],
"git-version": "package_version",
"main": "bin/climb",
"output": "climb.phar",
"compression": "GZ",
"stub": true
}

—
Reply to this email directly or view it on GitHub
#80 (comment).

…-project#80)

Displayed message: Phar build failed. If the error is "", you can try to increase the open file limit of your system: ulimit -Sn 4096 See box-project/box2#80 (comment)

The phar extension opens as many file descriptors as files it needs to compress during Phar::compressFiles(). See box-project/box2#80 (comment).

theofidry · 2018-04-08T19:10:58Z

If someone is still having that issue, please give a try to https://github.com/humbug/box. I couldn't reproduce that issue with it but I'm not 100% this is fixed so a reproducer is welcomed :)

lserni · 2018-04-08T22:19:33Z

I am not certain this fix actually addresses the problem - the temporary file design flaw is within the PHP code, so it will happen whenever you create a compressed PHAR with more than MAX_OPEN_FILES files inside (this is around 1024 or 4096 depending on systems). If you did set the ulimit to a value N and succeeded in creating a PHAR with N files then, unless you assemble the PHAR yourself bypassing the PHP generator, the problem is solved in the PHP version you're using. Cheers, Leonardo

…

On Sun, Apr 8, 2018 at 9:11 PM, Théo FIDRY ***@***.***> wrote: If someone is still having that issue, please give a try to https://github.com/humbug/box. I couldn't reproduce that issue with it but I'm not 100% this is fixed so a reproducer is welcomed :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGI6ngiBTO04d8b4cvw8q_L4TgP6Qu8fks5tmmDEgaJpZM4B1Fcn> .

theofidry · 2018-04-09T07:21:12Z

compressed PHAR with more than MAX_OPEN_FILES files inside

And how do you achieve this? Isn't creating a PHAR with N files and compressing the whole PHAR supposed to be the issue?

lserni · 2018-04-09T14:58:23Z

On Mon, Apr 9, 2018 at 9:21 AM, Théo FIDRY ***@***.***> wrote: compressed PHAR with more than MAX_OPEN_FILES files inside And how do you achieve this? Isn't creating a PHAR with N files and compressing the whole PHAR supposed to be the issue?

Yes, it is. I'm sorry, I have probably expressed myself badly. As far as I can see, your commit modifies the list of the files that will be included in the PHAR being built, is it not so? If that is the case, I believe this can't possibly fix, alone, the issue at hand, for it does not change the problem (the open files table in the PHP binary code). Nothing that we can do in a ".php" file can fix the problem (except manually compressing data a file at a time, which allows us to never invoke the problematic function in PHP but is very slow in comparison). Based on a VERY cursory examination of the new PHP code, *I believe that the issue has* *indeed been fixed*. I see several references to something called "PHAR_TMP" handles; I strongly suspect that phar code now uses some kind of indirect file mapping, to avoid hogging file handles, not unlike the fix I attempted, and failed, to make back in the day. https://github.com/php/php-src/blob/master/ext/phar/phar.c /* first, iterate over the manifest and close all PHAR_TMP entry fp handles, this prevents unnecessary unfreed stream resources */ Of course, it's also completely possible that I'm disastrously mistaken and am seeing things. It's been a while since I last waded inside PHP innards. Cheers, Leonardo

theofidry · 2018-04-09T15:05:47Z

If that just got fixed it's nice then :p I just try to reproduce it by creating a PHAR containing 100K files and compressing it whole but it didn't fail... So I wondered if I fixed that magically at some point or if that got fixed or if I did it wrong

ghost pushed a commit that referenced this issue Sep 5, 2014

Manual GC after opening 100 files (#80).

f895e5e

kherge added the PHP Bug label Mar 4, 2015

mbrodala mentioned this issue Dec 16, 2015

Enable gzip compression of .phar vinkla/climb#49

Merged

This was referenced Dec 18, 2015

Compression fails #124

Closed

'proc_open(): unable to create pipe Too many open files #126

Closed

remisauvat pushed a commit to remisauvat/box2 that referenced this issue Apr 7, 2016

Compress files on add instead of using Phar::compressFiles (fixes box…

3a4a3bd

…-project#80)

rjkip pushed a commit to ibuildingsnl/qa-tools that referenced this issue Feb 9, 2017

Increase open file limit during automatic build

f6b73d8

The phar extension opens as many file descriptors as files it needs to compress during Phar::compressFiles(). See box-project/box2#80 (comment).

allejo mentioned this issue Oct 1, 2019

Failed to open stream and cannot open phar archive box-project/box#433

Closed

cmb69 mentioned this issue Jan 26, 2021

Fix #53467: Phar cannot compress large archives php/php-src#6643

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error building phar with compression enabled #80

Error building phar with compression enabled #80

localgod commented Apr 24, 2014

ghost commented Apr 24, 2014

ghost commented May 17, 2014

ghost commented Sep 5, 2014

localgod commented Sep 6, 2014

ghost commented Sep 6, 2014

kingcrunch commented Feb 5, 2015

kherge commented Feb 5, 2015

ghost commented Feb 5, 2015

lserni commented Mar 1, 2015

kherge commented Mar 1, 2015

lserni commented Mar 4, 2015

kherge commented Mar 4, 2015

ghost commented Mar 4, 2015

kherge commented Mar 4, 2015

ghost commented Mar 4, 2015

lserni commented Mar 5, 2015

ghost commented Mar 5, 2015

jameshalsall commented Jul 24, 2015

vinkla commented Dec 21, 2015

lserni commented Dec 21, 2015

theofidry commented Apr 8, 2018

lserni commented Apr 8, 2018 via email

theofidry commented Apr 9, 2018

lserni commented Apr 9, 2018 via email

theofidry commented Apr 9, 2018

Error building phar with compression enabled #80

Error building phar with compression enabled #80

Comments

localgod commented Apr 24, 2014

ghost commented Apr 24, 2014

ghost commented May 17, 2014

ghost commented Sep 5, 2014

localgod commented Sep 6, 2014

ghost commented Sep 6, 2014

kingcrunch commented Feb 5, 2015

kherge commented Feb 5, 2015

ghost commented Feb 5, 2015

lserni commented Mar 1, 2015

kherge commented Mar 1, 2015

lserni commented Mar 4, 2015

kherge commented Mar 4, 2015

ghost commented Mar 4, 2015

kherge commented Mar 4, 2015

ghost commented Mar 4, 2015

lserni commented Mar 5, 2015

ghost commented Mar 5, 2015

jameshalsall commented Jul 24, 2015

vinkla commented Dec 21, 2015

lserni commented Dec 21, 2015

theofidry commented Apr 8, 2018

lserni commented Apr 8, 2018 via email

theofidry commented Apr 9, 2018

lserni commented Apr 9, 2018 via email

theofidry commented Apr 9, 2018