Error: EMFILE, too many open files

[daaain]

I'm working on a project with quite a good number of files (298 as picked up by Istanbul to be precise) and it doesn't seem to be able to deal with them :(

Setup is OS X 10.7.5, Node v0.10.0, Istanbul 0.1.32 and using via Guard-Jasmine 1.13.2.

fs.js:413
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: EMFILE, too many open files 'path-removed.js'
    at Object.fs.openSync (fs.js:413:18)
    at Object.fs.readFileSync (fs.js:270:15)
    at LookupStore.Store.mix.get (/usr/local/share/npm/lib/node_modules/istanbul/lib/store/fslookup.js:39:19)
    at HtmlReport.Report.mix.writeDetailPage (/usr/local/share/npm/lib/node_modules/istanbul/lib/report/html.js:358:71)
    at /usr/local/share/npm/lib/node_modules/istanbul/lib/report/html.js:435:26
    at Object.FileWriter.writeFile (/usr/local/share/npm/lib/node_modules/istanbul/lib/util/file-writer.js:24:9)
    at /usr/local/share/npm/lib/node_modules/istanbul/lib/report/html.js:434:24
    at Array.forEach (native)
    at HtmlReport.Report.mix.writeFiles (/usr/local/share/npm/lib/node_modules/istanbul/lib/report/html.js:428:23)
    at /usr/local/share/npm/lib/node_modules/istanbul/lib/report/html.js:430:22

[gotwarlost]

Could you give me the following info?

• What is the high-level use case? What specific command are you running? Is it istanbul cover ?
• Can you turn off reporting as part of running your tests and use the istanbul report command to produce the reports as a separate command? (see istanbul help for details). I want to make sure that it is not the test runner that is opening too many files and istanbul just happens to tip the # of open files over the limit.
• Could you run the original command in node 0.8 and see if that makes any difference? Want to make sure it is not because of the node version.

Thanks for reporting the issue.

[daaain]

My high-level use case is manually generating instrumented JS files with istanbul and then running Jasmine tests using Guard-Jasmine which will then feeds istanbul the results somehow and generates reports. I tried to read the code (relevant commit), but it's a bit too much for me to understand so early in the morning :)

Tried to run istanbul cover without Guard-Jasmine, but got stuck with how to make it run Jasmine tests, I'm not using node at all just plain client side JS. It can very well be the case that there's something with the way Guard-Jasmine is using istanbul, if you think that's the case I can open an issue there.

It's also happening with node 0.8.22.

Edit: Actually, there's one more piece to the puzzle I just realised, I do get a report on the CLI from Guard-Jasmine:

Spec coverage summary:

Statements   : 37.97% ( 6903/18178 )
Branches     : 26.18% ( 2210/8443 )
Functions    : 28.97% ( 1123/3876 )
Lines        : 38.04% ( 6899/18135 )

Which implies that the coverage bit is completely fine, it's the HTML report generation which seems to break. This also lines up with what's in the stack trace, but thought I should mention.

[gotwarlost]

This is good information - it proves that the reporter is opening too many files at the same time and it is not a side effect of the test runner.

I'm totally tied up with my day job right now so may not be able to get to a fix soon.

This is how I think you can get unblocked:

• Fork this repo
• Change this line: https://github.com/gotwarlost/istanbul/blob/master/lib/command/report.js#L88 to read

reporter.writeReport(collector, true); /* sync mode */

• Use your fork in your package.json

Let me know if this unblocks you. I will get to a permanent fix when my current day-job deadlines are met.

[gotwarlost]

Yeah, I can reproduce it and, yeah, that fix should work.

I don't know what I was smoking when I wrote the async version of the reporter but it will take some time to untangle :)

Let me know if this is a workable solution for you for the time being.

[daaain]

That worked perfectly, thanks 👍 Glad I could help with the report :)

It's quite fast as well, so this is not a bad fix :)

I don't even have a package.json (it's a pure client side project with some Ruby gems for compilation sprinkled on top), but a little bit of Googling gave me the quick and dirty way: npm install git://github.com/daaain/istanbul.git -g

Edit: Oh yeah, and by the way it works with node 0.10.0!

[gotwarlost]

Cool. I'll roll in a fix in the next week or two.

[gotwarlost]

@daaain - Fix is available in v0.1.34.

Please try it and let me know if it looks ok. Thanks!

[daaain]

It works perfectly (tested with node 0.10.2), thanks a lot for the fix!