"too many open files" in add-dir

[jefft0]

go-ipfs version: 0.4.5-dev-8566ce2
Repo version: 4
System version: amd64/linux
Golang version: go1.7.3

Bug

Description:
I'm now running on Ubuntu 16.04 (not Windows) but I still have my giant store of webcam videos. In a fresh IPFS installation, I tried the add-dir script and got the following errors. Any quick idea whether the Python script is leaving files or subprocesses open?

checking for files to add...  
adding 152029 files...
Error: open /public/cameras/2016/201607/20160725/camera3.20160725_150000.mp4: too many open files
added 1890 files,  150139 more to go.
Error: open /public/cameras/2016/201609/20160907/camera4.20160907_080000.mp4: too many open files
added 1890 files,  148249 more to go.

plus many more similar

[kevina]

I don't think the Python script is keeping subprocesses open ipfs is known for keeping lots of files open at once.

The easiest fix is to increase the number of file descriptors. See if you can increase it with

ulimit -n 65536

If you get an error try a lowering the number to what is outputted to

cat /proc/sys/fs/file-max

Also google "increase number of open files linux" for additional advice.

The other thing to try is to decrease MAX_CMD_LEN in the script. You might need to use "1890". This limits the number of files the script tries to add at once.

Finally note the very latest git version of master will likely break this script so I would stick will the version you are using for now.

[jefft0]

Thanks for the quick feedback. cat /proc/sys/fs/file-max already says 1579062, so I tried your other suggestion to set MAX_CMD_LEN to 1890. Now it's cranking. I'll let you know when it's done. (Could be days.)

[jefft0]

The script finished adding the 152029 files after about 56 hours. So decreasing MAX_CMD_LEN seems to be the solution. Is it worth making this a command line argument for the script, or should I just keep modifying the code for my use case (lots of files)?