'ipfs add -r' (on windows) running out of memory for folder containing 10'000 files of 100KB

[SCBuergel]

I am adding a large folder [from a network drive with 15'000 files at a total of 700MB] and I am running out of disk space. It is clear that this does not work but the way it crashes here could be improved. I post the complete error message and output below [using ipfs version 0.3.9]

---

fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw(0xf38b40, 0x30)
/usr/local/go/src/runtime/panic.go:527 +0x97
runtime.sysMap(0xc157300000, 0x100000, 0x1305c01, 0x1325e98)
/usr/local/go/src/runtime/mem_windows.go:112 +0x90
runtime.mHeap_SysAlloc(0x1305c60, 0x100000, 0x12fe1a0)
/usr/local/go/src/runtime/malloc.go:423 +0x167
runtime.mHeap_Grow(0x1305c60, 0x8, 0x0)
/usr/local/go/src/runtime/mheap.go:628 +0x6a
runtime.mHeap_AllocSpanLocked(0x1305c60, 0x7, 0x100)
/usr/local/go/src/runtime/mheap.go:532 +0x5f8
runtime.mHeap_Alloc_m(0x1305c60, 0x7, 0x100000000, 0x41e356)
/usr/local/go/src/runtime/mheap.go:425 +0x1ba
runtime.mHeap_Alloc.func1()
/usr/local/go/src/runtime/mheap.go:484 +0x48
runtime.systemstack(0x6fe50)
/usr/local/go/src/runtime/asm_amd64.s:278 +0xb5
runtime.mHeap_Alloc(0x1305c60, 0x7, 0x10100000000, 0xc08201aa00)
/usr/local/go/src/runtime/mheap.go:485 +0x6a
runtime.largeAlloc(0xd720, 0xc000000000, 0x4622d7)
/usr/local/go/src/runtime/malloc.go:745 +0xba
runtime.mallocgc.func3()
/usr/local/go/src/runtime/malloc.go:634 +0x3a
runtime.systemstack(0x12fe1e0)
/usr/local/go/src/runtime/asm_amd64.s:262 +0x7e
runtime.mstart()
/usr/local/go/src/runtime/proc1.go:674

goroutine 37 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc082405d80 sp=0xc082405d7
8
runtime.mallocgc(0xd720, 0xc9c740, 0x0, 0x730639)
/usr/local/go/src/runtime/malloc.go:635 +0xa03 fp=0xc082405e50 sp=0xc082
405d80
runtime.newarray(0xc9c740, 0x1ae4, 0xc1571598c0)
/usr/local/go/src/runtime/malloc.go:777 +0xd0 fp=0xc082405e90 sp=0xc0824
05e50
runtime.makeslice(0xb3d6a0, 0x0, 0x1ae4, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/slice.go:32 +0x16c fp=0xc082405ee0 sp=0xc08240
5e90
github.com/ipfs/go-ipfs/merkledag.(_Node).RemoveNodeLink(0xc082101560, 0xc152efd
b00, 0x75, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/merkledag/node.go:132 +0x70 fp=0xc082405
f90 sp=0xc082405ee0
github.com/ipfs/go-ipfs/merkledag/utils.addLink(0x3c30108, 0xc0825e5d00, 0x3c33c
20, 0xc0820cc260, 0xc082101560, 0xc152efdb00, 0x75, 0xc15236bc80, 0xc154a81be0,
0x0, ...)
/go/src/github.com/ipfs/go-ipfs/merkledag/utils/utils.go:44 +0x1d0 fp=0x
c082406020 sp=0xc082405f90
github.com/ipfs/go-ipfs/merkledag/utils.insertNodeAtPath(0x3c30108, 0xc0825e5d00
, 0x3c33c20, 0xc0820cc260, 0xc082101560, 0xc154a81be0, 0x1, 0x1, 0xc15236bc80, 0
xfa3988, ...)
/go/src/github.com/ipfs/go-ipfs/merkledag/utils/utils.go:68 +0xbd fp=0xc
0824060d0 sp=0xc082406020
github.com/ipfs/go-ipfs/merkledag/utils.(_Editor).InsertNodeAtPath(0xc0835cafc0,
0x3c30108, 0xc0825e5d00, 0xc152efdb00, 0x75, 0xc15236bc80, 0xfa3988, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/merkledag/utils/utils.go:58 +0x107 fp=0x
c082406168 sp=0xc0824060d0
github.com/ipfs/go-ipfs/core/commands.(_adder).addNode(0xc0835cc690, 0xc15236bc8
0, 0xc152efdb00, 0x75, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:365 +0x115 fp=0xc08
24061e8 sp=0xc082406168
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3adb860,
0xc1571f2f40, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:415 +0x887 fp=0xc08
2406318 sp=0xc0824061e8
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc15677e5a0, 0x5d, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2406458 sp=0xc082406318
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc15677e5a0, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2406588 sp=0xc082406458
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc154555130, 0x53, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
24066c8 sp=0xc082406588
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc154555130, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
24067f8 sp=0xc0824066c8
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc154555090, 0x4e, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2406938 sp=0xc0824067f8
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc154555090, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2406a68 sp=0xc082406938
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc14908e370, 0x44, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2406ba8 sp=0xc082406a68
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc14908e370, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2406cd8 sp=0xc082406ba8
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc13cec8a50, 0x3d, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2406e18 sp=0xc082406cd8
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc13cec8a50, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2406f48 sp=0xc082406e18
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc0835d6b40, 0x34, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2407088 sp=0xc082406f48
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc0835d6b40, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
24071b8 sp=0xc082407088
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc09a521cc0, 0x2b, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
24072f8 sp=0xc0824071b8
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc09a521cc0, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2407428 sp=0xc0824072f8
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc09c6fe050, 0x22, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2407568 sp=0xc082407428
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc09c6fe050, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2407698 sp=0xc082407568
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc087768af0, 0x1b, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
24077d8 sp=0xc082407698
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc087768af0, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2407908 sp=0xc0824077d8
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc0d5627e50, 0x10, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2407a48 sp=0xc082407908
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc0d5627e50, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2407b78 sp=0xc082407a48
github.com/ipfs/go-ipfs/core/commands.(_adder).addDir(0xc0835cc690, 0x3c30038, 0
xc08225e000, 0x1, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:432 +0x4a5 fp=0xc08
2407cb8 sp=0xc082407b78
github.com/ipfs/go-ipfs/core/commands.(_adder).addFile(0xc0835cc690, 0x3c30038,
0xc08225e000, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:382 +0x2ea fp=0xc08
2407de8 sp=0xc082407cb8
github.com/ipfs/go-ipfs/core/commands.glob.func2.1(0x3c30088, 0xc082104040, 0x0,
0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:152 +0x104 fp=0xc08
2407e58 sp=0xc082407de8
github.com/ipfs/go-ipfs/core/commands.glob.func2.3(0x3c30088, 0xc082104040, 0x0,
0x0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:171 +0x89 fp=0xc082
407f18 sp=0xc082407e58
github.com/ipfs/go-ipfs/core/commands.glob.func2.4(0xc0821015c0, 0xc0835ce000, 0
x3ad1120, 0xc0821263c0, 0x3ad94c0, 0xc0821547e0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:194 +0x84 fp=0xc082
407f60 sp=0xc082407f18
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc082407f68 sp=0xc08
2407f60
created by github.com/ipfs/go-ipfs/core/commands.glob.func2
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:199 +0x834

goroutine 1 [chan receive]:
github.com/ipfs/go-ipfs/core/commands.glob.func3(0x3ad1120, 0xc0821263c0, 0x3ad9
4c0, 0xc0821547e0)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:247 +0x4b0
main.callCommand(0x3c30108, 0xc0821040c0, 0x3ad1120, 0xc0821263c0, 0x12fdb20, 0x
12ede60, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:346 +0x748
main.(*cmdInvocation).Run(0xc0820d1a40, 0x3c30108, 0xc0821040c0, 0x0, 0x0, 0x0,
0x0)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:184 +0x1ec
main.main()
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:149 +0x662

goroutine 17 [chan receive]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics.init.1
.func2()
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahal
e/metrics/metrics.go:321 +0x7a
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/met
rics.init.1
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahal
e/metrics/metrics.go:328 +0x82

goroutine 18 [syscall, 46 minutes]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x1f
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x3e

goroutine 34 [chan receive, 46 minutes]:
main.(_IntrHandler).Handle.func1(0xc082102180, 0xc082264080)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:540 +0xa1
created by main.(_IntrHandler).Handle
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:545 +0x9c

goroutine 23 [select]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b/util.(*BufferPool).drain(0xc08226e460)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/util/buffer_pool.go:205 +0x278
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb/util.NewBufferPool
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/util/buffer_pool.go:236 +0x272

goroutine 24 [select, 21 minutes]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b.(*DB).compactionError(0xc0823c5080)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db_compaction.go:120 +0x483
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb.openDB
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db.go:126 +0x803

goroutine 25 [select]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b.(*DB).mpoolDrain(0xc0823c5080)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db_state.go:82 +0x152
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb.openDB
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db.go:127 +0x825

goroutine 26 [select, 21 minutes]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b.(*DB).tCompaction(0xc0823c5080)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db_compaction.go:812 +0x7cf
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb.openDB
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db.go:130 +0x871

goroutine 27 [select, 21 minutes]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b.(*DB).mCompaction(0xc0823c5080)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db_compaction.go:759 +0x25a
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb.openDB
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db.go:131 +0x893

goroutine 28 [select, 21 minutes]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveld
b.(*DB).jWriter(0xc0823c5080)
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db_write.go:37 +0x1a9
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/golev
eldb/leveldb.openDB
/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/
goleveldb/leveldb/db.go:132 +0x8b5

[SCBuergel]

I am trying to get more helpful insights into this. When adding this huge folder again after it crashes I see how my IPFS process is accumulating more and more memory and eventually runs out of memory. Windows also displays a warning message accordingly.

This is my RAM utilization (the dip is when IPFS crashes):

...memory leak?

[djdv]

Prior to this commit e007d1e, ipfs add -r did not work as intended on Windows, it would add a lot of malformed listings and seemed to crawl the same directories multiple times.
Can you see if using the latest builds from master allow this transfer to complete, I noticed unusually high memory usage in 0.3.9 myself but have not had that issue since, I recently added several gigabytes in one pass without problems. The only other add problem on Windows should be this
#2135 right now.

[SCBuergel]

I am running the latest master version and did the following test with ipfs add folder -r - p on a set of files with variable amounts of random binary data:

• folder with 1000 files of 1MB -> all ok, ~150MB of memory usage
• folder with 10000 files of 100KB -> memory usage goes ballistic with the above described crash pattern

[whyrusleeping]

this is a known issue, it should be addressed in 0.4.0. If you don't mind, could you run the same tests there?

[SCBuergel]

Sure I can do the test once 0.4.0 is out (whats the order-of-magnitude time-frame?). If its a known issue lets link it here and close this one? I just didnt find it when I was looking for it.

[jbenet]

You can test with 0.4.0 now-- binaries here: https://gobuilder.me/github.com/ipfs/go-ipfs/cmd/ipfs?branch=v0.4.0-dev (just triggered a build)

…

--- @whyrusleeping you can trigger a gobuilder build by: ``` git pull --tags git tag -f v0.4.0-dev dev0.4.0 git push --tags -f gobuilder-cli --repo=github.com/ipfs/go-ipfs/cmd/ipfs@v0.4.0-dev build ```

[SCBuergel]

You guys are great! Resolved in current build of 0.4.0 (70MB max ram usage). Theres a tiny display problem for which I opened another issue but this is resolved I guess.

[jbenet]

Would be great to see the graph again with dev0.4.0

[SCBuergel]

The problem is that the 70MB of memory utilization is below the resolution of the magnificent windows task manager, the bumps that you do see are most likely noise from other processes. I'd call that good enough!

[jbenet]

great work @whyrusleeping :)