add uses a lot of ram and cpu

[whyrusleeping]

an add operation on one of my servers (single core, 1GB ram) is taking all the CPU up, and as much ram as it can eat. watching memory usage from expvarmon, i see it allocating large amounts of memory, then running a GC, over and over again.

I suspect some of this is eventlog, i saw similar behaviour in bitswap due to eventlog, i will take a look at some profiles and try to get more info.

[whyrusleeping]

the RAM usage is primarily because we allocate memory for ever file read in, and while it does get 'released' the gc is slower than the allocator, causing large portions of your available RAM to be eaten. This can be helped by having a buffer pool like we do in go-msgio.

I havent noticed much CPU usage while testing, but as we get closer to the ~200MB/s limit of sha256 hashing, i beleive we will start cranking that.

The primary bottleneck right now as far as time though, is the datastore, we do directory syncs for every write and this slows us down a lot. If we could batch together multiple writes, it would save us a lot of time.

[davidar]

This bug is particularly bad on a smaller VPS, where ipfs add tends to trigger the OOM killer almost immediately.

[whyrusleeping]

Add has been improved significantly since this. I'm gonna call it closed, open new issues for new add perf issues