Daemon crashes on EC2 instance after a few days

[ianopolous]

I have an IPFS daemon running on an EC2 instance, (https://demo.peergos.net) and it keeps crashing. This is some of the latest goroutine dump upon crash:
ipfs,dump.txt

It is built from source from master. It has been happening since 0.4.0. The current git hash of the version that caused this dump is 79e9bd8...

[whyrusleeping]

@ianopolous how is it crashing? the stack trace you provided doesnt indicate a panic or OOM issue.

[ianopolous]

All I know is it terminates, and that was the last part of the goroutine dump. I've started it again to capture the entire output, but it will take a few days to repeat at least. I would be surprised if it was an OOM issue, but we'll see.

[ianopolous]

Turns out it was an OOM error. The machine has about 800 MiB of RAM free and is under extremely low usage (was uploading a 2MiB file). Do you want the full goroutine dump?

[ianopolous]

@whyrusleeping It has crashed again, but this time not an OOM,
runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailable
pthread_create failed: Resource temporarily unavailable
SIGABRT: abort

[ianopolous]

It just crashed again, still under almost no load, this time another OOM.

[Kubuxu]

We are working on it: #2848 #2848

[ianopolous]

This is still happening, though it took a week this time. This is on the master branch, at commit: 4fa85cc from July 1st.
ipfs.oom.log.txt

[jbenet]

Try rc3?
On Fri, Jul 15, 2016 at 03:43 Ian Preston notifications@github.com wrote:

This is still happening, though it took a week this time. This is on the
master branch, at commit: 4fa85cc
4fa85cc
from July 1st.
ipfs.oom.log.txt
https://github.com/ipfs/go-ipfs/files/365420/ipfs.oom.log.txt

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2864 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcobZecStblik6Vqkx46HW2JBSv2wAks5qVzoNgaJpZM4I4sGn
.

[ianopolous]

I'm not in a position where I can upgrade it at the moment as we're half way through rewriting our front end. I'm happy to close it for now, as I don't notice it, after putting it in upstart. We've also moved from EC2 to Linode.