Out of Memory Error (Possible Memory Leak in Scala?)

[jonfroehlich]

The server went down last night with an out-of-memory error. See below for details. This is a task for @Manaswis or @kotarohara (if he has time since Manaswi is traveling until late Jan).

---

It looks like your application ran out of memory a few times last night, which may be a contributing factor. [0] We can increase the available memory on the host, but will need to reboot the machine for that change to take effect. I see you currently are logged into sidewalk, so let us know when a good time to do this would be.

--Shawn

[0]

2017-01-05 20:22:24,655 - [ERROR] - from akka.actor.ActorSystemImpl in play-akka.actor.default-dispatcher-20538
Uncaught error from thread [play-akka.actor.default-dispatcher-20539] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

--

2017-01-10 00:15:04,271 - [ERROR] - from akka.actor.ActorSystemImpl in play-akka.actor.default-dispatcher-12229
Uncaught error from thread [play-akka.actor.default-dispatcher-12232] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

--

2017-01-10 00:24:33,507 - [ERROR] - from akka.actor.ActorSystemImpl in play-akka.actor.default-dispatcher-13
Uncaught error from thread [play-akka.actor.default-dispatcher-9] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

--

2017-01-10 00:26:08,005 - [ERROR] - from akka.actor.ActorSystemImpl in play-akka.actor.default-dispatcher-4
Uncaught error from thread [play-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

[root@sidewalk sidewalk]# free -m
total used free shared buffers cached
Mem: 2887 2809 78 0 3 1547
-/+ buffers/cache: 1257 1629
Swap: 607 66 541

[manaswisaha]

This happened again today:

[ERROR] [03/25/2017 09:28:16.476] [play-akka.actor.default-dispatcher-1977] [ActorSystem(play)] Uncaught error from thread [play-akka.actor.default-dispatcher-1977] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

[ [31merror [0m] a.a.ActorSystemImpl - Uncaught error from thread [play-akka.actor.default-dispatcher-1977] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space

Would have to figure out a solution soon.

[manaswisaha]

Increased the heap size to solve the problem
"-Xmx3072M", "-Xms2048M", "-XX:MaxPermSize=3072M"

Will have to see if that helps.

[jonfroehlich]

So, this seems reasonable and I think the previous settings of 1GB for heap size were likely too small; however, the difference between memory usage on deployment server vs. dev server is not accounted for. That is, we don't seem to actually know if there is an underlying problem here. So while increasing heap size may help address receiving this error (or the frequency of receiving this error), I still need we need to understand root cause.

I should note that if the problem frequency is reduced significantly with this "fix", then finding root cause does diminish in importance (e.g., you should focus on other TODOs and loop back to this after end of semester).

[manaswisaha]

Temp fix in #629

[misaugstad]

This has no longer been an issue since it was related to using the audit_task_interaction table. The queries to too long and large to be used in the actual tool. That table is going to be used only for data analysis from now on. We have not come across this problem since making that change, so I'm closing this.