Fix crashes

[saiichihashimoto]

We occasionally crash. What's going on?

[saiichihashimoto]

Adding loader.io into our workflow. Ran an initial test which loaded the homepage and made a zip with 3 random images 10,000 times over a minute. This triggered the error. Now we can start debugging properly.

[saiichihashimoto]

Converting the logos now caches in a CDN AND in redis and keeps nothing in memory once its done serving something.

Meanwhile, loading JUST the homepage enough triggers the crash. We're not just using a lot of memory with these conversions, there's got to be a legit memory leak somewhere.

[saiichihashimoto]

I'm at a loss about these damn things. Luckily, memory and heap profiling tools exist. I'm going to put them in place, push to master, run the load tests again, grab the heapdump and examine what's going on.

[saiichihashimoto]

Really hungry and upset and going to go eat, but it looks like this github issue might point me in the right direction.

[saiichihashimoto]

I'm super lost, so I'm just going to start fixing things that could fix it or will point me in the right direction. Right now, I'm integrating rollbar which just happens to give me everything I've ever wanted logging to give me, so I'll be using it with everything now.

[saiichihashimoto]

So I've got rollbar set up and I think it'll help with figuring things out. One thing I can't figure out is having it capture Heroku platform errors, which is where our issue currently is being reported. When there's not a lot of concurrent requests, it does just fine and doesn't time out or anything. It might very well not be a memory issue but simply hitting the limits of the Heroku hobby dyno. It might be worth it to store the logos themselves somewhere else, but the CDN should be taking care of that.

I'm going to leave this open but, for now, move on. I need more data to figure anything out. I've set up some stuff to give me data so I've done what I can.

[saiichihashimoto]

We seemed to have stopped crashing. This is probably because we have less people on the site, but there's one other thing: this issue sucks. It addresses a concern, but nothing is actually identified. Just chaotically trying to fix things. There's no real good way of determining if its solved or what solving it actually means. With rollbar creating github issues for us on errors (and we'll get everything once we complete #51), we can actually address specific issues rather than this catch all "fix things".

I'm closing this.