Memory issues

[RobbieTheWagner]

I am getting a bunch of critical memory errors with my app. Perhaps workers are being spawned infinitely until all the memory is eaten up?

[Dhaulagiri]

/cc @hone for any insights here. @rwwagner90 if you have any more detail you could about your setup or the specific errors you're seeing that might be helpful.

[RobbieTheWagner]

@Dhaulagiri I am not sure how to see more details about the errors. I just seem to have 500+ "critical memory errors" per day. I would be glad to provide more details, if someone could assist me with where to look for those details?

[hone]

sorry for the delay here. Are you using fastboot or a typical ember app?
If you're using the static buildpack, the number of workers should be
set. The static buildpack sets it to "auto":
https://github.com/heroku/heroku-buildpack-static/blob/master/scripts/config/templates/nginx.conf.erb.
This means nginx will try to autodetect the appropriate amount which MAY
but culprit, but I believe highly unlikely. If you're using fastboot,
you'll probably need to do some node debugging for memory.

https://devcenter.heroku.com/articles/log-runtime-metrics this might
help for watching memory growth on the dyno.

On 12/09/16 14:16, Robert Wagner wrote:

@Dhaulagiri https://github.com/Dhaulagiri I am not sure how to see
more details about the errors. I just seem to have 500+ "critical memory
errors" per day. I would be glad to provide more details, if someone
could assist me with where to look for those details?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABASQq1VgoXD7R0z8RRDX9pf1HmsMR9ks5qpaT8gaJpZM4J3iXQ.

[RobbieTheWagner]

@hone I am using fastboot. Please let me know what info you need from me, so we can resolve this. I had some other people suggest limiting the workers to 4 for fastboot, I believe, so that leads me to believe we may need to do that.

Is this buildpack using fastboot app server or what exactly? It seems like it is just serving from fastboot directly, and not using fastboot app server since this is what it runs: ember-fastboot dist --serve-assets-from dist --port $PORT

Shouldn't we be using this? https://github.com/ember-fastboot/fastboot-app-server

[hone]

We are using fastboot-app-server but through fastboot-cli:
heroku/heroku-buildpack-ember-cli-deploy@137c9c4.
fastboot-app-server bare requires you to roll your own server.js to run
it. You can of course do all of this yourself if you need be on heroku,
by just overriding the web process in your Procfile.

fastboot app server is using express-cluster under the hood and by
default it uses the number of CPUs:
https://github.com/Flipboard/express-cluster/blob/master/lib/index.js#L12.
On a standard-1X dyno, that should be 8. Cutting it down by half should
probably help. The fastboot apps I've been working with have been pretty
low memory, so I haven't quite run into that issue. You can adjust the
number of workers by just setting the WORKER_COUNT env var.

$ heroku config:set WORKER_COUNT=4

I think it might be helpful to see how much memory using the runtime
metrics feature flag a single worker uses and then figure out what a
good worker count is from there.

On 29/09/16 07:04, Robert Wagner wrote:

@hone https://github.com/hone I am using fastboot. Please let me know
what info you need from me, so we can resolve this. I had some other
people suggest limiting the workers to 4 for fastboot, I believe, so
that leads me to believe we may need to do that.

Is this buildpack using fastboot app server or what exactly? It seems
like it is just serving from fastboot directly, and not using fastboot
app server since this is what it runs: |ember-fastboot dist
--serve-assets-from dist --port $PORT|

Shouldn't we be using this?
https://github.com/ember-fastboot/fastboot-app-server

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#12 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABASTC84kaqw3G1Rq2WXjqyMx-nRaw2ks5qu6lPgaJpZM4J3iXQ.

[RobbieTheWagner]

@hone I had previously set NGINX_WORKERS = 4. It seems that I got that wrong, and it is WORKER_COUNT according to your suggestion. We should probably list this tip somewhere, so people know to tweak it.

[hone]

https://github.com/heroku/heroku-buildpack-emberjs/blob/master/buildpack/mrblib/buildpack/commands/detect.rb#L14-L22 This is the code that does the detect, it just looks for `ember-cli` in your `package.json`. The WORKER_COUNT is a good idea. It'd probably be nice if it was standardized across the two somehow, unless I start fronting the fastboot stuff with nginx. For more memory stuff, this trace addon recommended by @hunterloftis (our node language owner at Heroku) might help with some visibility: https://elements.heroku.com/addons/trace

[sgasser]

Same problem.
@rwwagner90 did you solve the problem?

Error R14 (Memory quota exceeded):

2016-10-06T21:34:35.483247+00:00 app[web.1]: 2016-10-06T21:34:35.483Z 200 OK /
2016-10-06T21:34:35.517804+00:00 heroku[router]: at=info method=GET path="/" host=hotel4conf.com request_id=fcf6c30a-7877-49dd-a9b1-aaa916330398 fwd="50.116.25.232" dyno=web.1 connect=0ms service=298ms status=200 bytes=32618
2016-10-06T21:34:39.191305+00:00 heroku[web.1]: source=web.1 dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51 sample#load_avg_1m=0.26 sample#load_avg_5m=0.21
2016-10-06T21:34:39.191457+00:00 heroku[web.1]: source=web.1 dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51 sample#memory_total=619.16MB sample#memory_rss=495.09MB sample#memory_cache=0.00MB sample#memory_swap=124.07MB sample#memory_pgpgin=667664pages sample#memory_pgpgout=540920pages sample#memory_quota=512.00MB
2016-10-06T21:34:39.191994+00:00 heroku[web.1]: Process running mem=619M(120.9%)
2016-10-06T21:34:39.191994+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2016-10-06T21:34:44.857355+00:00 app[web.1]: 2016-10-06T21:34:44.857Z 200 OK /
2016-10-06T21:34:44.887193+00:00 heroku[router]: at=info method=GET path="/" host=hotel4conf.com request_id=4758ccc7-c816-41c9-a94c-6e8c3d8ba2ab fwd="52.8.5.138" dyno=web.1 connect=1ms service=546ms status=200 bytes=32618
2016-10-06T21:34:46.409451+00:00 app[web.1]: 2016-10-06T21:34:46.409Z 200 OK /
2016-10-06T21:34:46.410858+00:00 heroku[router]: at=info method=GET path="/" host=hotel4conf.com request_id=e6ee02f5-8f09-4a11-b482-706436b37d56 fwd="52.64.61.27" dyno=web.1 connect=0ms service=487ms status=200 bytes=32618
2016-10-06T21:34:59.381801+00:00 heroku[web.1]: source=web.1 dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51 sample#load_avg_1m=0.19 sample#load_avg_5m=0.20
2016-10-06T21:34:59.381907+00:00 heroku[web.1]: source=web.1 dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51 sample#memory_total=626.67MB sample#memory_rss=503.84MB sample#memory_cache=0.00MB sample#memory_swap=122.83MB sample#memory_pgpgin=677917pages sample#memory_pgpgout=548935pages sample#memory_quota=512.00MB
2016-10-06T21:34:59.382595+00:00 heroku[web.1]: Process running mem=626M(122.4%)
2016-10-06T21:34:59.382712+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

[hone]

I definitely recommend setting the Worker to 1 and see if you're still
getting R14 errors. V8 is not known to conserve memory. It'd be nice to
know if there are too many processes being spawned or if fastboot is
just that memory intensive.

This GC tuning might help as well,
https://blog.heroku.com/node-habits-2016#7-avoid-garbage. It will kill
the node process if it exceeds instead of swapping and Heroku will spin
a new one up. How long does it take to get to the R14 state?

The other option is to just use a higher memory dyno. 2X dynos have 1gb
vs the 1X which has 512mb.

On 06/10/16 16:35, Stefan Gasser wrote:

Same problem.
@rwwagner90 https://github.com/rwwagner90 did you solve the problem?

Error R14 (Memory quota exceeded):

|2016-10-06T21:34:35.483247+00:00 app[web.1]: 2016-10-06T21:34:35.483Z
200 OK / 2016-10-06T21:34:35.517804+00:00 heroku[router]: at=info
method=GET path="/" host=hotel4conf.com
request_id=fcf6c30a-7877-49dd-a9b1-aaa916330398 fwd="50.116.25.232"
dyno=web.1 connect=0ms service=298ms status=200 bytes=32618
2016-10-06T21:34:39.191305+00:00 heroku[web.1]: source=web.1
dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51
sample#load_avg_1m=0.26 sample#load_avg_5m=0.21
2016-10-06T21:34:39.191457+00:00 heroku[web.1]: source=web.1
dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51
sample#memory_total=619.16MB sample#memory_rss=495.09MB
sample#memory_cache=0.00MB sample#memory_swap=124.07MB
sample#memory_pgpgin=667664pages sample#memory_pgpgout=540920pages
sample#memory_quota=512.00MB 2016-10-06T21:34:39.191994+00:00
heroku[web.1]: Process running mem=619M(120.9%)
2016-10-06T21:34:39.191994+00:00 heroku[web.1]: Error R14 (Memory quota
exceeded) 2016-10-06T21:34:44.857355+00:00 app[web.1]:
2016-10-06T21:34:44.857Z 200 OK / 2016-10-06T21:34:44.887193+00:00
heroku[router]: at=info method=GET path="/" host=hotel4conf.com
request_id=4758ccc7-c816-41c9-a94c-6e8c3d8ba2ab fwd="52.8.5.138"
dyno=web.1 connect=1ms service=546ms status=200 bytes=32618
2016-10-06T21:34:46.409451+00:00 app[web.1]: 2016-10-06T21:34:46.409Z
200 OK / 2016-10-06T21:34:46.410858+00:00 heroku[router]: at=info
method=GET path="/" host=hotel4conf.com
request_id=e6ee02f5-8f09-4a11-b482-706436b37d56 fwd="52.64.61.27"
dyno=web.1 connect=0ms service=487ms status=200 bytes=32618
2016-10-06T21:34:59.381801+00:00 heroku[web.1]: source=web.1
dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51
sample#load_avg_1m=0.19 sample#load_avg_5m=0.20
2016-10-06T21:34:59.381907+00:00 heroku[web.1]: source=web.1
dyno=heroku.56700560.9ecec0e5-b72e-4561-8fd1-4097ca6f2f51
sample#memory_total=626.67MB sample#memory_rss=503.84MB
sample#memory_cache=0.00MB sample#memory_swap=122.83MB
sample#memory_pgpgin=677917pages sample#memory_pgpgout=548935pages
sample#memory_quota=512.00MB 2016-10-06T21:34:59.382595+00:00
heroku[web.1]: Process running mem=626M(122.4%)
2016-10-06T21:34:59.382712+00:00 heroku[web.1]: Error R14 (Memory quota
exceeded) |

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#12 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABASfVGaca88cq6ABOE3xD-BkcB_u4Vks5qxWmwgaJpZM4J3iXQ.

[sgasser]

How can I set the Worker to 1?

No changes with
heroku config:set WORKER_COUNT=1

[RobbieTheWagner]

@sgasser I set WORKER_COUNT to 4 and it seems to have helped me.

[hone]

Talking with the ember team, it seems there are still a few memory leaks potentially in fastboot itself. Your best bet is to work with them to try to help track them down before 1.0. What might be helpful is grabbing heap dumps off your dynos. The newly announced heroku exec should help there! /cc @tomdale

[edmorley]

This buildpack has been deprecated (c06c1e3), and it's also no longer possible for changes to be made to it, due to the issues mentioned in #61.