How to use Parse at scale for e.g 1000 request\second or million users in app. most effective setup (cost $$$ and maintainability)

[agwl-saurabh]

This is more of a discussion question.

Its been more then one year for all of you for running parse server and now its time for most of you to want to have parse server as the scale like 100 RPS to 1000 RPS or more. I would welcome all of you to share your experience with parse server. and would like to discuss most effective (cost $$$ and maintainability) setup.

Below points would be great point to discus. please feel free to add more points.

1. how you are calculating the parse server request per sec or how to calculate if my current server are capable of handling x number of user.

Me :- still not able to calculate exact number of request but i used datadog free for 15 days to check my mongoDB and parse server (server uses). now process to setup grafana/prometheus will provide step to do it if i am able to complete this.

2. What exact server you guys are using (please tell exact the type of server also like t2, m3 or any other) to handle X million user app or RPS can handle and how many daily active users and what they are doing

Me :- I am using AWS Elastic beanstalk with T2.Medium server with 1-10 Auto Scale. with T2.large server with replica set for mongoDB 3.4 with Rock DB engine. if anyone need any help in DB setup I can help. Please note I tried Mlab and running my own DB is cheep option almost 50% cheep for such scale. My app has total 1.2-1.5 Million user and approx 300 thousand daily active user.

3. How you guys running Dashboard\livequery any other services like email or SMS verification or push notifications.
   Me:- I am running Dashboard on LocalHost on my Laptop and connect to production parse server as needed as i am only developer for my app and i don't need dashboard on could and i think its safest option right now. I still have issue with live Query on Aws elastic beanstalk so not able to make that work. for push i am using urbunship as i had issue with parse push.
   no email ans sms verification till date but i have plan to add these in future.

4. how you guys taking backup and monitoring the DB and images . or in other world do you have any backup plan for your apps if your server crash.

Me:- i am only taking back up of Mongo DB on my personal laptop from EC2 server and I am assuming S3 will automatically do there backup so my files are safe . S3 assumption is as per aws documents.

[flovilmart]

still not able to calculate exact number of request

If you're using elastic beanstalk, then you should use elastic load balancer, and AWS Cloud Watch should be able to monitor your load balancers, therefore you have access to the instant RPS without looking further

[stevestencil]

Are you @agwl-saurabh open to discussing how you have things setup in more depth? I currently do not have nearly as many users as you but I'm scaling fairly quickly. Maybe you're interested on collaborating on how and what we're using this library for?

[agwl-saurabh]

Can you please tell me more what you are looking for and what collaborating..??? . i am not able to understand what you need from me. and which library you are talking ..?? may be if you can provide the points on which you need help if i have information i can provide no issue.

[stevestencil]

@agwl-saurabh I'm interested in picking your brain on how to best optimize the AWS configuration for parse-server. Also, I'm always trying to network with other developers to bounce ideas/best practices off of. Let me know your thoughts?

[jcguarinpenaranda]

I have the same question, I want to able to scale parse-server to be able to process many requests per second. Right now I'm using Heroku, but off course it alone becomes a bottleneck when I start making lots of requests, and scaling it is manual unless you pay USD$250 servers and turn on auto scaling

App Engine

I need help scaling this out! I already tried Google App Engine, but I have found they scale the app based on CPU utilization, not memory. What has happened to me is that after getting ~10 requests per second on a 1vCPU and 1GB of RAM machine it hangs up, because the memory consumption goes up, and the cpu consumption stays low.

In the Calculator of Google I see this would be what I would have to pay, for having only one instance running on the flex environment

AWS Elastic Beanstalk

Also, tried to make it run on AWS Elastic Beanstalk but I get an error when the app first executes because it seems to make use of many resources.

Heroku + Hirefire.io

I have found that there is a service called Hirefire.io that helps you scale your Heroku servers, based on their response time. So far, I made it scale my servers when their response times are > 500ms. This was done with 1GB RAM servers from Heroku that cost USD$50/month, but I have a feeling this is too much, don't you think? Or, how much, approximately are you spending on this?

This is the test I made with Heroku, and I leave it here for the discussion

What happens here is that the memory consumption goes up. At some point the server starts answering the requests above the 500ms and that's where the autoscaling goes in adding more servers, but as you see the memory at some point uses more than 1GB, that is the limit for the USD$50/mo servers.

Now.sh

Also, I have uploaded my server to Now.sh, and it is pretty simple (very similar to Heroku), and auto scaling seems to be this simple (haven't tried it out yet): https://zeit.co/blog/scale

Note

I am not a cloud expert. I started using Parse when it was a service, and then started using Parse Server to continue having support for an app I am developing. Since then I have been learning about the different providers that i mentioned above and I would like to know if there's something I'm doing wrong, and how are you doing all the scaling process.

[jcguarinpenaranda]

I have also found Hyper.sh, but I don't know how it manages auto scaling

[AppOrchestra]

@jcguarinpenaranda Can you set your instances to have more memory then? What do you have in your app.yaml for the scaling configuration?

[jcguarinpenaranda]

Thank you,

Yesterday I read again the documentation for Elastic Beanstalk and saw that they charge you only for the EC2 instances you use, contrary to Google, that handles a different way to charge you when you select the flex environment.

I ended up using t2.small instances with double the RAM of Heroku, at a third of the price ($16.56 vs $50), so I'm very happy now. Also, with more memory I have been able to make the same test without any problem. I just had to change script postinstall, to prestart, because Elastic Beanstalk does not execute the first.

[agwl-saurabh]

Hi @jcguarinpenaranda ,

I was very busy so not able to help on this form. As i mentioned in my main comments I found Elastic Beanstalk best and cheapest option on all the option available right now. for large app. for small app digitalOcean is best. Please let me know if you need any specific help needed for the setup.

Please let us know wt test you are running so it will be helpful for all. and in which script you made changes postinstall, to prestart .

Please share your finding about memory issue as i am using parse server 2.6.5 without memory issue. with good number of request.(approx 175-220 per second) with two t2.medium server(max on loadblancer)

@flovilmart , Thanks for your info about RPS some how i missed your comment. I was not looking at "Cloud Watch" and trying to enable third party tools which was dumb idea. you are awesome 👍

[jcguarinpenaranda]

Hi @agwl-saurabh. The test I'm making is the same test I was showing in the graph of Heroku, with around 4 rps. I'm using Typescript, so on postinstall I was running tsc ., and I had to change it to prestart, because I saw Elastic Beanstalk does not support the first one.

[jjdp]

@agwl-saurabh
did you change your elb nginx or linux configs or just the defaults?

[agwl-saurabh]

@jjdp i am using default config and default config have nginx . let me know if you have any issue. or if this query is for any specific query.

[Rioner123]

Hi @agwl-saurabh

This is a very interesting topic for me as we scale up. How many aftersave or before save hooks do you use? We currently have about 1000 users that access our server every 5 minutes and we have both beforesave and aftersave hooks. We find that there is a huge stress on our server where we start to get a lot of 4xx and 5xx errors and the health of our server degrades. Any tips? Do you think it could be our cloud hooks?

[agwl-saurabh]

@Rioner123,

I don't think 1000 user per 5 minute is very large user its very small user base . i think i have almost 5000-1000 user per minute as per the analytics(at anytime i can see active user in analytics is minimum 3k for me) i am using in my app and my server never goes above 20% CPU uses. so i don't think you have user issue its look like some server configuration issue . provide me full details of your server set up with all details possible.

i have AWS paid support see below the reply which i receive recently it may help you also.

_I understand that you are getting an 'Environment health has transitioned from Ok to Warning' error because 1.0% of requests to the ELB are failing with 5XX and you are having a high latency when connecting to your application. Please correct me if I am wrong.

While looking into the ParseServer environment, I was able to see that it had two events on 2017-12-12. The first event started at 08:47:58 (UTC) and ended at 08:58:58. The second one started at 14:07:24 and ended at 14:30:37.

During these events, there was a high network usage, which triggered the autoscaling group and new instances were launch and then they were terminated when the high network usage dropped.

The i-08aa9 instance that you mentioned was launched at 2017-12-12 14:08:17 (UTC) and was removed at 2017-12-12 14:31:15 (UTC). The other instances that were launched during the event were:

• i-0a688
• i-01185

The i-01185d one was the one that continued on your Autoscaling group, the other one (i-0a688cf) was terminated at 2017-12-12 14:21:55.

Please note that is the correct behavior and does not indicate an error. The 1% of 5xx errors that the ELB received happened during the time you instance was booting up and your application was starting.

I also checked your application network usage and metrics, and I was not able to spot any issues. Do you mind to share your benchmarks with us so I can dig further on the high latency issue?

In the meantime, I see that your environment is configured to use a t2.Medium instance, which has only two vCPU. Depending on the usage of your application, that may be the cause of the high latency.

Please find some documentation about EB health status and reporting on the references section._

[andrewjedi]

@agwl-saurabh are you still using parse server?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mobappsbr]

Hi @agwl-saurabh, we can talk by email or linkedin? I want to talk with you more about scale parse server