Server crashes when you use more than three services

[aninja5]

I created four different services using thesample models. When I try to access all four of the services , the server exits with the message "killed". It happens when I am consequently hitting the services. It works fine with three services. But when I use the fourth one, the server crashes. I am running deepdetect on a Ubuntu 64 Bit VM with 8GB memory.

[beniz]

Hi, this is very likely that your system is simply running out of memory and server gets killed by kernel. I guess you are using a CPU, and thus you need to increase your system's RAM.

[aninja5]

Working with 12GB now. Will let you know what happens.

[Isaacpm]

@aninja5 check the system logs, if the process is being killed by the OS there should be an entry there

[aninja5]

@beniz @Isaacpm You guys are right. I checked kern.log. There is an out of memory entry for deep detect process. What is the recommended configuration? Any work around? I am thinking perhaps I could move the system to AWS.

[Isaacpm]

There is this value when you start training a job: Memory required for data: 82504000, which should give you the amount of memory required so you can plan accordingly.
For AWS you can check this issue:
https://github.com/beniz/deepdetect/issues/5

What I did was to get a cheap ovh.co.uk server with 32gb of ram, which is cheaper than AWS for long term running, and it works for me. But my jobs aren't that resource intensive.

[beniz]

User @revilokeb did an outstanding review of dd performances on AWS along with AMI and recommendations, I'd suggest to use that, it is in issue #5.

[aninja5]

@beniz @Isaacpm thanks! I'll follow up if I find issues!