Server crashed with src/zsock.c:563: zsock_attach: Assertion 'self' failed.

[rockychan]

(Describe the issue here)

• Skygear Server Date/Version: 0.14.0
• Attached logs, screenshots

https://gist.github.com/rockychan/11526bc2fc2f842c60124db99aedb64a

[rickmak]

@rockychan may I have your changeset on last working deployment and current one?

[rockychan]

last 2 update on server on 27/9 and 28/9
https://github.com/oursky/village/commit/2da01dec79de3031132f85135f456d4213c273f6
https://github.com/oursky/village/commit/8d95614ad5f1ad7ddd7ba9cab91187d941b5d978

[rickmak]

According to the log provided, the program is hitting the default limit of zmq socket. Which is 1024 ref: http://api.zeromq.org/3-2:zmq-ctx-set

To get around your issue quick, you should raise the following figure to handle more concurrency

• zmq socket limit
  Add ZSYS_MAX_SOCKETS: 10240 to your skygear-server environment. If you are using docker, add it to docker-compose.override.yml.
• ulimit in your OS
  Modify /etc/security/limits.conf, add
  soft nofile 10240 hard nofile 10240 to your setting
• ulimit in docker
  Ensure limit nofile specific /etc/init/docker.conf have large value (>10240 in our case). Restart the docker daemon to apply the new OS limit to docker container. Normally with
  sudo service docker stop && sudo service docker start

We will reuse the inproc zmq socket or change to use go chan. And to be address in separate issue.