New option max_fork to allow allow simultaneously (parallel) session handling (multi core)

[TomFreudenberg]

From @Davidk01

Describe the changes
This change adds an extra parameter to skip starting the accepting thread. The reason is that I would like to bind the TCP connection, fork several processes, and then in each process start the accept loops. This pattern is often used to let the operating system load balance connections to several processes without the need of proxies like HAProxy.

Test case
The change is backwards compatible but the way to test that it works would be to initialize the server and then start the accepting threads in forked processes, e.g.

server = MidiSmtpServer::Smtpd.new(...)
skip_accept = true
server.start(skip_accept)
n_procs = 2
children = (0...n_procs).map do |_|
  fork do
    server.tcp_servers.each { |s| server.start_accept_thread(s) }
    server.join
  end
end
children.each { |pid| Process.waitpid(pid) }

Expected behavior
Previous functionality should continue to work as expected and new functionality should be enabled as described above by passing in a parameter to skip starting the acceptance thread.

System tested on (please complete the following information):

• OS: Ubuntu 20.04.4
• Ruby: 3.0.0

Additional information
This is a draft proposal so if there are any changes that need to be made please let me know and I'm happy to make them. Being able to fork and then accept connections would be very useful for reducing operational overhead like deploying HAProxy so I'm motivated to get this merged, I think it would be helpful for other people as well who are interested in deploying a Ruby SMTP server to simplify the management of email servers and workloads.

Here is a blog post with a good explanation about pre-forking servers that accept connections in child processes: Linux Applications Performance: Part III: Preforked Servers. The relevant section for this PR is the following:

If there are 10 child processes waiting on accept(), the operating system will, with fairness, equally try to distribute incoming connections among them.

So in pseudo-code this is what happens:

parent process binds to a socket
children = n.times do
  fork {
    further set up
    forked child process starts accepting connections
  }
end
wait for children to finish

In each child process the number of active connection is respected the same way it would have been in the parent process if it was running by itself but the benefit of the pre-forking model is that once the parent process binds to the socket then that socket can be shared among any number of child processes handling the accept loop and the operating system will fairly distribute the connections among the forked processes. Ruby has a global interpreter lock so if a server has 10 cores then to fully utilize all the cores will require spawning more than one process to listen for connections because threads are not mapped to a process. Here is a good article explaining some of the implementation details of Ruby's threads: Ruby threads worth it?:

GIL (Global Interpreter Lock) or GVL (Global VM Lock), act as a lock around Ruby code.
It allows concurrent execution of Ruby code but prevents parallel execution.
Each instance of MRI has a GIL. If some of these MRI processes spawn multiple threads, GIL comes in action, preventing parallel execution.

Current implementation limits

In the moment midi-smtp-server will use 1 (one) core only regardless an available multi-core cpu and running threads per this one CORE only, handled by Ruby GIL and executed one thread action after next thread action.

Implementation target

We want to append an option to enable more cores by forking processes and let them handle a number of threads per each core (forked process) to handle simultaneously (parallel) sessions.

[TomFreudenberg]

Implementation of the preforking feature directly handled by midi-smtp-server library.

Append an additional config option like (proposal):

# +max_processings+:: maximum number of simultaneous processed connections PER CORE/FORK, this does not limit the number of concurrent TCP connections. Default value = DEFAULT_SMTPD_MAX_PROCESSINGS
# +max_connections+:: maximum number of connections PER CORE/FORK, this does limit the number of concurrent TCP connections (not set or nil => unlimited)

# +max_fork+:: maximum number of child processes (FORK) to handle connections preforked and in parallel on multiple processes/cores orchestrated by operating system (not set or nil => single process) 

[TomFreudenberg]

Hey @Davidk01

once implemented would you be so kind help testing things working like expected?

Cheers,
Tom

[TomFreudenberg]

Hi @gencer

Maybe an interesting feature for your environment as well.

Cheers,
Tom

[gencer]

@TomFreudenberg

Yes, This can be considered on my environment as you already know what I am doing.

I believe this is NOT implemented yet, right?

P.S.: Sorry for the late reply. I got the message same day as you send and already added to my tasks (backlog). I just away from GitHub for a while (except few things) but I'm back now 🥳

Thanks,
Gencer.

[TomFreudenberg]

Hi @gencer same on my side - wasn't able until today doing the implementation. Guess will postpone this to beginning of July.

I will make an update / notice here when start with the enhancement.

@Davidk01 - pls sorry for the delay - it's now on my July schedule

[gencer]

Hi @TomFreudenberg

Great to hear from you!

Awaiting your update here. 🥇

Gencer.

[TomFreudenberg]

Hi @Davidk01

I have finished the implementation and be currently doing some testing.

Therefor I run into a problem / case while testing on OSX (my dev system)

The Sockets are not balanced between the forked processes. Instead having a clean pid 1-2-3-4 1-2-3-4 1-2-3-4 situation I got some random like 1-1-1-2 2-1-3-3 1-1-3-4 and so on.

I have searched over the internet and found this article:

https://relaxdiego.com/2017/02/load-balancing-sockets.html

On OSX it is not working as expected. I opened an issue to the author at:

relaxdiego/relaxdiego.github.com#30

Could you please step in and try the simple ruby source and test from the page. While there is a bug in the example, please try this code:

#!/usr/bin/env ruby

require 'socket'

# Open a socket
socket = TCPServer.open('0.0.0.0', 9999)
puts "Server started ..."

# For keeping track of children pids
wpids = []

# Forward any relevant signals to the child processes.
[:INT, :QUIT].each do |signal|
  Signal.trap(signal) {
    wpids.each { |wpid| Process.kill(:KILL, wpid) }
  }
end

5.times {
  wpids << fork do
    loop {
      connection = socket.accept
      connection.puts "Hello from #{ Process.pid }"
      connection.close
    }
  end
}

Process.waitall

and this in a second terminal for testing connections when server is started:

for i in {1..10}; do nc -d localhost 9999; done

Thanks for sharing your results.

Tom

---

P.S.: @gencer: Hi :-), could you please try once the same test?

[TomFreudenberg]

Hi @gencer @Davidk01

I have checked in the new pre-release with enhancement of pre-fork.

To use please have a look at https://github.com/4commerce-technologies-AG/midi-smtp-server/blob/master/examples/midi-smtp-server-pre-fork-example.rb and take the changes in trap and at_exit to handle parent and child processes.

Any comment is welcome.

@gencer the component should work as usual without any breakings when not touching parameter pre_fork

[ghost]

Thanks @TomFreudenberg. I'll take a look later tonight and follow up but that examples looks ok to me. Thanks for putting it together.

[TomFreudenberg]

Please be aware of the posted issue on MacOSX - also asked on StackOverflow https://stackoverflow.com/questions/73704741/ruby-strange-forked-processes-behaviour-on-macos-vs-debian

Unless there is a reliable solution, it is not possible to handle a valid usage limitation adjusted by max_connections and max_processings properties. But that's the only negative on OSX.

On a Windows system fork is not available in general.

[ghost]

Thanks Tom, this is good to know. I always assumed the distribution would be fair but it looks like that's not the case. I tested in a virtual machine and there is definitely some imbalance in how the requests are distributed. Below are the results for 10k 20k requests on localhost for Ubuntu 20.04.4 LTS w/ WSL:

   4194 Hello from 4152
   3940 Hello from 4153
   4303 Hello from 4154
   4025 Hello from 4155
   3539 Hello from 4156

[gencer]

On my environment, Linux is the only OS used with MidiSmtpServer. Therefore, I'll not be able to test it on OSX or Win (as already fork is not available). @TomFreudenberg Thanks for those updates I'll test tonight and let you know the results.

🥳

[TomFreudenberg]

Hi all,

in case of not "fair balancing" the processes from OS, I consider to implement a communication pipe or unix socket between the parent and the child processes to handle a valid load usage limitation.

@Davidk01 some experiences in that?

Here a some links to share and for information:

• https://www.sitepoint.com/forking-ipc-ruby-part-ii/
• https://engineering.appfolio.com/appfolio-engineering/2018/4/20/ruby-coordinators-and-process-groups-for-worker-processes
• https://gist.github.com/ryanlecompte/1619490
• https://workingwithruby.com/wwup/ipc/
• https://zeromq.org/languages/ruby/

As from first reading I will have a look on unix sockets in conjunction with zeromq and on the threading and forking example from ryans gist

Any other suggestions so far?

[TomFreudenberg]

In addition to the terms I will change the internal wording from parent to master and from child to worker.

[TomFreudenberg]

From the above zeromq in general looks very interesting stuff but in case of compiling and binding additional sources when require gem install zmq I would name that solution a bit to oversized for the current issue.

So I will follow the approach from the linked gist and create an coordinator thread with unix sockets when activating pre-fork. That will only be used to communicate the current load and connections of the master. It enables to control the system load by max_connections and max_processings while systems process balancing is not fair in a reliable manner.

[ghost]

This sounds very familiar to a load balancer but I don't have much experience with implementing one in Ruby with zeromq. I often use HAProxy for that use case so their documentation might be helpful: http://www.haproxy.org/.

[ghost]

For interprocess communication without any dependencies dRuby might be a good option: https://www.druby.org/sidruby/the-druby-book.html.

[TomFreudenberg]

@Davidk01 drb looks pretty nice for our case and is already there in standard.

drbunix:// uri allows usage of unix sockets so that communication between processes is fast.

Not sure how much speed is taken from overhead of marshalling objects between the processes, maybe raw UNIXSockets is a bit faster. Will try to do some benchmarks for that.

[gencer]

On my environment, Linux is the only OS used with MidiSmtpServer. Therefore, I'll not be able to test it on OSX or Win (as already fork is not available). @TomFreudenberg Thanks for those updates I'll test tonight and let you know the results.

🥳

Thousands of emails processed without any issue! (Linux only)

[TomFreudenberg]

Great, thanks @gencer

I guess I will push the update with the coordinator thread around weekend. Will let you know when things are available.

[TomFreudenberg]

• Add some tests for pre_fork option