Reduce ssh sessions

[eflumerf]

We should look into ways to reduce the number of ssh sessions being spawned by the daq during startup. Ideally, there would be just one per host (excluding the run control host) to start the controllers, which could then start the other DAQ processes without further use of SSH.

Thoughts?

[plasorak]

Is that triggering a particular problem? We did think of including a little wrapper recently.

[jcpunk]

The creation of 14 ssh sessions within a single second to connect to the same host is "odd"

[alessandrothea]

What is the definition of odd and why is this an issue?

[plasorak]

Is this related to: #304 ?

[eflumerf]

Part of what we are seeing on this particular host is an unusual amount of CPU usage by systemd when closing ssh sessions. This issue leads to failures of the integration tests. We are debugging from the sysadmin side as well, but the number of ssh sessions the DAQ is starting is a concern for system scalability.

[jcpunk]

What is the definition of odd and why is this an issue?

From man sshd_config

   MaxSessions
           Specifies the maximum number of open shell, login or subsystem (e.g. sftp) sessions  permitted
           per  network connection.  Multiple sessions may be established by clients that support connec‐
           tion multiplexing.  Setting MaxSessions to 1 will effectively  disable  session  multiplexing,
           whereas  setting it to 0 will prevent all shell, login and subsystem sessions while still per‐
           mitting forwarding.  The default is 10.

Thus 14 connections to the same server in the span of 1 second can lead to failed connections and retries.