Execute commands directly on machine

[skabashnyuk]

Problem description:
Currently we run command via calling Machine API method placed on master node.
In a Docker case physically it means using docker exec() call (which is remote in common case).
In a case of big loading (Codenvy case) there are big command output which routes through master node which may cause:

1. Unexpected loss of output
2. UI freezes
3. Loss of server connection

Sub-tasks:

• Separate commands execution API from main API Separate commands execution API from main API #1946
• Adapt clients to use machine API deployed on a separate instance (remote to the main API) Adapt clients to use machine API deployed on a separate instance (remote to the main API) #1945
• Create Exec agent Create Exec agent #1943
• Rework clients to use new Exec agent Rework clients to use new command API of new exec agent #1947
• Rework agent launching SPI Rework agents launching SPI #2626

Task 2 depends on 1.
Task 3 can be started in parallel with task 1, but finished after task 1.
Task 4 depends on 3.

Proposal
To create special type of agent that will be injected into machine. That Agent will expose API of process execution and logs streaming as REST or RPC call.
Here is list of pros we can get from that agent in comparison to current state of API:

• Agent saves logs of processes locally in the machines. So main API doesn't have to care about scalability of logs storage.
• Agent stream logs from local OS so it is not possible that we will lose connection with process and won't be able to stop it without workarounds.
• Streaming of huge amount of logs from that agent can't overload main API because clients will connect to it directly.
• Currently execs that we uses for command execution are poorly supported by docker.
• Execs are not portable to other machines implementations (other containers, VMs, local PC, SSH machines, etc)
• To be able to stream logs from execs we create persistent connections, so we can consume all available threads on the instance where main API is located

How proposed agent will work:

• On machine start we deploy agent into the machine
• Main API shows url of machine agent in the machine configuration
• Clients use that url to create processes and stream logs

[bmicklea]

@skabashnyuk please take in the next sprint.