-
Notifications
You must be signed in to change notification settings - Fork 5
Streams YARN Client
The Streams YARN Client is the user facing end of the system. In consists of two parts: 1) A Python script to enable simple, centralised control, and 2) A Java class, StreamsClient -- which is also an Apache Thrift client, to launch the Streams Application Manager and also to communicate with it post-launch.
streams-on-yarn is a Python script that performs sanity checks on the commands, sets up environment variables and the Java classpath, and interfaces with the Streams Client.
The StreamsClient serves two primary roles: 1) It interfaces with the Resource Manager and registers the Streams Application Manager by acquiring an Application ID and then submitting the StreamsAM Jar, and 2) Once the Streams Application Manager has been launched, it acts as a Thrift client to connect to the StreamsAM (which is a Thrift server) and enables the user to manipulate the Streams deployment, such as starting an instance. Details of the Thrift interface can be found here.