Protocol
The Leap Motion controller provides a protocol for communicating high-level hand data to applications using Websockets. This document outlines the usage of this protocol.
leapd is a service which interprets data coming from the Leap Motion
controller and reconstructs hand position. That hand data is made available
in via a web socket server within leapd. This web socket communication is
done through an local service on your computer.
A websocket client can connect and communicate with leapd via this web socket
connection. Several clients exist already for this data, for example:
- https://github.com/leapmotion/leapjs - Javascript client for Nodejs and browsers.
- https://github.com/logotype/LeapMotionAS3 - Flash client
The data sent by the websocket server is versioned. As new fields and features are added, the version of data sent gets incremented. Currently there are four versions available. The requested version is encoded in the URI used to connect.
The current URL format is /{version number}.{format}. The only supported format
is json.
As of LeapJS 0.6.2 and Leap Service 2.0.6, Leap supports connection over Secure Websockets.
On start, the Leap Motion control panel installs a self-signed root certificate called "Leap Motion Local Certificate". Both browsers and the Leap service have access to this certificate, allowing secure communication.
LeapJS will automatically detect https websites, and attempt to connect to the Leap Service over wss:// port 6436. This will fail for any users with earlier versions of the Leap Service, and LeapJS will attempt once to connect over an unsecure port as a fallback.
To test the validity of an installed certificate, connect to https://127.0.0.1:6436. Chrome or Firefox will throw an error here for invalid certificates, or render a blank white page (with green lock icon) on success. Currently, browsers have poor support for checking certificate validity over wss:// directly.
To confirm the SSL is working in the wild, visit our test page over a secure connection: https://leapmotion.github.io/leapjs/examples/dumper.html
As of now (November 2013) the following protocols are supported. Each is described in a specific documentation sheet.
- Version 1: accessed by '/' or '/v1.json'
- Version 2: accessed by '/v2.json
- Version 3: accessed by '/v3.json'
- Version 4: accessed by '/v4.json'
- Version 5: accessed by '/v5.json' deviceEvents (streamingStart/streamingStop, etc)
- Version 6: accessed by '/v6.json' skeletal tracking
When you send this first message, you get back a single JSON object echoing the version that the leapd will be using in its communications with your client: {version: 4}
And from then on, you will receive frame data and events over that webSocket channel, as detailed below.
One leapd can have zero, one or many clients. It may only have one piece of hardware that it reports for. Whether a given client receives frame and gesture information depends on which client has focus and which clients have elected to listen in the background; this is detailed below. It also relies on the leapd's connection to a Leap Motion Controller hardware.
In a windowed GUI, mouse state is "blocked" by overlapping windows. You would not expect to receive mouse events if another window was wholly covering your own. With leap, however, every client expects to receive all hand/finger/pointer data continuously. There is no automatic blocking of leapd data when you (in your window/mouse environment "defocus" an application window, cover it with another window, minimize it, etc.
There may not even BE an active window or visible application tied to a particular client.
It is up to each client to decide when and if to start and stop listening to the stream when the application context changes.
You can tell the leapd server one of four things:
- "I have gained focus" -- _place_holder;{focused: true}. Take focus away from the last "on top" client and send me frames.
- "I have lost focus" -- {focused: false}. I do not want to be on top. this will NOT _place_holder;give any other client the focus but it will un-register this client as the focused client.
- "I always want to get frames" -- {background: true }. I want to receive frame data no matter which client has the focus.
- "I only want to get frames when in focus" -- {background: false}. I do not want to receive frame data unless I have the focus. (note - this is the default status of all clients until you explicitly send {background: true}.
The client is not told which state they are in _place_holder;by the leapd engine, except if the hardware is disconnected or reconnected. However beneath the surface, each client has the following possible states:
Connected Focused Listening Gets Frames
YES YES YES YES
YES YES NO YES
YES NO YES YES
YES NO NO NO
NO YES YES NO
NO YES NO NO
NO NO YES NO
NO NO NO NO
In any of these scenarios, the device (or in future devices) also has state, and when any of it changes the client will receive a 'deviceEvent' message. When a client is connected, it will receive a single deviceEvent for each device which is currently attached to the computer, regardless of if it is currently being used.
If hardware is then plugged in, they will receive another message letting them know that a new device is attached.
State events about connected/disconnected hardware are not repeated. They are only sent once to each client for each state change; that is, when the Leap hardware is plugged in/detected/begins tracking or when the hardware is unplugged/loses connection/stops tracking.
Almost all of the communication between the leapd and the client is one-way. The leapd broadcasts frames or state changes to the listening clients.
The client to leap communication happens when the client wants to change its state or turn gestures on and off.
Messages regarding focus and background from ONE client may interfere with ANOTHER client's leapd communication.
When a client grabs focus by setting {focused: true} to leapd, other clients will no longer receive frames. However other clients may also grab focus; there is no way to "Lock" the focus of the leapd to your connection.
The leapd will NOT inform the client when it has lost focus. However you can infer that this is so when you are in a client state that would ordinarily be getting frames and you don't get frames.
Clients can elect whether or not to receive gestures. This choice is at a per- client basis; if one client elects to get gestures, it will not affect whether another client does or does not receive gestures.
By default, you will not receive gestures unless you "opt in".
The array of gestures may be empty, have one gesture, or have many gestures.
Some gestures are only connected with a single event. Some are continuous, such as the circle gesture. Continuous gestures' states will be one of three (string) values: 'start', 'continue' or 'stop'.
You opt in or out of gestures by sending {enableGestures: true} or {enableGestures:false} to leapd. (it will not give you any feedback in response. )
As of November, 2013, this is the most current schema that leapd provides.
There are two sets of messages -- leapd to client and client to leapd
As described in the leapd documentation, the client establishes connection to the leapd by connecting to '/v4'. This connection is made to the leapd daemon at port 6347.
The server will respond with {version: 4}. It will then send a series of frames (see below).
If the server CANNOT support the requested api version it sends back a message with a status code of 1001. No JSON block is provided if the version is not supported.
By sending {background:true} or {background:false} to leapd you can elect to listen to frames regardless of which client has focus.
no direct response -- but you may see a change in your frame stream.
By sending {focused: true} or {focused: false} you can attempt to commandeer the stream of frames from other clients of the leapd. This is detailed in the leapd communication.
no direct response -- but you may see a change in your frame stream.
Events inform the client about changes to the state of the connection, etc. There is currently only one event: deviceEvent.
state is actually a json hash containing information about the device that triggered the deviceEvent. The onus is on the client to determine which fields have changed, though it is guaranteed that at least one will have.
{
event: {
type: "deviceEvent",
state: {
attached: true,
id: 0,
streaming: true,
type: "peripheral"
}
}
}
| Property | Description |
|---|---|
| attached | The physical connection of the device. This is only ever false once, when disconnecting. |
| id | A leapd-session unique serially assigned identifier for the device. These are not guaranteed to be the same between sessions, but they will uniquely identify a particular device, even between disconnect events for as long as leapd is active. |
| streaming | True if the device is currently contributing frame data. |
| type | The type of the device. Currently values will be one of "peripheral", "laptop", or "keyboard". |
Frames are streamed to active or listening clients as JSON blocks. The schema of the json block is as follows. Note, the "r", "s" and "t" values for all elements described below are for internal use and should not be used in your application.
a unique identifier of the frame
a Unix timestamp.
an array of hand data
the dimensions of the measurable area of the Leap Motion Controller hardware
An array of all your pointables: fingers, sticks, from all hands in the interaction box.
an array of 0..many Gestures currently detected
r: floats
(rotation) a 4 x 4 matrix; mainly for internal use, reflects overall motion of detected objects.
s: float:3
(scale) a 3 value array of floats indicating relative size of the space occupied by detected objects; for internal use
t: float:3
(translation) a three value array of the net relative motion of detected objects, for internal use
the identity of the hand
Euler angles of the hand
Euler angles of the palm
the position of the center of the hand, in millimeters, relative to the Controller hardware
The change in position of the palm relative to the last measured position of this hand.
The position of an imaginary "ball" being held by the hand, in millimeters, relative to the Controller hardware
The distance from the imaginary "ball" to your palm
r: [float]
rotation relative to last known position of this hand; for internal use.
s: float
size relative to the last frame of the extent of the hand; for internal use
relative motion of the hand and its fingers since the last frame; for internal use
The interaction box describes the measured area of space in which Leap can detect things. It is not going to change over time.
the position, in millimeters, of the origin relative to the Controller hardware
the extent of the Controller detection area in millimeters. The three values are width, height, and depth.
Note, it is not in reality a square; the closer you get to the Controller, the narrower the detection region gets.
the position, in millimeters, of the gesture. For circular gestures it is the position of the center of the gesture.
the number of microseconds that the gesture has been maintained.[1]
the ids of the hands making the gestures; at this point most gestures are made by a signle hand.
the ids of the pointables making the gestures
for the circle gesture, the approximate distance in millimeters between the center and the pointable (or the mean location of the pointables).
Whether the gesture has started or stopped this frame. Not all gestures will express in all three states -- one-frame gestures may only express 'stop'.
the type of gesture.
Leap Protocol Version 3 does not feature background events. You cannot expect to change your frame stream by sending
{focused: true} or {focused: false} to the leapd server. Likewise, {background: true} and {background: false}
have no effect or use in version 3. Leap version 3 also features the heartbeat described below.
In this version a client establishes unique control of the Leap by sending {heartbeat: true} messages within 100ms
(1/10th of a second) often. The heartbeat is present in version 3 but removed in version 4.
Leap Version 2 does not have focus or background messages.
No heartbeat, focus or background messages.
[1] _place_holder; _place_holder;(1 million microseconds == 1 second).