diff --git a/docs/Features/Audio.md b/docs/Features/Audio.md index c610d9dcaa..3749d4f62d 100644 --- a/docs/Features/Audio.md +++ b/docs/Features/Audio.md @@ -9,16 +9,19 @@ Unlike screen updates which are sent as discrete events, audio compression proce If you want to turn off speaker forwarding, use the option `speaker=off` in your system wide `xpra.conf` (to disable it globally) or in the per-user [configuration](../Usage/Configuration.md) file, or on the command line ## Screenshots -* Audio information displayed on session info (with speaker enabled and running and microphone disabled): +* Audio information displayed on session info (with speaker enabled and running and microphone disabled): ![session-info-audio screenshot](../images/session-info-sound.png) -* A Linux client's pavucontrol showing the Xpra application connected to the local pulseaudio server: +* A Linux client's pavucontrol showing the Xpra application connected to the local pulseaudio server: ![pavucontrol-client screenshot](../images/pavucontrol-client.png) -* pavucontrol running within the xpra session ("on the server"), showing xpra recording the session's audio: +* pavucontrol running within the xpra session ("on the server"), showing xpra recording the session's audio: ![pavucontrol-server screenshot](../images/pavucontrol-server.png) ## Options + +For low level implementation details, see [audio subsystem](../Subsystems/Audio.md). +
Main options diff --git a/docs/Features/Clipboard.md b/docs/Features/Clipboard.md index bfaa051c51..ec87796342 100644 --- a/docs/Features/Clipboard.md +++ b/docs/Features/Clipboard.md @@ -49,17 +49,16 @@ just add `-d clipboard` to your xpra command line. ## Useful Pointers +For low level implementation details, see [clipboard subsystem](../Subsystems/Clipboard.md). + * [How does X11 clipboard handle multiple data formats?](http://stackoverflow.com/questions/3571179/how-does-x11-clipboard-handle-multiple-data-formats) * [x11-clipboard.cpp](http://www.virtualbox.org/svn/vbox/trunk/src/VBox/GuestHost/SharedClipboard/x11-clipboard.cpp) from `VirtualBox` * [operating system specific clipboards](http://en.wikipedia.org/wiki/Clipboard_(computing)#Operating_system-specific_clipboards) on wikipedia * [The X11 clipboard](http://pvanhoof.be/files/Problems%20of%20the%20X11%20clipboard.pdf) _An overview of it's problems and a proposed solution_ And here is a good quote from it: - _Clipboard sharing and network transparency: It's nearly impossible to make the clipboard shared across different desktop computers. In fact it is possible, but such an implementation would be needlessly difficult and complex. The same can be said + _Clipboard sharing and network transparency: It's nearly impossible to make the clipboard shared across different desktop computers. In fact it is possible, but such an implementation would be needlessly difficult and complex. The same can be said of support for virtualization (Qemu, Xen, VMWare). Sharing the clipboard between a virtual machine and the desktop itself is painfully difficult to implement correctly (in case X11 is running on the host operating system)._ -### Source code -[xpra/clipboard](https://github.com/Xpra-org/xpra/tree/master/xpra/clipboard/) -
Related tickets diff --git a/docs/Network/Protocol.md b/docs/Network/Protocol.md new file mode 100644 index 0000000000..2436bcbf6f --- /dev/null +++ b/docs/Network/Protocol.md @@ -0,0 +1,117 @@ +# ![Protocol](../images/icons/connect.png) Protocol + +See also: [network layer](../Network/). + +Every packet exchanged between xpra servers and clients +must follow exactly the same format: an 8 byte header followed by +the packet data encoded using [rencodeplus](https://github.com/Xpra-org/xpra/tree/master/xpra/net/rencodeplus), +optionally compressed using `lz4`. +Only when a connection uses a protocol which cannot be identified, +xpra may respond with a plain-text error message without any packet header. + + +## Framing + +When connecting over websockets, SSL, SSH or QUIC, the transport layer +will obviously add its own framing. +Please refer to these protocols for the information on the packet framing used +by each of these network protocols. +Only xpra's own application layer protocol is documented here. + +--- + +## Packet Header + +The [packet header](https://github.com/Xpra-org/xpra/blob/master/xpra/net/protocol/header.py) +is made of 8 bytes: + +| Index | Type | Length in Bytes | Contents | +|-----------|------|-----------------|-------------------| +| 0 | Byte | 1 | `P` | +| 1 | Byte | 1 | protocol flags | +| 2 | Byte | 1 | compression level | +| 3 | Byte | 1 | chunk index | +| 4 | Long | 4 | payload size | + +### Protocol Flags + +The _protocol flags_ is an 8-bit bitmask value. +It must contain the value `16` for `rencodeplus` packet data +or `4` for `yaml` data (unsupported). +This value can then be ORed with: +* `8` to set the `flush` flag which notifies the packet layer that there aren't any other packets immediately following this one +* `2` to set the `cipher` flag for [AES encrypted packets](./AES.md) + +### Compression Level + +If the compression level is zero, the payload is not compressed. +Or at least, not compressed at the network layer: pixel and audio data may still be compressed but this is handled +by their respective subsystem and not the transport layer. + +This bitmask value is used to indicate how the packet payload is compressed. +The lower 4 bits indicate the compression level. +The higher 4 bits indicate which compressor was used: +* `16` for `lz4` +* `64` for `brotli` + +### Chunk Index + +The chunk index is used for sending large payloads and bypassing the packet encoder. +Packet chunks do not normally use a packet encoder or compressor. +The receiver must replace the item found at _chunk index_ in the main packet. + +Example for sending a hypothetical packet `("example-large-packet", "foo", 20MB data)`, send 2 chunks for better performance: +* send the `20MB data` with a chunk index of 2 (zero based) +* send `("example-large-packet", "foo", "")` with a chunk index of 0 (0 is the main packet) + +The receiver must reassemble the original packet from these two chunks. + + +--- + +## Payload + +The main payload has a chunk index of zero. +Once decompressed according to the _compression level_ flag if needed, +it must be decoded according to the _protocol flags_ using `rencodeplus`. + +It consists of a list of items. +The first item in that list is the packet-type string. +The packet type can be followed by a variable number of optional arguments. + +### Packet Type + +The packet-type is a string which is used for dispatching +the packet to the correct handler. +Each [subsystem](../Subsystems) should use the same prefix for all its packet types. + +The packet-type may also be sent as an integer once the `hello` packet +has been processed by the peer if the `hello` packet contains an `aliases` capability +containing the mapping from packet-type to integer. + +### Arguments + +The only data types that should be used are: +* integers +* booleans +* dictionaries +* iterables: lists and tuples +* byte arrays +* strings + +Floating point numbers and `None` values can be encoded but should be avoided. + +--- + +## Hello + +The `hello` packet is the initial packet used as handshake. +The connection is now fully established until both ends have received a `hello` packet. + +The `hello` packet contains a single argument which is a dictionary +containing all the capabilities advertised by the peer. + +For example, `hello` packets are expected to contain a `version` capability +containing the version string. +They may also include a `username` value, which can be used for [authentication](../Usage/Authentication.md). +Each [subsystem](../Subsystems) will also add its own attributes. diff --git a/docs/Network/README.md b/docs/Network/README.md index 4cd118f9b9..f3d4687141 100644 --- a/docs/Network/README.md +++ b/docs/Network/README.md @@ -1,7 +1,6 @@ # ![Network](../images/icons/connect.png) Network -See also: [authentication](../Usage/Authentication.md), [encryption](./Encryption.md) and [multicast DNS](./Multicast-DNS.md) - +See also: [protocol](./Protocol.md), [authentication](../Usage/Authentication.md), [encryption](./Encryption.md) and [multicast DNS](./Multicast-DNS.md) ## Connection Types | Type | Bind option | Availability | Information | @@ -19,7 +18,7 @@ See also: [authentication](../Usage/Authentication.md), [encryption](./Encryptio `TCP` sockets can also be upgraded transparently to (`Secure`) `WebSocket`, `SSL`, `SSH` and `RFB`, so a single `TCP` port can support 6 different protocols automatically.\ Unencrypted modes like plain-`TCP` and plain-`WebSocket` can also be secured with [AES](./AES.md).\ -All the sockets that can be accessed via a network connection (all but `vsock` and `named-pipe`) will usually be published via [multicast DNS](./Multicast-DNS.md). On Posix, `unix-domain-sockets` are exposed as `SSH` as we assume that a local SSH server is always available. +All the sockets that can be accessed via a network connection (all but `vsock` and `named-pipe`) will usually be published via [multicast DNS](./Multicast-DNS.md). On Posix, `unix-domain-sockets` are exposed as `SSH` as we assume that a local SSH server is always available. See also: [Security Considerations](../Usage/Security.md) diff --git a/docs/Subsystems/Audio.md b/docs/Subsystems/Audio.md new file mode 100644 index 0000000000..6b8eddec06 --- /dev/null +++ b/docs/Subsystems/Audio.md @@ -0,0 +1,62 @@ +# ![sound](../images/icons/sound.png) Audio Subsystem + +For usage related information, see [audio feature](../Features/Audio.md). + +## Common + +[xpra.audio](https://github.com/Xpra-org/xpra/tree/master/xpra/audio/) contains the components used for capturing and playing back audio streams +using [GStreamer](https://gstreamer.freedesktop.org/). +In order to avoid interfering with the performance of the main thread, +all audio processing is done in a separate process. +For historical reasons, this is done using a [subprocess wrapper](https://github.com/Xpra-org/xpra/tree/master/xpra/audio/wrapper.py) +rather than the builtin [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) module. + +### Pulseaudio + +[xpra.audio.pulseaudio](https://github.com/Xpra-org/xpra/tree/master/xpra/audio/pulseaudio) is often used for playback on Linux systems. +This is also the prefered backend for audio capture in server sessions. +The xpra server will usually start a pulseaudio instance hidden away +in a per-session user prefix so that multiple sessions can forward audio streams +independently. + +## Capabilities + +The client and server should expose the following capabilities in their `hello` packet +with the `audio` prefix: + +| Capability | Type | Purpose | +|------------|-----------------|----------------------------------------------------| +| `decoders` | List of strings | The audio formats that can be received and decoded | +| `encoders` | List of strings | The audio formats that can be encoded and sent | +| `send` | boolean | If sending audio is enabled | +| `receive` | boolean | If receiving audio is enabled | + +The lists of `decoders` and `encoders` contain strings such as: `mp3`, `opus+ogg`, `vorbis`... +You can run [xpra.audio.gstreamer_util](../../xpra/audio/gstreamer_util.py) to see which +encoders and decoders are available on the system. + + +## Client + +[xpra.client.mixins.audio](../../xpra/client/mixins/audio.py) + + +## Server + +[xpra.server.mixins.audio](../../xpra/server/mixins/audio.py) + +## Client connection + +[xpra.server.source.audio](../../xpra/server/source/audio.py) + + +## Network Packets + +This protocol is identical in both directions. +Audio being forwarded from the server to the client (aka "_speaker forwarding_") +uses the same packets as audio coming from the client to the server (aka "_microphone forwarding_"). + +| Packet Type | Arguments | Purpose | Information | +|----------------------|----------------------------------------------------------------------------------------------|----------------------------|----------------------------------------------------------------------------------------------------------------------------------| +| `sound-data` | `codec` : string
`data` : bytes
`attributes` : dictionary | Audio stream data | The initial and final packets may omit the `data` argument and should set the `start-of-stream` / `end-of-stream` attributes | +| `sound-control` | `subcommand` : string
(ie: `start`, `stop`, `sync`, `new-sequence`)
`argument` : Any | Send a request to the peer | diff --git a/docs/Subsystems/Clipboard.md b/docs/Subsystems/Clipboard.md new file mode 100644 index 0000000000..3fe9069eec --- /dev/null +++ b/docs/Subsystems/Clipboard.md @@ -0,0 +1,76 @@ +# ![Clipboard](../images/icons/clipboard.png) Clipboard + +For usage related information, see [clipboard feature](../Features/Clipboard.md). + +## Implementations + +[xpra.clipboard](https://github.com/Xpra-org/xpra/tree/master/xpra/clipboard/) contains the platform independent base class +used by all the backends. +It contains common features such as basic configuration, scheduling, filtering, etc. + +| Platform | Link | +|----------|-----------------------------------------------------------------------------------| +| `x11` | [xpra.x11.gtk_x11.clipboard](../../xpra/x11/gtk_x11/clipboard.py) | +| `win32` | [xpra.platform.win32.clipboard](../../xpra/platform/win32/clipboard.py) | +| `MacOS` | [xpra.platform.darwin.osx_clipboard](../../xpra/platform/darwin/osx_clipboard.py) | +| others | [xpra.gtk_common.gtk_clipboard](../../xpra/gtk_common/gtk_clipboard.py) | + + +## Capabilities + +The client and server should expose the following capabilities in their `hello` packet +with the `clipboard` prefix: + +| Capability | Value | Information | +|---------------------|-----------------------------|--------------------------------------------------------------------------------------| +| `enabled` | `enabled` : boolean | Whether clipboard support is enabled | +| `notifications` | `enabled` : boolean | Request the peer to send `clipboard-pending-requests` packets | +| `want_targets` | `enabled` : boolean | Request the peer to send `target`s with `clipboard-token` packets | +| `greedy` | `enabled` : boolean | Request the peer to send clipboard data with `clipboard-token` packets | +| `preferred-targets` | `targets` : list of strings | The `target`s that the peer should try to use | +| `direction` | `direction`: string | Optional, which direction is supported, ie: `none`, `to-client`, `to-server`, `both` | + +Notes: +* `MacOS` clients set the `want_targets` flag +* both `MacOS` and `MS Windows` clients set the `greedy` flag + +## Client + +[xpra.client.mixins.clipboard](../../xpra/client/mixins/clipboard.py) + + +## Server + +[xpra.server.mixins.clipboard](../../xpra/server/mixins/clipboard.py) + +## Client connection + +[xpra.server.source.clipboard](../../xpra/server/source/clipboard.py) + + +## Network Packets + +This protocol is identical in both directions, +as either end can send and receive clipboard events. + +| Packet Type | Arguments | Optional Arguments | Information | +|-------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `set-clipboard-enabled` | `enabled` : boolean | `reason` : string | Either end is free to enable or disable the clipboard at any time and should notify the peer. | +| `clipboard-enable-selections` | list of `selection`s | | The selections that the peer wants to synchronize with | +| `clipboard-token` | `selection` | list of `target`s, `target`, `data-type`, `data-format` and `data` | Notify the peer of a clipboard state change event for the given `selection`, this may include the new clipboard contents if known and / or if the client is known to be _greedy_ | +| `clipboard-request` | `request-id`, `target` | | Request clipboard contents from the peer | +| `clipboard-contents` | `request_id`, `selection`, `data-type`, `data-format`, `data` | | Response to a `clipboard-request` | +| `clipboard-contents-none` | | | Empty response to a `clipboard-request` | +| `clipboard-pending-requests` | `pending-requests` : integers | | The number of clipboard requests waiting | + + +Clipboard data format details: + +| Argument | Data type | Information | +|----------------|-----------|---------------------------------------------------------------| +| `selection` | `string` | X11 supports 3 different _clipboards_, known as selections | +| `request-id` | `integer` | Each `clipboard-request` should use a new unique identifier | +| `target` | `string` | A clipboard format, ie: `STRING`, `UTF8_STRING`, `text/plain` | +| `data-type` | `string` | The type of the contents, ie: `bytes` or `ATOM` | +| `data-format` | `integer` | The number of bits used by each item | +| `data` | variable | Typically, `bytes` that need to be decoded |