FRSC: Supports sub-millisecond latency, replaceable fast channel technology.

[winlinvip]

https://github.com/winlinvip/simple-rtmp-server/issues/120

Supports sub-100 millisecond latency, replaceable fast channel technology
Support <500ms latency, FRSC(fast RTMP-compatible stream channel tech)

Sub-100 millisecond latency requires improvement on the existing RTMP mechanism:

1. Consider using UDP instead of TCP for encoder transmission, server internal communication, and client transmission. Support switchable UDP/TCP mechanism, for example, only the client uses TCP/RTMP while others use UDP private protocols.
2. Consider non-interlaced media stream transmission: The encoder BLE supports non-interlaced output, where the audio stream timestamp increases monotonically, the video stream increases monotonically, but they are not mixed together. Mixing them would increase latency. The server does not mix them internally (interlaced) either, instead, the mixing of video and audio is done at the edge server or on the client side (own client).
3. Consider using IO mechanism instead of timeout mechanism: Currently, the server logic waits for 300 milliseconds for client control messages and then sends all pending packets together to the client. This introduces a delay of 300 milliseconds, which increases at each level (according to average probability). If the 300-millisecond timeout mechanism is replaced with an IO mechanism, i.e., the "write" event for packet transmission becomes an IO "read" event, it can wait for two file descriptors to determine whether there is a message from the client or a message to be written.
4. Packet loss mechanism: Consider supporting packet loss at the edge or in the encoder to control the delay within the sub-500 millisecond range, ideally around 300 milliseconds for Flash clients. If the server controls the delay within 200 milliseconds, the overall system delay can be controlled within 500 milliseconds. If the client is a custom client, the delay can be even lower (server-side control within 200 milliseconds).
5. Rollback to standard mechanism: Support standard RTMP, for example, within the encoder/server cluster, use low-latency distribution mechanism. If it is detected that the client is Flash-based, the edge should mix the video and audio streams and send them to the client with monotonically increasing timestamps.

In summary, supporting sub-100 millisecond latency requires a comprehensive set of mechanisms and customization on top of RTMP. Compatibility with standard RTMP should also be considered. Therefore, the main goal of the system is to establish a fast channel, control the latency within the sub-100 millisecond range, and support integration with universal solutions at any point in the fast channel.

TRANS_BY_GPT3

[lileiseven]

This is awesome, when will it be scheduled for development?

TRANS_BY_GPT3

[winlinvip]

The player may not be able to support streaming playback at the millisecond level.

TRANS_BY_GPT3

[lileiseven]

According to this train of thought, can Flash player achieve it?
For interactive live streaming in online education, low latency is required, ideally around 1 second.

TRANS_BY_GPT3

[winlinvip]

From encoding to distribution to playback, it should be impossible for Flash to achieve within 1 second.

TRANS_BY_GPT3

[winlinvip]

It's impossible for normal TCP to make it, there are some possible ways:

1. Use TCP BBR.
2. Use UDP protocol.

[winlinvip]

Fixed by #307 WebRTC, SRS 4.0.

[lidp888]

https://github.com/winlinvip/simple-rtmp-server/issues/120

Support for sub-millisecond latency, replaceable fast channel technology.
Support <500ms latency, FRSC(fast RTMP-compatible stream channel tech)

Sub-millisecond latency, improvement of existing mechanisms on the RTMP protocol is required:

1. Consider using UDP instead of TCP for encoding, server internal communication, and client transmission. Support switchable UDP/TCP mechanisms, for example, only the client uses TCP/RTMP while others use UDP private protocols.
2. Consider non-interlaced media stream transmission: The encoder BLE supports non-interlaced output, which means the audio stream timestamp in the encoder increases monotonically, the video stream increases monotonically, but they are not mixed together. Generally, encoder mixing increases latency, and the server does not mix (interlace) internally, but instead mixes video and audio at the edge server or on the client side (own client).
3. Consider using IO mechanism instead of timeout mechanism: Currently, the server's packet sending logic is to wait for the client's control message for 300 milliseconds, and then send all pending packets to the client together. This causes a delay of 300 milliseconds, which increases at each level (according to average probability). If the 300-millisecond timeout mechanism is changed to an IO mechanism, i.e., the "write" event of packet sending becomes an IO "read" event, it can wait for two file descriptors (fd) to determine whether there is a message from the client or a message to be written.
4. Packet loss mechanism: Consider supporting packet loss at the edge or in the encoder to control the delay within the sub-millisecond level, i.e., within 500 milliseconds. If the client is Flash, the delay can be around 300 milliseconds, and if the server controls the delay within 200 milliseconds, the system delay can be controlled within 500 milliseconds. If the client is the own client, the delay can be even lower (server-side control within 200 milliseconds).
5. Rollback to standard mechanism: Consider supporting standard RTMP, for example, using low-latency distribution mechanisms within the encoder/server cluster. If it is detected that the client is Flash, the edge should mix the video and audio streams and send them to the client after ensuring the monotonic increase of timestamps.

In summary, supporting sub-millisecond level latency requires a complete set of mechanisms and customization of certain parts of the system based on RTMP. At the same time, compatibility with standard RTMP should be considered. Therefore, the main goal of the system is to establish a fast channel, control the latency within the sub-millisecond level, and support integration with universal solutions at any point in the fast channel.

Can the third point here be modified to be lower? Where specifically can it be changed? Performance can be sacrificed as a replacement.

TRANS_BY_GPT3