Support Multiple-CPUs(or Threads) to improve concurrency. #2188

winlinvip · 2021-02-05T01:19:07Z

Remark

For now, let's put the multi-threading preparation on hold. Although ST already supports multi-threading and the RTC multi-threading part has been almost completed, there are several factors that make me think we should reconsider whether multi-threading is necessary at this stage.

First of all, the multi-threading branch has been deleted from the SRS repository, but it is still preserved in my repository feature/threads, which mainly includes the following commits:

ST: Simplify it, only Linux/Darwin, epoll/kqueue, single process Simplify ST, only support limited platforms.
ST: Support thread-local state-threads Optimize ST, support multi-threading.
Threads: Support Hybrid+API+LOG threads Transform SRS to support multi-threading.
Threads: Remove reload for streams Disable some Reload, multi-threading Reload is complex.

The main reasons for reconsidering multi-threading support are:

RTC cascading can extend SRS's capabilities, making it simple and reliable to expand with the Edge/Origin mechanism for live streaming.
Single-process concurrency is currently between 1000 and 1200 concurrent connections. After further optimizing QoS, it is estimated that it can maintain around 1000 concurrent connections, which is sufficient for open-source use.
Only ST and RTC have more comprehensive multi-threading support, while live streaming and API support are not yet perfect. It requires more effort and is not suitable for our time-limited community to do.
The main goal of multi-threading is to expand the single-machine concurrency to the tens of thousands level, which is generally used in large-scale commercial systems. Generally, open-source SFUs do not have this scenario (besides, SRS can use cascading for expansion).

However, simplifying ST and improving its performance can still be considered for merging, including:

Summary

SRS's support for multi-threading is a significant architectural upgrade, essentially aimed at addressing performance issues.

Regarding performance issues, the following points can be expanded:

In the live streaming playback scenario, a single-process single-thread can run up to 3000 concurrent connections or higher, as live streaming has no encryption and can directly distribute audio and video data in a one-to-many scenario. Moreover, it can be horizontally scaled through Edge.
In the live streaming push scenario, it is impossible to achieve complete horizontal scaling, and the source station cluster scale cannot be very large, as referenced in Cluster: Origin Cluster for Fault Tolarence and Load Balance. #464. If it is a high-bitrate stream, a single-process single-thread can hardly achieve more than 1000 connections.
In the RTC scenario, in addition to encryption and QoS, the performance of UDP sending and receiving will also be lower, and it is roughly estimated that it is difficult to reach 500 concurrent connections. This makes SRS no longer a purely IO-intensive server, but an IO and CPU-intensive server.

Why is this issue important?

In the RTC scenario, if a single process can only reach 100 or 300 concurrent connections, then 10 times the number of cores is needed. The amount of traffic between these servers is also ten times higher. At this point, the concurrency capability is entirely insufficient.
In the live streaming scenario, the current multi-process, Origin, and Edge cluster expansion capabilities can be supported by multi-threading, allowing a single machine to achieve high concurrency, reducing the number of machines to manage, and lowering system complexity.
In the monitoring scenario, it is no longer a case of a single stream being consumed by many people, but rather many streams requiring encryption. If a single process does not support enough connections, it will cause many process issues.

Therefore, the multi-threading architecture can be considered a revolution after the multi-coroutine architecture, but this time it is a self-revolution.

Arch

The previous SRS single-threaded architecture (SRS/1/2/3/4):

Single-process single-threaded architecture, live streaming supports Origin-Edge cluster, and if deployed on a single machine, it is a multi-process single-threaded architecture.
RTC is computationally and IO-intensive, so multi-core capabilities are essential, and the main problem is the issue described in this post.
Despite this, SRS4 has also made many single-core performance optimizations, as referenced in b431ad7, c5d2027, 14bfc98, and 36ea673

The ultimate goal architecture is horizontally scalable Hybrid threads, also known as low-lock multi-threaded structure (SRS/v5.0.3):

Merge SRTP thread into Hybrid thread, both threads call OpenSSL simultaneously, posing a risk of Crash.
UDP RECV and SEND threads are optional, disabled by default, and Hybrid threads are responsible for sending and receiving data.
ST is changed to a thread-local structure (ST#19), with each thread having its isolated ST. It is crucial to ensure that the ST structure of one thread does not pass to another thread, i.e., not reading or writing other threads' FDs.
Hybrid threads default to 1, which is almost identical to SRS4's completely single-threaded structure, maintaining structural simplicity. If high performance is needed, multiple threads can be enabled, and the architecture remains essentially unchanged (from one thread to multiple independent threads).
Hybrid threads support horizontal scaling, using a multi-port approach. RTC returns different ports through SDP, while RTMP and HTTP require 302 redirection. The specific implementation depends on the documentation.
When Hybrid scales horizontally, it still locks connections to a single thread based on the stream, enabling push stream expansion. It can almost scale horizontally to hundreds or thousands of cores, but as the number of cores increases, communication between threads increases, and it is not entirely cost-free, but the cost is relatively small.
When Hybrid scales horizontally, in general, multiple push streams and multiple play streams can be well supported, such as 1000 push streams with 10 play streams each, totaling 10,000 streams.

The disadvantages of this architecture:

Single-stream playback expansion issue: When Hybrid scales horizontally, since each stream is locked to a single thread, the number of play streams for a single stream is limited by the number of connections supported by a single thread. Solution: In the future, cascading will be used to solve the downstream expansion issue, such as supporting 1000 connections for a single stream on a single machine (regardless of the number of cores), and cascading 1000 servers to support 1 million play streams.
Global variable and static variable cleanup issue: Although multi-threading is isolated, they still have a chance to make mistakes through global and static variables. Therefore, all global variables must be checked and modified, either changed to thread-local or thread-safe. This brings risks to stability and is often difficult to troubleshoot when problems arise. Solution: By default, only 1 thread is enabled, allowing for a long enough transition and improvement period.
Library thread safety issue: For example, OpenSSL has multi-threading issues, OpenSSL 1.1 claims to be thread-safe, requires modifying the compilation script, and does not use the option -no-threads, but sometimes it may forget to change this option and cause problems. Solution: By default, only 1 SRTP thread is enabled, allowing for a long enough transition and improvement period.

Single-stream playback expansion issue: If you must modify multi-threading to allow a single machine with multiple cores to support playback horizontal scaling, you can have the push stream thread broadcast to the pull stream thread. This change is acceptable, but open-source does not entirely pursue performance. It still needs to maintain a very simple architecture with performance optimization. Personally, I think it is reasonable for a single machine to support 1000 play streams and use cascading to solve expansion, so this optimization will not be used in the future.

Note: Image source is here.

Communication Mechanism

There are two ways for threads to communicate: the first is locked chan, and the second is passing fd. The second can rely on the first.

Both methods should avoid passing audio and video data. Of course, they can be passed, but it is not efficient. For example, you can start a transcoding thread, communicate with chan, and it doesn't require much concurrency.

SRS will have multiple ST threads, which communicate through chan, but they do not pass audio and video data, only some coordination messages.

Currently, SRS's thread communication uses pipe implementation to avoid locks. Therefore, when using it, be aware that it is a low-efficiency mechanism and should not directly pass audio and video packets. It is mainly used for communication between the Master (API) thread and the Hybrid (service) thread, with Hybrid returning SDP to the API.

Thread Types

Each thread will have its ST, and ST is thread-local, i.e., independent and isolated ST.

Remark: The most critical risk and change is to avoid passing FD (or other ST resources) created by one thread to another thread for processing, which will definitely cause problems.

In the end, there will be several types of threads:

Main thread. It mainly manages configuration, manages threads, passes messages, and listens to APIs.
Log thread. Read logs from the queue and write them to disk. To avoid disk IO blocking ST, recording and HLS writing to disk can also be placed in this thread.
One or more Hybrid threads. Listen to network FDs, create epoll, start ST, and serve as the main audio and video business thread. Threads do not communicate with each other. Live streaming uses REUSE PORT, and RTC uses multi-port isolation.
Optional, SRT thread. As it is now, an independent SRT, pushing RTMP to the ST thread via local socket.
Optional, if implementing the SRT protocol yourself, the ST thread can handle SRT clients.
Optional, transcoding or AI thread, pulling audio and video data from the ST thread via local socket, or passing data through Chan, implementing capabilities such as mixing and streaming.

Milestones

4.0 will not enable multi-threading, maintaining single-threaded capabilities.

5.0 will implement most of the multi-threading capabilities, including improving ST's thread-local capabilities. However, Hybrid will only default to 1 thread, and although the process has multiple threads, the overall difference from the previous single-thread is not significant.

6.0 will enable as many threads as there are CPU cores by default, completing the entire multi-threaded architecture transformation.

Differences from Go

Go's multi-threading overhead is too high, and its performance is not sufficient, as it is designed for general services.

With multiple cores, such as 16 cores, Go has about 5 cores for switching. This is because there are locks and data copying between multiple threads, even though chan is used.

In addition, Go is genuinely multi-threaded, requiring constant consideration of competition and thread switching, while SRS is still genuinely single-threaded. Go is more complicated to use, while SRS can still maintain the simplicity of single-threading.

SRS is a multi-threaded and coroutine-based architecture optimized for business, essentially still single-threaded, with threads being essentially unrelated.

Relationship with Source

A single ST thread will have multiple sources.

A source, which is a push stream and its corresponding consumer (playback), is only in one ST thread.

In this way, both push and play are completed in a single ST thread, without the need for locks or switching.

Since the client's URL is unknown when connecting, it is also unknown which stream it belongs to, so it may be accepted by the wrong ST thread, requiring FD migration.

Migrating FD between multiple threads is relatively simple. The difficulty lies in ST, which needs to support multi-threading and consider rebuilding the FD in the new ST thread's epoll when migrating FD. However, this is not particularly difficult, and it is much easier than multi-process.

Why not Multi-process

FD migration between multi-processes is too difficult to implement, and communication between processes is not as easy as communication between threads, nor is it as efficient as threads.

The reason why Nginx uses multi-process is that there is no need for FD migration between multiple processes. So when doing live streaming, NginxRTMP processes push streams to each other, which is too difficult to maintain.

If not migrated, audio and video packets need to be forwarded, and it is definitely better and more suitable for streaming media to migrate FD based on the stream.

Thread Local

Each thread has its own ST, which can be referred to as the Envoy Threading Model, using the C++ thread_local keyword to indicate variables.

I wrote an example SRS: thread-local.cpp, with the following results:

$ ./thread-local
PFN1: tl_g_nn(0x7fbd59504080)=1, g_obj(0x7fbd59504084)=1, gp_obj(0x7fbd59504088,0x7fbd595040a0)=1, gp_obj2(0x7fbd59504090,0x7fbd595040b0)=1
PFN2: tl_g_nn(0x7fbd59604080)=2, g_obj(0x7fbd59604084)=2, gp_obj(0x7fbd59604088,0x7fbd596040a0)=2, gp_obj2(0x7fbd59604090,0x7fbd596040b0)=2
MAIN: tl_g_nn(0x7fbd59704080)=100, g_obj(0x7fbd59704084)=100, gp_obj(0x7fbd59704088,0x7fbd597040a0)=100, gp_obj2(0x7fbd59704090,0x7fbd597040b0)=100

It can be used to modify global variables:

// Global thread local int variable.
thread_local int tl_g_nn = 0;

Including global pointers:

thread_local MyClass g_obj(0);
thread_local MyClass* gp_obj = new MyClass(0);
thread_local MyClass* gp_obj2 = NULL;
MyClass* get_gp_obj2()
{
    if (!gp_obj2) {
        gp_obj2 = new MyClass(0);
    }
    return gp_obj2;
}

The addresses and values of these pointers are different in each thread.

GCC __thread

GCC has extended the keyword __thread, which has the same effect as C++11's thread_local.

A multi-threaded version of ST has been implemented before, using gcc's __thread keyword, referring to toffaletti and ST#19.

UDP Binding

Note: Please note that we ultimately chose to implement RTC multi-thread isolation with multiple ports instead of using UDP binding, so I have collapsed related comments by default.

RTC's UDP is connectionless, and multiple threads can reuse the fd through REUSE_PORT to receive packets sent to the same port.

The kernel will perform a five-tuple binding. When the kernel delivers to a certain listen fd, it will continue to deliver to this fd. Refer to udp-client and udp-server:

Start 3 workers, at 127.0.0.1:8000
listen at 127.0.0.1:8000, fd=3 ok
listen at 127.0.0.1:8000, fd=4 ok
listen at 127.0.0.1:8000, fd=5 ok
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!
fd #5, peer 127.0.0.1:50331, got 13B, Hello world!

Note: There are three fds listening on port 8000 above. After the client 50331 delivers to fd=5, it will continue to deliver to this fd.

UDP Migration

If we receive a client packet from a certain fd, such as 3, and find that this client should be received by another fd, such as 4, we can use connect to bind the delivery relationship.

Refer to the example udp-connect-client.cpp and udp-connect-server.cpp. The server receives the packet and continuously uses other fds to connect. The performance is different on different platforms.

CentOS 7 server, listening on 0.0.0.0:8000, as shown below, can achieve migration twice:

Start 2 workers, at 0.0.0.0:8000
listen at 0.0.0.0:8000, fd=4, migrate_fd=3 ok
listen at 0.0.0.0:8000, fd=3, migrate_fd=4 ok

fd #3, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #3 to #4, r0=0, errno=0

fd #4, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #4 to #3, r0=0, errno=0

fd #3, peer 172.16.239.217:37846, got 13B, Hello world!
Transfer 172.16.239.217:37846 from #3 to #4, r0=0, errno=0
fd #3, peer 172.16.239.217:37846, got 13B, Hello world!

CentOS 7 server, if bound to a fixed address, such as eth0 or lo, will not migrate:

Start 2 workers, at 172.16.123.121:8000
listen at 172.16.123.121:8000, fd=4, migrate_fd=3 ok
listen at 172.16.123.121:8000, fd=3, migrate_fd=4 ok

fd #3, peer 120.227.88.168:43015, got 13B, Hello world!
Transfer 120.227.88.168:43015 from #3 to #4, r0=0, errno=0

fd #3, peer 120.227.88.168:43015, got 13B, Hello world!
Transfer 120.227.88.168:43015 from #3 to #4, r0=0, errno=0
fd #3, peer 120.227.88.168:43015, got 13B, Hello world!

Note: Linux multiple migrations will not return an error, but will not take effect.

Mac server, regardless of which address is bound, will migrate once:

Start 2 workers, at 127.0.0.1:8000
listen at 127.0.0.1:8000, fd=4, migrate_fd=3 ok
listen at 127.0.0.1:8000, fd=3, migrate_fd=4 ok

fd #3, peer 127.0.0.1:61448, got 13B, Hello world!
Transfer 127.0.0.1:61448 from #3 to #4, r0=0, errno=0

fd #4, peer 127.0.0.1:61448, got 13B, Hello world!
Transfer 127.0.0.1:61448 from #4 to #3, r0=-1, errno=48
fd #4, peer 127.0.0.1:61448, got 13B, Hello world!

Note: On Mac, multiple migrations will return an error [ERRNO 48] ADDRESS ALREADY IN USE.

After connecting with @wasphin, we don't want to migrate. Instead, we hope to bind the 5-tuple to this fd after connecting, so as to avoid other FDs receiving packets.

In this case, a more suitable thread model is:

The UDP port is listened and packets are sent and received by default by the public thread, and the processing thread sends and receives packets through the public thread.
If the processing thread finds that this 5-tuple is not what it should handle, it will transfer the processing to other threads through the message queue.
If the processing thread finds that the packets of this 5-tuple are to be processed by itself, it will open the FD and connect to this address, so that only this thread will directly send and receive packets in the future.

This model is actually a hybrid model:

Most of the time, there is no need to pass packets through locks and inter-thread communication queues.
In a few cases, especially when a new address has just started, packets can be passed through the message queue.
In the early stage of architecture evolution, it is possible not to connect, which means that packets will be passed between threads.

This hybrid model does not rely on UDP connect, but the performance will be very high when Connect works.

In addition, the encryption and decryption problem can also be solved by a similar hybrid model:

Start multiple independent encryption and decryption threads and pass packets through the queue.
If the performance of the working thread is sufficient, it can directly encrypt and decrypt by itself.
In the early stage of architecture evolution, there can be independent encryption and decryption threads, which means that packets will be passed between threads.

What's special is the disk IO thread, which will definitely use the queue to send messages:

The log writing thread collects logs from other threads through the queue and writes them to the disk.
The TS, FLV, and MP4 file writing threads, also known as recording threads, collect the file content to be written through the queue and write the content to the disk.

In the early days, we will still pass packets between multiple threads and divide different threads according to the business. As the evolution progresses, we will gradually eliminate the communication and dependencies between threads and turn them into independent threads that do not rely on each other, achieving higher performance.

The text was updated successfully, but these errors were encountered:

winlinvip · 2021-02-05T02:07:44Z

See #2188 (comment)

winlinvip · 2021-02-05T03:37:55Z

See #2188 (comment)

winlinvip · 2021-02-23T08:43:32Z

See #2188 (comment)

1. Use SrsThreadPool to execute the hybrid server. 2. Right now, directly run in primordial thread, that is no threads. 3. Define the main APIs of SrsThreadPool.

1. Create thread when execute by thread pool. 2. The primordial thread check all threads status. 3. Have not complete the cleanup and stop.

1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK. 2. If dead lock, the pthread_mutex_lock return EDEADLK. 3. We assert fail if lock failed.

1. Use SrsThreadPool to execute the hybrid server. 2. Right now, directly run in primordial thread, that is no threads. 3. Define the main APIs of SrsThreadPool.

1. Create thread when execute by thread pool. 2. The primordial thread check all threads status. 3. Have not complete the cleanup and stop.

1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK. 2. If dead lock, the pthread_mutex_lock return EDEADLK. 3. We assert fail if lock failed.

1. Remove support for reload log configs. 2. Add config for thread pool cycle interval. 3. Add config for log flush interval. 4. Create a SrsAsyncLogManager to create writers. 5. Create a SrsAsyncFileWriter to write file async.

1. Wrap std::vector to thread-safe queue. 2. Keep API compatible with std::vector. 3. SrsAsyncFileWriter use thread-safe queue instead.

1. Create dual queue, the coroutine queue and thread queue. 2. The coroutine queue cache logs does not require lock. 3. When need to flush, flush the logs from coroutine-queue to thread-queue. 4. Finally, flush thread-queue to disk.

1. App/User controls the interval to flush coroutine-queue. 2. Use srs_update_system_time to get time for log. 3. Stat the thread sync in us, in SrsThreadPool. 4. Change default interval for thread to 5s.

1. Dual queue for async logs, exists risk. 2. Flush the logs is too slow, because it depends on logs and interval.

1. It exists delay for multiple threads. 2. There is overlay for cache of coroutine queues. 3. Risk when other threads write logs.

1. If configed the async srtp, use a new object. 2. Allow sync and async srtp, by config.

commit f4872e5 Author: winlin <winlin@vip.126.com> Date: Fri Feb 26 09:13:21 2021 +0800 ST: For #2188: Remove sendmmsg from ST. commit aaeb891 Author: winlin <winlin@vip.126.com> Date: Thu Mar 11 08:09:54 2021 +0800 ST: Refine utest script. commit d1ac9da Author: winlin <winlin@vip.126.com> Date: Wed Mar 3 11:02:25 2021 +0800 ST: Support fast utest and coverage commit 8400115 Author: winlin <winlin@vip.126.com> Date: Fri Feb 26 07:02:19 2021 +0800 ST: Always use unserialized accept for linux or darwin commit c3686f2 Author: winlin <winlin@vip.126.com> Date: Fri Feb 26 06:54:05 2021 +0800 ST: Refine ARFLAGS by disable the verbose log commit aaa5c4f Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 08:58:46 2021 +0800 ST: Stack always grows from top to down. commit dddd466 Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 08:51:31 2021 +0800 ST: Ignore process fork, for single process only commit 7906cb5 Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 08:50:59 2021 +0800 ST: Fix build warnings commit d94921b Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 07:27:45 2021 +0800 ST: Remove select and poll support, only epoll and kqueue commit 76d2025 Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 07:10:47 2021 +0800 ST: Remove multiple OS support, except Linux and Darwin. commit 13c4ba3 Author: winlin <winlin@vip.126.com> Date: Thu Feb 25 06:59:35 2021 +0800 ST: Remove __ia64__ CPU support commit 46c06e4 Author: winlin <winlin@vip.126.com> Date: Wed Feb 24 11:37:27 2021 +0800 ST: Remove unused files for ST

commit Remove SEND/RECV/SRTP threads. Enable generate_streams by default. Support disable circuit breaker. commit 6c83e89 Author: winlin <winlin@vip.126.com> Date: Mon Apr 26 09:38:19 2021 +0800 Threads: Refine circuit breaker commit 614a781 Author: winlin <winlin@vip.126.com> Date: Sun Apr 25 19:52:13 2021 +0800 Threads-Hybrid: Fix rebase bugs commit 9c845c5 Author: winlin <winlin@vip.126.com> Date: Wed Apr 7 15:53:14 2021 +0800 Threads-Hybrid: Support multiple hybrid threads, 5.0.3 1. Support multiple hybrid threads. 2. Update benchmark data for 1/4/8/32 CPUs. 3. Update benchmark for Janus. commit 10edbb5 Author: winlin <winlin@vip.126.com> Date: Fri Apr 9 11:45:59 2021 +0800 Threads-Hybrid: Always response channel with error information commit 5074a4d Author: winlin <winlin@vip.126.com> Date: Fri Apr 9 11:21:45 2021 +0800 Threads-Hybrid: Use ST wait and IO for RECV/SEND thread. commit c8f2ff0 Author: winlin <winlin@vip.126.com> Date: Fri Apr 9 10:21:05 2021 +0800 Threads-Hybrid: Merge api to master thread. commit 50f03ad Author: winlin <winlin@vip.126.com> Date: Wed Apr 7 13:26:04 2021 +0800 Threads-Hybrid: Config auto generate stream config for hybrids. commit 11aada1 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 20:00:16 2021 +0800 Threads-Hybrid: Process api request one by one commit 963a0fa Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 19:46:53 2021 +0800 Threads-Hybrid: Change pps stat to thread-local. commit 9eb9b19 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 15:21:48 2021 +0800 Threads-Hybrid: Move acquire pid file from hybrid to pool. commit 97f6684 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 12:53:17 2021 +0800 Threads-Hybrid: Schedule connection to the sample hybrid by url commit eab4f89 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 12:20:38 2021 +0800 Threads-Hybrid: Change global variables to thread-local or with lock 1. Thread-Safe: Config subscribes, subscribe or unsubscribe. 2. Global-Shared: Async SRTP/RECV/SEND/Log use thread-safe objects. 3. Global-Shared: SRTP and DTLS certificate, without critical data. 4. Thread-Local: Blackhole, ResourceManager, StreamManager, ObjectCache, by design. 5. Global-Shared: Log and context, which use thread-safe objects. 6. Thread-Local: Pithy print for each thread. commit 7dbac15 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 09:10:16 2021 +0800 Threads-Hybrid: Add TODO as be thread-local commit babe8e6 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 08:22:48 2021 +0800 Threads-Hybrid: Start multiple hybrid threads commit 3367956 Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 07:59:37 2021 +0800 Threads-Hybrid: Support mulitple hybrid/stream servers 1. For hybrid RTMP/HTTP/RTC servers. 2. Config hybrids in threads. 3. Get hybrid server config with index. commit 6e2de90 Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 21:48:25 2021 +0800 Threads-Hybrid: Support communicate between threads by chan and slot 1. Hybrid thread is responder, API thread is initiator. 2. Responder read message from initiator-slot, write message to responder-slot. 3. Initiator write message to initiator-slot, read message from responder-slot. 4. Responder start a coroutine to consume requests and response it. commit 281350d Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 18:53:32 2021 +0800 Threads-Hybrid: Extract pipe and pair to communicate between threads 1. Pipe can be open by any threads, because it's only os FDs. 2. If pipe is open read/write, it's associated by ST, so we MUST free it by the same thread. 3. If open pipe in one thread, it's ok to free it directly, without close pipe. 4. If open read in a thread, then open write in another thread, user MUST close it correctly. commit 40e59ef Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 16:16:23 2021 +0800 Threads-Hybrid: Enable RTC play API with bugs commit 9f4fbdb Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 12:39:28 2021 +0800 Threads-Hybrid: Init ST for each threads 1. ST is thread-local now. 2. MUST init st for each threads. 3. Do it as early as possible. commit 2fa3aca Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 10:22:13 2021 +0800 Threads-Hybrid: Extract API server and threads. commit dd7c7ad Author: winlin <winlin@vip.126.com> Date: Sun Apr 4 20:44:07 2021 +0800 Threads-Hybrid: Refine conf file commit 246d7f5 Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 11:55:10 2021 +0800 Threads-Hybrid: Research for extern and __thread commit 1a0456a Author: winlin <winlin@vip.126.com> Date: Fri Apr 9 22:44:56 2021 +0800 Threads: TODO: Sort the packets, to avoid NACK commit b11f958 Author: winlin <winlin@vip.126.com> Date: Wed Mar 31 18:37:39 2021 +0800 Threads: Support multiple threads with locks, #2188. 5.0.2 1. Threads-Log: Use thread to write and reopen logs. 2. Threads-SRTP: Support decrypt RTP by async SRTP. 3. Threads-RECV: Support dedicate thread to recv UDP packets. 4. Threads: Support cpu affinity for threads. 5. Threads-RECV: Drop received packet if exceed max queue size. 6. Threads: Use coroutine to consume recv/srtp packets. 7. Threads: Support Circuit-Breaker to work in storms. 8. Threads-SRTP: Support async decrypt RTCP 9. Threads-SEND: Support async send UDP packets 10. Threads-SRTP: Use async encrypt SRTP packet 11. Threads-SEND/RECV: Bind handler to listener to support multiple ports. 12. Threads-RECV: Support tunnel for recv-srtp. 13. Threads-SEND: Support tunnel for srtp-send. 14. Threads: Support circuit-breaker dying threshold commit 8cf7ead Author: winlin <winlin@vip.126.com> Date: Tue Apr 6 20:54:36 2021 +0800 Threads: Enable threads support for openssl. commit 28d8f1a Author: winlin <winlin@vip.126.com> Date: Wed Apr 7 10:29:18 2021 +0800 Threads: Support multiple SRTP/SEND/RECV threads. 1. Move received and cooked packets queue to thread entry, that is, each thread has its own queue. 2. In RECV thread, push received packet to queue of source thread, in thread listener. 3. In SRTP thread, push cooked packet to queue of source thread, in asyn SRTP task. 4. In hybrid thread, directly and only consume the packets of self thread. 5. Sync between SRTP task by lock. commit a505b6b Author: winlin <winlin@vip.126.com> Date: Mon Apr 5 16:17:08 2021 +0800 Threads: Directly use hybrid thread to consume messages. commit 14bc7bc Author: winlin <winlin@vip.126.com> Date: Tue Mar 30 16:38:30 2021 +0800 Threads: Keep alive when got RTP plaintext commit e15737f Author: winlin <winlin@vip.126.com> Date: Mon Mar 29 17:18:04 2021 +0800 Threads: Support circuit-breaker dying threshold commit d6a92cb Author: winlin <winlin@vip.126.com> Date: Tue Mar 30 14:14:26 2021 +0800 Threads-SEND: Support tunnel for srtp-send. commit d282ccd Author: winlin <winlin@vip.126.com> Date: Mon Mar 29 12:02:41 2021 +0800 Threads-RECV: Support tunnel for recv-srtp. commit eb7ce7f Author: winlin <winlin@vip.126.com> Date: Mon Mar 22 19:09:04 2021 +0800 Threads-RECV: Reset the cache buffer when copy commit 1235b33 Author: winlin <winlin@vip.126.com> Date: Mon Mar 22 18:45:12 2021 +0800 Threads-SEND/RECV: Bind handler to listener to support multiple ports. commit 615da26 Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 18:52:11 2021 +0800 Threads-SRTP: Use async encrypt SRTP packet 1. Async SRTP support protect RTCP. 2. Send packet ignore when encrypt size is 0. 3. Callback to send packet if encrypt done. commit a26b926 Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 18:43:08 2021 +0800 Threads-SRTP: Use async encrypt SRTP packet 1. Async SRTP support protect RTP. 2. Send packet ignore when encrypt size is 0. 3. Callback to send packet if encrypt done. commit 56ffc28 Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 18:03:40 2021 +0800 Threads-SEND: Support async send UDP packets 1. Support async send UDP by SrsAsyncSendManager. 2. Copy UDP packet by SrsAsyncUdpPacket. 3. Support SrsUdpMuxSocket raw sendto. 4. Config the async send by async_send. commit b0800f5 Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 16:16:09 2021 +0800 Threads-SRTP: Support async decrypt RTCP 1. SrsAsyncSRTP support unprotect_rtcp packet. 2. Extract on_rtcp_plaintext from on_rtcp. commit 949c80e Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 14:58:34 2021 +0800 Threads-RECV: Change UDP recv max size from 6k to 1500 bytes. commit 198dca5 Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 13:48:36 2021 +0800 Threads: Merge recv and srtp consume to one timer. commit 57b771a Author: winlin <winlin@vip.126.com> Date: Fri Mar 19 11:30:46 2021 +0800 Threads: Refine variables and do dispose 1. Rename packets to srtp or received packets. 2. Add task dispose API, cleanup in future. 3. If got packets before init AsyncSRTP, return error. 4. Never free the SRTPTask, dispose it instead. commit 9a05d24 Author: winlin <winlin@vip.126.com> Date: Thu Mar 18 17:33:51 2021 +0800 Threads: Support Circuit-Breaker to work in storms. 1. Config the recv queue, drop packet if exceed. 2. Config the high and critical threshold and pulse of water level. 3. If critical water level, disable NACK and TWCC. 4. If high water level, ignore for NACK insert and send. 5. Support read the CPU of thread. 6. Refine SrsPps to support r1s sample. commit 1c90497 Author: winlin <winlin@vip.126.com> Date: Wed Mar 17 13:53:55 2021 +0800 Threads: Use coroutine to consume recv/srtp packets. commit 957034e Author: winlin <winlin@vip.126.com> Date: Wed Mar 17 08:08:17 2021 +0800 Threads: Use thread-local buffer for log 1. Call SrsThreadPool::setup() in main(),or each thread starting. 2. Initialize the thread-local object in SrsThreadPool::setup(). 3. Change shared log buffer to thread-local. commit 185359f Author: winlin <winlin@vip.126.com> Date: Tue Mar 16 20:29:35 2021 +0800 Threads-RECV: Show the dropped packets pps. commit 6fca411 Author: winlin <winlin@vip.126.com> Date: Tue Mar 16 20:26:41 2021 +0800 Threads-RECV: Drop received packet if exceed max queue size. 1. Print the number of recv/srtp queue packets. 2. Drop packet if exceed max recv queue size. commit 9e554ca Author: winlin <winlin@vip.126.com> Date: Tue Mar 16 19:26:44 2021 +0800 Threads: Support cpu affinity for threads. 1. Config cpu_affinity in threads. 2. Default to not set the cpu affinity. 3. Support set by cpu range 0-63. commit fa4c9f9 Author: winlin <winlin@vip.126.com> Date: Tue Mar 16 11:42:32 2021 +0800 Threads: Set the threads name display in top. commit 1c6e941 Author: winlin <winlin@vip.126.com> Date: Tue Mar 16 08:30:21 2021 +0800 Threads-RECV: Refine the stat for SNMP UDP recv/error 1. Remove the delta of _srs_snmp_udp_stat. 2. Use _srs_pps_rloss for receive loss rate. 3. Use _srs_pps_sloss for send loss rate. commit 85ea39d Author: winlin <winlin@vip.126.com> Date: Mon Mar 15 22:51:58 2021 +0800 Threads-RECV: Support dedicate thread to recv UDP packets. 1. Use SrsUdpMuxSocket::raw_recvfrom to read, without ST. 2. Start a UDP recv thread, to recv packets. 3. Consume UDP packets in RTC server timer. commit e3686a4 Author: winlin <winlin@vip.126.com> Date: Mon Mar 15 21:15:28 2021 +0800 Threads: Fix bug for SRTP and Log thread nanosleep. commit 88165ba Author: winlin <winlin@vip.126.com> Date: Mon Mar 15 20:36:20 2021 +0800 Threads-SRTP: Support decrypt RTP by async SRTP. 1. Create dedicate thread for async srtp. 2. SrsSecurityTransport use async srtp if config on. 3. Cook SRTP packets in async srtp. 4. Consume cooked SRTP packets in RTC server timer. commit f068540 Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 21:39:36 2021 +0800 Threads-SRTP: Config and add files for the async-srtp 1. If configed the async srtp, use a new object. 2. Allow sync and async srtp, by config. commit 2367483 Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 22:11:02 2021 +0800 Threads-Log: Refine stat for sync wait, in log thread. commit 6a1d6a0 Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 18:43:54 2021 +0800 Threads-Log: Remove dual queue for sys logs. 1. It exists delay for multiple threads. 2. There is overlay for cache of coroutine queues. 3. Risk when other threads write logs. commit 6ddbc5f Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 18:30:23 2021 +0800 Threads-Log: Refine comments for global variable. 1. Dual queue for async logs, exists risk. 2. Flush the logs is too slow, because it depends on logs and interval. commit 6718c38 Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 10:42:09 2021 +0800 Threads-Log: Refine dual queue for log thread. 1. App/User controls the interval to flush coroutine-queue. 2. Use srs_update_system_time to get time for log. 3. Stat the thread sync in us, in SrsThreadPool. 4. Change default interval for thread to 5s. commit 37aee44 Author: winlin <winlin@vip.126.com> Date: Sun Mar 14 08:55:17 2021 +0800 Threads-Log: Support dual queue cache for async logs. 1. Create dual queue, the coroutine queue and thread queue. 2. The coroutine queue cache logs does not require lock. 3. When need to flush, flush the logs from coroutine-queue to thread-queue. 4. Finally, flush thread-queue to disk. commit 4918160 Author: winlin <winlin@vip.126.com> Date: Sat Mar 13 07:09:56 2021 +0800 Threads-Log: Support thread-safe queue SrsThreadQueue. 1. Wrap std::vector to thread-safe queue. 2. Keep API compatible with std::vector. 3. SrsAsyncFileWriter use thread-safe queue instead. commit 3810f5f Author: winlin <winlin@vip.126.com> Date: Fri Mar 12 18:59:14 2021 +0800 Threads-Log: Remove utest for reload log configs. commit bf95fdf Author: winlin <winlin@vip.126.com> Date: Fri Mar 12 18:48:03 2021 +0800 Threads-Log: Use thread to write and reopen logs. 1. Remove support for reload log configs. 2. Add config for thread pool cycle interval. 3. Add config for log flush interval. 4. Create a SrsAsyncLogManager to create writers. 5. Create a SrsAsyncFileWriter to write file async. commit 0cea382 Author: winlin <winlin@vip.126.com> Date: Fri Mar 12 12:02:36 2021 +0800 Threads-Log: Refine thread lock type to ERRORCHECK. 1. Set lock attribute type to PTHREAD_MUTEX_ERRORCHECK. 2. If dead lock, the pthread_mutex_lock return EDEADLK. 3. We assert fail if lock failed. commit 668c9c1 Author: winlin <winlin@vip.126.com> Date: Fri Mar 12 08:45:10 2021 +0800 Threads-Log: Run hybrid server in thread. 1. Create thread when execute by thread pool. 2. The primordial thread check all threads status. 3. Have not complete the cleanup and stop. commit 831b77b Author: winlin <winlin@vip.126.com> Date: Thu Mar 11 18:06:16 2021 +0800 Threads-Log: Refine API and main workflow. 1. Use SrsThreadPool to execute the hybrid server. 2. Right now, directly run in primordial thread, that is no threads. 3. Define the main APIs of SrsThreadPool.