Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support async logging #2413

Merged
merged 5 commits into from
Jan 23, 2024
Merged

Support async logging #2413

merged 5 commits into from
Jan 23, 2024

Conversation

chenBright
Copy link
Contributor

What problem does this PR solve?

Issue Number:

Problem Summary:

What is changed and the side effects?

Changed:

使用ExecutionQueue的相关技术支持异步日志

Side effects:

  • Performance effects(性能影响):

  • Breaking backward compatibility(向后兼容性):


Check List:

  • Please make sure your changes are compilable(请确保你的更改可以通过编译).
  • When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
  • Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

src/butil/logging.cc Outdated Show resolved Hide resolved
src/butil/logging.cc Outdated Show resolved Hide resolved
src/butil/logging.cc Show resolved Hide resolved
@wwbmmm
Copy link
Contributor

wwbmmm commented Oct 25, 2023

LGTM

@yanglimingcn
Copy link
Contributor

请问这个日志名字是怎么定义的,日志会根据大小切分吗

@chenBright
Copy link
Contributor Author

chenBright commented Dec 21, 2023

日志名通过butil::InitLogging LoggingSettings设置

brpc/src/butil/logging.h

Lines 244 to 280 in 2098dd3

struct BUTIL_EXPORT LoggingSettings {
// The defaults values are:
//
// logging_dest: LOG_DEFAULT
// log_file: NULL
// lock_log: LOCK_LOG_FILE
// delete_old: APPEND_TO_OLD_LOG_FILE
LoggingSettings();
LoggingDestination logging_dest;
// The three settings below have an effect only when LOG_TO_FILE is
// set in |logging_dest|.
const LogChar* log_file;
LogLockingState lock_log;
OldFileDeletionState delete_old;
};
// Implementation of the InitLogging() method declared below.
BUTIL_EXPORT bool BaseInitLoggingImpl(const LoggingSettings& settings);
// Sets the log file name and other global logging state. Calling this function
// is recommended, and is normally done at the beginning of application init.
// If you don't call it, all the flags will be initialized to their default
// values, and there is a race condition that may leak a critical section
// object if two threads try to do the first log at the same time.
// See the definition of the enums above for descriptions and default values.
//
// The default log file is initialized to "<process-name>.log" on linux and
// "debug.log" otherwise.
//
// This function may be called a second time to re-direct logging (e.g after
// loging in to a user partition), however it should never be called more than
// twice.
inline bool InitLogging(const LoggingSettings& settings) {
return BaseInitLoggingImpl(settings);
}

@chenBright
Copy link
Contributor Author

chenBright commented Dec 21, 2023

目前框架不支持日志滚动,需要用户自己实现日志滚动逻辑,然后在LoggingLock的保护下更换log_file。

不过,很多日志库都支持日志滚动。框架支持日志滚动特性应该是个合理的需求。

@chenBright
Copy link
Contributor Author

  1. 每次都会消费完所有日志,所以传入IsComplete的肯定是singular node,同时也不需要获取cur_tail。已删除相关逻辑。
  2. 进程退出时候,将异步日志队列的日志落盘,并退出异步日志线程。如果在停异步日志线程的时候,有异步日志入队,则有第一个日志所在的线程/协程负责将队列的日志落盘。停止后的异步日志回退为同步日志。为了保证最后的异步日志能全部落盘,在异步日志线程停止(Join结束)后,sleep一小段时间。

@wwbmmm
Copy link
Contributor

wwbmmm commented Jan 9, 2024

LGTM

@chenBright
Copy link
Contributor Author

chenBright commented Jan 14, 2024

当日志打印频率比较低的时候,异步日志有可能退化为每个日志都包含两个动作:signal异步日志线程、异步日志线程落盘。
针对这种场景,采用Socket::Write的策略,同步落盘队头的日志,队列的其他日志通过异步日志线程落盘,减少signal的消耗。对于日志打印频率比较高的场景,可通过开关支持异步日志线程来落盘全部日志。

@wwbmmm
Copy link
Contributor

wwbmmm commented Jan 17, 2024

LGTM

@wwbmmm wwbmmm merged commit 313258e into apache:master Jan 23, 2024
18 checks passed
@chenBright chenBright deleted the async_logging branch January 23, 2024 06:34
jiangdongzi pushed a commit to jiangdongzi/brpc that referenced this pull request Jan 31, 2024
* Support async logging

* Set max async log queue size

* Flush async log before exit

* Use sync log for the first LogRequest

* Support usec of logging for macOS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants