feat: support host monitor #1890

Abingcbc · 2024-11-18T02:08:43Z

打通进程元信息采集链路

TODO：

支持更多字段
可观测指标

core/common/timer/HostMonitorTimerEvent.cpp

core/models/PipelineEventGroup.h

core/runner/sink/http/HttpSink.cpp

core/host_monitor/collector/MockCollector.cpp

core/host_monitor/collector/CollectorManager.cpp

messixukejia · 2024-11-25T07:20:08Z

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

+const std::string ProcessorHostMetaNative::sName = "processor_host_meta_native";
+
+bool ProcessorHostMetaNative::Init(const Json::Value& config) {
+    auto hostType = ToString(getenv(DEFAULT_ENV_KEY_HOST_TYPE.c_str()));


这个机制也不合理。应该有个全局的管理中心，各业务方直接读值即可。这个事情日会上提下，讨论下。

新pr的实现也是每次再全新获取，跟之前提到的实现也不符合。

core/constants/EntityConstants.cpp

core/host_monitor/SystemInformationTools.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.cpp

core/common/FileSystemUtil.h

core/common/StringTools.cpp

core/common/timer/HostMonitorTimerEvent.cpp

core/plugin/input/InputHostMeta.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/HostMonitorInputRunner.cpp

messixukejia · 2024-11-30T10:28:11Z

core/common/timer/Timer.h

+    Timer& operator=(const Timer&) = delete;
+    Timer& operator=(Timer&&) = delete;
+    ~Timer() = default;
+    static Timer* GetInstance() {


改成单例，之前prometheus等使用的地方没有问题？

messixukejia · 2024-11-30T10:30:06Z

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

+const std::string ProcessorHostMetaNative::sName = "processor_host_meta_native";
+
+bool ProcessorHostMetaNative::Init(const Json::Value& config) {
+    auto hostType = ToString(getenv(DEFAULT_ENV_KEY_HOST_TYPE.c_str()));


新pr的实现也是每次再全新获取，跟之前提到的实现也不符合。

core/runner/sink/http/HttpSink.cpp

messixukejia · 2024-11-30T10:37:29Z

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

+    targetEvent->SetContent("binary", sourceEvent.GetContent(DEFAULT_CONTENT_KEY_PROCESS_BINARY));
+    targetEvent->SetContent("arguments", sourceEvent.GetContent(DEFAULT_CONTENT_KEY_PROCESS_ARGUMENTS));
+    targetEvent->SetContent("language", sourceEvent.GetContent(DEFAULT_CONTENT_KEY_PROCESS_LANGUAGE));
+    targetEvent->SetContent("containerID", sourceEvent.GetContent(DEFAULT_CONTENT_KEY_PROCESS_CONTAINER_ID));


需要统一到constans里字段。需要避免同一场景一会下划线，一会大小写的问题。

messixukejia · 2024-11-30T10:42:06Z

core/host_monitor/collector/ProcessCollector.h

+class ProcessCollector : public BaseCollector {
+public:
+    ProcessCollector() : mProcessSilentCount(INT32_FLAG(process_collect_silent_count)) {
+        mName = "process";


多处使用的字符串，避免直接使用字符串

messixukejia · 2024-11-30T11:12:22Z

core/host_monitor/collector/ProcessCollector.cpp

+    }
+}
+
+void ProcessCollector::SortProcessByCpu(std::vector<ProcessStatPtr>& processStats, size_t topN) {


这里面实现并不只是sort还有获取，函数名不匹配

messixukejia · 2024-11-30T11:13:51Z

core/host_monitor/collector/ProcessCollector.cpp

+
+    int readCount = 0;
+    WalkAllProcess(PROCESS_DIR, [&](const std::string& dirName) {
+        if (++readCount > mProcessSilentCount) {


这里要看下进程超多时，对timer框架的影响
1、是否影响其他场景
2、单次执行时间大于调度时间的策略

messixukejia · 2024-11-30T11:16:53Z

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

+
+namespace logtail {
+
+const std::string ProcessorHostMetaNative::sName = "processor_host_meta_native";


inner processor存在的必要性是什么？为什么不能collect时直接补充字段

messixukejia · 2024-11-30T11:19:44Z

core/host_monitor/collector/ProcessCollector.cpp

+    const auto now = steady_clock::now();
+
+    // TODO: more accurate cache
+    auto prev = GetPreProcessStat(pid);


mPrevProcessStat 使用场景很局部，都是内部使用，保持统一。

messixukejia · 2024-11-30T11:20:51Z

core/host_monitor/collector/ProcessCollector.cpp

+
+    int readCount = 0;
+    WalkAllProcess(PROCESS_DIR, [&](const std::string& dirName) {
+        if (++readCount > mProcessSilentCount) {


个数控制是不是也在控制缓存里的？不应该只是控制去直接操作系统交互的吗

Abingcbc force-pushed the host_monitor branch 6 times, most recently from 6150e52 to bfdd9c2 Compare November 19, 2024 06:27

messixukejia requested changes Nov 25, 2024

View reviewed changes

henryzhx8 requested changes Nov 26, 2024

View reviewed changes

feat: support host monitor

3e12438

Abingcbc force-pushed the host_monitor branch from 4110e3d to 3e12438 Compare November 27, 2024 06:22

Abingcbc added 2 commits November 29, 2024 00:01

fix

8f9f8ee

Merge remote-tracking branch 'alibaba/main' into host_monitor

0455a6c

messixukejia requested changes Nov 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support host monitor #1890

feat: support host monitor #1890

Abingcbc commented Nov 18, 2024

messixukejia Nov 25, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024

messixukejia Nov 30, 2024


		namespace logtail {

		const std::string ProcessorHostMetaNative::sName = "processor_host_meta_native";

feat: support host monitor #1890

Are you sure you want to change the base?

feat: support host monitor #1890

Conversation

Abingcbc commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment