ilogtail采集数据中缺乏采集文件路径

[shalousun]

下面是通过ilogtail采集容器标准输出日志后控制台的打印

{
	"Time": 1658828611,
	"Contents": [{
		"Key": "content",
		"Value": "2022-07-26 17:21:48.544 INFO [springboot-docker] [main] org.quartz.core.SchedulerSignalerImpl : Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl"
	}, {
		"Key": "_time_",
		"Value": "2022-07-26T09:21:48.545694203Z"
	}, {
		"Key": "_source_",
		"Value": "stdout"
	}, {
		"Key": "_image_name_",
		"Value": "registry.cn-shanghai.aliyuncs.com/shalousun/springboot:2.6.6-tomcat"
	}, {
		"Key": "_container_name_",
		"Value": "springboot-tomcat"
	}, {
		"Key": "_pod_name_",
		"Value": "springboot-tomcat-deployment-fd7fcd86b-fsrbm"
	}, {
		"Key": "_namespace_",
		"Value": "default"
	}, {
		"Key": "_pod_uid_",
		"Value": "4c52f763-06aa-424d-a0a2-ecfc67832263"
	}, {
		"Key": "_container_ip_",
		"Value": "172.30.166.182"
	}, {
		"Key": "time",
		"Value": "2022-07-26 17:21:48.544"
	}, {
		"Key": "level",
		"Value": "INFO"
	}, {
		"Key": "appname",
		"Value": "springboot-docker"
	}, {
		"Key": "thread",
		"Value": "main"
	}, {
		"Key": "class",
		"Value": "org.quartz.core.SchedulerSignalerImpl"
	}, {
		"Key": "message",
		"Value": "Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl"
	}]
}

对比其它开源例如fluent bit会展示容器的标准输出采集文件的路径

{
	"@timestamp": 1659247019.12391,
	"log_file_path": "/var/log/containers/springboot-tomcat-deployment-6857974976-svj6m_default_springboot-tomcat-7eb2033b4750551effcfe525c19323247b71804ecb996f28ce1caeaa234f8503.log",
	"log": "{\"date\":\"2022-07-31 13:56:59.123\",\"appname\":\"springboot-docker\",\"thread\":\"http-nio-8080-exec-2\",\"level\":\"ERROR\",\"class\":\"com.benchmark.springboot.controller.LogController\",\"msg\":\"error log\"}\n",
	"stream": "stdout",
	"log_processed": {
		"date": "2022-07-31 13:56:59.123",
		"appname": "springboot-docker",
		"thread": "http-nio-8080-exec-2",
		"level": "ERROR",
		"class": "com.benchmark.springboot.controller.LogController",
		"msg": "error log"
	},
	"kubernetes": {
		"pod_name": "springboot-tomcat-deployment-7d477c67d8-dctld",
		"namespace_name": "default",
		"pod_id": "4e219086-a1b5-473a-87ea-10db2945acd3",
		"labels": {
			"app": "springboot-tomcat",
			"pod-template-hash": "7d477c67d8",
			"tier": "backend",
			"version": "1.0.0"
		},
		"annotations": {
			"cni.projectcalico.org/podIP": "172.30.166.167/32",
			"cni.projectcalico.org/podIPs": "172.30.166.167/32",
			"co.elastic.logs/multiline.match": "after",
			"co.elastic.logs/multiline.negate": "true",
			"co.elastic.logs/multiline.pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}",
			"co.elastic.logs/multiline.type": "pattern",
			"prometheus.io/path": "/benchmark/actuator/prometheus",
			"prometheus.io/port": "8080",
			"prometheus.io/scrape": "true"
		},
		"host": "node1",
		"container_name": "springboot-tomcat",
		"docker_id": "67954e7dbae0a259467031a7fafd866b78afe05e89b4f9e1dc56fa809048f5b6",
		"container_hash": "registry.cn-shanghai.aliyuncs.com/shalousun/springboot@sha256:e9d252826eb94b5e6baa9824857e3df0fc96ef19d76c6f389eeaa5bf068adc54",
		"container_image": "registry.cn-shanghai.aliyuncs.com/shalousun/springboot:2.6.6-tomcat"
	}
}

fluent bit或者是filebeat会把当时采集文件的路径/var/log/containers/springboot-tomcat-deployment-6857974976-svj6m_default_springboot-tomcat-7eb2033b4750551effcfe525c19323247b71804ecb996f28ce1caeaa234f8503.log 返回。

当然对于容器标准日志输出的才加也可以不要返回采集路径，可以看社区未来是否有人需要。

[messixukejia]

1、首先需要明确下iLogtail内部数据流涉及的数据结构，顶层结构为LogGroup，子字段中Topic、LogTags、Logs都是跟日志相关的一些字段，具体可见链接。这个数据结构是跟SLS存储模型对应的，日志公共的一些信息会存在LogTags里，因此Logs跟LogTags组合才是完整的日志内容。

2、标准输出，虽然没有问题中提到的log_file_path（这个路径里有很多业务属性不太强的字符），但是ilogtail是有一些等价信息的：pod name、容器名、类型是stdout还是stderr，而这些信息是业务强关联的，所以友好型更强一些。

下图是标准输出采集到SLS后的结果（pod name、容器名、类型都是有的），因为SLS存储会去感知LogTags、Logs所以内容自然就展开了。

之所以上面输出到kafka的例子没有这些信息，是因为这些信息作为日志公共字段存在LogTags里了。所以kafka flusher里解下LogTags拼到日志里就可以了。

3、文件采集，日志中是有文件路径的：