filebeat 6.6.1 invalid CRI format with nginx logs

[ITBlogger]

• Version: 6.6.1
• Operating System: ubuntu 18.04
• Steps to Reproduce:

Getting errors of:
2019-02-27T05:00:20.577Z ERROR log/harvester.go:282 Read line error: invalid CRI log format; File: /var/lib/docker/containers/363dbdf50946a40b72b60937ab73a63a1f533998d04338abe89157467ac5a94d/363dbdf50946a40b72b60937ab73a63a1f533998d04338abe89157467ac5a94d-json.log

With nginx logs such as:
{"log":"10.50.131.251 - - [27/Feb/2019:05:00:28 +0000] "GET /status HTTP/1.1" 200 8 "-" "kube-probe/1.12"\n","stream":"stdout","time":"2019-02-27T05:00:28.617850486Z"}

This is with kubernetes auto-discover feature enabled.
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations: '*'

Should be noted that we have annotations to exclude these lines:
co.elastic.logs/exclude_lines: '/status|Sent\s200\sin\s[0-9]+(?:\P{M}\p{M}*)+s$'

So not sure why filebeat is complaining instead of just ignoring the lines.

[ElfoLiNk]

i'm also experiecing the same with filebeat 6.6.2 on kubernetes

2019-03-28T15:42:33.334Z	INFO	log/harvester.go:255	Harvester started for file: /var/lib/docker/containers/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985-json.log
2019-03-28T15:42:33.334Z	ERROR	log/harvester.go:282	Read line error: invalid CRI log format; File: /var/lib/docker/containers/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985-json.log

$ sudo head /var/lib/docker/containers/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985/e9658c7be9b468a8b9366f330c42df0ec8ec8205944e5a8c1dd3520be3ff2985-json.log
{"log":"time=\"2019-03-28T05:01:33Z\" level=info msg=\"Traefik version v1.7.9 built on 2019-02-11_11:36:32AM\"\n","stream":"stdout","time":"2019-03-28T05:01:33.786505734Z"}
{"log":"time=\"2019-03-28T05:01:33Z\" level=info msg=\"\\nStats collection is disabled.\\nHelp us improve Traefik by turning this feature on :)\\nMore details on: https://docs.traefik.io/basics/#collected-data\\n\"\n","stream":"stdout","time":"2019-03-28T05:01:33.786561691Z"}

[kaiyan-sheng]

@ElfoLiNk Have you gave combine_partial a try?

[exekias]

We have fixed something similar to this in the past, did you upgrade from a previous version or is this a clean install? If you were using 6.6.0 or lower before upgrading the registry file could contain wrong offsets, explaining this.

[odacremolbap]

@ITBlogger @ElfoLiNk

I've tried to reproduce with no success, using kubernetes 1.14 and filebeat 6.6 / 6.7
As posted originally, I'm using filebeat kubernetes autodiscover + hints + an annotation applied to an nginx pod.

Would you mind sharing full config yamls items for filebeat?

[odacremolbap]

I think I was able to reproduce for a very short while when switching versions, 6.7 to 7.0 and back, but It is being hard to reproduce again.

[odacremolbap]

Closing this ticket since there have been some improvements in recent releases regarding CRI
#12162