Helm chart logsCollection Failed to process entry #718
Labels
Comments
TylerHelmuth
added
the
chart:collector
|
@jonaskello please share the log that caused the error. |
|
Yes, the whole file it's quite big but here is a sample of it: |
|
It appears your timestamps have too many digits in after the |
|
The logs are from a worker node in the k8s cluster where the chart is deployed. The worker node has this config: So the logs are from containerd. These are the datetime setting on the node (not sure if it is relevant): |
|
The regex for containerd logs which is here seems to assume So this string does not match: Because of the |
|
Interesting. From the original error we can see that the filelog receiver routed the log to routes:
- output: parser-docker
expr: 'body matches "^\\{"'
- output: parser-crio
expr: 'body matches "^[^ Z]+ "'
- output: parser-containerd
expr: 'body matches "^[^ Z]+Z"'So it is expecting containerd logs to use I believe there are a couple possibilities here:
The cri-o parser would be able to handle your timezone offset if the millisecond precision matched. @djaglowski @dmitryax do you see anything out of the ordinary with our operators? |
|
While researching this issue I found this which may be of interest:
It gives this example:
Original found linked in this issue. |
|
Found this in one of the worker's logs:
Something weird seems to be going on with the way containerd is creating the logs on our workers. Sometimes, for some entries the number of chars for milliseconds is 8 instead of 9. So I think otel-collector is correct here, it is chosing the cri logging format which is also what containerd is using. Got thrown off by the |
|
I researched the format for the Comparing that to the format layout used in the helm chart here which is: I could see they did not exactly look the same so I switched the one in the otel collector config to look exactly like the one defined in golang time pacakge, and now the error is not appearing anymore. I have no clue why, but I'm happy it is working now :-) |
|
@jonaskello, thanks for digging into this. It looks like a change we should make in the chart. Using Using |
jonaskello
added a commit
to jonaskello/opentelemetry-helm-charts
that referenced
this issue
Mar 31, 2023
|
Made a PR in #721 with the change that worked for me. |
jonaskello
added a commit
to jonaskello/opentelemetry-collector-contrib
that referenced
this issue
Mar 31, 2023
See related discussion and PR in open-telemetry/opentelemetry-helm-charts#718
TylerHelmuth
pushed a commit
that referenced
this issue
Apr 3, 2023
* Update time layout for cri-o log format Fixes #718 * bump chart version and generate examples
TylerHelmuth
pushed a commit
to open-telemetry/opentelemetry-collector-contrib
that referenced
this issue
Jun 7, 2023
* Fix cri-o log format time layout See related discussion and PR in open-telemetry/opentelemetry-helm-charts#718 * Create fix-cri-o-log-format-time-layout.yaml Add changelog file * Update fix-cri-o-log-format-time-layout.yaml Remove external issue as it seems it does not validate * Update fix-cri-o-log-format-time-layout.yaml Add issue number
Caleb-Hurshman
pushed a commit
to observIQ/opentelemetry-collector-contrib
that referenced
this issue
Jul 6, 2023
* Fix cri-o log format time layout See related discussion and PR in open-telemetry/opentelemetry-helm-charts#718 * Create fix-cri-o-log-format-time-layout.yaml Add changelog file * Update fix-cri-o-log-format-time-layout.yaml Remove external issue as it seems it does not validate * Update fix-cri-o-log-format-time-layout.yaml Add issue number
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When using the helm chart and enabling value:
The following error is logged in the otel pods:
Seems related to #376 which was never resolved.
The text was updated successfully, but these errors were encountered: