Grafa/Loki/Promtail : syslog configuration, how I can visualize timestamp from the syslog message

[BriceNeo]

Describe the bug
Hello, I use promtail to distributed syslog message to loki and visualize them by Grafana. In the syslog configuration of promtail, we can't extract timestamp of syslog message. The timestamp transmit to loki is the intern timestamp of promtail not the timestamp from the syslog message. I tried many configuration but I don't find solution.

To Reproduce
Steps to reproduce the behavior:

1. Launch loki
2. Take this configuration to launch promtail :
   server:
   http_listen_port: 9080
   grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:

• url: http://loki:3100/loki/api/v1/push

scrape_configs:

• job_name: syslog
  syslog:
  listen_address: 0.0.0.0:1514
  idle_timeout: 3600s

  relabel_configs:

  • source_labels: ['__syslog_message_hostname']
    target_label: host
  • source_labels: ['__syslog_message_severity']
    target_label: level
  • source_labels: ['__syslog_message_app_name']
    target_label: app_name
  • source_labels: [__syslog_message_facility]
    target_label: facility

3. Send syslog message in RFC5424 in TCP on the port 1514
4. Check the timestamp of your syslog message and the timestamp display in Grafana isn't the same

Expected behavior
I expect that I can at least visualize the timestamp from the syslog message in Grafana and if possible remplace the timestamp used in Grafana by the timestamp from my message

Thanks for your help

[slim-bean]

Hey @BriceNeo! It just so happens I was looking at this exact code last week to troubleshoot a different issue.

I noticed that it is the case that the current code forces a time.Now() on every syslog message:

loki/pkg/promtail/targets/syslog/syslogtarget.go

Line 232 in b24a143

if err := t.handler.Handle(msg.labels, time.Now(), msg.message); err != nil {

I think this could be improved to also try to use the timestamp of the log message, the biggest concern is issues with out of order timestamps, I'm not sure how much of a problem this would actually be though and we should support using the syslog timestamp in Loki.

[BriceNeo]

Thanks for your quick reply,
Yes, I understand the concern, maybe it could be configurable in promtail to choose which timestamp we want to use, with a warning to explain this concern and the possible log lost when we choose the timestamp from log.