promtail/journal: cannot seek back to saved position

[fatpat]

Describe the bug
When promtail is configured to scrape logs from journald it is supposed to remember the position of the logs in the position file. When promtail is restarted it can't read back saved position from journal with the following error:

level=error ts=2020-05-21T05:53:29.024009194Z caller=journaltarget.go:219 msg="received error reading saved journal position" err="failed to get realtime timestamp: cannot assign requested address"

Note: on ubuntu, the error is not cannot assign requested address but 99 which is the same reality (error message vs error code).

The side effect is that is then reread logs from journal up to journal.max_age which can cause some troubles like:

• burst of level=error ts=2020-05-21T05:58:14.901208619Z caller=client.go:247 component=client host=172.30.0.101:6902 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry with timestamp 2020-05-21 04:40:42.937458 +0000 UTC ignored, reason: 'entry out of order' for stream: {host=\"node1.novalocal\", job=\"systemd-journal\", log_type=\"access\"},"
• and loki could then complains for too much requests with level=warn ts=2020-05-21T06:15:39.861839553Z caller=client.go:242 component=client host=172.30.0.101:6902 msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Ingestion rate limit exceeded (limit: 4194304 bytes/sec) while attempting to ingest '333' lines totaling '102198' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased"

To Reproduce
Steps to reproduce the behavior:

1. Started Loki 1.5.0
2. Started Promtail 1.5.0
3. check error logs

Expected behavior
promtail should be able to detect where it stopped to fetch logs and start again from there.
Instead it starts over again logs that have already been pushed

Environment:

• Infrastructure: baremetal Centos7 or Ubuntu 18.04
• Deployment tool: release binary from github or local compilation
• configuration file:

server:
  http_listen_address: 172.30.0.111
  http_listen_port: 6922
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yml

clients:
  - url: "http://172.30.0.101:6902/loki/api/v1/push"

scrape_configs:
  - job_name: journal
    journal:
      json: false
#      max_age: 1s
      labels:
        job: systemd-journal

[fatpat]

• from go-systemd: https://github.com/coreos/go-systemd/blob/master/sdjournal/journal.go#L739
• from systemd: https://github.com/systemd/systemd/blob/master/src/journal/sd-journal.c#L2178

and EADDRNOTAVAIL is #defined as 99