Promtail stops tailing statefulset pod on kubernetes after pod is deleted and rescheduled

[rsteneteg]

Describe the bug
When deleting a statefulset pod on Kubernetes the pod and its logs gets deleted causing Promtail to stop the tailer, the pod is then rescheduled with a different path to the logfile since the pods uid changed, yet the pod name is the same causing Promtail to think it is the same target as before the restart. This prevents Promtail from creating a new target for the new logfile, since it already have a target with the same labels, but that target no longer have any tailer running, so no more logs is read from the pod

To Reproduce
Steps to reproduce the behavior:

1. Deploy a statefulset
2. Have Promtail v2.2.0 run on the same node as the statefulset pod
3. Query the logs from the statefulset which at think point should work
4. Delete the statefulset pod
5. Query the logs from the statefulset which now would have stopped at the last line of the previous pod.

Expected behavior
Once Promtail has finished reading the last logs of the deleted pod, Promtail should pickup reading the logs from the new rescheduled pod

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
Relabel config for the __path__

relabel_configs:
  - action: replace
    replacement: /var/log/pods/*$1/*.log
    separator: /
    source_labels:
    - __meta_kubernetes_pod_uid
    - __meta_kubernetes_pod_container_name
    target_label: __path__

in the /var/log/pods folder the path to the logs look like
/var/log/pods/<namespace>_<podname>_<uid>/<containername>/0.log