Unable to find any logs to tail makes pods unhelthy #808

alikhil · 2019-07-29T10:47:11Z

Describe the bug
Promtail respond with 500 Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc. if there are no logs to scrape which match scrape condition.

To Reproduce
Steps to reproduce the behavior:

Started loki (master-f2bec3b0)
Started promtail (master-f2bec3b0) to tail pods with label (monitor: spring-app)
Start deployment with number of pods less than number of nodes in your cluster
Check list of promtail pods and see that in nodes where is no deployment matching label selector promtail pods are failed to start

Expected behavior
All pods in all existing minion nodes will be up and running.

Environment:

Infrastructure: Kubernetes
Deployment tool: helm

Screenshots, promtail config, or terminal output
Logs from failing pod:

level=warn ts=2019-07-29T10:17:59.433772066Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511098591Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511380636Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511513947Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T10:17:59.511627719Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=info ts=2019-07-29T10:17:59.512196582Z caller=kubernetes.go:192 component=discovery discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-07-29T10:17:59.514231267Z caller=server.go:120 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-07-29T10:17:59.514633991Z caller=main.go:55 msg="Starting Promtail" version="(version=master-f2bec3b0, branch=master, revision=f2bec3b0)"
level=warn ts=2019-07-29T10:18:10.670280612Z caller=logging.go:49 msg="GET /ready (500) 151.423µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "
level=warn ts=2019-07-29T10:18:20.669893429Z caller=logging.go:49 msg="GET /ready (500) 122.642µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "
level=warn ts=2019-07-29T10:18:30.670121286Z caller=logging.go:49 msg="GET /ready (500) 342.837µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.11; "

Logs from running pod:

level=warn ts=2019-07-29T09:45:36.376487811Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.37702662Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377624682Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377787544Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=warn ts=2019-07-29T09:45:36.377903092Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
level=info ts=2019-07-29T09:45:36.378411566Z caller=kubernetes.go:192 component=discovery discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2019-07-29T09:45:36.466269594Z caller=server.go:120 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-07-29T09:45:36.467184037Z caller=main.go:55 msg="Starting Promtail" version="(version=master-f2bec3b0, branch=master, revision=f2bec3b0)"
level=info ts=2019-07-29T09:45:41.668433219Z caller=filetargetmanager.go:257 msg="Adding target" key="{app=\"my-app\", category=\"backend\", container_name=\"my-app\", env=\"development\", instance=\"my-app-857d7c5d56-6vmsw\", job=\"development/my-app\", monitor=\"spring-app\", namespace=\"development\", pod_template_hash=\"4138371812\"}"
level=info ts=2019-07-29T09:45:41.669612293Z caller=tailer.go:73 msg="start tailing file" path=/var/log/pods/a44eb032-ad23-11e9-b35e-fa163ecb059f/my-app/0.log
2019/07/29 09:45:41 Seeked /var/log/pods/a44eb032-ad23-11e9-b35e-fa163ecb059f/my-app/0.log - &{Offset:0 Whence:0}

$ kubectl get pods 
promtail-4j9vx   1/1     Running   0          1h
promtail-6zfx2   1/1     Running   0          1h
promtail-f968n   1/1     Running   0          57m
promtail-fjk8c   0/1     Running   0          1m
promtail-fp8jc   1/1     Running   0          57m
promtail-h6vg6   1/1     Running   0          1h
promtail-qjjzf   0/1     Running   0          1m
promtail-ztktd   1/1     Running   0          1h

The text was updated successfully, but these errors were encountered:

steven-sheehy · 2019-07-29T18:13:57Z

That's expected. You need to fix your scrape config to be able to scrape at least one pod. At minimum, it should be able to scrape itself.

alikhil · 2019-07-30T09:14:55Z

@steven-sheehy what if I don't want to scrape logs from all the pods in the cluster and scrape only pods of selected apps (which can be deployed only on some nodes, not all)? Seems like such a usecase is missed now.

I think it can be solved by passing some flag like IGNORE_NO_LOGS.

pennpeng · 2019-08-05T07:01:17Z

+1

geekodour · 2019-08-09T08:26:05Z

@alikhil
these two links might be useful to get you scrape_config right, you can select what you want to scrape using relabel configs:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

alikhil · 2019-08-13T07:38:34Z

@geekodour thank you! Could you tell me what is mean to be right for scrape_config so that promtail will not fail if there is nothing to scrape?

stale · 2019-09-12T08:05:08Z

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

alikhil · 2019-09-12T09:04:36Z

Do not stale

stale · 2019-10-12T09:20:44Z

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

DonnieBwin · 2019-11-18T12:28:00Z

hello it is that been solved?

HuJake · 2019-12-04T16:35:22Z

I have the same problem, how can I solve it？

NAME                 READY   STATUS    RESTARTS   AGE
test-loki-0           1/1     Running   0          5h49m
test-promtail-bd9h6   0/1     Running   0          3m42s
test-promtail-bpfsc   0/1     Running   0          3m42s
test-promtail-chrcv   0/1     Running   0          3m42s
test-promtail-dq2jw   0/1     Running   0          3m42s
test-promtail-j4sxg   0/1     Running   0          3m42s

caller=logging.go:49 msg="GET /ready (500) 40.895µs Response: \"Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: kube-probe/1.14; "

I found entering the pod for observation,
found positions.yaml:
filename: /run/positions.yaml is empty
Is there a problem?

slim-bean · 2020-02-28T19:52:56Z

Generally we would suggest at least tailing the promtail log to avoid this issue.

If you don't want to do this I suggest just removing the readiness probe entirely.

slim-bean · 2020-04-09T18:52:51Z

For anyone who was following this ticket #1920 introduced a flag to essentially always make the /ready endpoint return 200

imranrazakhan · 2020-08-11T23:16:17Z

@slim-bean In my case docker log driver was set as journald and i changed it to "log-driver": "json-file" and it start working.

fabiopaiva · 2023-10-18T08:51:02Z

Following @slim-bean suggestions, I got it working with the following configuration:

config:
  snippets:
    extraRelabelConfigs:
      # Combine labels in order to keep logs from NGINX Ingress and Promtail # https://github.com/grafana/loki/issues/808#issuecomment-592698307
      - source_labels: [ __meta_kubernetes_pod_label_app_kubernetes_io_instance, __meta_kubernetes_pod_label_app_kubernetes_io_name ]
        separator: ';'
        target_label: combined_labels
      - source_labels: [ combined_labels ]
        action: keep
        regex: alb-ingress-nginx;.*|.*;promtail

cyriltovena added component/agent type/question labels Jul 29, 2019

stale bot added the stale A stale issue or PR that will automatically be closed. label Sep 12, 2019

stale bot removed the stale A stale issue or PR that will automatically be closed. label Sep 12, 2019

stale bot added the stale A stale issue or PR that will automatically be closed. label Oct 12, 2019

cyriltovena added the keepalive An issue or PR that will be kept alive and never marked as stale. label Oct 12, 2019

stale bot removed the stale A stale issue or PR that will automatically be closed. label Oct 12, 2019

slim-bean closed this as completed Feb 28, 2020

mhoyer mentioned this issue Nov 29, 2022

Promtail kubernetes host log & drop namespace #748

Closed

reitermarkus mentioned this issue Dec 10, 2023

promtail is never ready when no related Mayastor pod is running on a given node. openebs/mayastor#1560

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to find any logs to tail makes pods unhelthy #808

Unable to find any logs to tail makes pods unhelthy #808

alikhil commented Jul 29, 2019

steven-sheehy commented Jul 29, 2019

alikhil commented Jul 30, 2019

pennpeng commented Aug 5, 2019

geekodour commented Aug 9, 2019 •

edited

Loading

alikhil commented Aug 13, 2019

stale bot commented Sep 12, 2019

alikhil commented Sep 12, 2019

stale bot commented Oct 12, 2019

DonnieBwin commented Nov 18, 2019

HuJake commented Dec 4, 2019 •

edited

Loading

slim-bean commented Feb 28, 2020

slim-bean commented Apr 9, 2020

imranrazakhan commented Aug 11, 2020 •

edited

Loading

fabiopaiva commented Oct 18, 2023

Unable to find any logs to tail makes pods unhelthy #808

Unable to find any logs to tail makes pods unhelthy #808

Comments

alikhil commented Jul 29, 2019

steven-sheehy commented Jul 29, 2019

alikhil commented Jul 30, 2019

pennpeng commented Aug 5, 2019

geekodour commented Aug 9, 2019 • edited Loading

alikhil commented Aug 13, 2019

stale bot commented Sep 12, 2019

alikhil commented Sep 12, 2019

stale bot commented Oct 12, 2019

DonnieBwin commented Nov 18, 2019

HuJake commented Dec 4, 2019 • edited Loading

slim-bean commented Feb 28, 2020

slim-bean commented Apr 9, 2020

imranrazakhan commented Aug 11, 2020 • edited Loading

fabiopaiva commented Oct 18, 2023

geekodour commented Aug 9, 2019 •

edited

Loading

HuJake commented Dec 4, 2019 •

edited

Loading

imranrazakhan commented Aug 11, 2020 •

edited

Loading