Loki component of k8s Helm chart does not work on Docker Desktop kubernetes

[scarolan]

Steps to reproduce:

• Install Docker Desktop and enable Kubernetes
• Install the Grafana Helm chart for k8s. I used the recommended config from Grafana Cloud
• Logs are unable to be found, due to what looks like some broken symlinks

2024-02-14 17:20:24 ts=2024-02-14T22:20:24.878324657Z level=error msg="error getting os stat" component=local.file_match.pod_logs path=/var/log/pods/kube-system_coredns-76f75df574-wgv5q_fc8e8dd6-8500-4871-a5a8-b90f93929cf7/coredns/1.log err="stat /var/log/pods/kube-system_coredns-76f75df574-wgv5q_fc8e8dd6-8500-4871-a5a8-b90f93929cf7/coredns/1.log: no such file or directory"
2024-02-14 17:20:24 ts=2024-02-14T22:20:24.878398234Z level=error msg="error getting os stat" component=local.file_match.pod_logs path=/var/log/pods/kube-system_coredns-76f75df574-wgv5q_fc8e8dd6-8500-4871-a5a8-b90f93929cf7/coredns/1.log err="stat /var/log/pods/kube-system_coredns-76f75df574-wgv5q_fc8e8dd6-8500-4871-a5a8-b90f93929cf7/coredns/1.log: no such file or directory"
2024-02-14 17:20:24 ts=2024-02-14T22:20:24.878485951Z level=error msg="error getting os stat" component=local.file_match.pod_logs path=/var/log/pods/kube-system_kube-proxy-nlrqz_e0fec916-2c5c-4cfe-b08c-9bf3baef9559/kube-proxy/1.log err="stat /var/log/pods/kube-system_kube-proxy-nlrqz_e0fec916-2c5c-4cfe-b08c-9bf3baef9559/kube-proxy/1.log: no such file or directory"
2024-02-14 17:20:24 ts=2024-02-14T22:20:24.879219894Z level=error msg="error getting os stat" component=local.file_match.pod_logs path=/var/log/pods/default_grafana-k8s-monitoring-prometheus-node-exporter-9qtsm_7a320f8c-bed4-41aa-ab1c-a0e3025dca39/node-exporter/1.log err="stat /var/log/pods/default_grafana-k8s-monitoring-prometheus-node-exporter-9qtsm_7a320f8c-bed4-41aa-ab1c-a0e3025dca39/node-exporter/1.log: no such file or directory"

Shell commands from the logs pod

root@grafana-k8s-monitoring-grafana-agent-logs-vjfvs:/# ls -l /var/log/pods/default_grafana-k8s-monitoring-prometheus-node-exporter-9qtsm_7a320f8c-bed4-41aa-ab1c-a0e3025dca39/node-exporter/
total 4
lrwxrwxrwx 1 root root 165 Feb 14 22:18 1.log -> /var/lib/docker/containers/fe015412ac81470dd1e31cabdeec8c16a874db11d2a70a20f3843c39d0d30e44/fe015412ac81470dd1e31cabdeec8c16a874db11d2a70a20f3843c39d0d30e44-json.log
root@grafana-k8s-monitoring-grafana-agent-logs-vjfvs:/# ls /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory

[scarolan]

The fix is here in case anyone else runs into this:

https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/Troubleshooting.md#pod-log-files-in-varlibdockercontainers