How to best debug logs/Loki ?

[mortenson]

Hi all, I just started using Grafana Agent with Grafana Cloud (free tier) and have very little experience with Loki.

I've installed and configured the agent using this guide: https://grafana.com/docs/grafana-cloud/infrastructure-as-code/ansible/ansible-grafana-agent-linux/

And this config:

logs:
  configs:
    - clients:
      - basic_auth:
          password: redacted
          username: redacted
        url: https://redacted.grafana.net/loki/api/v1/push
      name: default
      positions:
        filename: /tmp/positions.yaml
      scrape_configs:
        - job_name: journal
          journal:
            json: false
            max_age: 12h
            matches: SYSLOG_IDENTIFIER=my_service
            labels:
              job: systemd-journal
          relabel_configs:
            - source_labels: ['__journal__systemd_unit']
              target_label: 'unit'
      target_config:
        sync_period: 10s

But am not seeing any logs in Grafana Cloud when checking the default query in "Explore" for the grafanacloud-x-logs data source.

How can I best troubleshoot this logging pipeline? Thanks!

My debugging steps so far:

• I can tell my config is loaded because I had a syntax error I could see in journal logs for the agent service
• systemctl status grafana-agent.service looks good, no errors
• journalctl SYSLOG_IDENTIFIER=my_service returns the logs I'm interested in
• Checked /tmp/positions.yaml, which only contains positions: {}

[mortenson]

I got this working by using the newest release and the cloud-config command.