Understanding Loki observation

[matteodisabatino]

I deployed Loki with promtail into my server with this configuration:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
client:
  url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: myapp
  static_configs:
  - targets:
    - localhost
    labels:
      job: platform-logs
      env: test
      agent: promtail
      host: myhost
      __path__: "/var/log/*/*.log"
  pipeline_stages:
  - match:
      selector: '{agent="promtail"}'
      stages:
      - json:
          expressions:
            statusCode: statusCode
            method: method
            url: url
            path: path
            level: level
            message: message
            timestamp: timestamp
            service: service
            host: host
      - labels:
          statusCode: statusCode
          method: method
          url: url
          path: path
          level: level
          message: message
          timestamp: timestamp
          service: service
          host: host
      - timestamp:
          source: timestamp
          format: RFC3339Nano

After I added loki to my prometheus targets, like so:

- job_name: "loki"
    scrape_interval: 5s
    static_configs:
      - targets: ["loki:3100"]

And I installed the dashboard 10880 but I encountered a problem because I could not see any graph: the dashboard require loki_ingester_chunk_entries_count, loki_distributor_ingester_appends_total, promtail_file_bytes_total and promtail_encoded_bytes_total metrics to be exported but none of these where in the list of Loki exported metrics.

So I searched in the official documentation and I made some tests changing my configuration like that:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
client:
  url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: myapp
  static_configs:
  - targets:
    - localhost
    labels:
      job: platform-logs
      env: test
      agent: promtail
      host: myhost
      __path__: "/var/log/*/*.log"
  pipeline_stages:
  - match:
      selector: '{agent="promtail"}'
      stages:
      - json:
          expressions:
            statusCode: statusCode
            method: method
            url: url
            path: path
            level: level
            message: message
            timestamp: timestamp
            service: service
            host: host
      - labels:
          statusCode: statusCode
          method: method
          url: url
          path: path
          level: level
          message: message
          timestamp: timestamp
          service: service
          host: host
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - metrics:
          log_lines_total:
            type: Counter
            description: "total number of log lines"
            config:
              match_all: true
              action: inc
          log_bytes_total:
            type: Counter
            description: "total bytes of log lines"
            config:
              match_all: true
              count_entry_bytes: true
              action: add

With this configuration I'm able to see the first two graphs (those who refers to metrics loki_ingester_chunk_entries_count and loki_distributor_ingester_appends_total) but I cannot see the last two.

After that I also installed the dashboard 10004 and I understood that none of the metrics that starts with the word "promtail" are exported.

Hence my question are:

1. Why the metrics loki_ingester_chunk_entries_count and loki_distributor_ingester_appends_total have been exported only when I added my custom metrics? Is there a way to export them "naturally"?
2. Why none of the metrics that starts with the word "promtail" are exported? Is there a way to export them?

Thanks in advance to anyone that will answer.

[slim-bean]

I bit embarrassed to say this but we don't actually publish any official dashboards on grafana.com :(

The dashboards you found were community contributed and may not be current.

In regards to the promtail metrics they would be exported by promtail itself so you would need to scrape your promtail(s) with prometheus to see them.

We do have some official dashboards for Loki and Promtail in ksonnet

Some of them are generated from code, a few are just static json (mostly because they got complex enough it was too time consuming to build them from code).

One challenge though is they are based on our opinionated prometheus scrape config which includes a job label in the same format we export the job label from the promtail kubernetes scrape config in our ksonnet and helm charts.

This is an area we really want/need to improve, Thanks to @brancz for opening #1978 to start the discussion but we just haven't had a chance to really dig into this.

[matteodisabatino]

@slim-bean Thanks for your answer and for the clarification: I'll wait for official dashboard but per now the two I installed are sufficient :-)
And really thanks for have said me what was wrong with promtail metrics :-).
But there is a thing that I still cannot understand: as I asked in my question 1, why do I need to write custom metrics to let loki export metrics loki_ingester_chunk_entries_count and loki_distributor_ingester_appends_total? Why aren't they exported "naturally"?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.