Promtail configuration support for environment variable interpolation not working as documented

[shairozan]

Describe the bug
Documentation for promtail on grafana indicates that variable interpolation happens before the config file is read

scrape_configs:
  - job_name: varlogs
ec2_sd_configs:
  - role_arn: ${INSTANCE_PROFILE_ARN}

The above section of the config is for a scrape with EC2 service discovery. SInce we're using service profiles on EC2, it makes sense to just use the instance profile which we can set from the env.

Attempting to start promtail works, but authentication for the SD fails:

is not authorized to perform: sts:AssumeRole on resource: ${INSTANCE_PROFILE_ARN}

Whenever the EC2 authorization happens, it should have already interpolated and read those values from

To Reproduce
Steps to reproduce the behavior:

1. Started Promtail (Version 2.0.0) specifying config file in dry-run mode referencing config file
2. Immediately get EC2 auth errors
3. Logging for other scrape targets continues unimpeded

Expected behavior
The config would have already interpolated the INSTANCE_PROFILE_ARN from the environment (or used a null string) per the documentation and not actually just try to use that raw text as the ARN.

Environment:

• Infrastructure: EC2 Instance
• Deployment tool: promtail binary from 2.0.0 release

Screenshots, Promtail config, or terminal output

Terminal output

Clients configured:
----------------------
url: https://<REDACTED>/loki/api/v1/push
batchwait: 10s
batchsize: 102400
tls_config:
  ca_file: /etc/loki/ca.crt
  cert_file: /etc/loki/client.crt
  key_file: /etc/loki/client.key
  server_name: <REDACTED>
  insecure_skip_verify: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
external_labels: |
  ami_name: ${AMI_NAME}
  node_type: ${NODE_TYPE}
timeout: 10s
tenant_id: ""

level=info ts=2020-12-02T15:33:06.360175051Z caller=server.go:225 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-12-02T15:33:06.361160507Z caller=main.go:108 msg="Starting Promtail" version="(version=2.0.0, branch=HEAD, revision=6978ee5d)"
2020-12-02T14:37:32	{job="systemd-journal"}pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
2020-12-02T14:37:37	{job="systemd-journal"}pam_unix(sudo:session): session closed for user root
2020-12-02T15:17:01	{job="systemd-journal"}pam_unix(cron:session): session opened for user root by (uid=0)
2020-12-02T15:17:01	{job="systemd-journal"}(root) CMD (   cd / && run-parts --report /etc/cron.hourly)
2020-12-02T15:17:01	{job="systemd-journal"}pam_unix(cron:session): session closed for user root
2020-12-02T15:31:20	{job="systemd-journal"}  ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=./promtail-linux-amd64 -config.file=/etc/loki/promtail-config.yaml --dry-run
2020-12-02T15:31:20	{job="systemd-journal"}pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
2020-12-02T15:31:22	{job="systemd-journal"}pam_unix(sudo:session): session closed for user root
2020-12-02T15:32:55	{job="systemd-journal"}  ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/vim /etc/loki/promtail-config.yaml
2020-12-02T15:32:55	{job="systemd-journal"}pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
2020-12-02T15:33:03	{job="systemd-journal"}pam_unix(sudo:session): session closed for user root
2020-12-02T15:33:06	{job="systemd-journal"}  ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=./promtail-linux-amd64 -config.file=/etc/loki/promtail-config.yaml --dry-run
2020-12-02T15:33:06	{job="systemd-journal"}pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
level=error ts=2020-12-02T15:33:06.435743867Z caller=refresh.go:79 component=discovery discovery=ec2 msg="Unable to refresh target groups" err="could not describe instances: AccessDenied: User: arn:aws:sts::<REDACTED>:assumed-role/<REDACTED>/<REDACTED> is not authorized to perform: sts:AssumeRole on resource: ${INSTANCE_PROFILE_ARN}\n\tstatus code: 403, request id: 2f96df6e-b1ca-47b2-a405-f52b16e8ad79"

I've dug into the source code for V2 and I don't even see the interpolation happening for any variables. Was this removed in 2.0 and the docs not updated?

At this point Promtail is created and getting ready to be run, but the config still has the interpolation strings, not null values or values from the environment

[cyriltovena]

I don't think it's in 2.0.0

see 5793c49

[shairozan]

#2837

It's not. I can verify now that working off of master that it does work if you include the config item that's also not documented. The problem is that the latest docset pulls from master, which includes items not yet wrapped into a release.

[shairozan]

Trying to use master, however breaks for us. Something changed in journald scraping that broke what was previously working. We've moved to just templating the file as opposed to working with this feature.

[cyriltovena]

#2837

It's not. I can verify now that working off of master that it does work if you include the config item that's also not documented. The problem is that the latest docset pulls from master, which includes items not yet wrapped into a release.

Yes I'm aware of that, we need a versioned documentation.

[cyriltovena]

Trying to use master, however breaks for us. Something changed in journald scraping that broke what was previously working. We've moved to just templating the file as opposed to working with this feature.

Tell me more about that issue with journald, I made a change there recently.

[shairozan]

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 168h
      path: /var/log/journal
      labels:
        job: systemd-journal
    relabel_configs:
      - action: replace
        source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
  - job_name: varlogs
    ec2_sd_configs:
      - profile: arn:aws:iam::<REDACTED>:instance-profile/<REDACTED>
    static_configs:
    - labels:
        job: varlogs
    relabel_configs:
      - action: replace
        replacement: /var/log/**.log
        target_label: __path__
      - action: labelmap
        regex: __meta_ec2_(.+)
      - action: replace
        source_labels: [__meta_ec2_private_dns_name]
        regex: "(.*).ec2.internal"
        replacement: $1
        target_label: __host__

The above worked for us in version 2.0, but when testing with the master branch journal doesn't even show up

[cyriltovena]

Any logs from promtail ?

[perrie625]

@shairozan hey, just pass -config.expand-env flag when start promtail
doc is missing mention abou this.

[tristan-tsl]

i meet the same probleam yet, i will try it, thank you

[tristan-tsl]

in this doc page, already have it
https://grafana.com/docs/loki/latest/configuration/

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.