Label job on file_sd_configs produces unknown errors on ADOT

[pcolazurdo]

Hi team,
I was trying to follow this guide to configure Kafka monitoring into Prometheus using ADOT and found that using labels named "job" produced some unknown errors.

...
    "labels": {
      "job": "jmx"
    },
...

produced the following errors:

{2021-09-28 17:43:13.531622039 +0000 UTC m=+100.477388245, Level:info, Caller:github.com/prometheus/prometheus@v1.8.2-0.20210621150501-ff58416a0b02/scrape/scrape.go:1460, Message:Unexpected error, Stack:}
{2021-09-28 17:43:13.531747554 +0000 UTC m=+100.477513738, Level:info, Caller:github.com/prometheus/prometheus@v1.8.2-0.20210621150501-ff58416a0b02/scrape/scrape.go:1248, Message:Append failed, Stack:}
{2021-09-28 17:43:13.531781892 +0000 UTC m=+100.477548094, Level:info, Caller:github.com/prometheus/prometheus@v1.8.2-0.20210621150501-ff58416a0b02/scrape/scrape.go:1203, Message:Appending scrape report failed, Stack:}

changing the label name to "kafka_type" for example, worked fine.

I've also tried the original configuration in a standalone prometheus server and it worked fine with the "job" label, so this looks like something particular in ADOT code.

My config.yaml looks like

...
        - job_name: "broker"
          scrape_interval: 60s
          file_sd_configs:
            - files:
              - '/opt/aws/aws-otel-collector/etc/targets.json'
...

More info:

# sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a status
{
  "status": "running",
  "starttime": "2021-09-28T17:52:39+0000",
  "version": "v0.13.0"
}

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stall for 30 days with no activity.