Prometheus is not enabled since its retention time is not positive

[Anashaikh]

3 Consul servers are created using statefulset inside Kubernetes. All three of them have joined with each other and selected one of them as a leader and consul UI works fine. I would like to monitor them through prometheus/grafana so have added telemetry to enable prometheus metrics. However when accessing localhost:8500/v1/agent/metrics?format=prometheus within the the pod, get this message
"Prometheus is not enabled since its retention time is not positive"
the retention_time is set to 24h, however i have changed this to 1min/1hour/24 hour but still get the same message. If i try accessing localhost:8500/v1/agent/metrics without ?format=prometheus than it does shows the metrics but not in prometheus format. Kindly help

Overview of the Issue

When accessing Consul metrics, getting the following message
https://xxx.xxx.com/v1/agent/metrics?format=prometheus
"Prometheus is not enabled since its retention time is not positive"

Reproduction Steps

Steps to reproduce this issue, eg:

curl localhost:8500/v1/agent/metrics?format=prometheus
Prometheus is not enabled since its retention time is not positive/

1. View error

Consul info for Server

$consul version
Consul v1.5.0

Operating system and Environment details

uname -a

Linux vault-consul-2 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 Linux

Consul configuration file:
cat config.json:
{
"server": true,
"leave_on_terminate": true,
"bind_addr": "0.0.0.0",
"ca_file": "/consul/tls/ca.crt",
"cert_file": "/consul/tls/tls.crt",
"client_addr": "0.0.0.0",
"data_dir": "/consul/data",
"key_file": "/consul/tls/tls.key",
"ports": {
"http": 8500,
"https": 8501
},
"datacenter": "abcd",
"primary_datacenter": "abcd",
"acl" : {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache"
},
"telemetry": {
"prometheus_retention_time": "24h",
"disable_hostname": true
},
"verify_outgoing": true,
"disable_anonymous_signature": true,
"ui": true,
"disable_update_check": true
}

Log Fragments

Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the gist of the log instead of posting it in the issue. Use -log-level=TRACE on the client and server to capture the maximum log detail.

[pierresouchay]

Not a bug. To activate prometheus, you need to set the option prometheus_retention_time in the agent. I would advise you to put a high value here (such as "240h" or "72h"), so the stats are not reset too often and prometheus can work well with it

[Anashaikh]

Not a bug. To activate prometheus, you need to set the option prometheus_retention_time in the agent. I would advise you to put a high value here (such as "240h" or "72h"), so the stats are not reset too often and prometheus can work well with it

Scenario 1: (not working)
Initially i have used the prometheus_retention_time value of 30s when starting up the consul cluster (3 consul servers in kubernetes) . However i have now changed the value to 24h and did a consul reload on all three pods, however still get the same message when accessing the prometheus metrics.
"Prometheus is not enabled since its retention time is not positive"
It seems like the value change from 30s to 24h is somehow not being loaded or not being picked up.
The problem is, I cannot delete all the pods and re-deploy them again with 24h or 72h value

Scenario 2: (Working)
However in the test environment when i have deployed a brand a new consul cluster (3 consul servers) with prometheus_retention_time as 24h and i can access the prometheus metrics from there.

[Anashaikh]

Scenario 1:
The prometheus metrics started working only once i was able to run kill -9 -1 within the pod.

[ikimi]

Scenario 1: The prometheus metrics started working only once i was able to run kill -9 -1 within the pod.

I met the same problem, MUST restart the consul agent...