Thanos rule metrics missing after upgrade to 0.12.0.

[karlism]

Thanos, Prometheus and Golang version used:

# thanos --version            
thanos, version 0.12.0 (branch: HEAD, revision: 69f9c4e604d20b2e640f790957414cf3d19ecafa)
  build user:       root@aab79ed2e8d5
  build date:       20200415-09:34:28
  go version:       go1.13.6

Object Storage Provider:
irrelevant

What happened:
Thanos rule metrics endpoint is not reachable after Thanos upgrade to 0.12.0

# ps aux | grep "thanos rule"
prometh+   1149  0.0  2.4 220684 94420 ?        Ssl  Apr15   3:26 /usr/bin/thanos rule --data-dir=/var/lib/thanos/rule --http-address=0.0.0.0:19902 --grpc-address=0.0.0.0:19901 --rule-file=/etc/prometheus/thanos_alerts.yml --query=localhost:19192 --alertmanagers.url=https://alerts.example.com --alert.query-url=https://metrics.example.com/all --web.external-prefix=https://metrics.example.com/thanos/query

# netstat -tulpn | grep thanos
tcp6       0      0 :::19192                :::*                    LISTEN      1151/thanos         
tcp6       0      0 :::19901                :::*                    LISTEN      1149/thanos         
tcp6       0      0 :::19902                :::*                    LISTEN      1149/thanos         
tcp6       0      0 :::19092                :::*                    LISTEN      1151/thanos    

# curl localhost:19902/       
<a href="https:/metrics.example.com/thanos/query/alerts">Found</a>.

# curl --max-time 10 localhost:19902/metrics
curl: (28) Operation timed out after 10001 milliseconds with 0 out of -1 bytes received

What you expected to happen:
Metrics to be returned

How to reproduce it (as minimally and precisely as possible):
See above

Full logs to relevant components:
Nothing in logs (with info log level), I can try debug logging if necessary.

Anything else we need to know:
Previously system was running Thanos 0.11.0 and metrics were working fine. The same system has Thanos query running and metrics for that service are returned just fine.

Environment:

# cat /etc/*release
CentOS Linux release 7.7.1908 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.7.1908 (Core)
CentOS Linux release 7.7.1908 (Core)
# uname -a
Linux hostname 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[karlism]

@yeya24, thanks for looking into this!

Additionally to the original issue, it seems that URL is rendered incorrectly here (only 1 slash before hostname):

# curl localhost:19902/       
<a href="https:/metrics.example.com/thanos/query/alerts">Found</a>.

[bwplotka]

Thanks for report and details!

Looks like a duplicate of #2486

[yeya24]

Hi @karlism could you please try the latest release and see if your issue still exists.

But to me, it seems this issue is unrelated to #2484. You only configure --web.external-prefix=https://metrics.example.com/thanos/query and didn't configure --web.route-prefix.

Also, the missing endpoint is /metrics, which is unrelated to --web.route-prefix as well.

[karlism]

@yeya24, thanks! It is working now and /metrics endpoint returns metrics properly.
I'm not certain --web.route-prefix is necessary in my case as --web.external-prefix points to NGINX load balancer that forwards connections to multiple Thanos servers in different datacenters.

[yeya24]

Thanks for confirming!