Grafana Alerting Issue: tsdb.HandleRequest() error bad_response: bad response code 502

[kamrajshahapure]

While creating any alerts on Grafana, I am facing an error "tsdb.HandleRequest() error bad_response: bad response code 502" every single time, tried to find out but couldn't find the solution since nobody else had this issue in the past. Here is the screenshot for the same

Edit:

1. Grafana version: 5.1.4
2. Datasource: Prometheus 2.3.1

Thanks in Advance :)

[marefr]

What grafana version? What datasource and how have you configured it? Any proxy configured to be used for datasource?

[kamrajshahapure]

Hi,
I am using

1. Grafana version: 5.1.4
2. Datasource: Prometheus 2.3.1

(Added same info as an edit)

[marefr]

And my other questions? Screenshot of datasource configuration? Any proxy? Any special authentication?

[kamrajshahapure]

The configuration of Prometheus is straightforward with no authentication and we have organization level proxy but Prometheus and Grafana are on same host though.

[marefr]

So what happens if you change access to Server? I guess it would fail. Server access mode is kind of how prometheus are accessed in when evaluating alerting rules

[kamrajshahapure]

Yes I got the same bad gateway error,but later I tried my datasource to work on server access instead of browser access, and changed the servers address in the form of "http://ipaddress:port" and then the datasource started working fine.

After that I tried out alerting again, and now alerts are appearing on my Telegram channel
It was a trivial ignorance which caused issue :(
Glad that got a quick help, Thanks a ton

So the takeaway from this is "Always add datasources' access as server because that is how alerts access prometheus while evaluating alert rules"

[marefr]

Glad you worked it out. Browser access mode should only be used in the case of you cannot use server access mode due to special authentication or similar.