Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"malformed HTTP request \"\"" #118

Open
scythargon opened this issue Sep 22, 2021 · 5 comments
Open

"malformed HTTP request \"\"" #118

scythargon opened this issue Sep 22, 2021 · 5 comments

Comments

@scythargon
Copy link

scythargon commented Sep 22, 2021 •
edited
Loading

Hi!
I'm having troubles running PushProx on my staging server.

The most informative error I see is from proxy client receiving bad request:

$ ./pushprox-client --proxy-url=http://prom-proxy-stg.***.com/ --fqdn=`hostname` --metrics-addr=":9368"
level=info ts=2021-09-22T10:26:52.905Z caller=main.go:245 msg="URL and FQDN info" proxy_url=http://prom-proxy-stg.***.com/ fqdn=argon-u1910
level=error ts=2021-09-22T10:27:53.467Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""
level=error ts=2021-09-22T10:28:55.319Z caller=main.go:205 msg="Error reading request:" err="malformed HTTP request \"\""

And then probably because of that pushprox-proxy fails to retrieve data from the client:

caller=coordinator.go:137 msg=WaitForScrapeInstruction fqdn=argon-u1910
caller=main.go:149 msg="Error WaitForScrapeInstruction:" err="request is expired"

Some more details:

The client gets successfully registered at proxy/clients page: [{"targets":["argon-u1910"],"labels":null}]

And Prometheus is able to see that client live, but fails to scrape.

Prometheus config files and dashboard screenshots:
Selection_696

If I run proxy-server locally - everything is fine. So the problem is probably somewhere in the stack on which I'm trying to host it on cloud my servers. Should there be some additional ports open or something? Might the problem be in some http/https redirect where the request body is lost?

Any help will be much appreciated!

Thank you!
@scythargon
Copy link
Author

The problem lies somewhere in reverse proxy configuration which is placed in front of PushProx.
I believe there is one at my cloud service provider side which I can't really configure.
Then I've tried to install PushProx in my AWS EC2 instance behind Nginx - this didn't work either - was receiving err="malformed HTTP request \"<html>\" error instead.
But it works it if just reach to PushProx listening port directly. (Had to make in available in AWS firewall).
So if you can point me on what nginx configuration would be suitable here - that would be nice:)
Thanks!

@bingtw
Copy link

bingtw commented Apr 28, 2022

I got the same issue when my PushProx service is located behind AWS ALB. (client <-> AWS ALB <-> server)
But it works properly when I use public IP address of my PushProx service in client. (client <-> server)

@nitinakash1989
Copy link

I am also facing the same issue

@ipsecguy
Copy link

Same here on local infra. I have to encrypt the traffic (Security will reject otherwise, and probably rightfully so).

Same error happens regardless whether I specify a valid URL (http://server1:9100/metrics for node exporter, an invaldi URL or an invalid path. So I suspect the proxy client does not even try to open a connection or the port/path is lost on the way even though it is displayed correctly in the "Got scrape request" log on the client.

@walkure walkure mentioned this issue Jan 23, 2023
Open
@menvol3
Copy link

menvol3 commented Mar 19, 2024

I faced the same issue. Does anyone know how to resolve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@scythargon @nitinakash1989 @bingtw @ipsecguy @menvol3