vmagent doesn't follow redirects without stream_parse #1945

vbichov · 2021-12-14T13:53:56Z

Describe the bug
vmagent doesn't follow redirects without setting stream_parse: true
for details see slack thread:
https://victoriametrics.slack.com/archives/CGZF1H6L9/p1639404018163100

Expected behavior
it should follow redirects in a way that is similar to prometheus

VM agent config

global:
  scrape_interval: 1m0s
scrape_configs:
- job_name: emr
  follow_redirects: true
  relabel_configs:
  - source_labels: [__meta_ec2_private_ip]
    target_label: __metrics_path__
    replacement: /bi-emr-reverse-logs/$1:4040/metrics/executors/prometheus/
  - source_labels: [__meta_ec2_tag_Name]
    target_label: instance
  - target_label: __address__
    replacement: bo.XX.com
  ec2_sd_configs:
  - region: us-east-1
    access_key: XXX
    secret_key: <secret>
    filters:
    - name: tag:aws:elasticmapreduce:instance-group-role
      values:
      - MASTER
    - name: tag:Name
      values:
      - airflow-shared-persistent-emr-6.3.0

Version
tried v1.7.0 and v1.68.0 and latest docker version


**Logs**
`unexpected status code returned when scraping "http://bo.XXX.com:80/bi-emr-reverse-logs/10.207.109.205:4040/metrics/executors/prometheus/": 301; expecting 200; response body: "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body>\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>openresty/1.19.3.1</center>\r\n</body>\r\n</html>\r\n"`

**Workaround**
the issue was resolved by adding stream_parse: true

The text was updated successfully, but these errors were encountered:

…ault See #1945

valyala · 2021-12-15T22:18:28Z

vmagent was following only a single redirect up to v1.70.0 . It should follow up to 5 redirects when built from the commit a3adf24 . @vbichov , could you build vmagent from this commit according to these docs and verify whether this allows scraping targets without stream_parse: true mode?

Note that vmagent expects that redirects are performed to the original hostname when stream_parse: true isn't set in scrape config.

valyala · 2021-12-20T18:23:33Z

vmagent should follow for up to 5 redirects starting from v1.71.0. @vbichov , could you check whether this helps resolving the issue?

valyala · 2022-02-12T18:03:25Z

@vbichov , could you check whether the issue persists in vmagent v1.72.0?

earendilfr · 2022-12-14T09:50:18Z

Hello,

For information, I still have the issue with vmagent v1.79.5...

2022-12-14T09:46:18.261Z        warn    VictoriaMetrics/lib/promscrape/scrapework.go:385        cannot scrape "http://xxxxxxx:80/snmp?module=icx&target=xxxxxxx" (job "Switches-foundry", labels {datacenter="XX",env="XX",instance="XX",job="Switches-foundry",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/snmp?module=icx&target=xxxxxxx": too many redirects
2022-12-14T09:46:18.280Z        warn    VictoriaMetrics/lib/promscrape/scrapework.go:385        cannot scrape "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx" (job "Switches-ping", labels {datacenter="XX",env="XX",instance="XX",job="Switches-ping",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx": too many redirects
2022-12-14T09:46:18.379Z        warn    VictoriaMetrics/lib/promscrape/scrapework.go:385        cannot scrape "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx" (job "Routers-ping", labels {datacenter="XX",env="XX",instance="XX",job="Routers-ping",monitor="prometheus04",site="XX"}): error when scraping "http://xxxxxxx:80/probe?module=icmp&target=xxxxxxx": too many redirects

hagen1778 · 2023-04-27T07:48:41Z

@earendilfr do you know how many redirects are there? It could be checked via browser...

earendilfr · 2023-05-10T15:36:09Z

@earendilfr do you know how many redirects are there? It could be checked via browser...

@hagen1778 : sorry, I have changed of jobs so hard to me to have this information but, if I remember correctly, I have only one redirect because, in this case, it's was just a redirect from http to https on the same server.

See VictoriaMetrics/VictoriaMetrics#1945

yrka5180 · 2023-06-14T09:40:25Z

@earendilfr do you know how many redirects are there? It could be checked via browser...

@hagen1778 : sorry, I have changed of jobs so hard to me to have this information but, if I remember correctly, I have only one redirect because, in this case, it's was just a redirect from http to https on the same server.

hi, have you solved this？
I am facing the same problem that it shows too many redirect while directing from http to https.

hagen1778 · 2023-06-14T13:05:14Z

@yrka5180 have you tried enabling/disabling stream parse?

jnadler · 2023-08-21T17:58:56Z

we just ran into this same issue: a single hop redirect http -> https fails with too many redirects

we enabled streamParse for all of vmagent via arguments, but i don't feel we should have to do this

and after enabling streamParse we see some new skipping duplicate scrape target with identical labels errors for some of our targets; does streamParse change how relabelling works?

hagen1778 · 2023-08-25T11:32:52Z

we just ran into this same issue: a single hop redirect http -> https fails with too many redirects

Is it possible to compile docker-compose env where this issue can be reproduced? This would simplify debug from our side.

and after enabling streamParse we see some new skipping duplicate scrape target with identical labels errors for some of our targets; does streamParse change how relabelling works?

Not that I'm aware of. Have you checked https://docs.victoriametrics.com/vmagent.html#troubleshooting ?

jnadler · 2023-08-25T16:42:16Z

I just turned off streamParse and confirmed we're still facing this problem. Here's what the redirect looks like:

The metrics target is cockroachDB. I'll see if I can reproduce in docker-compose or even better, narrow down the problem in the code.

jnadler · 2023-08-25T17:28:49Z

After reading some code, perhaps this behavior is unfortunate but expected.

VictoriaMetrics/lib/promscrape/client.go

Line 291 in fb9b8f6

// It is expected that the redirect is made on the same host.

I expect that http->https qualifies as changing the host, but fasthttp is reusing the existing connection to port 80, so of course this will be a redirect loop as the redirect was not followed at all.

jnadler · 2023-08-25T17:48:30Z

I believe it will be possible to fix this case, but @valyala may not love it

If we detect this case on redirect in promscrape client.go and create new c.hc = &fasthttp.HostClient{... with IsTLS now set appropriately, it will work thereafter.

jnadler · 2023-08-25T20:01:24Z

Here's a PR (the code is awful! it's just to communicate the idea) that shows it is possible to fix this behavior #4897

hagen1778 · 2023-08-29T09:32:33Z

I believe, the issue would be fixed if we completely migrate vmagent to native HTTP lib. @zekker6 I recall you was working on that some time ago... wdyt?

zekker6 · 2023-08-29T11:06:00Z

@hagen1778 When I've been testing this native HTTP lib was consuming more memory than fasthttp, and we decided to keep it as is in order to avoid increasing resource usage during scrapes.
It seems to me like it will be better to add a logic to re-create fasthttp.HostClient in case of redirects as it will create much smaller memory footprint than switching to native HTTP lib.

jnadler · 2023-08-29T17:52:41Z

Agreed, moving away from fasthttp would probably hurt performance in a way that might surprise users. This change gets it a little closer to the normal behavior expected from the native client.

Note that in my example here I'm only handling http->https redirects but "different hostname" redirects could also be handled with the same approach.

If you'd like me to do any more work down this road, or clean up this code a bit, just let me know! I factored the creation of the fasthttp client out of the new method, and could do the same for the streamparse client for consistency (not that there's any reason to recreate that one at runtime).

This will allow to handle redirect easily as fasthttp.Client creates new fasthttp.HostClient on demand and caches created instances per URL. See: #1945 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

ndevenish · 2023-09-29T09:08:08Z

I ran into this with federation targets. This specifically seems to be a problem because it looks like prometheus doesn't let you write https:// in the target, but VM defaults to http and fails on the redirect.

I think this means that you can't write a completely compatible configuration that works on both (this is probably only an issue while transitioning)

ptimofee · 2023-12-04T19:15:52Z

I can confirm that at least vmagent v1.93.4 does follow redirects only when stream_parse is true, no matter if follow_redirects is set to true or false. I. e. follow_redirects has no effect at all.

…Client for scraping targets in non-streaming mode While fasthttp.Client uses less CPU and RAM when scraping targets with small responses (up to 10K metrics), it doesn't work well when scraping targets with big responses such as kube-state-metrics. In this case it could use big amounts of additional memory comparing to net/http.Client, since fasthttp.Client reads the full response in memory and then tries re-using the large buffer for further scrapes. Additionally, fasthttp.Client-based scraping had various issues with proxying, redirects and scrape timeouts like the following ones: - #1945 - #5425 - #2794 - #1017 This should help reducing memory usage for the case when target returns big response and this response is scraped by fasthttp.Client at first before switching to stream parsing mode for subsequent scrapes. Now the switch to stream parsing mode is performed on the first scrape after reading the response body in memory and noticing that its size exceeds the value passed to -promscrape.minResponseSizeForStreamParse command-line flag. Updates #5567 Overrides #4931

valyala · 2024-01-30T18:13:29Z

FYI, the commit bc7cf49 should fix following redirects in stream parsing mode when the redirect is performed to different hostname. This commit will be included in the next release. In the mean time it is possible to build vmagent from this commit according to these docs and verify whether follow_redirects: true option in scrape_configs works as intended.

Updates #1945

#1945

valyala · 2024-01-30T22:47:56Z

vmagent should properly follow redirects at scrape targets when stream parsing mode is disabled starting from v1.97.0. Closing the issue as fixed.

YuriKravetc added the bug Something isn't working label Dec 14, 2021

YuriKravetc assigned valyala Dec 14, 2021

valyala added the vmagent label Dec 15, 2021

valyala added a commit that referenced this issue Dec 15, 2021

lib/promscrape: allow up to 5 redirects when scraping a target by def…

a3adf24

…ault See #1945

valyala added a commit that referenced this issue Dec 15, 2021

lib/promscrape: allow up to 5 redirects when scraping a target by def…

5bd4e47

…ault See #1945

MatthewJohn added a commit to MatthewJohn/vault-nomad-consul-terraform that referenced this issue Jun 10, 2023

Enable stream parse for vault to fix redirect following

d6e453a

See VictoriaMetrics/VictoriaMetrics#1945

hagen1778 unassigned valyala Jun 14, 2023

laconi51 mentioned this issue Jun 27, 2023

[Snyk] Upgrade sass from 1.57.1 to 1.62.1 laconi51/VictoriaMetrics#5

Open

zekker6 mentioned this issue Aug 31, 2023

lib/promscrape/client: replace fasthttp.HostClient with fasthttp.Client #4931

Closed

valyala mentioned this issue Sep 1, 2023

Fix 'too many redirects' on HTTP to HTTPS redirect by scrape target #4897

Closed

ptimofee mentioned this issue Dec 4, 2023

VMAgent does not attach bearer_token when following redirects #5425

Closed

valyala added a commit that referenced this issue Jan 30, 2024

docs/CHANGELOG.md: document the bugfix addressed by the commit bc7cf49

ea2752c

Updates #1945

valyala added a commit that referenced this issue Jan 30, 2024

docs/CHANGELOG.md: document the bugfix addressed by the commit bc7cf49

1678741

Updates #1945

valyala added a commit that referenced this issue Jan 30, 2024

docs/CHANGELOG.md: refer to the related pull request for the bugfix for

17f8ed8

#1945

valyala added a commit that referenced this issue Jan 30, 2024

docs/CHANGELOG.md: refer to the related pull request for the bugfix for

dc9f4f0

#1945

valyala closed this as completed Jan 30, 2024

chenrui333 mentioned this issue Jan 31, 2024

victoriametrics 1.97.0 Homebrew/homebrew-core#161378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vmagent doesn't follow redirects without stream_parse #1945

vmagent doesn't follow redirects without stream_parse #1945

vbichov commented Dec 14, 2021

valyala commented Dec 15, 2021

valyala commented Dec 20, 2021

valyala commented Feb 12, 2022

earendilfr commented Dec 14, 2022

hagen1778 commented Apr 27, 2023

earendilfr commented May 10, 2023

yrka5180 commented Jun 14, 2023

hagen1778 commented Jun 14, 2023

jnadler commented Aug 21, 2023

hagen1778 commented Aug 25, 2023

jnadler commented Aug 25, 2023

jnadler commented Aug 25, 2023

jnadler commented Aug 25, 2023 •

edited

Loading

jnadler commented Aug 25, 2023

hagen1778 commented Aug 29, 2023

zekker6 commented Aug 29, 2023

jnadler commented Aug 29, 2023

ndevenish commented Sep 29, 2023

ptimofee commented Dec 4, 2023 •

edited

Loading

valyala commented Jan 30, 2024

valyala commented Jan 30, 2024

vmagent doesn't follow redirects without stream_parse #1945

vmagent doesn't follow redirects without stream_parse #1945

Comments

vbichov commented Dec 14, 2021

valyala commented Dec 15, 2021

valyala commented Dec 20, 2021

valyala commented Feb 12, 2022

earendilfr commented Dec 14, 2022

hagen1778 commented Apr 27, 2023

earendilfr commented May 10, 2023

yrka5180 commented Jun 14, 2023

hagen1778 commented Jun 14, 2023

jnadler commented Aug 21, 2023

hagen1778 commented Aug 25, 2023

jnadler commented Aug 25, 2023

jnadler commented Aug 25, 2023

jnadler commented Aug 25, 2023 • edited Loading

jnadler commented Aug 25, 2023

hagen1778 commented Aug 29, 2023

zekker6 commented Aug 29, 2023

jnadler commented Aug 29, 2023

ndevenish commented Sep 29, 2023

ptimofee commented Dec 4, 2023 • edited Loading

valyala commented Jan 30, 2024

valyala commented Jan 30, 2024

jnadler commented Aug 25, 2023 •

edited

Loading

ptimofee commented Dec 4, 2023 •

edited

Loading