Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

with per_host in metrics, how about compatible with old version #6604

Closed
ta2013 opened this issue Oct 3, 2024 · 10 comments · Fixed by #6606
Closed

with per_host in metrics, how about compatible with old version #6604

ta2013 opened this issue Oct 3, 2024 · 10 comments · Fixed by #6606
Assignees
@mohammed90
Labels
bug 🐞 Something isn't working

Comments

@ta2013
Copy link

ta2013 commented Oct 3, 2024 •
edited
Loading

current caddyfile is using:

{
	servers {
		metrics 
	}
}

if i switch to caddy@master (with per_host support in metrics), then above config will cause panic!! because missing :port (ie servers :80, servers :443 )

the full log:

panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc00068ddb0, {0xc000812f20?, 0x1d1de87?, 0x4?})
        github.com/prometheus/client_golang@v1.19.1/prometheus/registry.go:405 +0x66
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeVec({{0x22edd30?, 0xc00068ddb0?}}, {{0x1d1fa3f, 0x5}, {0x1d1de87, 0x4}, {0x1d51a67, 0x12}, {0x1da476e, 0x34}, ...}, ...)
        github.com/prometheus/client_golang@v1.19.1/prometheus/promauto/auto.go:308 +0x163
github.com/caddyserver/caddy/v2/modules/caddyhttp.initHTTPMetrics({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:46 +0x130
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler.func1()
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:121 +0x38
sync.(*Once).doSlow(0x1?, 0x3?)
        sync/once.go:76 +0xb4
sync.(*Once).Do(...)
        sync/once.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/metrics.go:120 +0xb3
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware({{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:321 +0x99
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Route).ProvisionHandlers(0xc000978a88, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:167 +0x2e5
github.com/caddyserver/caddy/v2/modules/caddyhttp.RouteList.ProvisionHandlers({0xc000978a88, 0xb, 0xc000975190?}, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/routes.go:212 +0xa9
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision(0xc0006f3680, {{0x22f2478, 0xc00080ee70}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, ...}, ...})
        github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/app.go:354 +0xe1e
github.com/caddyserver/caddy/v2.Context.LoadModuleByID({{0x22f24b0, 0xc00068de00}, 0xc0004ae360, 0xc0000bed20, {0xc000645150, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/context.go:389 +0x695
github.com/caddyserver/caddy/v2.Context.App({{0x22f24b0, 0xc00068de00}, 0xc0004ae360, 0xc0000bed20, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
        github.com/caddyserver/caddy/v2@v2.8.4/context.go:465 +0xee
github.com/caddyserver/caddy/v2.provisionContext.func3(...)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:545
github.com/caddyserver/caddy/v2.provisionContext(0x1?, 0x0)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:550 +0x53c
github.com/caddyserver/caddy/v2.run(0x1942f40?, 0x0)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:400 +0x78
github.com/caddyserver/caddy/v2.Validate(0xc0000bed20)
        github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:714 +0x5a
github.com/caddyserver/caddy/v2/cmd.cmdValidateConfig({0x0?})
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/commandfuncs.go:551 +0x155
github.com/caddyserver/caddy/v2/cmd.init.1.func8.WrapCommandFuncForCobra.1(0xc000924908, {0x1d1de5b?, 0x4?, 0x1d1ddc7?})
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/cobra.go:141 +0x2f
github.com/spf13/cobra.(*Command).execute(0xc000924908, {0xc0006a3660, 0x2, 0x2})
        github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00098a608)
        github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/caddyserver/caddy/v2/cmd.Main()
        github.com/caddyserver/caddy/v2@v2.8.4/cmd/main.go:75 +0x1dd
main.main()
        caddy/main.go:26 +0xf
ta2013 referenced this issue Oct 3, 2024
@mohammed90 @hussam-almarzoq @hairyhenderson
metrics: scope metrics to active config, add optional per-host metrics (
41f5dd5 
#6531)

* Add per host config

* Pass host label when option is enabled

* Test per host enabled

* metrics: scope metrics per loaded config

* doc and linter

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* inject the custom registry into the admin handler

Co-Authored-By: Dave Henderson <dhenderson@gmail.com>

* remove `TODO` comment

* fixes

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* refactor to delay metrics admin handler provision

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

---------

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Hussam Almarzooq <me@hussam.io>
Co-authored-by: Dave Henderson <dhenderson@gmail.com>
@mohammed90
Copy link
Member

because missing :port (ie servers :80, servers :443 )

I don't think that's related. Can you share your full config file? I need to replicate it, because I don't see the same issue on my side.

@ta2013
Copy link
Author

ta2013 commented Oct 3, 2024

I don't think that's related. Can you share your full config file? I need to replicate it, because I don't see the same issue on my side.

Here is the configuration file that can be replicated.

{
        servers {
                metrics
        }

}

*.example.com {
        respond "fallback"
}

http://nossl.example.com {
        respond "nossl"
}

@steffenbusch
Copy link
Contributor

steffenbusch commented Oct 3, 2024 •
edited
Loading

Note

I have the same panic while trying to test v2.9.0-beta.2 with my existing Caddyfile from v.2.8.4.

panic: duplicate metrics collector registration attempted
goroutine 1 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0003d39f0, {0xc0008d36f0?, 0x1b21117?, 0x4?})
        github.com/prometheus/client_golang@v1.19.1/prometheus/registry.go:405 +0x66
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeVec({{0x1ffd3c0?, 0xc0003d39f0?}}, {{0x1b229f7, 0x5}, {0x1b21117, 0x4}, {0x1b450a6, 0x12}, {0x1b9>
        github.com/prometheus/client_golang@v1.19.1/prometheus/promauto/auto.go:308 +0x163
github.com/caddyserver/caddy/v2/modules/caddyhttp.initHTTPMetrics({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...}>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:46 +0x130
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler.func1()
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:121 +0x38
sync.(*Once).doSlow(0x1?, 0x3?)
        sync/once.go:76 +0xb4
sync.(*Once).Do(...)
        sync/once.go:67
github.com/caddyserver/caddy/v2/modules/caddyhttp.newMetricsInstrumentedHandler({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/metrics.go:120 +0xb3
github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware({{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x1}, {0x0, 0x0, 0x0}, ...},>
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/modules/caddyhttp/routes.go:321 +0x99
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Route).ProvisionHandlers(0xc00062a288, {{0x2001aa8, 0xc000643bc0}, 0xc0004e0810, 0xc000284840, {0xc00073c1f0, 0x1, 0x>
...
github.com/spf13/cobra.(*Command).execute(0xc000980608, {0xc0008fee70, 0x3, 0x3})
        github.com/spf13/cobra@v1.8.0/command.go:983 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000980008)
        github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/caddyserver/caddy/v2/cmd.Main()
        github.com/caddyserver/caddy/v2@v2.9.0-beta.2/cmd/main.go:75 +0x1dd
main.main()
        caddy/main.go:20 +0xf

@mohammed90
Copy link
Member

mohammed90 commented Oct 3, 2024 •
edited
Loading

Okay, I figured the root cause. This is hairy. The shared caddy.Context applies to the entire process, while the metrics key may exist in multiple servers within the same configuration object, so they both try to register themselves and end up registering the same collectors.

I need to control the registration flow within the caddy context. I wonder if it's better to swap the control and have the context register the collectors instead of the registry being given to the HTTP app.

@mohammed90 mohammed90 added the bug 🐞 Something isn't working label Oct 3, 2024
@mohammed90 mohammed90 self-assigned this Oct 3, 2024
mohammed90 added a commit that referenced this issue Oct 5, 2024
@mohammed90
metrics: move metrics up, outside servers
d5c6e6c 
This change moves the metrics configuration from per-server level to a single config knob within the `http` app. Enabling `metrics` in any of the configured servers inside `http` enables metrics for all servers.

Fix #6604

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
@mohammed90 mohammed90 mentioned this issue Oct 5, 2024
Merged
@mohammed90
Copy link
Member

@ta2013 @steffenbusch, can you test the CI artifacts of #6606 (here) to validate it works? I've tested it locally, and the panic is gone.

@ta2013
Copy link
Author

ta2013 commented Oct 6, 2024

Appreciate the swift fix; everything's running smoothly now. 👍

@steffenbusch
Copy link
Contributor

I have used xcaddy with the branch from #6606 (--with github.com/caddyserver/caddy/v2=github.com/caddyserver/caddy/v2@promote-metrics) and all other plugins I use. The panic is gone 👍

@ta2013
Copy link
Author

ta2013 commented Oct 7, 2024 •
edited
Loading

@mohammed90

I observed that r.Host="example.com" differs from r.Host="exAmple.com". Shouldn't the metrics for these labels be treated as identical?

@mohammed90
Copy link
Member

Shouldn't the metrics for these labels be treated as identical?

Yep. I've pushed a commit to normalize them.

@ta2013
Copy link
Author

ta2013 commented Oct 8, 2024

Shouldn't the metrics for these labels be treated as identical?

Yep. I've pushed a commit to normalize them.

Thanks. It works!

mholt pushed a commit that referenced this issue Oct 18, 2024
@mohammed90
metrics: move metrics up, outside servers (#6606)
388c7e8 
* metrics: move `metrics`  up, outside `servers`

This change moves the metrics configuration from per-server level to a single config knob within the `http` app. Enabling `metrics` in any of the configured servers inside `http` enables metrics for all servers.

Fix #6604

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* normalize domain name

---------

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mohammed90 mohammed90

Labels
bug 🐞 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

metrics: move metrics up, outside servers caddyserver/caddy
3 participants
@ta2013 @mohammed90 @steffenbusch