Skip to content

Conversation

@mergify
Copy link
Contributor

@mergify mergify bot commented Oct 14, 2024

Motivation/summary

Fix panic due to monitoring middleware concurrent map write.

fatal error: concurrent map writes

goroutine 21913 [running]:
github.com/elastic/apm-server/internal/beater/middleware.(*monitoringMiddleware).getCounter(0xc008a53530, {0x32a4b68?, 0xc0073d1c00?})
	/home/carson/projects/apm-server/internal/beater/middleware/monitoring_middleware.go:87 +0xb8
github.com/elastic/apm-server/internal/beater/middleware.MonitoringMiddleware.(*monitoringMiddleware).Middleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/monitoring_middleware.go:49 +0x59
github.com/elastic/apm-server/internal/beater/api.apmMiddleware.RecoverPanicMiddleware.func2.1(0x3617ce0?)
	/home/carson/projects/apm-server/internal/beater/middleware/recover_panic_middleware.go:60 +0x43
github.com/elastic/apm-server/internal/beater/middleware.TimeoutMiddleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/timeout_middleware.go:35 +0x35
github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1(0xc00521a000)
	/home/carson/projects/apm-server/internal/beater/middleware/log_middleware.go:47 +0xaa
github.com/elastic/apm-server/internal/beater/api.NewMux.(*ContextPool).HTTPHandler.func11({0x3632730, 0xc0047620e0}, 0xc0003f88c0)
	/home/carson/projects/apm-server/internal/beater/request/context_pool.go:47 +0xec
net/http.HandlerFunc.ServeHTTP(...)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/server.go:2220
go.elastic.co/apm/module/apmhttp/v2.(*handler).ServeHTTP(0xc006b7de40, {0x3632730, 0xc0047620e0}, 0xc0003f88c0)
	/home/carson/apm-server-gopath/pkg/mod/go.elastic.co/apm/module/apmhttp/v2@v2.6.2/handler.go:74 +0xc9
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0070bd2c0, {0x3632730, 0xc0047620e0}, 0xc0003f8640)
	/home/carson/apm-server-gopath/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x2d7
github.com/elastic/gmux.ConfigureServer.(*mux).withGRPCInsecure.func3({0x3632730?, 0xc0047620e0?}, 0x2fe25c0?)
	/home/carson/apm-server-gopath/pkg/mod/github.com/elastic/gmux@v0.3.2/mux.go:119 +0x105
net/http.HandlerFunc.ServeHTTP(...)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/server.go:2220
net/http.serverHandler.ServeHTTP({0x362c8e8?}, {0x3632730?, 0xc0047620e0?}, 0x6?)
	/home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/server.go:3210 +0xa2
net/http.(*conn).serve(0xc0078d7440, {0x3639618, 0xc00bcae120})
	/home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/server.go:2092 +0x610
created by net/http.(*Server).Serve in goroutine 21909
	/home/carson/apm-server-gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/net/http/server.go:3360 +0x485

Alternative to this, we could have populated the maps with all the possible keys, but we may risk missing some possible map keys.

Checklist

For functional changes, consider:

  • Is it observable through the addition of either logging or metrics?
  • Is its use being published in telemetry to enable product improvement?
  • Have system tests been added to avoid regression?

How to test these changes

Related issues


This is an automatic backport of pull request #14335 done by [Mergify](https://mergify.com).

Fix panic due to monitoring middleware concurrent map write.

(cherry picked from commit 5cd8b7d)
@mergify mergify bot requested a review from a team as a code owner October 14, 2024 07:41
@mergify mergify bot added the backport label Oct 14, 2024
@mergify
Copy link
Contributor Author

mergify bot commented Oct 14, 2024

This pull request has not been merged yet. Could you please review and merge it @carsonip? 🙏

@mergify mergify bot merged commit ddf91da into 8.x Oct 15, 2024
@mergify mergify bot deleted the mergify/bp/8.x/pr-14335 branch October 15, 2024 09:59
@kruskall kruskall mentioned this pull request Oct 21, 2024
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants