Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Replace mutex with channel for TelemetryCounter #1981

Open
wants to merge 2 commits into
base: release/v1.65.0
Choose a base branch
from

Conversation

shazlehu
Copy link
Contributor

@shazlehu shazlehu commented Nov 19, 2024

Proposed Change

Users are seeing agents stuck at restart after reconfiguring, and my working hypothesis is that the agent is having trouble with a pipeline that includes 16 count_telemetry processors, each of which uses a mutex. The obvious problem is that the agent defers unlocking the mutex until the next consumer in the ConsumeLogs chain returns, but it seemed like a good idea to rewrite the processor using channels instead of mutexes, the preferred Go synchronization method

From a mutex profile of a stuck agent:

image
Checklist
  • Changes are tested
  • CI has passed

@shazlehu shazlehu force-pushed the samhazlehurst/bpop-1242-gap-agents-stuck-in-configuring branch 2 times, most recently from d8bf13c to d08caab Compare November 20, 2024 14:18
@shazlehu shazlehu force-pushed the samhazlehurst/bpop-1242-gap-agents-stuck-in-configuring branch from d08caab to 73787ab Compare November 20, 2024 14:22
@shazlehu shazlehu marked this pull request as ready for review November 20, 2024 14:31
@shazlehu shazlehu requested review from dpaasman00 and a team as code owners November 20, 2024 14:31
@shazlehu shazlehu force-pushed the samhazlehurst/bpop-1242-gap-agents-stuck-in-configuring branch from 343c097 to 62d4482 Compare November 20, 2024 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant