Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/loadbalancing] Fix panic on a sub-exporter shutdown #31456

Merged
merged 1 commit into from
Mar 5, 2024

Conversation

dmitryax
Copy link
Member

@dmitryax dmitryax commented Feb 27, 2024

Fix panic when a sub-exporter is shut down while still handling requests. This change wraps exporters with an additional working group to ensure that exporters are shut down only after they finish processing data.

Fixes #31410

It has some small related refactoring changes. I can extract them in separate PRs if needed.

@dmitryax dmitryax requested a review from jpkrohling as a code owner February 27, 2024 21:56
@dmitryax dmitryax requested a review from a team February 27, 2024 21:56
@dmitryax dmitryax force-pushed the fix-load-balancing-exp branch 3 times, most recently from 46b1931 to d21aeb9 Compare February 27, 2024 22:07
@dmitryax dmitryax marked this pull request as draft February 27, 2024 23:47
@dmitryax dmitryax force-pushed the fix-load-balancing-exp branch from d21aeb9 to e2959eb Compare February 28, 2024 07:08
@dmitryax dmitryax marked this pull request as ready for review February 28, 2024 07:08
@dmitryax dmitryax force-pushed the fix-load-balancing-exp branch 2 times, most recently from dd571b0 to c582e34 Compare February 28, 2024 17:37
@crobert-1
Copy link
Member

Failing CI/CD test build-and-test / build-package (deb) (pull_request) is frequency of #31443

@dmitryax dmitryax force-pushed the fix-load-balancing-exp branch from c582e34 to 6332e6a Compare March 1, 2024 18:57
@@ -287,6 +287,58 @@ func TestLogsWithoutTraceID(t *testing.T) {
assert.Len(t, sink.AllLogs(), 1)
}

// this test validates that exporter is can concurrently change the endpoints while consuming logs.
func TestConsumeLogs_ConcurrentResolverChange(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test passes on main as well, it might not trigger the error condition this PR is aiming to fix.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how it passed for you. Here is the draft PR with the tests only #31566

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the confirmation! Something certainly went wrong when backporting it to main on my side. LGTM!

Fix panic when a sub-exporter is shut down while still handling requests
@dmitryax dmitryax force-pushed the fix-load-balancing-exp branch from 6332e6a to 366b03b Compare March 5, 2024 04:27
@jpkrohling jpkrohling merged commit c3efe18 into open-telemetry:main Mar 5, 2024
142 checks passed
@github-actions github-actions bot added this to the next release milestone Mar 5, 2024
dmitryax added a commit that referenced this pull request Mar 7, 2024
DougManton pushed a commit to DougManton/opentelemetry-collector-contrib that referenced this pull request Mar 13, 2024
…elemetry#31456)

Fix panic when a sub-exporter is shut down while still handling
requests. This change wraps exporters with an additional working group
to ensure that exporters are shut down only after they finish processing
data.

Fixes
open-telemetry#31410

It has some small related refactoring changes. I can extract them in
separate PRs if needed.
DougManton pushed a commit to DougManton/opentelemetry-collector-contrib that referenced this pull request Mar 13, 2024
XinRanZhAWS pushed a commit to XinRanZhAWS/opentelemetry-collector-contrib that referenced this pull request Mar 13, 2024
…elemetry#31456)

Fix panic when a sub-exporter is shut down while still handling
requests. This change wraps exporters with an additional working group
to ensure that exporters are shut down only after they finish processing
data.

Fixes
open-telemetry#31410

It has some small related refactoring changes. I can extract them in
separate PRs if needed.
XinRanZhAWS pushed a commit to XinRanZhAWS/opentelemetry-collector-contrib that referenced this pull request Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic using the load balancing exporter
6 participants