Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Proxy] Log warning when opening connection to broker fails #14710

Merged
merged 1 commit into from
Mar 17, 2022
Merged

[Proxy] Log warning when opening connection to broker fails #14710

merged 1 commit into from
Mar 17, 2022

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Mar 16, 2022 •
edited
Loading

Motivation

The Pulsar proxy silently fails when establishing the broker connection fails. It's better to log a warning when opening of the connection to the broker fails.

I'm currently investigating an issue where connections between proxies and brokers fail with logs like this:

19:56:00.520 [pulsar-proxy-io-2-1] WARN  org.apache.pulsar.common.protocol.PulsarHandler - [[id: 0x00010ad9, L:/10.5.0.221:40136 - R:pulsar-broker.pulsar.svc.cluster.local/172.20.192.198:6650]] Pulsar Handshake was not completed within timeout, closing connection
19:56:00.520 [pulsar-proxy-io-2-1] WARN  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x00010ad9, L:/10.5.0.221:40136 ! R:pulsar-broker.pulsar.svc.cluster.local/172.20.192.198:6650]] Connection handshake failed: org.apache.pulsar.client.api.PulsarClientException: Connection already closed

The real reason is hidden, and this PR intends to improve logging.

The PR #14078 added connection timeout for proxy connections. The default is 10000 ms. It's possible that in environments with many connections, the default timeout is too short. Adding more logging will help detect the root cause.

Modifications

Add logging at warn level.

@lhotari lhotari added area/proxy doc-not-needed Your PR changes do not impact docs labels Mar 16, 2022
@lhotari lhotari self-assigned this Mar 16, 2022
eolivelli
eolivelli approved these changes Mar 16, 2022
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui added this to the 2.11.0 milestone Mar 16, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Mar 16, 2022
@lhotari @nicoloboschi
[Proxy] Log warning when opening connection to broker fails apache#14710
dbc3fa3
@codelipenghui codelipenghui merged commit 13c4562 into apache:master Mar 17, 2022
aparajita89 pushed a commit to aparajita89/pulsar that referenced this pull request Mar 21, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (apache#1…
8bf1a20 
…4710)
lhotari added a commit to datastax/pulsar that referenced this pull request Mar 22, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (apache#1…
30783d0 
…4710)

(cherry picked from commit 13c4562)
lhotari added a commit to datastax/pulsar that referenced this pull request Mar 22, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (apache#1…
f1f6121 
…4710)

(cherry picked from commit 13c4562)
(cherry picked from commit 30783d0)
@lhotari lhotari mentioned this pull request Mar 24, 2022
Open
lhotari added a commit that referenced this pull request Mar 24, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (#14710)
dae653f 
(cherry picked from commit 13c4562)
lhotari added a commit that referenced this pull request Mar 24, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (#14710)
b472453 
(cherry picked from commit 13c4562)
lhotari added a commit that referenced this pull request Mar 24, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (#14710)
1307e98 
(cherry picked from commit 13c4562)
@lhotari lhotari added cherry-picked/branch-2.8 Archived: 2.8 is end of life cherry-picked/branch-2.9 Archived: 2.9 is end of life labels Mar 24, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails (apache#1…
4dc3986 
…4710)
lhotari added a commit that referenced this pull request Jun 1, 2022
@lhotari
[Proxy] Log warning when opening connection to broker fails #14710
03fb6ed 
(cherry picked from commit dbc3fa3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaeljmarshall michaeljmarshall michaeljmarshall approved these changes

@nicoloboschi nicoloboschi nicoloboschi approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@eolivelli eolivelli eolivelli approved these changes

@dlg99 dlg99 dlg99 approved these changes

@addisonj addisonj Awaiting requested review from addisonj

Assignees

@lhotari lhotari

Labels
area/proxy cherry-picked/branch-2.8 Archived: 2.8 is end of life cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-not-needed Your PR changes do not impact docs release/2.8.4 release/2.9.3 release/2.10.1
Projects
None yet
Milestone
2.11.0
Development

Successfully merging this pull request may close these issues.
6 participants
@lhotari @dlg99 @eolivelli @codelipenghui @nicoloboschi @michaeljmarshall