fix: Connection Prematurely Closed issue 대응 작업 #987
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
slack 서버의 네트워크 관련 구성, 설정등을 알 수 없음. 따라서, Reactor Netty Docs(https://projectreactor.io/docs/netty/1.0.21/reference/index.html\#faq.connection-closed) 의 내용을 토대로 추정하여 대응한다.
sakjung
force-pushed
the
fix/985-connection-prematurely-closed
branch
from
d2a2e6b to
216c33f
Compare
xrabcde
approved these changes
Apr 1, 2024
Comment on lines
+53
to
+54
| slackWebClient.post() | ||
| .uri(slackUrl) |
|
상용 배포는 내일 할게요~ |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
구현 기능
현재 작업으로 문제가 해결되지 않을 가능성도 있으므로 배포 후 장기간 (N month) 모니터링 필요
공유하고 싶은 내용
slack 서버의 네트워크 관련 구성, 설정등을 알 수 없음. 따라서, Reactor Netty Docs 의 내용을 토대로 원인을 추정하여 대응한다.
추정 원인
1. idle timeout (the connection is closed when there is no incoming data for a certain period of time)
즉, target server (i.e. Slack) 의 keep-alive timeout config 와 우리의 connection idle timeout config 가 맞지않아 (호환되지 않게끔 설정) 발생하는 케이스. tcpdump 와 로그등을 분석해본 결과, 이 케이스가 가장 유력하다고 보임.
tcpdump 를 보았을 때, 3초간 3번 정도의 TCP keep alive check (probe packet) 통신 후 target server 측에서 보낸 RST 패킷을 통해 커넥션이 premature 하게 끊어짐. 따라서, 우리 측에서는
maxIdelTimeout설정값을 2초 정도로 잡아서 우리가 먼저 connection 을 알아서 끊도록 함.2. max keep alive requests (the connection is closed when the requests reach the configured maximum number)
원인이 아니겠지만, docs 를 보다보니 관련 설정도 해주는게 좋을것 같아서 수정함. 트래픽이 많지 않은데 connection pool 을 크게 가져갈 필요가 없는 것 같아서
maxConnections값 10으로 설정.Close #985