(Customer 39) OrientDB Server shutdown doesn't complete in a timely manner #2736
Comments
|
For what it's worth, our OrientDB servers are running: Java Version: OS: Linux and here's our /etc/init.d/orientdb file's stop() function that
|
|
Hi this fixed in version 1.7.7 , could you try on latest version and then close issue if it is not reproduced. |
|
Hi @iaa Which git commit within ODB 1.7.7 fixes this bug? Please let us know; we want to review it; thanks. |
|
@lloydchang We suggest you to stay updated with hotfixes we release. You can find the changelog in GitHub of each release. |
|
Hi @lvca @Laa @enisher @maggiolo00 @luigidellaquila @tglman We are still experiencing this issue after trying a newer ODB version. Hence as git committers for ODB 1.7.7, perhaps you can help us? Thank you for your time! We skimmed git commits at https://github.com/orientechnologies/orientdb/commits/1.7.7 and we cannot identify the ODB 1.7.7 git commit that fixes this bug. Request: Would you be able to identify which git commit within ODB 1.7.7 fixes this bug? |
|
Hi, seems you are right. It is different issue. |
|
Hi @lvca @Laa @enisher Summary: This, #2736, is a new issue — It is a different shutdown issue than #2239 and #2449 Trying steps @lvca suggested, this issue is still reproducible 👎
|
|
@lloydchang do you have Skype/Hangout to contact you ? )) |
|
@Laa Thank you for the response. What is your Skype and/or Google Hangout name? I asked @henryzhao81 and @hcmwork to Live Skype and/or Google Hangout with you, and execute a technical deep dive and code review, at line-by-line source code level. I recall that you and @henryzhao81 executed 1:1 technical conversations previously about OrientDB. |
|
Hi my skype is lomakin_andrey |
|
Hi Andrey: my skype id is hemalcm I have sent you a contact request. |
|
Hi @Laa My skype ID is lchang.proteusdh -- I have sent you a contact request as well. Regarding incident 1 described, I see the following warning: Question 1:
I found "1000" hard-coded(?) in multiple OrientDB versions' branch source code; here's the source code from OrientDB 1.7.9 https://github.com/orientechnologies/orientdb/blob/1.7.9/core/src/main/java/com/orientechnologies/orient/core/config/OGlobalConfiguration.java#L415 and the warning message Question 2:
Chris @chrishuttonch wrote, "I have ensured that I close my connection for every one opened but still find that some threads aren't closed. This causes the OrientDB server to eventually hit the 1000 open connections limit (WARN Reached maximum number of concurrent connections (1000), reject incoming connection from /IP:60350 [OServerNetworkListener])." Luca @lvca replied, "As work around use a connection pool: it's faster and don't leaves pending connections." "The configurations is very simple, just 2 parameters:
Comparatively, our server configuration sets following values: Therefore, I don't understand where the 1000 is coming from. Questions 3, 4, 5, and 6:
After searching through logs on server 1, I found two incidents of "concurrent connections (1000)" warning message ... that happened during shutdown / service orientdb stop attempts on one of our server running single server mode: and it happened a second time: Reading through the log entry and source code linked above, I think the 10.224.176.11 originates from OServerShutdownMain.java#L75 networkAddress = l.ipAddress; https://github.com/orientechnologies/orientdb/blob/1.7.9/server/src/main/java/com/orientechnologies/orient/server/OServerShutdownMain.java#L75 And OServerShutdownMain requires a remote connection from localhost 127.0.0.1 to 10.224.176.11 (the server's 10.x.x.x IP address) to be accessible ... However, if the server is already at 1000 concurrent connections (even though client.channel.maxPool = 5000), then shutdown will not complete in a timely manner, and the warning message will appear, followed by "WARN Low free heap memory" messages. Questions 7 and 8:
Looking forward to your thoughts; thanks. |
|
@lloydchang My answers to your Questions:
|
|
Hi @lvca @Laa Thank you; status update about resolution:
References |
|
Hi, On Mon, Oct 6, 2014 at 8:15 AM, lloydchang notifications@github.com wrote:
Best regards, Orient Technologies |
|
The fix is on releasing of connections |
|
hi @lloydchang for 2.0 is here 0609439 |
|
Hi @tglman @lvca @Laa Today, we observed following log snippet Therefore, would you consider re-opening this issue #2736 until following scenario is resolved? I don't know how to reproduce it, but I want to bring this to your attention:
|
|
Hi @tglman @lvca @Laa @enisher
|
|
hi i found a problem with shutdown code introduced in orientdb. Surely it can create a lot of problems in a orientdb cllient hosting in more open system but also in server. |
|
We used signals to catch any possible case to avoid to break the DB. This works well with server and usually clients. Are you using an App Server? |
|
yes i m using orientdb in a server that is making a specific task (but it is a classical web container with a war ... i m using tomcat now. There is also a strange loop when you close application not completely started . The loop show continuely this message : com.orientechnologies.common.log.OLogManager.log Removing disconnected network channel '127.0.0.1:2424/AAA'... |
|
in orientdb version 2.1.1 the file bin/shutdown.sh does not work. The only thing that I can get to work is kill . |
|
@dcarr178 Please could you open a new issue for that? |
|
@lvca I dropped my db and started over which seemed to correct the problem so I cannot reproduce anymore. Will create a new issue if I can reproduce again. The weirdest part was the log file entry that said Error:null. |
Hi @lvca @enisher @Laa
CC @mattaylor @henryzhao81 @hcmwork @stuartking @pmoorhead
Version: OrientDB 1.7.4
Frequency: 5 incidents across 2 servers in 2 days
Steps to Reproduce:
which effectively executes...
INCIDENT 1 happened on Server 1 running OrientDB 1.7.4
INCIDENT 2 happened on Server 1 running OrientDB 1.7.4
INCIDENT 3 happened on Server 1 running OrientDB 1.7.4
INCIDENT 4 happened on Server 1 running OrientDB 1.7.4
INCIDENT 5 happened on Server 2 running OrientDB 1.7.4
The text was updated successfully, but these errors were encountered: