[STORM-2754] Not killing on exceptions in other threads #2341
Conversation
|
Same thought on this: fail-fast if we can't handle the exception. But I'm OK to see others' opinions as well. |
|
@Ethanlm I think we want to combine STORM-2753 with this into a single pull request. I think I need to give a little background on this so it makes since, and I am not sure this is the best solution which is why I would like to see the two combined into a single pull request. We ran into some issues where pacemaker servers were being killed because of a client being shot at the wrong time, or security scans, etc. The issue was that the uncaught exception handler was not what we wanted for the pacemaker server. We then adjusted some things and ended up with an end user that had their topology crashing periodically because an exception that was being thrown on a background thread, and ignored previously, was now fatal. So the goal with these was to find a balance. We want the default uncaught exception handler for all threads to be a little bit more lenient. But for daemon threads we already handle exceptions very strictly so the default UncaughtExceptionHandler is not needed here. But thinking about it now perhaps what we want is something that is a bit more explicit about what is happening. For example we can perhaps have a separate default exception handler for the worker from everything else. |
| public void uncaughtException(Thread thread, Throwable thrown) { | ||
| try { | ||
| handleUncaughtException(thrown); | ||
| handleUncaughtExceptionWithoutKillingProcess(thrown); |
There was a problem hiding this comment.
This will swallow all the Throwables including Errors excluding OOME. Is it intended? We may still want to make it crash when there's Error thrown from somewhere, given that it normally indicates unrecoverable state.
| } | ||
|
|
||
| if(allowedExceptions.contains(t.getClass())) { | ||
| if(alwaysSwallow || allowedExceptions.contains(t.getClass())) { |
There was a problem hiding this comment.
Even alwaysSwallow is set to true, we may still want to throw Error on Errors.
|
@HeartSaVioR Sorry for the late reply. I made the changes. Could you please review again? This also fixes a potential NPE issue. |
|
+1 |
| LOG.error("Exception thrown while handling uncaught exception " + error.getCause()); | ||
| Runtime.getRuntime().exit(1); | ||
| } | ||
| LOG.info("Received error in netty thread.. terminating server..."); |
There was a problem hiding this comment.
If a thread hits the exception handler will it still die? If so who is responsible for replacing it? As far as I can tell this handler is used by a HashedWheelTimer in Context, and it doesn't look to me like that class will replace a dead thread.
There was a problem hiding this comment.
It looks like the HashedWheelTimer will not be replaced - I'll take a second pass at this to confirm
| public void uncaughtException(Thread thread, Throwable thrown) { | ||
| try { | ||
| handleUncaughtException(thrown); | ||
| handleUncaughtExceptionWithoutKillingProcess(thrown); |
There was a problem hiding this comment.
Same question, if a thread gets an exception and the process is not shut down, will the thread be replaced?
We are closing stale Pull Requests to make the list more manageable. Please re-open any Pull Request that has been closed in error. Closes apache#608 Closes apache#639 Closes apache#640 Closes apache#648 Closes apache#662 Closes apache#668 Closes apache#692 Closes apache#705 Closes apache#724 Closes apache#728 Closes apache#730 Closes apache#753 Closes apache#803 Closes apache#854 Closes apache#922 Closes apache#986 Closes apache#992 Closes apache#1019 Closes apache#1040 Closes apache#1041 Closes apache#1043 Closes apache#1046 Closes apache#1051 Closes apache#1078 Closes apache#1146 Closes apache#1164 Closes apache#1165 Closes apache#1178 Closes apache#1213 Closes apache#1225 Closes apache#1258 Closes apache#1259 Closes apache#1268 Closes apache#1272 Closes apache#1277 Closes apache#1278 Closes apache#1288 Closes apache#1296 Closes apache#1328 Closes apache#1342 Closes apache#1353 Closes apache#1370 Closes apache#1376 Closes apache#1391 Closes apache#1395 Closes apache#1399 Closes apache#1406 Closes apache#1410 Closes apache#1422 Closes apache#1427 Closes apache#1443 Closes apache#1462 Closes apache#1468 Closes apache#1483 Closes apache#1506 Closes apache#1509 Closes apache#1515 Closes apache#1520 Closes apache#1521 Closes apache#1525 Closes apache#1527 Closes apache#1544 Closes apache#1550 Closes apache#1566 Closes apache#1569 Closes apache#1570 Closes apache#1575 Closes apache#1580 Closes apache#1584 Closes apache#1591 Closes apache#1600 Closes apache#1611 Closes apache#1613 Closes apache#1639 Closes apache#1703 Closes apache#1711 Closes apache#1719 Closes apache#1737 Closes apache#1760 Closes apache#1767 Closes apache#1768 Closes apache#1785 Closes apache#1799 Closes apache#1822 Closes apache#1824 Closes apache#1844 Closes apache#1874 Closes apache#1918 Closes apache#1928 Closes apache#1937 Closes apache#1942 Closes apache#1951 Closes apache#1957 Closes apache#1963 Closes apache#1964 Closes apache#1965 Closes apache#1967 Closes apache#1968 Closes apache#1971 Closes apache#1985 Closes apache#1986 Closes apache#1998 Closes apache#2031 Closes apache#2032 Closes apache#2071 Closes apache#2076 Closes apache#2108 Closes apache#2119 Closes apache#2128 Closes apache#2142 Closes apache#2174 Closes apache#2206 Closes apache#2297 Closes apache#2322 Closes apache#2332 Closes apache#2341 Closes apache#2377 Closes apache#2414 Closes apache#2469
We are closing stale Pull Requests to make the list more manageable. Please re-open any Pull Request that has been closed in error. Closes apache#608 Closes apache#639 Closes apache#640 Closes apache#648 Closes apache#662 Closes apache#668 Closes apache#692 Closes apache#705 Closes apache#724 Closes apache#728 Closes apache#730 Closes apache#753 Closes apache#803 Closes apache#854 Closes apache#922 Closes apache#986 Closes apache#992 Closes apache#1019 Closes apache#1040 Closes apache#1041 Closes apache#1043 Closes apache#1046 Closes apache#1051 Closes apache#1078 Closes apache#1146 Closes apache#1164 Closes apache#1165 Closes apache#1178 Closes apache#1213 Closes apache#1225 Closes apache#1258 Closes apache#1259 Closes apache#1268 Closes apache#1272 Closes apache#1277 Closes apache#1278 Closes apache#1288 Closes apache#1296 Closes apache#1328 Closes apache#1342 Closes apache#1353 Closes apache#1370 Closes apache#1376 Closes apache#1391 Closes apache#1395 Closes apache#1399 Closes apache#1406 Closes apache#1410 Closes apache#1422 Closes apache#1427 Closes apache#1443 Closes apache#1462 Closes apache#1468 Closes apache#1483 Closes apache#1506 Closes apache#1509 Closes apache#1515 Closes apache#1520 Closes apache#1521 Closes apache#1525 Closes apache#1527 Closes apache#1544 Closes apache#1550 Closes apache#1566 Closes apache#1569 Closes apache#1570 Closes apache#1575 Closes apache#1580 Closes apache#1584 Closes apache#1591 Closes apache#1600 Closes apache#1611 Closes apache#1613 Closes apache#1639 Closes apache#1703 Closes apache#1711 Closes apache#1719 Closes apache#1737 Closes apache#1760 Closes apache#1767 Closes apache#1768 Closes apache#1785 Closes apache#1799 Closes apache#1822 Closes apache#1824 Closes apache#1844 Closes apache#1874 Closes apache#1918 Closes apache#1928 Closes apache#1937 Closes apache#1942 Closes apache#1951 Closes apache#1957 Closes apache#1963 Closes apache#1964 Closes apache#1965 Closes apache#1967 Closes apache#1968 Closes apache#1971 Closes apache#1985 Closes apache#1986 Closes apache#1998 Closes apache#2031 Closes apache#2032 Closes apache#2071 Closes apache#2076 Closes apache#2108 Closes apache#2119 Closes apache#2128 Closes apache#2142 Closes apache#2174 Closes apache#2206 Closes apache#2297 Closes apache#2322 Closes apache#2332 Closes apache#2341 Closes apache#2377 Closes apache#2414 Closes apache#2469
5 participants
JIRA-STORM-2754
We probably don't want to kill the process if the exceptions are from other threads