Skip to content

Shutdown problems related to Tomcat service being stopped before app context is #4396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
trisberg opened this issue Nov 6, 2015 · 3 comments
Closed

Shutdown problems related to Tomcat service being stopped before app context is #4396

trisberg opened this issue Nov 6, 2015 · 3 comments
Milestone
1.2.8

Comments

@trisberg
Copy link
Member

trisberg commented Nov 6, 2015

Have some issues where the change to stop Tomcat service before stopping the app context (issue #4130) seems to cause other problems.

Starting a simple Boot web app (https://github.com/trisberg/boot-shutdown) and shutting it down via the actuator shutdown endpoint I see the following:

2015-11-06 10:51:30.960  INFO 20904 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2015-11-06 10:51:30.962  INFO 20904 --- [           main] com.example.DemoApplication              : Started DemoApplication in 2.703 seconds (JVM running for 3.014)
2015-11-06 10:51:54.828  INFO 20904 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
2015-11-06 10:51:54.829  INFO 20904 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
2015-11-06 10:51:54.840  INFO 20904 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 11 ms
2015-11-06 10:51:55.412  INFO 20904 --- [       Thread-4] o.apache.catalina.core.StandardService   : Stopping service Tomcat
2015-11-06 10:51:55.414  INFO 20904 --- [ost-startStop-2] o.a.c.c.C.[Tomcat].[localhost].[/]       : Destroying Spring FrameworkServlet 'dispatcherServlet'
2015-11-06 10:51:55.417  WARN 20904 --- [ost-startStop-2] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
 java.util.concurrent.FutureTask.get(FutureTask.java:191)
 org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:972)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.StandardService.stopInternal(StandardService.java:501)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:790)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.startup.Tomcat.stop(Tomcat.java:347)
 org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:228)
 org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:295)
 org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.doClose(EmbeddedWebApplicationContext.java:149)
 org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:840)
 org.springframework.boot.actuate.endpoint.ShutdownEndpoint$1.run(ShutdownEndpoint.java:70)
 java.lang.Thread.run(Thread.java:745)
2015-11-06 10:51:55.417  WARN 20904 --- [ost-startStop-2] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [Tomcat-startStop-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
 java.util.concurrent.FutureTask.get(FutureTask.java:191)
 org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:972)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
 org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
 java.util.concurrent.FutureTask.run(FutureTask.java:266)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
2015-11-06 10:51:55.422  INFO 20904 --- [       Thread-4] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@74120a2a: startup date [Fri Nov 06 10:51:28 EST 2015]; root of context hierarchy
2015-11-06 10:51:55.423  INFO 20904 --- [       Thread-4] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2015-11-06 10:51:55.424  INFO 20904 --- [       Thread-4] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

Also when developing a new module for spring-cloud-stream-modules there is an exception when beans in the context are trying to do some cleanup work due to the Tomcat classloader saying it is closed for business and reporting "Illegal access". (spring-attic/spring-cloud-stream-modules#58)

2015-11-06 09:47:56.729  INFO 19503 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2015-11-06 09:47:56.737  INFO 19503 --- [           main] o.s.c.s.m.d.sink.DatasetSinkApplication  : Started DatasetSinkApplication in 8.361 seconds (JVM running for 9.306)
2015-11-06 09:47:58.337  INFO 19503 --- [hannel-adapter1] o.k.d.spi.filesystem.FileSystemWriter    : Opened output appender AvroAppender{path=hdfs://localhost:8020/tmp/hdfs-dataset-sink/data/string/.5daf2ad3-94a0-41eb-b27c-817a3cb95f51.avro.tmp, schema="string", fileSystem=DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1746917339_9, ugi=trisberg (auth:SIMPLE)]], enableCompression=true, dataFileWriter=org.apache.avro.file.DataFileWriter@1aeaaa12, writer=org.apache.avro.reflect.ReflectDatumWriter@2443918c} for hdfs://localhost:8020/tmp/hdfs-dataset-sink/data/string/5daf2ad3-94a0-41eb-b27c-817a3cb95f51.avro
2015-11-06 09:48:03.372  INFO 19503 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
2015-11-06 09:48:03.372  INFO 19503 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
2015-11-06 09:48:03.398  INFO 19503 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 25 ms
2015-11-06 09:48:03.989  INFO 19503 --- [      Thread-12] o.apache.catalina.core.StandardService   : Stopping service Tomcat
2015-11-06 09:48:03.991  INFO 19503 --- [ost-startStop-2] o.a.c.c.C.[Tomcat].[localhost].[/]       : Destroying Spring FrameworkServlet 'dispatcherServlet'
2015-11-06 09:48:03.996  WARN 19503 --- [ost-startStop-2] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [Thread-12] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
 java.util.concurrent.FutureTask.get(FutureTask.java:187)
 org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:972)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.StandardService.stopInternal(StandardService.java:501)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:790)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.startup.Tomcat.stop(Tomcat.java:347)
 org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:236)
 org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:297)
 org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.doClose(EmbeddedWebApplicationContext.java:150)
 org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
 org.springframework.boot.actuate.endpoint.ShutdownEndpoint$1.run(ShutdownEndpoint.java:70)
 java.lang.Thread.run(Thread.java:745)
2015-11-06 09:48:03.996  WARN 19503 --- [ost-startStop-2] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [Tomcat-startStop-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
 java.util.concurrent.FutureTask.get(FutureTask.java:187)
 org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:972)
 org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
 org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
 org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
 java.util.concurrent.FutureTask.run(FutureTask.java:262)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 java.lang.Thread.run(Thread.java:745)
2015-11-06 09:48:04.002  INFO 19503 --- [      Thread-12] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@8247262: startup date [Fri Nov 06 09:47:49 EST 2015]; root of context hierarchy
2015-11-06 09:48:04.003  INFO 19503 --- [      Thread-12] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 2147482647
2015-11-06 09:48:04.004  INFO 19503 --- [      Thread-12] o.s.i.store.MessageGroupStoreReaper      : Expiring all messages from message group store: org.springframework.integration.store.SimpleMessageStore@10b53376
2015-11-06 09:48:04.004  INFO 19503 --- [      Thread-12] .s.i.a.AbstractCorrelatingMessageHandler : Expiring MessageGroup with correlationKey[java.lang.String]
2015-11-06 09:48:04.010 ERROR 19503 --- [      Thread-12] o.s.i.store.SimpleMessageStore           : Exception in expiry callback

org.springframework.messaging.MessageHandlingException: error occurred in message handler [datasetSinkConfiguration.datasetSinkMessageHandler.serviceActivator.handler]; nested exception is java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.xml.parsers.DocumentBuilderFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:139) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:147) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:120) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:442) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:392) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:654) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:639) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.expireGroup(AbstractCorrelatingMessageHandler.java:616) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.forceComplete(AbstractCorrelatingMessageHandler.java:540) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler$ForceReleaseMessageGroupProcessor.processMessageGroup(AbstractCorrelatingMessageHandler.java:718) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler$1.execute(AbstractCorrelatingMessageHandler.java:163) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.AbstractMessageGroupStore.expire(AbstractMessageGroupStore.java:183) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.AbstractMessageGroupStore.expireMessageGroups(AbstractMessageGroupStore.java:122) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.destroy(MessageGroupStoreReaper.java:97) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.stop(MessageGroupStoreReaper.java:138) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.stop(MessageGroupStoreReaper.java:182) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:229) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:220) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:51) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:363) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:202) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:118) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:969) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.doClose(EmbeddedWebApplicationContext.java:151) [spring-boot-1.3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.boot.actuate.endpoint.ShutdownEndpoint$1.run(ShutdownEndpoint.java:70) [spring-boot-actuator-1.3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.xml.parsers.DocumentBuilderFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1355) ~[tomcat-embed-core-8.0.28.jar!/:8.0.28]
    at org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream(WebappClassLoaderBase.java:1088) ~[tomcat-embed-core-8.0.28.jar!/:8.0.28]
    at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94) ~[na:1.7.0_67]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
    at javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87) ~[na:1.7.0_67]
    at javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283) ~[na:1.7.0_67]
    at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255) ~[na:1.7.0_67]
    at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121) ~[na:1.7.0_67]
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2516) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1232) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:178) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) ~[hadoop-common-2.7.1.jar!/:na]
    at org.kitesdk.data.DatasetDescriptor$Builder.<init>(DatasetDescriptor.java:354) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.load(FileSystemMetadataProvider.java:118) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.AbstractDatasetRepository.load(AbstractDatasetRepository.java:40) ~[kite-data-core-1.0.0.jar!/:na]
    at org.springframework.data.hadoop.store.dataset.DatasetUtils.getOrCreateDataset(DatasetUtils.java:73) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AvroPojoDatasetStoreWriter.createWriter(AvroPojoDatasetStoreWriter.java:70) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AbstractDatasetStoreWriter.write(AbstractDatasetStoreWriter.java:75) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AvroPojoDatasetStoreWriter.write(AvroPojoDatasetStoreWriter.java:64) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.DatasetTemplate.write(DatasetTemplate.java:271) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.cloud.stream.module.dataset.sink.DatasetSinkConfiguration$1.handleMessage(DatasetSinkConfiguration.java:117) ~[hdfs-dataset-sink-1.0.0.BUILD-SNAPSHOT-exec.jar!/:1.0.0.BUILD-SNAPSHOT]
    at org.springframework.integration.config.annotation.ServiceActivatorAnnotationPostProcessor$ReplyProducingMessageHandlerWrapper.handleRequestMessage(ServiceActivatorAnnotationPostProcessor.java:98) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    ... 34 common frames omitted

2015-11-06 09:48:04.011 ERROR 19503 --- [      Thread-12] o.s.i.store.MessageGroupStoreReaper      : failed to stop bean

org.springframework.messaging.MessageHandlingException: error occurred in message handler [datasetSinkConfiguration.datasetSinkMessageHandler.serviceActivator.handler]; nested exception is java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.xml.parsers.DocumentBuilderFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:139) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:147) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:120) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:442) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:392) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:654) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:639) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.expireGroup(AbstractCorrelatingMessageHandler.java:616) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.forceComplete(AbstractCorrelatingMessageHandler.java:540) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler$ForceReleaseMessageGroupProcessor.processMessageGroup(AbstractCorrelatingMessageHandler.java:718) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler$1.execute(AbstractCorrelatingMessageHandler.java:163) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.AbstractMessageGroupStore.expire(AbstractMessageGroupStore.java:183) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.AbstractMessageGroupStore.expireMessageGroups(AbstractMessageGroupStore.java:122) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.destroy(MessageGroupStoreReaper.java:97) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.stop(MessageGroupStoreReaper.java:138) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.store.MessageGroupStoreReaper.stop(MessageGroupStoreReaper.java:182) [spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:229) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:220) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:51) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:363) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:202) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:118) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:969) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.doClose(EmbeddedWebApplicationContext.java:151) [spring-boot-1.3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928) [spring-context-4.2.3.BUILD-SNAPSHOT.jar!/:4.2.3.BUILD-SNAPSHOT]
    at org.springframework.boot.actuate.endpoint.ShutdownEndpoint$1.run(ShutdownEndpoint.java:70) [spring-boot-actuator-1.3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [META-INF/services/javax.xml.parsers.DocumentBuilderFactory]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1355) ~[tomcat-embed-core-8.0.28.jar!/:8.0.28]
    at org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream(WebappClassLoaderBase.java:1088) ~[tomcat-embed-core-8.0.28.jar!/:8.0.28]
    at javax.xml.parsers.SecuritySupport$4.run(SecuritySupport.java:94) ~[na:1.7.0_67]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67]
    at javax.xml.parsers.SecuritySupport.getResourceAsStream(SecuritySupport.java:87) ~[na:1.7.0_67]
    at javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:283) ~[na:1.7.0_67]
    at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255) ~[na:1.7.0_67]
    at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121) ~[na:1.7.0_67]
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2516) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2492) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1232) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:178) ~[hadoop-common-2.7.1.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) ~[hadoop-common-2.7.1.jar!/:na]
    at org.kitesdk.data.DatasetDescriptor$Builder.<init>(DatasetDescriptor.java:354) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.load(FileSystemMetadataProvider.java:118) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192) ~[kite-data-core-1.0.0.jar!/:na]
    at org.kitesdk.data.spi.AbstractDatasetRepository.load(AbstractDatasetRepository.java:40) ~[kite-data-core-1.0.0.jar!/:na]
    at org.springframework.data.hadoop.store.dataset.DatasetUtils.getOrCreateDataset(DatasetUtils.java:73) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AvroPojoDatasetStoreWriter.createWriter(AvroPojoDatasetStoreWriter.java:70) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AbstractDatasetStoreWriter.write(AbstractDatasetStoreWriter.java:75) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.AvroPojoDatasetStoreWriter.write(AvroPojoDatasetStoreWriter.java:64) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.data.hadoop.store.dataset.DatasetTemplate.write(DatasetTemplate.java:271) ~[spring-data-hadoop-store-2.3.0.RC1.jar!/:2.3.0.RC1]
    at org.springframework.cloud.stream.module.dataset.sink.DatasetSinkConfiguration$1.handleMessage(DatasetSinkConfiguration.java:117) ~[hdfs-dataset-sink-1.0.0.BUILD-SNAPSHOT-exec.jar!/:1.0.0.BUILD-SNAPSHOT]
    at org.springframework.integration.config.annotation.ServiceActivatorAnnotationPostProcessor$ReplyProducingMessageHandlerWrapper.handleRequestMessage(ServiceActivatorAnnotationPostProcessor.java:98) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.1.RELEASE.jar!/:na]
    ... 34 common frames omitted

2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] .s.i.r.i.RedisQueueMessageDrivenEndpoint : stopped inbound.input
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 2147482646
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 0
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.i.channel.PublishSubscribeChannel    : Channel 'application.errorChannel' has 0 subscriber(s).
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : stopped _org.springframework.integration.errorLogger
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase -2147482648
2015-11-06 09:48:04.014  INFO 19503 --- [      Thread-12] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase -2147483648
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : Removing {message-handler:datasetSinkConfiguration.aggregatorFactoryBean.serviceActivator} as a subscriber to the 'input' channel
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.integration.channel.DirectChannel    : Channel 'application.input' has 0 subscriber(s).
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : stopped datasetSinkConfiguration.aggregatorFactoryBean.serviceActivator
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : Removing {message-handler:datasetSinkConfiguration.datasetSinkMessageHandler.serviceActivator} as a subscriber to the 'toSink' channel
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.integration.channel.DirectChannel    : Channel 'application.toSink' has 0 subscriber(s).
2015-11-06 09:48:04.015  INFO 19503 --- [      Thread-12] o.s.i.endpoint.EventDrivenConsumer       : stopped datasetSinkConfiguration.datasetSinkMessageHandler.serviceActivator
2015-11-06 09:48:04.021  INFO 19503 --- [      Thread-12] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2015-11-06 09:48:04.021  INFO 19503 --- [      Thread-12] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
2015-11-06 09:48:04.024  INFO 19503 --- [      Thread-12] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'taskScheduler'
2015-11-06 09:48:04.026  INFO 19503 --- [      Thread-12] o.s.i.store.MessageGroupStoreReaper      : Expiring all messages from message group store: org.springframework.integration.store.SimpleMessageStore@10b53376
@philwebb
Copy link
Member

philwebb commented Nov 6, 2015

Is this with 1.2.7?

@philwebb
Copy link
Member

philwebb commented Nov 6, 2015

Ignore that. I just read the pom.xml from your sample app.

@philwebb philwebb added this to the 1.2.8 milestone Nov 6, 2015
@trisberg
Copy link
Member Author

trisberg commented Nov 6, 2015

It's with both 1.2.7 and 1.3.0.RC1 or 1.3.0.BUILD-SNAPSHOT

@wilkinsona wilkinsona mentioned this issue Aug 9, 2017
Closed
@wilkinsona wilkinsona mentioned this issue May 31, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.2.8
Development

No branches or pull requests
2 participants
@trisberg @philwebb