Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReactorResourceFactory not working with CRaC onRefresh checkpoint #32945

Closed
frederikz opened this issue Jun 3, 2024 · 10 comments
Closed

ReactorResourceFactory not working with CRaC onRefresh checkpoint #32945

frederikz opened this issue Jun 3, 2024 · 10 comments
Assignees
@jhoeller
Labels
in: web Issues in web modules (web, webmvc, webflux, websocket) status: feedback-provided Feedback has been provided type: bug A general bug
Milestone
6.1.9

Comments

@frederikz
Copy link

I have a spring boot 3.3.0 application running with embedded tomcat which uses WebClient. When I try out CRaC and generate a checkpoint with -Dspring.context.checkpoint=onRefresh
I get errors during checkpoint creation. The cause is org.springframework.http.client.ReactorResourceFactory which is already started in afterPropertiesSet. I could prevent the errors by implementing a org.crac.Resource that stops the ReactorResourceFactory beforeCheckpoint. As I have some other problems I couldn't try if everything would be fine if I start it again in afterRestore.
Errors during checkpoint creation:

Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=8 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=9 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=10 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=11 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=12 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=13 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=14 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=15 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=16 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=17 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=18 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=19 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=20 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=21 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=22 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=23 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=24 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=25 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=26 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=27 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=28 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=29 type=unknown path=anon_inode:[eventpoll]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=30 type=unknown path=anon_inode:[eventfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted
Suppressed: jdk.crac.impl.CheckpointOpenResourceException: FD fd=31 type=unknown path=anon_inode:[timerfd]
		at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:122) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore1(Core.java:192) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:299) ~[na:na]
		at java.base/jdk.crac.Core.checkpointRestore(Core.java:278) ~[na:na]
		at java.base/javax.crac.Core.checkpointRestore(Core.java:73) ~[na:na]
		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
		at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
		at org.crac.Core$Compat.checkpointRestore(Core.java:141) ~[crac-1.4.0.jar!/:na]
		... 17 common frames omitted

I don't know what the reason is that ReactorResourceFactory is already started in afterPropertiesSet. Either don't start it already in afterPropertiesSet or provide a org.crac.Resource implementation that stops it before a checkpoint (at least when the checkpoint is made onRefresh as later after start phase it is alrady stopped by the lifecycle implementation) and starts again after restore.
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Jun 3, 2024
@sdeleuze sdeleuze self-assigned this Jun 3, 2024
@sdeleuze
Copy link
Contributor

sdeleuze commented Jun 3, 2024

ReactorResourceFactory needs to be declared as bean to handle properly the lifecycle as it implements Lifecycle. I have not checked on Spring Boot side if that the case or not when it is used only in the client. Could you please double check if that's the case, and if you still think there is a potential Spring Framework issue involved, can you please provide a reproduced as an attached archive or a link to a repository?

@sdeleuze sdeleuze added status: waiting-for-feedback We need additional information before we can continue in: web Issues in web modules (web, webmvc, webflux, websocket) labels Jun 3, 2024
@frederikz
Copy link
Author

The problem is that with -Dspring.context.checkpoint=onRefresh beans are not started so stop isn't called on Lifecyle beans. If you later after a start event do a manual jcmd JDK.checkpoint the lifecycle beans are stopped and everything is fine.
So currently with -Dspring.context.checkpoint=onRefresh CracResourceAdapter in DefaultLifecycleProcessor calls stopForRestart which does nothing as this.running is false. So even though there is a ReactorResourceFactory lifecycle bean stop is never called on it which makes somewhat sense as we haven't reached the start event and Spring hasn't called start on the lifecycle beans. The problem is that ReactorResourceFactory itself already internally called the lifecycle method start in afterPropertiesSet.
If you later manually do a JDK.checkpoint the application is already started so Sping called start on Lifcycle beans and the CracResourceAdapter with stopForRestart would also call stop on all Lifcycle beans including the ReactorResourceFactory.

So currently when you do a checkpoint with -Dspring.context.checkpoint=onRefresh Lifcycle beans don't take part in the beforeCheckpoint and afterRestore process of CRaC as we haven't reached the start point of Lifcycle.

Hope this helps to understand the problem without providing an example project?

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Jun 4, 2024
@sdeleuze
Copy link
Contributor

sdeleuze commented Jun 4, 2024

Thanks for the detailed explanation, but a repro is still welcomed, especially for this kind of issue where the devil is usually in the details.

@sdeleuze sdeleuze added status: waiting-for-feedback We need additional information before we can continue and removed status: feedback-provided Feedback has been provided labels Jun 4, 2024
@frederikz
Copy link
Author

I did a fork of your spring-boot-crac-demo: https://github.com/frederikz/spring-boot-crac-demo
Added a spring-boot-starter-webflux dependency and a WebClient needs to be created (but doesn't has to be a Bean like in the example). Changed your checkpoint.sh so that it executes in the foreground and you can see the error.
Thanks for investigating.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Jun 4, 2024
@jhoeller
Copy link
Contributor

jhoeller commented Jun 4, 2024 •
edited
Loading

@sdeleuze looking at the ReactorResourceFactory implementation, it was made lifecycle-aware after initially relying on init and destroy methods... and in the course of that revision, the init/destroy methods were retained as internally delegating to start/stop. This seems suboptimal since the container lifecycle management is not aware of these internal calls.

Could we simply remove the InitializingBean/DisposableBean methods there and rather implement SmartLifecycle which implies an auto-startup call within the container's lifecycle management?

@jhoeller jhoeller added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Jun 4, 2024
@jhoeller jhoeller added this to the 6.1.9 milestone Jun 4, 2024
@sdeleuze sdeleuze assigned jhoeller and unassigned sdeleuze Jun 4, 2024
@sdeleuze sdeleuze changed the title org.springframework.http.client.ReactorResourceFactory not working with CRaC ReactorResourceFactory not working with CRaC Jun 4, 2024
@jhoeller jhoeller changed the title ReactorResourceFactory not working with CRaC ReactorResourceFactory not working with CRaC onRefresh checkpoint Jun 4, 2024
@jhoeller
Copy link
Contributor

jhoeller commented Jun 4, 2024

A new ReactorResourceFactory revision will be available in the upcoming 6.1.9 snapshot, participating in standard lifecycle management when running within an ApplicationContext. Feel free to give it an early try!

@frederikz
Copy link
Author

It seems like calling the start method was required before the start lifecycle. For a quick test I directly replaced only the changed ReactorResourceFactory.class in my spring-web-6.1.8.jar and now I get this exception:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webClientBuilder' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/WebClientAutoConfiguration.class]: Failed to instantiate [org.springframework.web.reactive.function.client.WebClient$Builder]: Factory method 'webClientBuilder' threw exception with message: Error creating bean with name 'webClientHttpConnectorCustomizer' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webClientHttpConnectorCustomizer' parameter 0: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: ConnectionProvider not initialized yet
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:355) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 24 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.reactive.function.client.WebClient$Builder]: Factory method 'webClientBuilder' threw exception with message: Error creating bean with name 'webClientHttpConnectorCustomizer' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webClientHttpConnectorCustomizer' parameter 0: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: ConnectionProvider not initialized yet
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 36 common frames omitted
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webClientHttpConnectorCustomizer' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'webClientHttpConnectorCustomizer' parameter 0: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: ConnectionProvider not initialized yet
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:795) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:542) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1696) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1653) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveMultipleBeans(DefaultListableBeanFactory.java:1470) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1392) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.resolveStream(DefaultListableBeanFactory.java:2198) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.orderedStream(DefaultListableBeanFactory.java:2192) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.boot.autoconfigure.web.reactive.function.client.WebClientAutoConfiguration.webClientBuilder(WebClientAutoConfiguration.java:55) ~[spring-boot-autoconfigure-3.3.0.jar!/:3.3.0]
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 37 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webClientHttpConnector' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/function/client/ClientHttpConnectorAutoConfiguration.class]: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: ConnectionProvider not initialized yet
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 57 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.http.client.reactive.ClientHttpConnector]: Factory method 'webClientHttpConnector' threw exception with message: ConnectionProvider not initialized yet
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177) ~[spring-beans-6.1.8.jar!/:6.1.8]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 71 common frames omitted
Caused by: java.lang.IllegalStateException: ConnectionProvider not initialized yet
        at org.springframework.util.Assert.state(Assert.java:76) ~[spring-core-6.1.8.jar!/:6.1.8]
        at org.springframework.http.client.ReactorResourceFactory.getConnectionProvider(ReactorResourceFactory.java:146) ~[spring-web-6.1.8.jar!/:6.1.8]
        at org.springframework.http.client.reactive.ReactorClientHttpConnector.createHttpClient(ReactorClientHttpConnector.java:107) ~[spring-web-6.1.8.jar!/:6.1.8]
        at org.springframework.http.client.reactive.ReactorClientHttpConnector.<init>(ReactorClientHttpConnector.java:101) ~[spring-web-6.1.8.jar!/:6.1.8]
        at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory.createClientHttpConnector(ReactorClientHttpConnectorFactory.java:68) ~[spring-boot-autoconfigure-3.3.0.jar!/:3.3.0]
        at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory.createClientHttpConnector(ReactorClientHttpConnectorFactory.java:44) ~[spring-boot-autoconfigure-3.3.0.jar!/:3.3.0]
        at org.springframework.boot.autoconfigure.web.reactive.function.client.ClientHttpConnectorFactory.createClientHttpConnector(ClientHttpConnectorFactory.java:32) ~[spring-boot-autoconfigure-3.3.0.jar!/:3.3.0]
        at org.springframework.boot.autoconfigure.web.reactive.function.client.ClientHttpConnectorAutoConfiguration.webClientHttpConnector(ClientHttpConnectorAutoConfiguration.java:57) ~[spring-boot-autoconfigure-3.3.0.jar!/:3.3.0]
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140) ~[spring-beans-6.1.8.jar!/:6.1.8]
        ... 72 common frames omitted

@jhoeller
Copy link
Contributor

jhoeller commented Jun 5, 2024

We also got a similar phase mismatch with Boot's NettyWebServer. Reopening for another revision.

@jhoeller jhoeller reopened this Jun 5, 2024
@jhoeller
Copy link
Contributor

jhoeller commented Jun 5, 2024

I've pushed another revision. Please let me know whether the upcoming 6.1.9 snapshot works for your purposes now... either through upgrading the entire spring-web.jar or through patching ReactorResourceFactory as well as ReactorClientHttpConnector (which has corresponding changes as well now).

@frederikz
Copy link
Author

Thanks, works.

jhoeller added a commit that referenced this issue Jun 6, 2024
@jhoeller
Lazily start resources on demand (if necessary outside of lifecycle)
c0bef2c 
See gh-32945
@kzander91 kzander91 mentioned this issue Jun 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhoeller jhoeller

Labels
in: web Issues in web modules (web, webmvc, webflux, websocket) status: feedback-provided Feedback has been provided type: bug A general bug
Projects
None yet
Milestone
6.1.9
Development

No branches or pull requests
4 participants
@sdeleuze @jhoeller @spring-projects-issues @frederikz