CDI not scanning bean libraries in domain applib or lib directories/FISH-7871

[lprimak]

Brief Summary

Trying to create a layered payara domain and thin WARs by putting many large libraries into domain/lib or domain/applib and using asadmin deploy --libraries lib1.jar,lib2.jar xxx.war
When libraries contain CDI artifacts, they are not loaded, nor are they scanned by the Payara deployment process

Why do this?

Class loading takes a lot of time during application deployment.
In order to speed up deployment during development process, large application dependency JARs can be
put into Payara <domain>/lib directory. This prevents class-loading of dependencies during deployment more than once, and can speed up development time by orders of magnitude, and improves developer experience.

Expected Outcome

CDI beans are loaded from applib/ or lib/ in the domain
The following message should print in the Payara server.log:

[#|2023-09-27T19:01:00.719-0500|INFO|Payara 6.2023.9||_ThreadID=99;_ThreadName=admin-thread-pool::admin-listener(1);_TimeMillis=1695859260719;_LevelValue=800;|
  Application Started, Greeting Hello from Library
|#]

Current Outcome

CDI beans are not loaded and application cannot be deployed
Error Message:

[#|2023-09-27T18:44:56.431-0500|SEVERE|Payara 6.2023.9|javax.enterprise.system.core|_ThreadID=99;_ThreadName=admin-thread-pool::admin-listener(1);_TimeMillis=1695858296431;_LevelValue=1000;|
  Exception while loading the app : CDI deployment failure:WELD-001408: Unsatisfied dependencies for type Greeter with qualifiers @Default
  at injection point [BackedAnnotatedField] @Inject com.flowlogix.cdiscanning.app.GreeterApp.greeter
  at com.flowlogix.cdiscanning.app.GreeterApp.greeter(GreeterApp.java:0)
 -- WELD-001408: Unsatisfied dependencies for type Greeter with qualifiers @Default
  at injection point [BackedAnnotatedField] @Inject com.flowlogix.cdiscanning.app.GreeterApp.greeter
  at com.flowlogix.cdiscanning.app.GreeterApp.greeter(GreeterApp.java:0)
|#]

Full exception:

[#|2023-09-27T18:44:56.415-0500|SEVERE|Payara 6.2023.9|javax.enterprise.system.core|_ThreadID=99;_ThreadName=admin-thread-pool::admin-listener(1);_TimeMillis=1695858296415;_LevelValue=1000;_MessageID=NCLS-CORE-00026;|
  Exception during lifecycle processing
org.glassfish.deployment.common.DeploymentException: CDI deployment failure:WELD-001408: Unsatisfied dependencies for type Greeter with qualifiers @Default
  at injection point [BackedAnnotatedField] @Inject com.flowlogix.cdiscanning.app.GreeterApp.greeter
  at com.flowlogix.cdiscanning.app.GreeterApp.greeter(GreeterApp.java:0)
 -- WELD-001408: Unsatisfied dependencies for type Greeter with qualifiers @Default
  at injection point [BackedAnnotatedField] @Inject com.flowlogix.cdiscanning.app.GreeterApp.greeter
  at com.flowlogix.cdiscanning.app.GreeterApp.greeter(GreeterApp.java:0)

	at org.jboss.weld.bootstrap.Validator.validateInjectionPointForDeploymentProblems(Validator.java:367)
	at org.jboss.weld.bootstrap.Validator.validateInjectionPoint(Validator.java:285)
	at org.jboss.weld.bootstrap.Validator.validateGeneralBean(Validator.java:141)
	at org.jboss.weld.bootstrap.Validator.validateRIBean(Validator.java:162)
	at org.jboss.weld.bootstrap.Validator.validateBean(Validator.java:515)
	at org.jboss.weld.bootstrap.Validator.validateBeans(Validator.java:501)
	at org.jboss.weld.bootstrap.Validator.validateDeployment(Validator.java:476)
	at org.jboss.weld.bootstrap.WeldStartup.validateBeans(WeldStartup.java:492)
	at org.jboss.weld.bootstrap.WeldBootstrap.validateBeans(WeldBootstrap.java:94)
	at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:582)
	at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:437)
	at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
	at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:344)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:580)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:570)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:453)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:453)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
	at org.glassfish.admin.rest.resources.admin.CommandResource.executeCommand(CommandResource.java:409)
	at org.glassfish.admin.rest.resources.admin.CommandResource.execCommandSimpInMultOut(CommandResource.java:236)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:240)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:697)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:367)
	at org.glassfish.admin.rest.adapter.JerseyContainerCommandService$3.service(JerseyContainerCommandService.java:179)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
	at java.base/java.lang.Thread.run(Thread.java:1583)
|#]

Reproducer

https://github.com/flowlogix/cdi-scanning

The reproducer contains a library and an app that uses it (Greeter/GreeterApp)

To reproduce the failure:
Inside the cdi-scanning directory:

• mvn package - generates a 'skinny' WAR without the library inside
• copy library/target/library-1.x-SNAPSHOT.jar to <domain_dir>/lib directory (or use asadmin add-library, but this is irrelevant
• Start Payara server
• asadmin deploy app/target/app-1.x-SNAPSHOT.war

---

To get a working application, using standard WAR method:
Inside the cdi-scanning directory:

mvn package -Dskinny.war=false
asadmin deploy app/target/app-1.x-SNAPSHOT.war

Operating System

Any

JDK Version

Any

Payara Distribution

Payara Server Full Profile, Payara Server Web Profile (version 2023.9)

[felixif]

Hello @lprimak,

Thank you for raising this issue. Is there a reason why you are trying to load the libraries using the --libraries flag rather than using the ./asadmin add-library command? I will come back after I test the reproducer.

Best regards,
Felix

[lprimak]

Hi, Felix,

Actually add-library is irrelevant to the problem. The issue exists whether the libraries are in <domain>/lib or <domain>/applib, using add-library, --libraries or not, it doesn't matter.

Easiest test case is just to put a CDI library into <domain>/lib and try that. CDI artifacts aren't loaded no matter how the library gets into the domain.

Thank you!

[github-actions]

Greetings,
It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please?
We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[felixif]

Hello @lprimak,

Please ignore the automation. Apologies for not coming back earlier, other internal projects kept me from replying, I had a look at the reproducer, and in order for me to assess what went wrong, I would need more information: do I require any additional configuration besides an empty Postgres database and the data source?

The project is also targeted at Java 21. We do not officially support Java 21 on Payara 6 Community Edition, only Java 17 and 11. We plan on adding support for Java 21 in the future, but we do not officially support the platform yet. So if you could provide me with a reproducer targeting Java 17 or 11 and explain the additional configuration steps on the server side, I could raise the issue with the Platform Development team.

[lprimak]

Ok, I’ll create a branch

Just FYI the target doesn’t really make a difference but I do get the need tor an easier reproducer. I’ll get something going

[lprimak]

Hi, Felix,

I have created a much simpler reproducer and re-worked the issue description above.
Please take a look, it should be true http://sscce.org so you can easily see the issue.

Thank for your patience!

[github-actions]

Greetings,
It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please?
We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[github-actions]

Greetings,
It's been more than 5 days since this issue was identified as abandoned.
We have closed this issue due to inactivity, please feel free to re-open it if you have more information to share.

[lprimak]

Not abandoned. I can't even reopen the issue :(

[felixif]

Hello @lprimak,

Apologies for the automation closing the issue before it was done. I will have a look at the reproducer today and come back with an update

[felixif]

Hello @lprimak,

I have finished looking through the reproducer. I had to add a couple of things (namely the compile and source Java version, 11 in my case) and the maven-war-plugin with the property failOnMissingWebXml set to false.

In the case of -Dskinny.war=true, I can see the error that you are describing, the app is not even being able to be deployed.
In the case of -Dskinny.war=false, it works as expected.

Overall, your use case should be supported, as adding the library to the <domain_dir>/lib should be enough. I even tried <domain_dir>/lib/ext, with no success. I raised an internal issue with the internal reference FISH-7871, for the Platform Development team to investigate and fix the bug.

Thank you very much for the contribution and for the easy-to-follow reproducer!

Best regards,
Felix

[lprimak]

Thank you Felix. BTW your maven must be ancient for you to need to add anything. If you upgrade to latest maven it should work out of the box.

[felixif]

Hello @lprimak, no problem!

I am using Maven 3.8.6. So probably a year out of date, but you made me curious, so I will update my Maven and try it again with no modifications.

Thank you again for the many contributions you have made to our platform!

Kind regards,
Felix

[lprimak]

Since I am working on CDI integration, I will put a fix for this on my TODO list :)