JDBC driver org.apache.derby.jdbc.ClientDriver could not be loaded

[alwin-joseph]

Describe the bug

ant -f initdb.xml fails with below error for jpa tck (also for security TCK)

BUILD FAILED
/home/jenkins/agent/workspace/jakartaee-tck_master/persistence-tck/bin/initdb.xml:46: The following error occurred while executing this line:
/home/jenkins/agent/workspace/jakartaee-tck_master/persistence-tck/bin/initdb.xml:78: The following error occurred while executing this line:
/home/jenkins/agent/workspace/jakartaee-tck_master/persistence-tck/bin/initdb.xml:51: The following error occurred while executing this line:
/home/jenkins/agent/workspace/jakartaee-tck_master/persistence-tck/bin/initdb.xml:65: Class Not Found: JDBC driver org.apache.derby.jdbc.ClientDriver could not be loaded

Complete test run logs : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck/branches/master/runs/1243/nodes/72/steps/271/log/?start=0

To Reproduce
Download https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee9-eftl/staged-910/jakarta-persistence-tck-3.0.0.zip
Run jpa tck using docker/jpatck.sh

Additional context

Derby Information --------
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derby.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbytools.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbynet.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbyclient.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbyshared.jar] 10.15.2.0 - (1873585)

[scottmarlow]

It sounds classpath related.
From the standalone JPA tck console, I see classpath is set via:
classpath appears to be set via: sed -i 's#jdbc\.driver\.classes=.*#jdbc.driver.classes=/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbyclient.jar:/home/jenkins/agent/workspace/jakartaee-tck_master/persistence-tck/lib/dbprocedures.jar#g' ts.jte

The static configuration for JakartaEE Full Platform ts.jte.jdk11 has derby.classes.ri=${derby.home.ri}/lib/derbyclient.jar

This doesn't explain the failure as that jar does exist as per:

--------- Derby Information --------
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derby.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbytools.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbynet.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbyclient.jar] 10.15.2.0 - (1873585)
[/home/jenkins/agent/workspace/jakartaee-tck_master/glassfish6/javadb/lib/derbyshared.jar] 10.15.2.0 - (1873585)

[alwin-joseph]

I see similar issue in other tests too :

Platform TCK :
appclient :
/root/jakartaeetck/bin/xml/initdb.xml:505: Class Not Found: JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded
logs : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck/branches/master/runs/1248/nodes/98/steps/840/log/?start=0

ejb30/assembly :
/root/jakartaeetck/bin/xml/initdb.xml:505: Class Not Found: JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded
logs : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck/branches/master/runs/1248/nodes/118/steps/491/log/?start=0

[hussainnm]

Derby 10.15.2.0 requires derbyshared.jar and derbytools.jar to be added along with derbyclient.jar. The JDBC driver is present in derbytools.jar.

[alwin-joseph]

Derby 10.15.2.0 requires derbyshared.jar and derbytools.jar to be added along with derbyclient.jar. The JDBC driver is present in derbytools.jar.

I can confirm that this fixes the JPA tests. Thanks much for the tip!!

I will check the same for other failing suites also.

[alwin-joseph]

Derby 10.15.2.0 requires derbyshared.jar and derbytools.jar to be added along with derbyclient.jar. The JDBC driver is present in derbytools.jar.

I can confirm that this fixes the JPA tests. Thanks much for the tip!!

I will check the same for other failing suites also.

I could see this fixed tests in jpa, jstl, security tck tests with this change #630 .

I see similar issue in other tests too :

Platform TCK :
appclient :
/root/jakartaeetck/bin/xml/initdb.xml:505: Class Not Found: JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded
logs : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck/branches/master/runs/1248/nodes/98/steps/840/log/?start=0

ejb30/assembly :
/root/jakartaeetck/bin/xml/initdb.xml:505: Class Not Found: JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded
logs : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck/branches/master/runs/1248/nodes/118/steps/491/log/?start=0

The above exception is "JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded" . I was able to resolve this issue by adding derbyshared.jar and derbytools.jar along with derby.jar and derbyclient.jar. But I encounter below exception

/root/jakartaeetck/bin/xml/initdb.xml:528: java.sql.SQLNonTransientConnectionException: java.net.ConnectException : Error connecting to server localhost on port 1,527 with message Connection refused (Connection refused)
because the db was not up.

logs :
job : https://ci.eclipse.org/jakartaee-tck/blue/organizations/jenkins/jakartaee-tck-alw/detail/embeddeddriver_fix/4/pipeline/38
complete run log : https://ci.eclipse.org/jakartaee-tck/blue/rest/organizations/jenkins/pipelines/jakartaee-tck-alw/branches/embeddeddriver_fix/runs/4/nodes/38/steps/58/log/?start=0

#630 (review) are there any other known changes required for derby db config ?

[hussainnm]

Can you check the derby.log to see why it did not start.

[alwin-joseph]

derby.log

I dont see any errors in vi derby.log except
Database Class Loader started - derby.database.classpath=''

[arjantijms]

Hi, One thing I encountered on my local system, but not on the Jenkins instance (and it's specific docker image for the GF build), is that Derby runs with a security manager. This causes it to check a policy file with connection permissions, which on its turn activates a search for the java.net.DefaultInterface, which again on its turn does a reverse DNS lookup for all network interfaces on the system. Locally this causes a timeout in the ./asadmin start-database command, but since it starts Derby asynchronously in the background, Derby continues to start. It just takes (a lot) more time. There's an option in Derby to disable the security manager, but it needs to be passed from the asadmin process to Derby then, so needs some coding. You could check if this is indeed the case on the server you're trying by simply starting Derby separately using ./asadmin start-database and seeing if it takes too long or not. Kind regards, Arjan

…

On Tue, Mar 2, 2021 at 10:29 AM Alwin Joseph ***@***.***> wrote: derby.log <https://github.com/eclipse-ee4j/jakartaee-tck/files/6067551/derby.log> I dont see any errors in vi derby.log except Database Class Loader started - derby.database.classpath='' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXFOTTBSSECCUTETU55B43TBSVXPANCNFSM4YIEMGIQ> .

[scottmarlow]

From last nights Full Platform TCK TCK run ./root/run_cts.log in https://ci.eclipse.org/jakartaee-tck/job/jakartaee-tck/job/master/1275/artifact/ejb30_lite_tx-results.tar.gz:

ping.jdbc.pool.check:
     [echo] ts.home = /root/jakartaeetck/bin/xml/../..
     [echo] In a bundle
     [echo] The deliverable currently in use is /root/jakartaeetck/bin/xml/../../bin.

clean.derby.embedded:
     [echo] Deleting embedded derby DB located at /tmp/DerbyDB

derby.embedded.init:
     [echo] init.derby.embedded is true

init.derbyEmbedded:
     [echo] ts.home = /root/jakartaeetck/bin/xml/../..
     [echo] In a bundle
     [echo] The deliverable currently in use is /root/jakartaeetck/bin/xml/../../bin.

check.configure.cmp:

configure.basic:
     [echo] DB URL: jdbc:derby:/tmp/DerbyDB;create=true

BUILD FAILED
/root/jakartaeetck/bin/build.xml:441: The following error occurred while executing this line:
/root/jakartaeetck/bin/xml/impl/glassfish/javaee_vi.xml:107: The following error occurred while executing this line:
/root/jakartaeetck/bin/xml/initdb.xml:137: The following error occurred while executing this line:
/root/jakartaeetck/bin/xml/initdb.xml:505: Class Not Found: JDBC driver org.apache.derby.jdbc.EmbeddedDriver could not be loaded

[scottmarlow]

looks like we also need to add add derbyshared.jar + derbytools.jar to:

• install/jacc/bin/ts.jte.jdk11 (look for database.classes=${jdbc.lib.class.path}/derbyclient.jar)
• install/jakartaee/bin/ts.jte.jdk11 (look for derbyclient.jar)
• internal/docs/jpa/README.txt (look for derbyclient.jar)

An example that is correct is https://github.com/eclipse-ee4j/jakartaee-tck/blob/master/install/securityapi/bin/ts.jte.jdk11#L467

[scottmarlow]

email sent to GlassFish + TCK mailing list asking for a volunteer. Please comment here on the issue if you are volunteering to fix this. as per my previous comment. Thanks!

Also, I assigned to myself until we have a volunteer.

[suhridk]

Hi Scott, I would like to help. Could you assign this to me ?

[scottmarlow]

@suhridk you are assigned now, can you create the pull request now? Be sure to sign the commit with your github account that is associated with your Eclipse account that you signed the ECA with.

When you commit, be sure to use -s to sign it:
git commit -s -a -m"description"

[scottmarlow]

It also looks like GlassFish needs to add derbyshared.jar to com.sun.enterprise.admin.cli.optional.DatabaseCommand. I'll create an issue for that.

Hi, One thing I encountered on my local system, but not on the Jenkins instance (and it's specific docker image for the GF build), is that Derby runs with a security manager. This causes it to check a policy file with connection permissions, which on its turn activates a search for the java.net.DefaultInterface, which again on its turn does a reverse DNS lookup for all network interfaces on the system. Locally this causes a timeout in the ./asadmin start-database command, but since it starts Derby asynchronously in the background, Derby continues to start. It just takes (a lot) more time. There's an option in Derby to disable the security manager, but it needs to be passed from the asadmin process to Derby then, so needs some coding. You could check if this is indeed the case on the server you're trying by simply starting Derby separately using ./asadmin start-database and seeing if it takes too long or not. Kind regards, Arjan
…
On Tue, Mar 2, 2021 at 10:29 AM Alwin Joseph @.***> wrote: derby.log https://github.com/eclipse-ee4j/jakartaee-tck/files/6067551/derby.log I dont see any errors in vi derby.log except Database Class Loader started - derby.database.classpath='' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#627 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXFOTTBSSECCUTETU55B43TBSVXPANCNFSM4YIEMGIQ .

I started change to remove the security manager from Derby via https://github.com/scottmarlow/jakartaee-tck/tree/start_database in case that helps avoid the timeout issue mentioned.

[hussainnm]

@scottmarlow If it is not asadmin command, then it should be the classpath for embedded driver that is set in ts.jte

https://github.com/eclipse-ee4j/jakartaee-tck/blob/ef2de161d735e651c8ae374f7086638d97a41e5f/install/jakartaee/bin/ts.jte#L568

[scottmarlow]

I built your branch via https://ci.eclipse.org/jakartaee-tck/job/build-glassfish-scott/1/ which I am trying via https://ci.eclipse.org/jakartaee-tck/job/9.1/job/eftl-standalonetck-build-run-910/32/.

Otherwise, I'll try the ts.jte change you mentioned:

-derbyEmbedded.classes=${javaee.home}/../javadb/lib/derby.jar${pathsep}${ts.home}/lib/dbprocedures.jar
+derbyEmbedded.classes=${javaee.home}/../javadb/lib/derby.jar${pathsep}${ts.home}/lib/dbprocedures.jar${pathsep}${javaee.home}/../javadb/lib/derbyshared.jar${pathsep}${javaee.home}/../javadb/lib/derbytools.jar

[hussainnm]

I think derby.classpath.ri also needs to be updated

https://github.com/eclipse-ee4j/jakartaee-tck/blob/51ba10f22d2d83d789a3720d01d7e71161b28546/install/jakartaee/bin/ts.jte.jdk11#L905

This property is used in javaee_ri.xml to set use.derby.classpath

https://github.com/eclipse-ee4j/jakartaee-tck/blob/51ba10f22d2d83d789a3720d01d7e71161b28546/install/jakartaee/bin/xml/impl/glassfish/javaee_ri.xml#L48

which is later used to set the classpath to start the derby server

https://github.com/eclipse-ee4j/jakartaee-tck/blob/98fa8e8f0a619cf3de5401aa0c45695de6c23265/bin/xml/impl/glassfish/common.xml#L921-L927

[hussainnm]

I found the root cause to be a permission issue which is preventing the database from starting. I have raised a PR in glassfish to add the required permission which will resolve the Connection Refused error.

This permission needs to be added to https://github.com/eclipse-ee4j/jakartaee-tck/blob/master/install/jpa/bin/security.policy. I will raise a PR here for the same.