remote access using Java API broken in 2.2.2 in OSGi environment?

[grexe]

I'm running OrientDB 2.2.2 from the official docker image (with minor tweaks mainly preprocessing our environment etc.).
Running on Linux (Ubuntu 14.04) 64bit.
Accessing from Java API (Oracle JDK8, OSGi 4.3) using the client, core and graphdb JARs and the Blueprints 2.6.0 (wrapped as OSGi bundles using OPS4J Pax Tipi master POM).

The OSGi Java process is also running in a separate Docker container and accessing OrientDB over the network, using the URL remote:orientdb/t2database

Using 2.1.x, everything worked fine (but we faced some issues with concurrent access we need to track down, this is why we want to make sure to use the latest stable version to rule out OrientDB as a cause).

Now, after upgrading to 2.2.2, I always get this error (but can connect perfectly from the orientdb docker host using console):

the engine 'remote' was not found. URL was: remote:orientdb/t2database. Registered engines are: [memory, plocal]
    DB name="remote:orientdb/t2database"
    at com.orientechnologies.orient.core.Orient.loadStorage(Orient.java:476)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.<init>(ODatabaseDocumentTx.java:167)

Seems like the server doesn't bring up the remote engine, but I see no reason why, and no error in the log, only this:

Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot create the remote storage: t2database

and this (but seems to be #3073 which can supposedly be ignored):

INFO  [OVariableParser.resolveVariables] Error on resolving property: distributed [orientechnologies]

Any hints on what might be the cause?

There was #4883 but it's slightly different, since it never works for me from the Java client, and I have memory and plocal registered.

I am facing this issue since 2.2.0, before it worked.

[taburet]

The problem is in the usage of java.util.ServiceLoader to load OrientDB storage engines in OSGI environment. We will fix this in the next minor release.

As for now, the only workaround I can think of is manually calling Orient.instance().registerEngine(new OEngineRemote()) on startup. But since registerEngine is a protected method and Orient.instance() is a static shared instance you have to call it from the code under the com.orientechnologies.orient.core package:

package com.orientechnologies.orient.core;

import com.orientechnologies.orient.client.remote.OEngineRemote;

public class RemoteEngineFixer {
  public static void applyFix() {
    Orient.instance().registerEngine(new OEngineRemote());
  }
}

and call RemoteEngineFixer.applyFix() on initialisation of your application.

[taburet]

Sorry, RemoteEngineFixer probably will not work since RemoteEngineFixer and Orient will be loaded by different class loaders in OSGI, so package-private access style to the methods of Orient will be not allowed from RemoteEngineFixer. Updated version:

public class RemoteEngineFixer extends Orient {
  public static void applyFix() {
    new RemoteEngineFixer();
  }

  private RemoteEngineFixer() {
    registerEngine(new OEngineRemote());
  }
}

Hope it will work.

[taburet]

Sorry again, it will not work since registerEngine is called on different instance than Orient.instance() :(

[grexe]

thank you @taburet for your thorough analysis and help, I was suspecting OSGi class loading again, as it works from the console.
So I can only hope for the next hotfix release...

[taburet]

@grexe on the other side, you may invoke registerEngine using Reflection API magic :)

[grexe]

hehe we already use enough magic in our project;) and I had enough fun with OSGi already... I can imagine reflection magic won't play nice with the almighty OSGi class loader...

[grexe]

sadly still happens with 2.2.4; I was hoping the changes to RemoteEngineManager I saw in the changelog would help, but they did not:(

[taburet]

Hi @grexe! Unfortunately, I was working hard on other sever issues during 2.2.4. Hope I will find a free time slot to fix this in the next minor versions.

[evandor]

I am using 2.2.6 and have this issue as well... when I checked the source code, I came to the same conclusions (about this being an issue with the ServiceLoader in an OSGi enviroment). If I can be of any assistance with this issue (testing, checking, coding, ...) please let me know.

[taburet]

Hi guys! I started to work on a standalone OrientDB OSGi bundle, so basically it will wrap all the OrientDB modules and it's dependencies in an all-in-one OSGi bundle.

The only problem is OrientDB Spatial module which cannot be distributed in that uber-bundle due to licensing restrictions. And it depends on Apache Lucene non-OSGi jars, the uber-bundle also depends on them. I'm going to export Lucene packages from the uber-bundle to fix this.

WDYT, is it ok or should I repack Lucene jars as a separate OSGi bundle?

[evandor]

With "export Lucene packages" you mean "exclude"? I'm fine with the uber-bundle without Lucene, happy to give it a try ;)

[evandor]

For what it's worth, I was able to resolve the issue I had with creating an uber-bundle consisting of

orientdb-client
orientdb-enterprise
orientdb-graphdb
orientdb-object
orientdb-server
orientdb-tools
orientdb-core
blueprints-core
blueprints-orient-graph

together with the META-INF/serivces files.

(I am using 2.1.3 though).

Actually, the uber-bundle approach feels more like a workaround than a proper solution... Are there any plans to divert from the ServiceLoader path maybe?

[taburet]

The initial plan was to repackage everything in the uber-bundle, but since we cannot repack everything because of spatial module (licensing problems), I abandoned that path. My current thoughts are (1) repackage everything except spatial and enterprise into a single bundle, (2) package spatial and enterprise into their own bundles, (3) package shared non-OSGi deps into a separate bundle(s) and (4) create OSGi service to share OrientDB class loaders between OrientDB bundles.

The main problem is still shared non-OSGi deps. I may create bundles for them, but it will be impossible to get into public repos (Eclipse Marketplace, for example) with this approach. On the other side, it will take ages to convince original authors to create and maintain OSGi distributions for their libraries.

[taburet]

Guys, I created the first version of OrientDB OSGi distribution. Could you please try it in your setups to verify it's working in the wild.

1. git checkout osgi-2.2.x from orientdb repository
2. mvn clean verify it

The resulting uber bundle jar will be at osgi/osgi-core-bundle/target. Please drop it into your OSGi containers.

If you use spatial module:

1. git checkout osgi-2.2.x from orientdb-spatial repository
2. mvn clean verify it, the bundle jar will be at osgi/osgi-spatial-bundle/target

[evandor]

Thanks for your efforts! I checked out osgi-2.2.x and cleaned / verified (but without the tests, there were failures). Now I need to migrate my code to 2.2 (is: 2.1.3) as I - expectedly - have some issues; I'll let you know the outcome asap. (I have no use case for spatial though).

[taburet]

@evandor Could you please provide more details about test failures like logs/exceptions? Was it OSGi tests or regular ones?

[evandor]

I ran "mvn clean verify" and I got

2016-09-14 06:31:46:332 INFO - shutdown storage: OSystem... [Orient]
2016-09-14 06:31:46:350 INFO OrientDB Server is shutting down... [OServer]
2016-09-14 06:31:46:350 INFO Shutting down listeners: [OServer]
2016-09-14 06:31:46:350 INFO - ONetworkProtocolBinary /0.0.0.0:2425: [OServer]
2016-09-14 06:31:46:351 INFO - ONetworkProtocolHttpDb /0.0.0.0:2481: [OServer]
2016-09-14 06:31:46:351 INFO Shutting down protocols [OServer]
2016-09-14 06:31:46:351 INFO Shutting down plugins: [OServerPluginManager]
2016-09-14 06:31:46:351 INFO - studio [OServerPluginManager]
2016-09-14 06:31:46:351 INFO - jmx [OServerPluginManager]
2016-09-14 06:31:46:352 INFO - script-interpreter [OServerPluginManager]
2016-09-14 06:31:46:353 INFO Shutting down databases: [OServer]
2016-09-14 06:31:46:353 INFO Orient Engine is shutting down... [Orient]
2016-09-14 06:31:46:607 INFO OrientDB Engine shutdown complete [Orient]
2016-09-14 06:31:46:608 INFO OrientDB Server is shutting down... [OServer]
2016-09-14 06:31:46:609 INFO OrientDB Server shutdown complete
[OServer]
2016-09-14 06:31:46:610 INFO Shutting down listeners: [OServer]
Results :

Tests in error:
BinaryProtocolAnyResultTest.scriptReturnValueTest:39 » OStorage Cannot connect...
OLiveQueryShotdownTest.testShutDown:44->bootServer:32 » OStorage Cannot connec...
ORemoteImportTest.before:38 » OStorage Cannot connect to the remote server/dat...
RemoteIndexSupportTest.before:40 » OStorage Cannot connect to the remote serve...
TestNetworkSerializerIndipendency.createBinaryDatabaseConnectCsv:80->createDatabase:72 » OStorage
TestNetworkSerializerIndipendency.createCsvDatabaseConnectBinary:37->createDatabase:72 » OStorage

Tests run: 70, Failures: 0, Errors: 6, Skipped: 1

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] OrientDB .......................................... SUCCESS [ 1.934 s]
[INFO] OrientDB Test Commons ............................. SUCCESS [ 3.110 s]
[INFO] OrientDB Core ..................................... SUCCESS [03:27 min]
[INFO] OrientDB Client ................................... SUCCESS [ 5.181 s]
[INFO] OrientDB Object ................................... SUCCESS [ 14.827 s]
[INFO] OrientDB Tools .................................... SUCCESS [ 6.028 s]
[INFO] OrientDB Server ................................... FAILURE [ 29.546 s]
[INFO] OrientDB GraphDB .................................. SKIPPED
[INFO] OrientDB Tests .................................... SKIPPED

So it is in Server, not in the OSGi module.

[evandor]

When I ran the command with skipped tests, I got a 34,7 MB file orientdb-osgi-core-bundle-2.2.10-SNAPSHOT, which first looked ok, but I am having issues with it: The embedded files are jars (not even bundles); when I wanted to import any class in my own bundle, e.g.

import com.orientechnologies.orient.core.metadata.schema.OType;
import com.orientechnologies.orient.core.record.impl.ODocument;

they cannot be resolved (compile time), as java does not support nested jars... Am I supposed to use the usual orientdb bundles for building and the uber jar for running the application? And where in the build I'd find the usual orientdb bundles? (the jars don't seem to be OSGi'ified).

Thanks ;)

[taburet]

Please provide details about your development/build/runtime environment. A striped down version of the project that won't compile would be great.

I basically was targeting following environments:

1. Full-fledged OSGi dev/build/runtime environment like Eclipse+Tycho, it supports embedded jars and obeys OSGi rules in general.
2. IDEA+Maven for dev/build time, regular non-OSGi deps are used since they should be transitively visible from orientdb-osgi-core-bundle. As for runtime, any OSGi-compatible runtime environment should support embedded jars and allow import resolution from them.

[evandor]

I am using

Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T18:37:52+01:00)
Maven home: /Users/carsten/install/apache-maven-3.2.1
Java version: 1.8.0_74, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"

For development, I use gradle with bndtools (Eclipse). The project which won't compile is on github (not stripped down yet): https://github.com/evandor/skysail/tree/master/skysail.server.db

I have orientdb imports, e.g. in https://github.com/evandor/skysail/blob/master/skysail.server.db/src/io/skysail/server/db/DbService.java which cannot be resolved as the osgi uber bundle relies on nested jars (which is not a standard java/jar setup afaik).

In case you're interested, here's my uber-version (ca 5MB): https://github.com/evandor/skysail-repository/blob/master/release/io.skysail.bundled.orientdb-uber.jar (which works for me, build time and runtime). Required dependencies have to be added to OSGi "by hand", which I think is absolutely ok.

This jar was created with bndtools with just the following instructions:

Bundle-Version: 2.1.3.${tstamp}

-classpath:
jar/orientdb-client-2.1.3.jar,
jar/orientdb-enterprise-2.1.3.jar,
jar/orientdb-graphdb-2.1.3.jar,
jar/orientdb-object-2.1.3.jar,
jar/orientdb-server-2.1.3.jar,
jar/orientdb-tools-2.1.3.jar,
jar/orientdb-core-2.1.3.jar,
jar/blueprints-core-2.6.0.jar,
jar/blueprints-orient-graph-2.4.0.jar

Export-Package:
com.orientechnologies.orient.,
com.orientechnologies.common.,
com.orientechnologies.nio.,
com.tinkerpop.blueprints.

Import-Package:
!com.carrotsearch.hppc.,
!com.tinkerpop.gremlin.,
!javax.mail.*,
*

Include-Resource: META-INF/services=services/orientdb

Bundle-Name: orientdb-uber.jar

Hope that helps, let me know if you need more information and thanks for your efforts!

[taburet]

Hi @evandor! Maybe I missed something, but I don't see any references to orientdb-osgi-core-bundle in Gradle dependencies of the project you provided. Seems like you somehow tweaking the javac class path manually.

Please try following:

1. mvn clean install the ODB from osgi-2.2.x branch into local Maven repo.
2. Add dependency to orientdb-osgi-core-bundle into Gradle file(s).
3. Remove all other ODB for OSGi "magic" from the project.
4. Build it with Gradle.

[evandor]

Hi @taburet,

this time the mvn clean install succeeded without any errors.

But could you please clarify/confirm the following:

• the created jars (like orientdb/core/target/orientdb-core-2.2.10-SNAPSHOT.jar are not bundles any more
• the osgi uber-jar contains nested jars

As I said, in that case I have a problem, as my build tool (bndtools) does not support nested jars. This is only relevant at build time, as I have a bundle that imports ODB classes, which cannot be resolved.

Btw, I got in contact with the bndtools guys in this thread: https://groups.google.com/forum/#!topic/bndtools-users/pDveE9vafBk (maybe it contains some useful information for you).

About the project I was referencing to: I did not use orientdb-osgi-core-bundle there, I created my own uber-bundle (io.skysail.bundled.orientdb-uber.jar, just 5MB, based on 2.1.3) which works for me (build and runtime). I think an uber-bundle like orientdb-osgi-core-bundle (containing all its dependencies) is quite against the "OSGi spirit"...

So, long story short, with the

• provided orientdb uber bundle (2.2.10-SNAPSHOT)
• and bndtools (as OSGi tool chain)

I cannot build any bundle which references any orientdb class (as the code resides in embedded, unexploded jars) (and I don't want to abandon bndtools)

I might be able to use the uber jar at runtime, but I cannot test this as my bundle does not even compile. I hope orientdb will not stop shipping its jars as proper OSGi bundles (as it is still done in the 2.2.10 release).

As long as orientdb ships with proper bundle-jars, I always can create my own uber jar and will be happy with it. But it would be even better, if the official uber-jar could be used by anyone, independent of their choice of tooling. Furthermore I think that the uber-jar does not need to contain all of its dependencies; OSGi developers should be able to provide them in their runtime (with the help of a little documentation of what is needed).

Thanks for reading all of this ;) - I hope things became a bit more clear.

If I can be of any assistance, please let me know!

[taburet]

@evandor Thank you for the feedback. I will think what we can do about the problems you faced.

[grexe]

@taburet checked out the osi branch today and got this error compiling with Oracle JDK 8 (Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) on Linux x64 (Ubuntu 16.04, linux 4.8.0-040800rc8-generic)

Failed tests: 
  HashSaltTest.testSalt:23 expected [true] but found [false]
  OSecurityManagerTest.shouldCheckPlainPasswordAgainstHashWithSalt:46 expected:<[tru]e> but was:<[fals]e>

[taburet]

Merged a fresh 2.2.x into osgi-2.2.x, maybe that will solve the problem with the test.

[grexe]

@taburet there seems to be a problem with the last merge: Maven tries to resolve the wrong POM because of osgi vs. non-osgi difference in artifact names, I guess:

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for com.orientechnologies:orientdb-osgi:[unknown-version]: Could not find artifact com.orientechnologies:orientdb-parent:pom:2.2.10-SNAPSHOT in env-dev (http://repo1.maven.org/maven2) and 'parent.relativePath' points at wrong local POM @ line 27, column 13

[taburet]

Sorry, bumped the version. Pushed.

[grexe]

thanks @taburet for the super quick fix, that did it! Now testing...

[grexe]

now I get this error when running with tests, though, @taburet

2016-09-30 14:03:31:254 INFO  - shutdown storage: OSystem... [Orient]
2016-09-30 14:03:31:380 INFO  OrientDB Server is shutting down... [OServer]
2016-09-30 14:03:31:380 INFO  OrientDB Server is shutting down... [OServer]
2016-09-30 14:03:31:380 INFO  OrientDB Server is shutting down... [OServer]
2016-09-30 14:03:31:380 INFO  Shutting down listeners: [OServer]
2016-09-30 14:03:31:380 INFO  Shutting down listeners: [OServer]
2016-09-30 14:03:31:380 INFO  - ONetworkProtocolBinary /0.0.0.0:2426: [OServer]
2016-09-30 14:03:31:381 INFO  - ONetworkProtocolBinary /0.0.0.0:2425: [OServer]
2016-09-30 14:03:31:381 INFO  Shutting down listeners: [OServer]
2016-09-30 14:03:31:381 INFO  - ONetworkProtocolHttpDb /0.0.0.0:2482: [OServer]
2016-09-30 14:03:31:381 INFO  Shutting down protocols [OServer]
2016-09-30 14:03:31:381 INFO  - ONetworkProtocolHttpDb /0.0.0.0:2499: [OServer]
2016-09-30 14:03:31:381 INFO  - ONetworkProtocolHttpDb /0.0.0.0:2481: [OServer]
2016-09-30 14:03:31:381 INFO  Shutting down protocols [OServer]
2016-09-30 14:03:31:381 INFO  Shutting down plugins: [OServerPluginManager]
2016-09-30 14:03:31:382 INFO  - studio [OServerPluginManager]
2016-09-30 14:03:31:382 INFO  Shutting down protocols [OServer]
2016-09-30 14:03:31:383 INFO  - jmx [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  Shutting down plugins: [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  Shutting down plugins: [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  - jmx [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  - studio [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  - script-interpreter [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  - jmx [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  - script-interpreter [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  Shutting down databases: [OServer]
2016-09-30 14:03:31:383 INFO  Orient Engine is shutting down... [Orient]
2016-09-30 14:03:31:383 INFO  OrientDB Server shutdown complete
 [OServer]
2016-09-30 14:03:31:383 INFO  - script-interpreter [OServerPluginManager]
2016-09-30 14:03:31:383 INFO  Shutting down databases: [OServer]
Results :

Tests in error: 
  BinaryProtocolAnyResultTest.scriptReturnValueTest:39 ? OStorage Cannot connect...
  OLiveQueryShotdownTest.testShutDown:44->bootServer:32 ? OStorage Cannot connec...
  ORemoteImportTest.before:38 ? OStorage Cannot connect to the remote server/dat...
  RemoteIndexSupportTest.before:40 ? OStorage Cannot connect to the remote serve...
com.orientechnologies.orient.server.network.RemoteRidBagTest.testTwoClients(com.orientechnologies.orient.server.network.RemoteRidBagTest)
  Run 1: RemoteRidBagTest.before:63 ? OStorage Cannot connect to the remote server/data...
  Run 2: RemoteRidBagTest.after:85 NullPointer

  TestNetworkSerializerIndipendency.createBinaryDatabaseConnectCsv:80->createDatabase:72 ? OStorage
  TestNetworkSerializerIndipendency.createCsvDatabaseConnectBinary:37->createDatabase:72 ? OStorage

Tests run: 71, Failures: 0, Errors: 7, Skipped: 1

As expected, with skipping tests, the compile went through, so I can test the jar in my OSGi environment (standalone).

[taburet]

That's strange, the server involved in these tests should listen on any network interface (0.0.0.0) and the client is connecting to localhost. Maybe there are some rights restrictions in your system for listening on 0.0.0.0. Also, try nslookup localhost, just in case.

[grexe]

localhost is fine @taburet but I'm running a docker group on the same system, maybe that's causing trouble, however localhost is free, as docker runs on 10.0.0.100, on a virtual network card eth0:1:

➜  orientdb git:(osgi-2.2.x) nslookup localhost
Server:         192.168.0.1
Address:        192.168.0.1#53

Non-authoritative answer:
Name:   localhost
Address: 127.0.0.1

[tglman]

hi @grexe,

That test usually fail if there is a server running somewhere else while you run the tests, i hope it help ;).

Bye

[grexe]

thanks @taburet and @tglman it was indeed the docker daemon interfering, tests run through now.

[taburet]

Sorry guys, after internal discussion we have decided to leave ODB OSGi support in its present state, consider osgi-2.2.x branch unsupported. It's impossible to create a solution that will satisfy everyone at this point in time, mainly due to non-OSGi dependencies we have. Partial solution is not an option, since it will require too much effort on supporting it.

For now, I suggest you to repackage the original ODB distribution for your specific needs. We will return to the subject in the future, maybe OSGi support will receive a more widespread adoption among our dependencies at some time.