how to add jar://model using criterion #1333

JerryYanWan · 2021-10-31T10:55:02Z

JerryYanWan
Oct 31, 2021

hello, I want to use DJL to load a trained pytorch model and do the inference.

Here is the problem:

The program ran successfully in the IntelliJ tool, however it failed when I use command line tool.
It throws NullPointerError when I ran it in the command line.

Hope @frankfliu or other expert could help me to figure this problem?

Here is what i did:

packed my model into an archive format: "test_model.tgz" (test_model/model_tc_1.pt)
insert my model into resources directory
using the following codes as illustrated in this post: https://github.com/deepjavalibrary/djl/discussions/408?sort=top

 Criteria<Float[], Float[]> criteria = Criteria.builder()
                .setTypes(Float[].class, Float[].class)
                .optTranslator(new FloatToFloatTranslator())
                .optModelUrls("jar:///test_model.tgz")
                .optModelName("model_tc_1")
                .build();
        ZooModel<Float[], Float[]> model = ModelZoo.loadModel(criteria);

package the project using : mvn clean package -DskipTests
test the model by running the command:

java -cp target/model-1.0-SNAPSHOT.jar com.xx.xx.xx.Test

Problem guess

I think the path found by IntelliJ might be different from maven.

Here lists the differences:
the uriPath decoded by using this code:
uriPath = uri.getSchemeSpecificPart();
refer to this:
https://github.com/deepjavalibrary/djl/blob/v0.13.0/api/src/main/java/ai/djl/repository/RepositoryFactoryImpl.java#L119

commandline version:
file:/Users/xx/xx/project/target/model-1.0-SNAPSHOT.jar!/test_model.tgz

IntelliJ version:
/Users/xx/xx/project/target/classes/test_model.tgz

Answered by frankfliu

Nov 1, 2021

@JerryYanWan You need to use offline native library as dependency: https://docs.djl.ai/master/engines/pytorch/pytorch-engine/index.html#linux-cpu

If you don't mind the package size, you can add multiple platform specific dependencies.

You need to use 1.9.1 if you are 0.14.0-SNAPSHOT version.

View full answer

frankfliu · 2021-10-31T16:01:47Z

frankfliu
Oct 31, 2021

@JerryYanWan

I just tried it, it should just work. Can you check if the test_model.tgz is actually in the .jar file:

jar tvf target/model-1.0-SNAPSHOT.jar

Can you provide full stacktrace?

1 reply

JerryYanWan Nov 1, 2021
Author

I have uploaded my test files into this repo: https://github.com/JerryYanWan/djl-test
which includes a Test2.java, MyLoadedModel.java and a tar.gz model file.

When I run the file in IntelliJ, it gives the correct result:

However, when I package the project by using maven command, and type the following:
java -cp target/my-model-1.0-SNAPSHOT-jar-with-dependencies.jar com.xx.examples.Test2

It throws the error as:

I have double checked my Jar file and it did include the test_model.tgz file:

Could you help me to figure it out?

frankfliu · 2021-11-01T04:37:32Z

frankfliu
Nov 1, 2021

@JerryYanWan which djl version are you using? the line number doesn't match the latest code, see: https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/repository/JarRepository.java#L104

Can you share your project (pom.xml or build.gradle)?

1 reply

JerryYanWan Nov 1, 2021
Author

i use 0.12.0 as follows:

ai.djl api 0.12.0

I have uploaded my pom.xml into this: https://github.com/JerryYanWan/djl-test

frankfliu · 2021-11-01T06:19:02Z

frankfliu
Nov 1, 2021

This is a bug, I created a PR to fix this issue: #1336

2 replies

JerryYanWan Nov 1, 2021
Author

oh, i see. When the PR is merged, I will use the latest version. Billion Thanks!!

JerryYanWan Nov 1, 2021
Author

@frankfliu Thanks for the bug fix PR. I have downloaded your PR branch, compiled it in my own mac and gradle install it into my local .m2 repository. The program run successfully both using IntelliJ and command line tool.

Then it comes to the second problem:
I uploaded the assembly Jar onto my company's inner server which blocks the outer website. The program failed again. Now the error is as follows:

ai.djl.engine.EngineException: Failed to load PyTorch native library
	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:77)
	at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:40)
	at ai.djl.engine.Engine.getEngine(Engine.java:168)
	at ai.djl.Model.newInstance(Model.java:98)
	at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:187)
	at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:149)
	at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:166)
	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:123)
        .. 32 more
Caused by: java.lang.IllegalStateException: Failed to download PyTorch native library
	at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:283)
	at ai.djl.pytorch.jni.LibUtils.getLibName(LibUtils.java:84)
	at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:72)
	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:50)
	... 32 more
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
	at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at java.net.URL.openStream(URL.java:1045)
	at ai.djl.pytorch.jni.LibUtils.downloadPyTorch(LibUtils.java:374)
	at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:281)
	... 35 more

Then I check the source code and locate the following position in LibUtils.java, I think there is no local pytorch engine so the program tiggers the remote link downloading function.

 private static String downloadPyTorch(Platform platform, AtomicBoolean fallback) {
        version = platform.getVersion();
        String flavor = platform.getFlavor();
        String classifier = platform.getClassifier();
        String os = platform.getOsPrefix();
        if (Boolean.getBoolean("PYTORCH_PRECXX11")
                || Boolean.parseBoolean(System.getenv("PYTORCH_PRECXX11"))) {
            flavor += "-precxx11";
        }

        String libName = System.mapLibraryName(NATIVE_LIB_NAME);
        Path cacheDir = Utils.getEngineCacheDir("pytorch");
        Path dir = cacheDir.resolve(version + '-' + flavor + '-' + classifier);
        logger.debug("Using cache dir: {}", dir);
        Path path = dir.resolve(libName);
        if (Files.exists(path)) {
            return dir.toAbsolutePath().toString();
        }

        Matcher matcher = VERSION_PATTERN.matcher(version);
        if (!matcher.matches()) {
            throw new IllegalArgumentException("Unexpected version: " + version);
        }
        String link = "https://publish.djl.ai/pytorch-" + matcher.group(1);
        Path tmp = null;
        try (InputStream is = new URL(link + "/files.txt").openStream()) {
            // if files not found
            Files.createDirectories(cacheDir);

            List<String> lines = Utils.readLines(is);
            if (!lines.contains(flavor + '/' + os + "/native/lib/" + libName + ".gz")) {
                if (flavor.startsWith("cu")) {
                    logger.warn("No matching cuda flavor for {} found: {}.", os, flavor);
                    // fallback to CPU
                    flavor = "cpu";
                    fallback.set(true);

Thus, is there anything that I can do to included the PyTorch native library in my Jar with out extra download?

frankfliu · 2021-11-01T16:02:42Z

frankfliu
Nov 1, 2021

@JerryYanWan You need to use offline native library as dependency: https://docs.djl.ai/master/engines/pytorch/pytorch-engine/index.html#linux-cpu

If you don't mind the package size, you can add multiple platform specific dependencies.

You need to use 1.9.1 if you are 0.14.0-SNAPSHOT version.

1 reply

JerryYanWan Nov 3, 2021
Author

The program runs successfully. Thank you very much!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to add jar://model using criterion #1333

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

how to add jar://model using criterion #1333

JerryYanWan Oct 31, 2021

Here is the problem:

Here is what i did:

Problem guess

Replies: 4 comments · 5 replies

frankfliu Oct 31, 2021

JerryYanWan Nov 1, 2021 Author

frankfliu Nov 1, 2021

JerryYanWan Nov 1, 2021 Author

frankfliu Nov 1, 2021

JerryYanWan Nov 1, 2021 Author

JerryYanWan Nov 1, 2021 Author

frankfliu Nov 1, 2021

JerryYanWan Nov 3, 2021 Author

JerryYanWan
Oct 31, 2021

Replies: 4 comments 5 replies

frankfliu
Oct 31, 2021

JerryYanWan Nov 1, 2021
Author

frankfliu
Nov 1, 2021

JerryYanWan Nov 1, 2021
Author

frankfliu
Nov 1, 2021

JerryYanWan Nov 1, 2021
Author

JerryYanWan Nov 1, 2021
Author

frankfliu
Nov 1, 2021

JerryYanWan Nov 3, 2021
Author