Interaction between Tensorflow Java and JavaCPP Pointer deallocation

[skirdey]

I am trying to understand why when I use DJL + Tensorflow engine, there is a good amount of time spent in GC, while both DJL and Tensorflow Java seem to use pointerscope and do not relay on GC for object cleanup.

For example, I see JavaCPP Pointer.deallocator gets invoked https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Pointer.java#L666-L667

which is a heavy synchronized call.

Any help is appreciated.

[saudet]

If you're confident you don't need GC, we can set the "org.bytedeco.javacpp.nopointergc" system property to "true" to reduce overhead.

[skirdey]

I am not sure wether I need GC or not. Looking at the EagerSession class seems like it is managing memory through the pointer scope. The NDArrays I allocate through DJL/TF also use the same.

[skirdey]

I will try the system property you have mentioned and run my benchmarks, thanks for such a quick response!

[saudet]

You'll need to keep in mind though that neither TF nor DJL are particularly designed to reduce GC. JavaCPP is only a small part of the overall design, and there is a lot garbage that gets generated elsewhere.

[skirdey]

Still investigating how GC optimizations work in the DJL + Tensorflow Environment.
There seems to be also a blocking call when using several eager sessions in parallel for multi-threading, and when JavaCPP needs to run GC, it blocks all of the threads with sessions.

[saudet]

Like I said, make sure to set the "org.bytedeco.javacpp.nopointergc" system property to "true" to prevent JavaCPP from calling System.gc(). It should also avoid blocking calls, but let me know if you still see anything of concern there.

[skirdey]

I've tried the org.bytedeco.javacpp.nopointergc and very quickly ran out of memory :-)

[skirdey]

If I keep it "false", then in JVM with G1GC, I spend 25% of the time in GC on a high throughput inference use-case, with occasional blocking.

[skirdey]

stacktrace related to allocation

Exception in thread "ForkJoinPool-1-worker-7" java.lang.OutOfMemoryError: Cannot allocate new PointerPointer(1000): totalBytes = 3640M, physicalBytes = 1226M
at org.bytedeco.javacpp.PointerPointer.(PointerPointer.java:149)
at org.tensorflow.EagerOperationBuilder.execute(EagerOperationBuilder.java:270)
at org.tensorflow.EagerOperationBuilder.build(EagerOperationBuilder.java:66)
at org.tensorflow.EagerOperationBuilder.build(EagerOperationBuilder.java:55)
at ai.djl.tensorflow.engine.TfNDArray.(TfNDArray.java:81)
at ai.djl.tensorflow.engine.TfNDManager.create(TfNDManager.java:154)

[skirdey]

So I've tried it, the blocking calls are removed, the thread with JavaCPP Deallocator is also disappears and no-more blocking GCs. Only issue is I quickly run out of memory :-)

[skirdey]

There is a team discussion on DJL around Tensorflow Java and JavaCPP performance, if you are interested - deepjavalibrary/djl#625

[skirdey]

The GC behaviour I see when

org.bytedeco.javacpp.nopointergc=false
Xmx2G
Xms2G

maxPhysicalBytes=1G

when running multi-threaded inference locally on my laptop. It uses Tensorflow Eager Session.

[saudet]

So I've tried it, the blocking calls are removed, the thread with JavaCPP Deallocator is also disappears and no-more blocking GCs. Only issue is I quickly run out of memory :-)

Obviously, when not using the GC, you'll need to make sure to deallocate native memory some other way!

[skirdey]

True that, I've just tested the theory that PointerScope takes care of my specific use case for inference. But you are right, there is quite a lot of garbage besides that.

…

On Fri, Feb 19, 2021 at 4:33 PM Samuel Audet ***@***.***> wrote: So I've tried it, the blocking calls are removed, the thread with JavaCPP Deallocator is also disappears and no-more blocking GCs. Only issue is I quickly run out of memory :-) Obviously, when not using the GC, you'll need to make sure to deallocate native memory some other way! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHIJQDEMC63YUQQAQFNHWTS737NBANCNFSM4XD2EC2Q> .

[karllessard]

Eager execution does allocate a lot of native resources, as each operation and each of their outputs will need to be freed.

I don't know much about details the GC feature of JavaCPP we are using now but I can tell that in 1.14, the way it was working is that the GC listener was running in a separate thread and was trying to free resources referenced by deleted objects, but just as a "best effort" since GC is not entirely reliable when it comes to native memory. @saudet, this thread-based implementation was just listening to a phantom reference queue and was therefore non-blocking, is JavaCPP doing something similar?

Ultimately, resources should be cleaned up by closing the eager session enclosing some piece of code and that part is still true:

try (EagerSession s = EagerSession.create()) {
    Ops tf = Ops.create(s);
    // your eager computations
}

This ensure to release all resources independently from the GC. So while it is not enforced by the API, it is recommended to scope your eager operations into multiple eager session instead of only relying on the default one (i.e. the one used when simply invoking Ops.create() without any parameter).