Tensorflow.NET with Unity

[SestoAle]

Hi,

I'm not used to import external library in Unity and/or with C#. Is there an easy guide to import and use Tensorflow.NET in Unity?

Thanks!

[Oceania2018]

@SestoAle There is a blog about how to use TensorFlow.NET in Unity written in Chinese, you can use Google translate.

[SestoAle]

I read it but it isn't so detailed on how you can do it.

Following the blog, I compiled TensorFlow.NET with visual studio but I can't find the "bin/Debug/runtimes" folder that the blog uses.

Anyone that has already used this in Unity?

[Oceania2018]

Actually you don’t have to know runtime folder, the blog supposed you do it manually. A simpler approach is just do this: Install-Package SciSharp.TensorFlow.Redist in your project. It will automatically generate the runtimes folder.

[jvo3dc]

I've been able to get things running in Unity up to some level. (Linear) regression, a convolutional classification network and a GAN are all working fine. I've added a post on the Unity forum with some steps and results: Tensorflow dot Net experiences

I'm having serious difficulties getting a DCGAN to work though. That all lies in the deconvolution, what I've tried:

• tf.nn.conv2d_transpose (doesn't exist in Tensorflow.Net)
  (Following steps try to upscale the image and then convolving for the same effect.)
• Upscaling using tf.image.resize_images (NotImplementedException)
• Upscaling using tf.image.resize_bilinear (NullReferenceException during construction)
• Upscaling by splitting and concatting (NullReferenceException during training)

The splitting/concatting code is as follow:

private static Tensor Replicate(Tensor input, int axis, int count, int extraStart, int extraEnd)
{
    List<Tensor> slices = new List<Tensor>();
    Tensor[] split = tf.split(input, input.shape[axis], tf.constant(axis), "split");
    for (int index = 0; index < split.Length; index++)
    {
        int current = count;
        if (index == 0) current += extraStart;
        if (index == split.Length - 1) current += extraEnd;
        for (int i = 0; i < current; i++) slices.Add(split[index]);
    }
    return tf.concat(slices, axis);
}

So I call that as Replicate(Replicate(input, 1, 2, 1, 1), 2, 2, 1, 1) to try and do 2x2 nearest neighbour upscale with a padding of 1 around the image before applying a 3x3 convolution kernel.

(Part of) the stack trace is this:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object
at Tensorflow.Tensor.op_Explicit (Tensorflow.Tensor tensor) [0x00018] in <069e61b4965e459399a727a20f9e4a09>:0
at Tensorflow.Gradients.array_grad._ConcatGradHelper (Tensorflow.Operation op, Tensorflow.Tensor grad, System.Int32 start_value_index, System.Int32 end_value_index, System.Int32 dim_index) [0x00150] in <069e61b4965e459399a727a20f9e4a09>:0
at Tensorflow.Gradients.array_grad._ConcatGradV2 (Tensorflow.Operation op, Tensorflow.Tensor[] grads) [0x00004] in <069e61b4965e459399a727a20f9e4a09>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in :0

It seems to go wrong during backpropagation. I've seen another topic where the operation was null, which might also be the case here.

I'm running out of options here. Is there any example of how to build an image deconvolution step using Tensorflow.Net?

[jvo3dc]

No suggestions to perform a transposed convolution using TensorFlow.NET?

I've just tried applying a convolution with stride 1 and then reorganizing the tensor, but then again I run into issues, neither concat nor stack seem to work.

Idea is for example to convolve a ?x7x7xN tensor into a ?x14x14x16 tensor. So with a stride of 1 I do a convolution into a ?x7x7x64 tensor. Then I need to split it in two on axis 3 and interlace the result on axis 1. And then the same from axis 3 to axis 2 to finally get a ?x14x14x16 tensor. Should be possible with either split/concat or unstack/stack, but neither seem to work.

[akaimody123]

Hi, I have installed TensorFlow.NET and SciSharp.TensorFlow.Redist, and put their dll in my Unity Assets Plugins folder. But it gaves me following error when I try to run a linear regression model. I have checked other related issues about dllNotFound, but it is still not working.

DllNotFoundException: tensorflow
Tensorflow.tensorflow..ctor () (at <069e61b4965e459399a727a20f9e4a09>:0)
System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) (at <437ba245d8404784b9fbab9b439ac908>:0)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) (at <437ba245d8404784b9fbab9b439ac908>:0)
System.RuntimeType.CreateInstanceMono (System.Boolean nonPublic) (at <437ba245d8404784b9fbab9b439ac908>:0)
System.RuntimeType.CreateInstanceSlow (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) (at <437ba245d8404784b9fbab9b439ac908>:0)
System.RuntimeType.CreateInstanceDefaultCtor (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) (at <437ba245d8404784b9fbab9b439ac908>:0)
System.Activator.CreateInstance[T] () (at <437ba245d8404784b9fbab9b439ac908>:0)
Tensorflow.Binding.New[T] () (at <069e61b4965e459399a727a20f9e4a09>:0)
Tensorflow.Binding..cctor () (at <069e61b4965e459399a727a20f9e4a09>:0)
Rethrow as TypeInitializationException: The type initializer for 'Tensorflow.Binding' threw an exception.