tfjs.node.decodeImage run error in workthread

[mccoysc]

System information
any system，any node version 18.×

Describe the current behavior
in nodejs workThread:

ctx={tfjsnode:function(){return require("@tensorflow/tfjs-node")}};

code="imgData=fs.readFileSync("img.jpg");imgData=Uint8Array.from(imgData);tfjsnode().node.decodeImage(imgData)";

vm.runInNewContext(ctx,code);

the code would throw an error: decodeImage can only accept typedArray and so on.

i traced it into code and found:
isTypedArray(values) would always return false.
but really,values is a Uint8Array.

i don't know why.

Describe the expected behavior

Standalone code to reproduce the issue

Other info / logs

[mccoysc]

[mccoysc]

i am sure that the imgData is "Uint8Array".
and the code can be run good in non-workThread and non-contextfied node env.

[rthadur]

You may need to useimgData=Uint8Array(imgData)instead of imgData=Uint8Array.from(imgData) , can you please check here . Thank you!

[mccoysc]

You may need to useimgData=Uint8Array(imgData)instead of imgData=Uint8Array.from(imgData) , can you please check here . Thank you!

no，i don't only use from,i also use new Uint8Array

[mccoysc]

maybe the offical should re-write the function code for "isTypedArray",the orignal code use "instnaceof" to determine if the content is an instance of "typedarray".
but "instanceof" only checks if the object's constructor function is equal.

in fact,"tensorflow" has no need to be sure that the constructor functions are equal.
@rthadur

[mattsoulanille]

I can reproduce this issue. Here's the code I'm using:

const tfjs = require('@tensorflow/tfjs');
const tfjsnode = require('@tensorflow/tfjs-node');
const fs = require('fs');
const vm = require('node:vm');


console.log('Running in main context');
let imgData = fs.readFileSync('img.jpg');
imgData = Uint8Array.from(imgData);
console.log(tfjsnode.node.decodeImage(imgData));

const ctx = {
  tfjsnode: function() {
    return require('@tensorflow/tfjs-node');
  },
  fs: function() {
    return require('fs');
  },
  log: console.log,
};

code = `
log('Running in a new context');
let imgData = fs().readFileSync('img.jpg');
imgData = Uint8Array.from(imgData);
const decoded = tfjsnode().node.decodeImage(imgData);
log(decoded);
`;

vm.createContext(ctx);
vm.runInContext(code, ctx);

I'm not yet sure why this isn't working, but if you just need to load an image and run decodeImage on it, passing it directly instead of converting it to a Uint8Array seems to work in node 18:

const vm = require('node:vm');

const ctx = {
  tfjsnode: function() {
    return require('@tensorflow/tfjs-node');
  },
  fs: function() {
    return require('fs');
  },
  log: console.log,
};

code = `
log('Running in a new context');
let imgData = fs().readFileSync('img.jpg');
// imgData = Uint8Array.from(imgData); // <-- This seems to break it.
const decoded = tfjsnode().node.decodeImage(imgData);
log(decoded);
`;

vm.createContext(ctx);
vm.runInContext(code, ctx);

[mattsoulanille]

maybe the offical should re-write the function code for "isTypedArray",the orignal code use "instnaceof" to determine if the content is an instance of "typedarray". but "instanceof" only checks if the object's constructor function is equal.

in fact,"tensorflow" has no need to be sure that the constructor functions are equal. @rthadur

You're right, this seems to be the source of the issue. Thanks for the pointer!

Here's some test code that demonstrates it:

const vm = require('node:vm');

const ctx = {
  tfjsnode: function() {
    return require('@tensorflow/tfjs-node');
  },
  fs: function() {
    return require('fs');
  },
  log: console.log,
};

code = `
log('Running in a new context');
let imgData = fs().readFileSync('img.jpg');
imgData = Uint8Array.from(imgData);
log('imgData instanceof Uint8Array:', imgData instanceof Uint8Array);
const tfn = tfjsnode();
log('tfn.util.isTypedArray(imgData):', tfn.util.isTypedArray(imgData));
`;

vm.runInNewContext(code, ctx);

This prints the following:

imgData instanceof Uint8Array: true
tfn.util.isTypedArray(imgData): false

This is unexpected because isTypedArray returns true if its argument is an instance of Uint8Array. Maybe there's something I don't understand about vm.runInNewContext that's making it check the Uint8Array of the image from the new context against the Uint8Array from the main context, and that's why it's failing?

[mattsoulanille]

My guess is that this code:

const ctx = {
  tfjsnode: function() {
    return require('@tensorflow/tfjs-node');
  },
}

... is causing tfjs-node to be loaded in the main context while it should actually be loaded in the new context. That's why isTypedArray is returning false (since it's prototype is from the main context but it's checking an object from the new context). If we use the Uint8Array constructor from the main context as well, then it works:

const fs = require('fs');
const vm = require('node:vm');

const ctx = {
  tfjsnode: function() {
    return require('@tensorflow/tfjs-node');
  },
  fs: function() {
    return require('fs');
  },
  log: console.log,
  makeUint8Array: function(data) {
    return new Uint8Array(data);
  },
};

code = `
log('Running in a new context');
let imgData = fs().readFileSync('img.jpg');
imgData = makeUint8Array(imgData);
log('imgData instanceof Uint8Array:', imgData instanceof Uint8Array);
const tfn = tfjsnode();
log('tfn.util.isTypedArray(imgData):', tfn.util.isTypedArray(imgData));
const decoded = tfn.node.decodeImage(imgData);
log(decoded);
`;

vm.runInNewContext(code, ctx);

Is there a way to load tfjs-node from the new context instead of the main context? I think that would solve this issue better than the above approach. Otherwise, I don't have a good solution to this. TFJS uses TypedArrays extensively to represent Tensor values, and we need to check the type of a typed array when creating a tensor.

Maybe your problem can be solved with WebWorkers in node and Comlink instead of vm.runInNewContext.