Tfjs pose detection not working with newer Android devices (12, 13, 14)

[expolli]

Please see repro project: https://github.com/expolli/pose-test/tree/main

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 11
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Please see repro project for details.
• TensorFlow.js installed from (npm or script link): npm
• TensorFlow.js version (use command below): 4.19.0
• Browser version: Expo 51
• Tensorflow.js Converter Version: -

Describe the current behavior

Repro app working with older devices but not with new ones

Please see repro project for details.

Describe the expected behavior

Pose detection working on all devices.

Standalone code to reproduce the issue

https://github.com/expolli/pose-test/tree/main

Other info / logs Include any logs or source code that would be helpful to

Please see repro project.

[rachelwong44]

Tensorcamera is not working with expo-camera thanks to the SDK51 update

[expolli]

Thank you for the update @rachelwong44 . Do you know of a workaround to get the barebones app working?

App was not working with Expo 50 either with newer Androids, btw.

Not sure if tfjs is using the tensorcamera internally, but app itself is not using camera at all. It is just reading the one image from the assets and processing that.

[gaikwadrahul8]

Hi, @expolli

I apologize for the delayed response and thank you for bringing this issue to our attention, could you please give it try with these Expo versions 47.0.14 and 49.0.23 and see is it working as expected or not ? meanwhile I'll discuss this issue with our internal meeting and will update you soon.

Thank you for your cooperation and patience.

[expolli]

Hei @gaikwadrahul8 ,

Installed Expo 49.0.23 and unfortunately the result was the same. Older Android working but newer not.

Old Android: LOG {"poses": [{"keypoints": [Array], "keypoints3D": [Array], "score": 0.9999972581863403}]}
New Android: LOG {"poses": []}

These were the versions used (dropped the camera as it was not needed):

    "@tensorflow-models/pose-detection": "^2.1.3",
    "@tensorflow/tfjs": "^4.19.0",
    "@tensorflow/tfjs-react-native": "^1.0.0",
    "async-storage": "^0.1.0",
    "expo": "^49.0.23",
    "expo-asset": "~8.10.1",
    "expo-gl": "~13.0.1",
    "expo-status-bar": "~1.6.0",
    "react": "18.2.0",
    "react-native": "0.72.10",
    "react-native-fs": "^2.20.0"

Had some issues using expo-asset so used fetch instead to get the image from local web server:
const response = await fetch("https://my-webserver/image.jpg")

Please let me know if there is something else I can do to help in your investigations.

Thank you!

[Jersyfi]

Dear @gaikwadrahul8 , an update to the newest Expo SDK Version should be discussed instead on fucussing on old versions. With some changes it is possible to run the SDK 50 and for SDK 51 only the dependencies need to be updated first. Otherwise it isn't so complicated to update to the newest expo-camera release.
My intentions in sending pull-requests are pretty low because i know that no one will answer and release it. Is there a solution we can get progress on this?

At the moment i try to get it running with the following package.json and a custom modified @tensorflow/tfjs-react-native:

"dependencies": {
        "@hookform/resolvers": "^3.3.4",
        "@mediapipe/pose": "^0.5.1675469404",
        "@react-native-async-storage/async-storage": "1.23.1",
        "@react-native-community/datetimepicker": "8.0.1",
        "@react-native-firebase/app": "^20.0.0",
        "@react-native-firebase/auth": "^20.0.0",
        "@react-native-firebase/firestore": "^20.0.0",
        "@react-native-firebase/functions": "^20.0.0",
        "@react-native-firebase/storage": "^20.0.0",
        "@react-navigation/bottom-tabs": "^6.5.20",
        "@react-navigation/native": "^6.1.17",
        "@react-navigation/native-stack": "^6.9.26",
        "@tensorflow-models/pose-detection": "^2.1.3",
        "@tensorflow/tfjs": "^4.19.0",
        "@tensorflow/tfjs-backend-webgl": "^4.19.0",
        "@tensorflow/tfjs-backend-webgpu": "^4.19.0",
        "@tensorflow/tfjs-react-native": "^1.0.0",
        "expo": "^51.0.8",
        "expo-av": "~14.0.5",
        "expo-build-properties": "~0.12.1",
        "expo-camera": "~15.0.9",
        "expo-dev-client": "~4.0.14",
        "expo-device": "~6.0.2",
        "expo-file-system": "~17.0.1",
        "expo-gl": "~14.0.2",
        "expo-image": "~1.12.9",
        "expo-image-manipulator": "~12.0.5",
        "expo-image-picker": "~15.0.5",
        "expo-keep-awake": "~13.0.2",
        "expo-linear-gradient": "~13.0.2",
        "expo-linking": "~6.3.1",
        "expo-screen-orientation": "~7.0.5",
        "expo-splash-screen": "~0.27.4",
        "expo-status-bar": "~1.12.1",
        "expo-system-ui": "~3.0.4",
        "expo-web-browser": "~13.0.3",
        "lucide-react-native": "^0.378.0",
        "react": "18.2.0",
        "react-hook-form": "^7.51.1",
        "react-native": "0.74.1",
        "react-native-fs": "^2.20.0",
        "react-native-gesture-handler": "~2.16.1",
        "react-native-reanimated": "~3.10.1",
        "react-native-safe-area-context": "4.10.1",
        "react-native-screens": "3.31.1",
        "react-native-svg": "15.2.0",
        "react-native-ui-lib": "^7.22.0",
        "seedrandom": "^3.0.5"
    },
    "devDependencies": {
        "@babel/core": "^7.20.0",
        "@babel/plugin-proposal-nullish-coalescing-operator": "^7.18.6",
        "@babel/plugin-proposal-optional-chaining": "^7.21.0",
        "@babel/plugin-transform-arrow-functions": "^7.24.1",
        "@babel/plugin-transform-optional-chaining": "^7.24.1",
        "@babel/plugin-transform-shorthand-properties": "^7.24.1",
        "@babel/plugin-transform-template-literals": "^7.24.1",
        "@babel/preset-env": "^7.24.3",
        "@react-native/babel-preset": "*",
        "@types/react": "~18.2.79",
        "@typescript-eslint/eslint-plugin": "^6.12.0",
        "@typescript-eslint/parser": "^6.12.0",
        "eslint": "^8.45.0",
        "eslint-config-prettier": "^9.0.0",
        "eslint-config-universe": "^12.0.0",
        "eslint-plugin-react-native": "^4.1.0",
        "prettier": "3.0.0",
        "typescript": "~5.3.3"
    }

[expolli]

At the moment i try to get it running with the following package.json and a custom modified @tensorflow/tfjs-react-native:

@Jersyfi - sorry to bother but are you facing similar issues where tfjs pose detection is not working with latest devices?

[expolli]

Hello @gaikwadrahul8, Installed the official react-native expo app (expo SDK 46) from tfjs-examples: https://github.com/tensorflow/tfjs-examples/blob/master/react-native/pose-detection.

• Movenet lib detects poses with old and new devices.
• BlazePose fails to detect poses with newer devices.

Wondering if there is an official tfjs app for react-native (expo) utilizing BlazePose that works with newer Androids?

This is the expo go version for SDK 46: https://d1ahtucjixef4r.cloudfront.net/Exponent-2.25.1.apk matching the tfjs-example.

Note: BlazePose-MediaPipe does not support React Native but BlazePose-TensorFlow.js should. (Deleted my comment about this.)

[expolli]

Did some more testing with another expo app (SDK 51).

I was able to detect poses with newer Android device by adding the following line:
await tf.setBackend('cpu'). This does bring considerable time overhead but is working nevertheless on all devices.

Tried the same with the mimimal repro app provided and ended up with dozens of lines of warnings:
WARN The kernel 'Unpack' for backend 'webgl' is already registered WARN The kernel 'UnsortedSegmentSum' for backend 'webgl' is already registered WARN The kernel 'ZerosLike' for backend 'webgl' is already registered
and an error:
{"e": [Error: pixels passed to tf.browser.fromPixels() must be either an HTMLVideoElement, HTMLImageElement, HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData in webworker or {data: Uint32Array, width: number, height: number}, but was Tensor]}

At the time of writing not able to say what the actual difference is but must have something to do with libraries included and maybe the ordering of imports. Couldn't tell.

IMO this suggests the issue to lay in the tf backend processing side and how the gpu processing is done. With an older Android device the gpu is somehow supported but with newer ones there is incompatibility issue with gpu being used. Newer device just bails out without any information and returns an empty pose array.

[Jersyfi]

At the moment i try to get it running with the following package.json and a custom modified @tensorflow/tfjs-react-native:

@Jersyfi - sorry to bother but are you facing similar issues where tfjs pose detection is not working with latest devices?

Hello @expolli i am facing this issue with older devices. SDK 50 was running on all newer ones (iOS and Android). With Expo SDK 51 the package @tensorflow/tfjs-react-native can't run because the expo camera API was updated.

I will refer to an existing issue that discusses the issue in more detail: #8292

[Jersyfi]

Hello @gaikwadrahul8, Installed the official react-native expo app (expo SDK 46) from tfjs-examples: https://github.com/tensorflow/tfjs-examples/blob/master/react-native/pose-detection.

• Movenet lib detects poses with old and new devices.
• BlazePose fails to detect poses with newer devices.

Wondering if there is an official tfjs app for react-native (expo) utilizing BlazePose that works with newer Androids?

This is the expo go version for SDK 46: https://d1ahtucjixef4r.cloudfront.net/Exponent-2.25.1.apk matching the tfjs-example.

Note: BlazePose-MediaPipe does not support React Native but BlazePose-TensorFlow.js should. (Deleted my comment about this.)

You can use the package versions from my comment #8292 (comment) for Expo SDK 50.

[expolli]

You can use the package versions from my comment #8292 (comment) for Expo SDK 50.

Thank you @Jersyfi for the info about the other issue. I can see that you have done excellent work digging these expo issues. Unfortunately with the devices I have available it did not help on getting poses with newer Android. Just to clarify the intent is just to get poses from an image and not cam.

The poses array is empty even with the versions from your package.json, unfortunately. Setting the backend to cpu fixes the issue and poses are found from the image albeit taking very long time.

cpu backend, new android: 1min29s
cpu backend, old android: 5min16s
(default gpu), old android: 24s

[sepretf]

I’ve been having issues with integrating TensorFlow's pose detection models on iOS/Android for a couple of years too...
So, I shifted to a different approach a few months ago—using a Node.js API with tfjs that I call within an iframe to process the images. You have access to the exact same data than with detector.estimatePoses(...). Try it for free on PoseTracker.com !