Movinet fails in GPU mode #1122

bugsyb · 2024-04-07T19:40:43Z

Which version of recognize are you using?

6.11

Enabled Modes

Object recognition, Face recognition, Video recognition, Music recognition

TensorFlow mode

GPU mode

Downstream App

Memories App

Which Nextcloud version do you have installed?

28.0.4.1

Which Operating system do you have installed?

Ubuntu 20.0.4.4

Which database are you running Nextcloud on?

Postgres

Which Docker container are you using to run Nextcloud? (if applicable)

28.0.4.1

How much RAM does your server have?

32

What processor Architecture does your CPU have?

x86_64

Describe the Bug

Seems like after upgrade to NC 28.0.4.1 & Recognize 6.1.1 it started to report below.
It seems like it launches process, based on looking at nvidia-smi and it stays there though doesn't seem like it puts a load on GPU.

Classifier process output: Error: Session fail to run with error: 2 root error(s) found.
  (0) NOT_FOUND: could not find registered platform with id: 0x7fd379c7fae4
\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]
\t [[StatefulPartitionedCall/_1555]]
  (1) NOT_FOUND: could not find registered platform with id: 0x7fd379c7fae4
\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]
0 successful operations.
0 derived errors ignored.
    at NodeJSKernelBackend.runSavedModel (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/nodejs_kernel_backend.js:461:43)
    at TFSavedModel.predict (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/saved_model.js:341:43)
    at MovinetModel.predict (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:46:21)
    at /var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:95:24
    at /var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4559:22
    at Engine.scopedRun (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4569:23)
    at Engine.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4558:21)
    at Object.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:8291:19)
    at MovinetModel.inference (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:92:21)
    at runMicrotasks (<anonymous>)

At the same time it seems to have all what's needed (btw. it didn't raise this error prior upgrades):

./bin/node src/test_gputensorflow.js 
2024-04-07 21:37:06.584377: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-07 21:37:06.593729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:06.636405: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:06.636674: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.109813: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110064: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110196: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-04-07 21:37:07.110354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3393 MB memory:  -> device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0

And models seem to be there too:

du -shx ./models/*
50M	./models/efficientnet_lite4
794M	./models/efficientnetv2
22M	./models/landmarks_africa
41M	./models/landmarks_asia
41M	./models/landmarks_europe
41M	./models/landmarks_north_america
31M	./models/landmarks_oceania
41M	./models/landmarks_south_america
47M	./models/movinet-a3
31M	./models/musicnn

Unsure where to look for an issue.
It is after ffmpeg finishes its job.

Thanks!

Expected Behavior

Would just proceed to classify.

To Reproduce

Unsure.

Debug log

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2024-04-07T19:41:02Z

Hello 👋

Thank you for taking the time to open this issue with recognize. I know it's frustrating when software
causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at
and if possible solved.
I try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it.
Until then, please be patient.
Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation
to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can
collaborate to make this software better. For everyone.
Thus, if you can, you could also look at other issues to see whether you can help other people with your knowledge
and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and
try to fix the odd bug yourself. Everyone will be thankful for extra helping hands!
One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum,
to twitter or somewhere else. But this is a technical issue tracker, so please make sure to
focus on the tech and keep your opinions to yourself. (Also see our Code of Conduct. Really.)

I look forward to working with you on this issue
Cheers 💙

bugsyb · 2024-04-08T10:54:51Z

Could be related to:
#1060

In my case it is CUDA 11

#nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Thu_Feb_10_18:23:41_PST_2022
Cuda compilation tools, release 11.6, V11.6.112
Build cuda_11.6.r11.6/compiler.30978841_0

bugsyb · 2024-12-08T09:41:25Z

Upgraded to latest 30.0.3.2 and latest Recognize and same issue still.

Movinet fails and doesn't back off to CPU - leaves files with no tags
Other object and faces recognition works fine.

Could this be please looked at and at least CPU back-off provided?

Could it be that there is a HW limitation due to loading model into memory? Given objects, landmarks and faces recognition works fine, not sure why this one would fail.

Details about the system at the bottom, in short:

Software - container side

latest Cuda 11 tensorflow release from nVidia (nvcr.io/nvidia/tensorflow:24.10-tf2-py3)
Nc 30.0.3.2
Recognize 8.1.1

HW/docker host level

Driver Version: 535.183.01
CUDA Version: 12.2 - this is what is container host level, though within container it is all only version 11 and no complaints about libraries, etc. missing or wrong.
GPU Quadro M1200

Finishes ffmpeg successfully:

{
  "reqId": "eFc2QibSSKZLl8hSQd9x",
  "level": 0,
  "time": "2024-12-08T10:15:46+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: decoded 60/60 images\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}

The error:

{
  "reqId": "eFc2QibSSKZLl8hSQd9x",
  "level": 0,
  "time": "2024-12-08T10:15:46+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:15:46.966028: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at xla_ops.cc:296 : NOT_FOUND: could not find registered platform with id: 0x7f6d69c7fae4\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "eFc2QibSSKZLl8hSQd9x",
  "level": 0,
  "time": "2024-12-08T10:15:46+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: Error: Session fail to run with error: 2 root error(s) found.\n  (0) NOT_FOUND: could not find registered platform with id: 0x7f6d69c7fae4\n\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]\n\t [[StatefulPartitionedCall/_1555]]\n  (1) NOT_FOUND: could not find registered platform with id: 0x7f6d69c7fae4\n\t [[{{node movinet_classifier/movinet/stem/stem/conv3d/StatefulPartitionedCall}}]]\n0 successful operations.\n0 derived errors ignored.\n    at NodeJSKernelBackend.runSavedModel (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/nodejs_kernel_backend.js:461:43)\n    at TFSavedModel.predict (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-node-gpu/dist/saved_model.js:341:43)\n    at MovinetModel.predict (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:46:21)\n    at /var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:95:24\n    at /var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4559:22\n    at Engine.scopedRun (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4569:23)\n    at Engine.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:4558:21)\n    at Object.tidy (/var/www/html/custom_apps/recognize/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:8291:19)\n    at MovinetModel.inference (/var/www/html/custom_apps/recognize/src/movinet/MovinetModel.js:92:21)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}

Does not switch to CPU - probably just drops with no results and moves on.

{
  "reqId": "eFc2QibSSKZLl8hSQd9x",
  "level": 0,
  "time": "2024-12-08T10:15:46+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Result for /file.mp4) = []",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}

System details - Recognize output:

{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/pchmura/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.295470: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.395867: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.396576: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.459936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.460175: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.460380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}
{
  "reqId": "qAv5ALUmYNyKoYDBdTzo",
  "level": 0,
  "time": "2024-12-08T10:31:04+01:00",
  "remoteAddr": "",
  "user": "--",
  "app": "recognize",
  "method": "",
  "url": "/cron.php",
  "message": "Classifier process output: 2024-12-08 10:31:04.460515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3510 MB memory:  -> device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0\n",
  "userAgent": "--",
  "version": "30.0.3.2",
  "data": {
    "app": "recognize"
  }
}

Host level side (drivers):

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro M1200                   Off | 00000000:01:00.0 Off |                  N/A |
| N/A   41C    P8              N/A / 200W |      4MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2096      G   /usr/lib/xorg/Xorg                            2MiB |
+---------------------------------------------------------------------------------------+

bugsyb · 2024-12-08T10:08:22Z

Adding successful load of models, prior it went into ffmpeg transcoding/extraction of frames (not parsed by jq, as there is unrecognized character on which jq fails):

{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:43+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Running array (\n  0 => '/usr/bin/nice',\n  1 => '-0',\n  2 => '/var/www/html/custom_apps/recognize/bin/node',\n  3 => '/var/www/html/custom_apps/recognize/src/classifier_movinet.js',\n  4 => '-',\n)","userAgent":"--","version":"30.0.3.2","data":{
"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.666062: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-c
ritical operations:  AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.677758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.729578: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.729862: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.767991: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.768253: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.768468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:44+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:44.768580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3510 MB memory:  -> device: 0, name: Quadro M1200, pci bus id: 0000:01
:00.0, compute capability: 5.0\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.048773: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /var/www/html/custom_apps/recognize/models/movinet-a3/\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.233574: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }\n2024-12-08 09:37:47.233613: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if prese
nt) from: /var/www/html/custom_apps/recognize/models/movinet-a3/\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.233983: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.234247: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.234482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.234731: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.234969: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\
n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.235084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3510 MB memory:  -> device: 0, name: Quadro M1200, pci bus id: 0000:01
:00.0, compute capability: 5.0\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.657173: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:47+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:47.778523: I tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:49+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:49.619424: I tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: /var/www/html/custom_apps/recognize/models/movinet-a3/\n","userAgent":"--","version":"30.0.3.
2","data":{"app":"recognize"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:50+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: 2024-12-08 09:37:50.601217: I tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 3552449 microseconds.\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize
"}}
{"reqId":"8z5HH3gWkezclidr6PA4","level":0,"time":"2024-12-08T09:37:51+01:00","remoteAddr":"","user":"--","app":"recognize","method":"","url":"/cron.php","message":"Classifier process output: Starting transcoding\n","userAgent":"--","version":"30.0.3.2","data":{"app":"recognize"}}

bugsyb added the bug Something isn't working label Apr 7, 2024

marcelklehr changed the title ~~Movinet fails~~ Movinet fails in GPU mode May 8, 2024

github-project-automation bot added this to Recognize Aug 28, 2024

github-project-automation bot moved this to Backlog in Recognize Aug 28, 2024

marcelklehr added the priority: normal label Sep 2, 2024

bugsyb mentioned this issue Dec 8, 2024

Successfully loaded libtensorflow in Node.js, but couldn't load GPU. Make sure CUDA Toolkit and cuDNN are installed and accessible, or turn off GPU mode. #1060

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Movinet fails in GPU mode #1122

Movinet fails in GPU mode #1122

bugsyb commented Apr 7, 2024 •

edited

Loading

github-actions bot commented Apr 7, 2024

bugsyb commented Apr 8, 2024

bugsyb commented Dec 8, 2024

bugsyb commented Dec 8, 2024

Movinet fails in GPU mode #1122

Movinet fails in GPU mode #1122

Comments

bugsyb commented Apr 7, 2024 • edited Loading

Which version of recognize are you using?

Enabled Modes

TensorFlow mode

Downstream App

Which Nextcloud version do you have installed?

Which Operating system do you have installed?

Which database are you running Nextcloud on?

Which Docker container are you using to run Nextcloud? (if applicable)

How much RAM does your server have?

What processor Architecture does your CPU have?

Describe the Bug

Expected Behavior

To Reproduce

Debug log

github-actions bot commented Apr 7, 2024

bugsyb commented Apr 8, 2024

bugsyb commented Dec 8, 2024

bugsyb commented Dec 8, 2024

bugsyb commented Apr 7, 2024 •

edited

Loading