Inference error

[se0ngrae]

Hi! I am a graduate student who is working with the behavior of two mice.

I have a problem with inference procedures.

An error occurred in the middle or just the start of the inference step.

with these messages below:

Traceback (most recent call last):

  File "/Users/kimseonglae/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
    sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 4909, in main
    labels_pr = predictor.predict(provider)
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 449, in predict
    self._make_labeled_frames_from_generator(generator, data)
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2448, in _make_labeled_frames_from_generator
    for ex in generator:
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 380, in _predict_generator
    ex = process_batch(ex)
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 325, in process_batch
    preds = self.inference_model.predict_on_batch(ex, numpy=True)
  File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 988, in predict_on_batch
    outs = super().predict_on_batch(data, **kwargs)
  File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
    outputs = self.predict_function(iterator)
  File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/kimseonglae/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 4909, in main
      labels_pr = predictor.predict(provider)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 449, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2448, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 380, in _predict_generator
      ex = process_batch(ex)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 325, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 988, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2149, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2159, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 1981, in call
      out = self.keras_model(crops)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/kimseonglae/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 4909, in main
      labels_pr = predictor.predict(provider)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 449, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2448, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 380, in _predict_generator
      ex = process_batch(ex)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 325, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 988, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2149, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 2159, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/sleap/sleap/nn/inference.py", line 1981, in call
      out = self.keras_model(crops)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/kimseonglae/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
2 root error(s) found.
  (0) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
	 [[top_down_inference_model/find_instance_peaks/RaggedFromValueRowIds_1/RowPartitionFromValueRowIds/control_dependency/_342]]
  (1) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_4355]
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB


Process return code: 1

---

Could you kindly help me with this problem??

Thanks,

Regards,
Rae

[talmo]

Hi @se0ngrae,

Just to confirm: are you on an M1 Mac? If so, this looks like it might be the same problem at #1100.

We're working on it and you're welcome to subscribe to that issue thread for updates on our troubleshooting.

In the meantime, you can try running SLEAP on Google Colab or on another platform (Windows/Linux) if you have one available.

Sorry about that!

Talmo

[se0ngrae]

Thank you for your prompt reply. I keep following up on your troubleshooting!! Thanks again!

[Jaspermoray]

Hi. I just wanted to mention that I am using an M1 Mac and had the same exact issue.

[talmo]

Thanks for the report! This is happening because there are some chunks of frame during which no centroids are detected, which causes an error downstream in the centered instance model. This works in Windows/Linux, but due to the implementation of certain CUDA/TensorFlow operations on Apple Silicon, the behavior is different with 0-length tensors and it throws this error. We're trying to find a fix for it, but it's a bit tricky since Apple's tensorflow-metal is pretty experimental and under heavy development.

We'll post updates on that issue as we make progress on it.

[emackev]

I also am getting this issue on a mac M1. I'm new to using SLEAP, and posting a comment in case this has already gotten figured out. I'll try out google colab for now. Thanks, and excited to track some birds :D

[ayexander]

Hi @talmo,
I'm also getting the same error on my personal system (Apple M2 Max). I'm looking forward to when SLEAP will work fully on the M1/M2 GPUs - it will be so METAL ;)

[roomrys]

Hi @ayexander, @emackev,

I am curious if you either of you are on v1.3.1? We made some large dependency updates (including a conda package for Apple Silicon Macs). There is definitely a possibility that using updated packages in our dependencies may have solved this issue.

Thanks,
Liezl

[ayexander]

Hi Liezl (@roomrys)

Yes, I'm using v1.3.1. I just installed it in the last few days for the first time.

From the terminal on opening sleap-label:
Software versions:
SLEAP: 1.3.1
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-13.4-arm64-arm-64bit

Mac M2 Max, 38-core GPU, 96GB RAM.

My "Missing 0-th output from...." error happened on my first attempt ever at Inference over an 800,000 frame h264 video. Very probably I did not run the training enough to avoid long runs of frames with 0 instances predicted. I also did not optimize my video format or length etc. In my experiment the spiders do sometimes partially disappear from view behind hides for hours before rapid bursts of speed to eg. capture prey. However, I've been able to train the model to label those partially hidden instances in subsequent runs. I have also shortened my video files by 80-90% to only the relevant parts of the behavior. And I now convert my videos to the libx264 format which possibly has helped.

In subsequent runs since those fixes the inference step does work such that I can get full prediction and tracking over ~100k-200k frames on my Mac M2 Max.

So far SLEAP is amazing compared to other tracking software I've tried. It's my current favorite for its power, speed, intuitive and user friendly GUI, thorough documentation, ease of learning and support. Still hoping your team may eventually make it possible for it to access the Apple Silicon GPU's to speed it up on personal systems, but can wait for that as I have access to NVIDIA GPU equipped Linux systems in my lab.

Best and thanks for your support and active development and maintenance of a great software package!
Alex

[emackev]

I also got the error when I tried to do inference, and I also installed SLEAP within the week. I tried both the mambaforge version "mamba create -y -n sleap -c conda-forge -c anaconda -c sleap sleap=1.3.1" and also from the mac-specific environment "mamba env create -f environment_mac.yml -n sleap_mac". In both cases, I could label the birds, but it failed at the inference step.

I tried the colab version, and SLEAP is tracking my birds very impressively, which is great! They are outdoor videos in natural environments with multiple different species, so challenging to track -- I second Alex's comments about how SLEAP is awesome, well documented, etc.

Please keep me posted if/when it becomes possible to use SLEAP on Apple Silicon. Thanks!!

[tshindmarsh]

Hi! Appreciate all your awesome work! Unfortunately I am still running into this error on my Apple M2 Ultra. Has there been any updates or workarounds to get the inference running?

[talmo]

Hi all!

Apologies for the delay, but we finally do have an update that should solve this issue!! We're working on testing it out now across the other platforms to make sure we don't break more things than we fix, but if all goes well we should have a fix by the next release.

Subscribe for notifications on #1100 to hear about when we have a beta ready for early testing.

Cheers,

Talmo