ValueError: Expected boxes to be a Tensor with a final dimension of 4. Instead, got boxes.shape=(None, None, None) when using generator

[hugopi]

Dear kerasCV team,

I recently open a discussion on your github, talking about my project of Boat detection. You advise me to contact you when I got problems.

I am writing to you as I encountered an issue with my boat detection project while using your library. Specifically, when using my generator with the RetinaNet model, I encountered the following error:

ValueError: Expected boxes to be a Tensor with a final dimension of 4. Instead, got boxes.shape=(None, None, None)

To give you some background, I am working with a large dataset, and I am not able to load the entire dataset into memory. Therefore, I have implemented a generator. I found that when I take an output from my generator and feed it to the RetinaNet model directly, there is no problem. However, when I use my generator directly with the model, I encounter the aforementioned error.

I am hoping that you could provide some guidance on how to overcome this issue. For your reference, I have included a sample of the generator's output. It returns a tuple of (images, labels), where the images are in the form of a TensorFlow tensor.

I appreciate any help you could provide in resolving this issue.

Best regards,
Hugo

generator output (images, labels) :

images = tf.Tensor(
[[[[0.6547775  0.5771363  0.6547775 ]
   [0.6334593  0.5771363  0.6334593 ]
   [0.6126148  0.5771363  0.6126148 ]
   ...
   [0.6291439  0.5771363  0.6291439 ]
   [0.6316387  0.5771363  0.6316387 ]
   [0.6373456  0.5771363  0.6373456 ]]

  [[0.6552488  0.5771363  0.6552488 ]
   [0.6371163  0.5771363  0.6371163 ]
   [0.6201546  0.5771363  0.6201546 ]
   ...
   [0.62082434 0.5771363  0.62082434]
   [0.6426415  0.5771363  0.6426415 ]
   [0.6585576  0.5771363  0.6585576 ]]

  [[0.65501314 0.5771363  0.65501314]
   [0.64148647 0.5771363  0.64148647]
   [0.6291439  0.5771363  0.6291439 ]
   ...
   [0.6157083  0.5771363  0.6157083 ]
   [0.6472823  0.5771363  0.6472823 ]
   [0.6678632  0.5771363  0.6678632 ]]

  ...

  [[0.5913914  0.5771363  0.5913914 ]
   [0.59011614 0.5771363  0.59011614]
   [0.593096   0.5771363  0.593096  ]
   ...
   [0.6232862  0.5771363  0.6232862 ]
   [0.5952337  0.5771363  0.5952337 ]
   [0.5771363  0.5771363  0.5771363 ]]

  [[0.6075664  0.5771363  0.6075664 ]
   [0.6168169  0.5771363  0.6168169 ]
   [0.63005    0.5771363  0.63005   ]
   ...
   [0.6172609  0.5771363  0.6172609 ]
   [0.59288263 0.5771363  0.59288263]
   [0.5771363  0.5771363  0.5771363 ]]

  [[0.60931766 0.5771363  0.60931766]
   [0.62418383 0.5771363  0.62418383]
   [0.6428727  0.5771363  0.6428727 ]
   ...
   [0.6108541  0.5771363  0.6108541 ]
   [0.59459156 0.5771363  0.59459156]
   [0.579387   0.5771363  0.579387  ]]]], shape=(1, 512, 512, 3), dtype=float32)
   
labels = {'boxes': <tf.RaggedTensor [[[484.0, 293.0, 12.0, 29.0]]]>, 'classes': <tf.RaggedTensor [[1.0]]>}

The getitem of the generator

 def __getitem__(self, idx):
        """Gets batch at position `index`."""
        list_image_tensor = []
        list_box_tensor = []
        list_class_tensor = []

        batch_patch_paths = self.patch_paths[idx * self.batch_size: (idx + 1) * self.batch_size]
        for patch_path in batch_patch_paths:
            # read dataframe of the chip (containing one or multiple detections
            dataframe = pd.read_csv(os.path.join(patch_path.path, 'annotations.csv'))
            # transform dataframe information in keras inputs
            keras_input = self.data_to_keras_input(dataframe)

            # get each detection's image, box and class
            list_image_tensor.append(keras_input['images'])

            boxes = tf.squeeze(keras_input['bounding_boxes']['boxes'], axis=0)
            list_box_tensor.append(boxes.numpy())

            classes = tf.squeeze(keras_input['bounding_boxes']['classes'], axis=0)
            list_class_tensor.append(classes.numpy())

        # stack detections in one tensor
        stack_image_tensor = tf.concat(list_image_tensor, axis=0)
        stack_label_tensor = {'boxes': tf.ragged.constant(list_box_tensor, ragged_rank=1),
                              'classes': tf.ragged.constant(list_class_tensor, ragged_rank=1)}

        return stack_image_tensor, stack_label_tensor

Test lines:

>>>item = train_generator.__getitem__(0)

>>>model.fit(x=item[0],y=item[1], epochs=1)
 1/1 [==============================] - 17s 17s/step - loss: nan - box_loss: nan - classification_loss: inf - percent_boxes_matched_with_anchor: 0.0000e+00

>>>model.fit(train_generator, epoch=1)
 
 Traceback (most recent call last):
  File "/snap/pycharm-community/329/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)
TypeError: Model.fit() got an unexpected keyword argument 'epoch'
model.fit(train_generator, epochs=1)
Traceback (most recent call last):
  File "/snap/pycharm-community/329/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/tmp/__autograph_generated_filex33c4n0y.py", line 15, in tf__train_function
    retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
ValueError: in user code:
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/engine/training.py", line 1249, in train_function  *
        return step_function(self, iterator)
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/engine/training.py", line 1233, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras/engine/training.py", line 1222, in run_step  **
        outputs = model.train_step(data)
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras_cv/models/object_detection/retinanet/retinanet.py", line 443, in train_step
        y_for_label_encoder = bounding_box.convert_format(
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras_cv/bounding_box/converters.py", line 373, in convert_format
        boxes["boxes"] = convert_format(
    File "/home/hugo/miniconda3/envs/sinbad/lib/python3.10/site-packages/keras_cv/bounding_box/converters.py", line 384, in convert_format
        raise ValueError(
    ValueError: Expected `boxes` to be a Tensor with a final dimension of `4`. Instead, got `boxes.shape=(None, None, None)`.
    ```

[LukeWood]

Hey @hugopi ! Is there any way you can convert your data pipeline to a tf.data pipeline as we do in https://keras.io/guides/keras_cv/object_detection_keras_cv/?

With respect to the NaN loss I think I have this fixed in #1684

[hugopi]

Dear @LukeWood ,

Thanks a lot for your invaluable advice. Your suggestion to convert my generator to a tf.data pipeline was spot-on and made all the difference.

Moreover, your recent update to address the NaN loss issue has been helpful. I am now able to work with small batches with ease.

KerasCV Team prompt responsiveness and expertise have been truly remarkable, thanks for this kind assistance.

With warm regards,

Hugo Pires