Failed tests in the last release (v3.8.0) test suite

[gdrosa]

Summary:
While attempting to install Keras in editable mode and running the test suite, several tests failed. I now detail the steps I followed to reproduce the errors.

---

Steps to Reproduce:

1. Created a conda environment:

   conda create -n keras python=3.10
   conda activate keras

2. Cloned the Keras repository:

   git clone https://github.com/keras-team/keras.git
   cd keras

3. Moved to the latest release:

   git fetch --tags
   git checkout tags/v3.8.0

4. Installed the requirements:

   python3.10 -m pip install -r requirements.txt
   ./shell/api_gen.sh

5. Installed Keras in editable mode:

   python3.10 -m pip install -e .

6. Configured the backend:

   export KERAS_BACKEND="tensorflow"

7. Ran the test suite:

   python3.10 -u -m pytest --cov=keras --cov-report=html:htmlcov --cov-report=xml:coverage.xml > keras_golden_run

---

Observed Behavior:

• The test suite reported the following short summary:

  FAILED integration_tests/jax_custom_fit_test.py::test_custom_fit - TypeError: test_custom_fit.<locals>.CustomModel.train_step() missing 1 required positional argument: 'data'
  FAILED integration_tests/tf_distribute_training_test.py::test_model_fit - RuntimeError: Virtual devices cannot be modified after being initialized
  FAILED integration_tests/torch_custom_fit_test.py::test_custom_fit - AttributeError: 'CustomModel' object has no attribute 'zero_grad'
  FAILED integration_tests/torch_workflow_test.py::TorchWorkflowTest::test_keras_layer_in_nn_module - AssertionError: 0 != 2
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_notop_custom_input_shape_EfficientNetV2B0_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_notop_custom_input_shape_EfficientNetV2B1_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_notop_custom_input_shape_EfficientNetV2B2_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_notop_custom_input_shape_EfficientNetV2B3_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_pooling_EfficientNetV2B0_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_pooling_EfficientNetV2B1_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_pooling_EfficientNetV2B2_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  FAILED keras/src/applications/applications_test.py::ApplicationsTest::test_application_pooling_EfficientNetV2B3_channels_first - tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__BroadcastTo_device_/job:localhost/replica:0/task:0/device:CPU:0}} Unable to broadcast tensor of shape [3] to tensor of shape [1,3,1,1] [Op:BroadcastTo]
  = 12 failed, 12870 passed, 346 skipped, 8 xfailed, 2 xpassed in 1283.05s (0:21:23) =
  

• Errors include:

  • TypeError in integration_tests/jax_custom_fit_test.py.
  • RuntimeError in integration_tests/tf_distribute_training_test.py.
  • InvalidArgumentError in several applications_test.py cases involving EfficientNetV2.

---

Environment Details:

• Python Version: 3.10
• Anaconda Version: 4.10.1
• Operating System: Ubuntu 22.04.5 LTS

---

Additional Notes:

1. I avoided using pip_build.py as I wanted to install Keras in editable mode.
2. The error logs seem to point towards compatibility or configuration issues:
   • The integration_tests errors suggest issues with the test setup for jax and torch.
   • The applications_test.py errors may involve TensorFlow device configurations or input shapes for EfficientNetV2.

---

Request for Help:
I expect the test suite to pass without any errors, or at least there should be no critical errors, so:

• Should these failures be ignored, or are they indicative of underlying issues?
• Should I reconfigure the backend to run these tests?
• Are there additional dependencies or configurations required to run the full test suite?

I guess that I should use both Jax and PyTorch as backends to run these tests, but maybe I can just ignore them if I am using TensorFlow. Or am I missing something? However, any assistance in resolving this issue would be greatly appreciated.

Thank you.

[SamanehSaadat]

Looking at our GitHub actions here shows the backend specific tests. For example, integration_tests/jax_custom_fit_test.py should only be run with the JAX backend. Also application tests seem to be ignored here.

I think the only test that is unexpected to fail here is integration_tests/tf_distribute_training_test.py. I just ran this test on my local machine and it passed. Do you only see this issue if you install Keras in editable mode? Or do you see the same issue if you checkout the master and run pytest integration_tests/tf_distribute_training_test.py?

[gdrosa]

Looking at our GitHub actions here shows the backend specific tests. For example, integration_tests/jax_custom_fit_test.py should only be run with the JAX backend. Also application tests seem to be ignored here.

I think the only test that is unexpected to fail here is integration_tests/tf_distribute_training_test.py. I just ran this test on my local machine and it passed. Do you only see this issue if you install Keras in editable mode? Or do you see the same issue if you checkout the master and run pytest integration_tests/tf_distribute_training_test.py?

Thank you for the detailed feedback. I re-executed the command in editable mode and in the v3.8.1 tag like this:

python3.10 -u -m pytest keras \
  --ignore keras/src/applications \
  --cov=keras \
  --cov-config=pyproject.toml \
  --deselect integration_tests/jax_custom_fit_test.py::test_custom_fit \
  --deselect integration_tests/torch_custom_fit_test.py::test_custom_fit \
  --deselect integration_tests/torch_workflow_test.py::TorchWorkflowTest::test_keras_layer_in_nn_module \
  > keras_golden_run

I had no issues. Additionally, I ran the integration_tests/tf_distribute_training_test.py test individually, and it passed. I think it was a false failure due to the other failing tests.

[SamanehSaadat]

Awesome! That's great to hear! Thanks for re-running the command and isolating the test. Yeah, it likely was a false failure caused by other failing tests.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No