CI fails for Python 3.9: ValueError: zero-size array to reduction operation maximum which has no identity

[albertvillanova]

CI fails for Python 3.9: https://github.com/huggingface/trl/actions/runs/18605452485/job/53053774127

ValueError: zero-size array to reduction operation maximum which has no identity

FAILED tests/test_online_dpo_trainer.py::TestOnlineDPOTrainer::test_training_with_transformers_paged_0_standard_prompt_only - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/test_online_dpo_trainer.py::TestOnlineDPOTrainer::test_training_with_transformers_paged_1_conversational_prompt_only - ValueError: zero-size array to reduction operation maximum which has no identity

Stacktrace:

tests/test_online_dpo_trainer.py:453: in test_training_with_transformers_paged
    trainer.train()
.venv/lib/python3.9/site-packages/transformers/trainer.py:2325: in train
    return inner_training_loop(
.venv/lib/python3.9/site-packages/transformers/trainer.py:2674: in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
trl/trainer/online_dpo_trainer.py:1280: in training_step
    prompt_ids, prompt_mask, completion_ids, completion_mask = self._generate(model, prompts, images)
trl/trainer/online_dpo_trainer.py:1128: in _generate
    completion_ids = pad(completion_ids, padding_value=self.pad_token_id, padding_side="right")
trl/trainer/utils.py:348: in pad
    output_shape = np.max([t.shape for t in tensors], 0).tolist()
.venv/lib/python3.9/site-packages/numpy/_core/fromnumeric.py:2899: in max
    return _wrapreduction(a, np.maximum, 'max', axis, None, out,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = [], ufunc = <ufunc 'maximum'>, method = 'max', axis = 0, dtype = None
out = None
kwargs = {'initial': <no value>, 'keepdims': <no value>, 'where': <no value>}
passkwargs = {}

    def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
        passkwargs = {k: v for k, v in kwargs.items()
                      if v is not np._NoValue}
    
        if type(obj) is not mu.ndarray:
            try:
                reduction = getattr(obj, method)
            except AttributeError:
                pass
            else:
                # This branch is needed for reductions like any which don't
                # support a dtype.
                if dtype is not None:
                    return reduction(axis=axis, dtype=dtype, out=out, **passkwargs)
                else:
                    return reduction(axis=axis, out=out, **passkwargs)
    
>       return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
E       ValueError: zero-size array to reduction operation maximum which has no identity

.venv/lib/python3.9/site-packages/numpy/_core/fromnumeric.py:86: ValueError

[albertvillanova]

After investigation, I think the underlying issue is a @staticmethod decorator used in a module-level function (not within a class):

ERROR    ContinuousBatchingLogger:continuous_api.py:879 Error in generation loop: 'staticmethod' object is not callable
Traceback (most recent call last):
  File "/__w/trl/trl/.venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/continuous_api.py", line 837, in _run_generation_loop
    paged_attention_cache = PagedAttentionCache(
  File "/__w/trl/trl/.venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/cache.py", line 191, in __init__
    num_blocks, max_batch_tokens = memory_handler.infer_num_blocks_and_max_batch_tokens(
  File "/__w/trl/trl/.venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/cache.py", line 437, in infer_num_blocks_and_max_batch_tokens
    num_blocks, max_batch_tokens = self.compute_num_blocks_and_max_batch_tokens(
  File "/__w/trl/trl/.venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/cache.py", line 476, in compute_num_blocks_and_max_batch_tokens
    cache_memory = self.get_available_memory(max_memory_percent)
  File "/__w/trl/trl/.venv/lib/python3.9/site-packages/transformers/generation/continuous_batching/cache.py", line 411, in get_available_memory
    _, total, reserved, allocated = get_device_and_memory_breakdown()
TypeError: 'staticmethod' object is not callable

Note that:

• In Python 3.9 and earlier, @staticmethod produces a descriptor object that is not directly callable when defined outside a class.
• Starting with Python 3.10, CPython changed the behavior of staticmethod: staticmethod objects gained a __call__ method that delegates to the wrapped function, making them callable even outside a class context. However, this masked the underlying issue in newer Python versions.

I think this issue was introduced in transformers by this PR:

• Continuous batching refactor transformers#40426

I have opened a PR in transformers to fix it:

• Remove invalid @staticmethod from module-level get_device_and_memory_breakdown transformers#41747

[albertvillanova]

Fixed by huggingface/transformers#41747

[albertvillanova]

The fix needs being backported to the v4.57-release branch and being included in the next v4.57.2 patch release: huggingface/transformers#41747 (comment)

I reopen this issue to track this.