tracker: move prepare_inputs_for_generation into the generation mixin 🧹

[gante]

🧹 This is a tracker regarding the move of prepare_inputs_for_generation into the generation mixin 🧹

Why?

1. prepare_inputs_for_generation is not part of the core modeling, but rather a utility for generate
2. it should greatly reduce the need to touch modeling code, on generate changes. Fewer modeling changes -> improved model stability
3. greatly reduced number of lines of code 🙏

Tracker

Kinda ordered list of tasks:

• 1. Fix related slow tests before we start — all llama, generate, and cache_utils [except sink cache, broken atm] slow tests should be passing to ensure we don’t break anything (Llama: make slow tests green 🟢  #33138)
• 2. PreTrainedModel doesn't inherit from GenerationMixin, so that can_generate() becomes independent of prepare_inputs_for_generation being overwritten or not (Generation: deprecate PreTrainedModel inheriting from GenerationMixin  #33203)
• 3. Move llama’s prepare_inputs_for_generation to the generation mixin. This implies moving one function that prepares the 4D mask too (the one that is called there) (Generate: move llama prepare_inputs_for_generation to GenerationMixin #33677)
• 4. Add tests for the generalist prepare_inputs_for_generation — currently we don’t test it directly, and we should (decoder-only llms: Generate: remove most decoder-only LLMs prepare_inputs_for_generation #33870, encoder-decoder llms: Generate: move prepare_inputs_for_generation in encoder-decoder llms #34048)
• 5. Address the case of synced_gpus in generate: when synced_gpus and cache_positions is out of bounds, take the latest available input_ids for dummy computations (Generate: Fix modern llm generate calls with synced_gpus #34095)
• 6. Delete prepare_inputs_for_generation from as many models as possible. There may be merge conflicts here, due to the 4D mask function. Try to iron out as many trivial cases as possible (decoder-only llms: Generate: remove most decoder-only LLMs prepare_inputs_for_generation #33870, encoder-decoder llms: Generate: move prepare_inputs_for_generation in encoder-decoder llms #34048)
• 7. Change prepare_inputs_for_generation to forward **kwargs from its input to its output. With minimal changes, this should enable most VLMs to use the shared function -- they forward pixel_values from the input to the output (support for **kwargs Generate: remove most decoder-only LLMs prepare_inputs_for_generation #33870)
• 8. By this point most cases of prepare_inputs_for_generation should have been removed 🤗 We would need to check the others individually, there may be further simplification patterns available!

[gante]

@ydshieh edit the tracker above as soon as you start working on a task, so we don't risk doing redundant work 🤗 (e.g. with the link to a draft PR)

I'll do the same!

[ydshieh]

Thanks