[Tracking Issue]: Use `merge_by_field_config` for MM models

[DarkLight1337]

🚀 The feature, motivation and pitch

Implement #25676 for each MM model so input shapes are the same as their HF implementations, i.e. no need to use flatten_bn anymore.

I'm going through the list in mostly alphabetical order:

• [Model] Use merge_by_field_config for MM models (A-C) #26073
• [Model] Use merge_by_field_config for MM models (D-F) #26076
• [Model] Use merge_by_field_config for MM models (G) #26117
• [Model] Use merge_by_field_config for MM models (InternVL family) #26153
• [Model] Use merge_by_field_config for MM models (H-L) #26230
• [Model] Use merge_by_field_config for MM models (Llava family) #26280
• [Model] Use merge_by_field_config for MM models (M-N) #26710
• [Model] Use merge_by_field_config for MM models (O-P) #26776
• [Model] Use merge_by_field_config for MM models (Ovis family) #26308
• [Model] Use merge_by_field_config for MM models (M-N) #26710
• [Model] Define merge_by_field_config MM interface (R-T) #26260
• [Model] Define merge_by_field_config MM interface (U-Z) #26261

After all models have been migrated, I will try to merge TensorSchema with MultiModalFieldConfig so we can apply validation inside processor instead of model forward pass (which is the critical path)

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ayushsatyam146]

@DarkLight1337 can I please work on this? Can you assign this if possible?

[DarkLight1337]

Sure, please indicate which item you're working on so we don't overlap our efforts

[ayushsatyam146]

@DarkLight1337 I can start from the bottom of the list, I can take R-Z.

[DarkLight1337]

Great, thanks!