Optional bias for qwen2 model

[wavy-jung]

Feature request

bias of linear layers in qwen2 model is hard coded as following:

• transformers/src/transformers/models/qwen2/modeling_qwen2.py

  Lines 217 to 219 in 85345bb

  	self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
  	self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
  	self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)

• transformers/src/transformers/models/qwen2/modeling_qwen2.py

  Lines 271 to 274 in 85345bb

  	self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=True)
  	self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=True)
  	self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=True)
  	self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)

It would be good to make bias optionally configurable through a config file to ensure compatibility with the latest models. (e.g. llama)

Motivation

bias is optional in llama model as following:

• transformers/src/transformers/models/llama/modeling_llama.py

  Lines 286 to 288 in 85345bb

  	self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
  	self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=config.mlp_bias)
  	self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=config.mlp_bias)

Your contribution

I'll submit PR for this feature

[amyeroberts]

cc @ArthurZucker

[ArthurZucker]

Answered on the PR~