How did you get instructions to train Base Decoder on Chart Perception task

[nguyenquangtan]

Hi,
Congratulations on your great work! I am curious about how you created the instructions for the Chart Perception task since the component datasets included in this task (e.g. Chart2Text) do not originally come with the instructions. Did you set one fixed instruction for all samples or utilize a generative model to generate multiple instructions? If it is the latter case, did you generate different instructions for each sample or construct a set of instructions, like about 1000, and use them alternately?

Thank you for your attention to this matter.

[renqiux0302]

Hi @nguyenquangtan,

We set one fixed instruction for all summarization samples during traing auxiliary decoder.
However, to adapt various user instructions, we trained the instruction adapter to classify tasks according to user instructions. Hence, the initial input to the model (instruction adapter) is various user instruction, but the input to the auxiliary decoder is fixed task-specific instruction.

Best regards

[nguyenquangtan]

Hi @renqiux0302,
Thank you for your thoughtful replies! May I ask about the prompts that you use to generate the data for the instruction adapter?