Will support multi-turn conversations?

[dz1iang]

Multi-turn conversations are quite common scenarios in the process of fine-tuning.

[RdoubleA]

Yes, these are supported. In your message transform, you can create a list of messages with multiple user and assistant turns. Or you can use the chat dataset builder. See https://pytorch.org/torchtune/stable/basics/chat_datasets.html#chat-datasets for examples.

[dz1iang]

Yes, these are supported. In your message transform, you can create a list of messages with multiple user and assistant turns. Or you can use the chat dataset builder. See pytorch.org/torchtune/stable/basics/chat_datasets.html#chat-datasets for examples.

There should still be some processing involved in calculating the loss and conducting evaluations.

[RdoubleA]

If you are using a custom message transform, you can control the masked attribute for each Message which will control which ones participate in loss computation. There is a PR to make this easily configurable if you use the chat dataset builders: #2284

Could you describe what you mean by evaluations for multi turn conversations? These can vary quite widely - if there's a particular eval in the Eleuther eval harness around multiturn conversations then that's supported through our integrations. Other multiturn evals usually revolve around tool use. Since evals are very user and data dependent, this is usually done in a custom script.