Skip to content

Unitxt 1.7.0
Compare
Choose a tag to compare
@elronbandel elronbandel released this 05 Mar 14:11
· 776 commits to main since this release
1.7.0
f71d1bc

What Changed in Unitxt 1.7.0

This release introduces a few significant changes that modify existing conventions:

  1. Instructions renamed to system_prompts

This means that from now on, to define a new system-level instruction, you can use this code:

system_prompt = TextualSystemPrompt( # <<<< Class name has changed
    "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"
)

add_to_catalog(system_prompt, "system_prompts.models.alpaca", overwrite=True) # <<<< Catalog name has changed

It also means that all the system-level instructions were moved to the catalog under system_prompts instead of instructions.
This change is breaking old instruction but was necassry to enable the next very useful change.

  1. Templates can now (1) generate task specific instruction once at the head of the example, and (2) can add few words the model will say before the models' final prediction

This change was requested by many pepole.

For example here in this COLA dataset example:

User: Classify the grammatical acceptability of the following text to one of these options: unacceptable, acceptable. text: Fred watered the plants flat.
Agent: acceptable

User: Classify the grammatical acceptability of the following text to one of these options: unacceptable, acceptable. text: The pond froze solid.
Agent:

The instruction "Classify the ..." is reapted for every demonstration. Also with the current template there is no way to put few words that the agent will say before the prediciton for instance: "Agent: The class is ". With the new changes both of these important features are enabled.

If the old way for defining tempaltes for classification was:

add_to_catalog(
    InputOutputTemplate(
        input_format="Classify the {type_of_class} of the following {text_type} to one of these options: {classes}. {text_type}: {text}",
        output_format="{label}",
    ),
    "templates.classification.multi_class.default_no_instruction",
    overwrite=True,
)

It is now defined this way:

add_to_catalog(
    InputOutputTemplate(
        input_format="{text_type}: {text}", # <<<< Changed
        output_format="{label}",
        target_prefix="The {type_of_class} is ", # <<<< Added
        instruction="Classify the {type_of_class} of the following {text_type} to one of these options: {classes}.\n", # <<<< Added
    ),
    "templates.classification.multi_class.instruction",
    overwrite=True,
)

The new template fields instruction and target_prefix will produce this example:

Classify the grammatical acceptability of the following text to one of these options: unacceptable, acceptable.

User: text: Fred watered the plants flat.
Agent: The grammatical acceptability is acceptable

User: text: The pond froze solid.
Agent: The grammatical acceptability is

Notice how the instruction appears only once, and the target prefix is appearing after the 'Agent:'.

Read more in the tutorial on preparing templates.

  1. Loading from catalog with modifications

Now you can load an item from the catalog and change its fields. For example, if you want to use a task but with a different metric, you can use this syntax:

card = TaskCard(
    loader=LoadHF(path="glue", name="cola"),
    preprocess_steps=[...],
    task="tasks.classification.multi_class[metrics=[metrics.matthews_correlation]]", # <<<< Modified
    templates="templates.classification.multi_class.all",
)

add_to_catalog(card, "cards.cola", overwrite=True)

Read more in the tutorial on loading from the catalog.

  1. Renaming of additional_inputs to task_data

In an effort to more accurately represent the origin of certain fields within our system, we've renamed the additional_inputs parameter to task_data. This modification underscores the fact that these fields are derived directly from the task definition itself. This change is crucial for maintaining the integrity and reliability of metrics, as it ensures these fields are validated against the task schema. Consequently, developers crafting metrics for specific tasks can effortlessly ascertain which fields are accessible to them by simply referring to the task schema. This alignment between task definitions and metrics development fosters a more intuitive and efficient workflow for unitxt contributors.

Release Changes

BugFixes:

New Assets:

Enhancments

🚨 Breaking Changes 🚨

  • Rename answer_relevance to answer_reward by @assaftibm in #539
  • Migrate (task-related) Instruction into Template, and introduce (task independent) SystemPrompt by @dafnapension in #565
  • Rename additional_inputs to task_data and make it a simple json dumped by @elronbandel in #595
  • Add shuffling to banking77 and few more classification datasets @ilyashnil in #603 this was necassry in order to balance the classes in those dataset.

New Contributors

Full Changelog: 1.6.1...1.7.0

Contributors

yifanmai, marukaz, and 13 other contributors
Assets 2
Loading
0 Join discussion