feat: Preliminary implementation of self-instruct pipeline #1276
Conversation
Wendong-Fan
requested review from
raywhoelse,
Wendong-Fan,
Asher-hss and
CaelumF
There was a problem hiding this comment.
Thanks @AveryYay ! Overall looks great, left some comments below
camel/synthetic_datagen/self_instruct/filter/filter_function.py
Outdated
Show resolved
Hide resolved
camel/synthetic_datagen/self_instruct/filter/filter_function.py
Outdated
Show resolved
Hide resolved
camel/synthetic_datagen/self_instruct/filter/filter_function.py
Outdated
Show resolved
Hide resolved
| clf_prompt += " Only answer yes or no" | ||
| response = self.agent.step(clf_prompt) | ||
| result = response.msgs[0].content.strip().lower() | ||
| return result in ["yes", "true"] |
There was a problem hiding this comment.
we can make the response from agent more robust by using structured output by define a BaseModel like below
class AgentResponse(BaseModel):
answer: bool = Field(..., description="Indicates whether the task is classification (True/False).")
There was a problem hiding this comment.
Sure I will take a look into this!
There was a problem hiding this comment.
Yeah ideally we replace all string splitting on LLM-produced text with structured output. Potentially relevant is my approach here #1289 though it's still under review
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
AveryYay
changed the title
There was a problem hiding this comment.
Hi Avery, great job! Just a small question: why are you using data where is_classification is set to false?
There was a problem hiding this comment.
The seed data contains classification instructions as well, but most of the data points have is_classification set to false. The code you're looking at is an example case where I only sampled 6 datapoints, so it's likely that all 6 samples happened to have is_classification as false. If we want to deal with the classification instructions, we could insert more datapoints into the seed.
| r"""Filters instructions based on their word count. | ||
|
|
||
| Args: | ||
| r"""Filters instructions based on their word count. | ||
|
|
||
| Args: | ||
| min_len (int): The minimum word count required for an instruction. | ||
| (default::obj:`5`) | ||
| max_len (int): The maximum word count allowed for an instruction. | ||
| (default::obj:`200`) | ||
| """ | ||
| """ |
There was a problem hiding this comment.
docstring here need to be fixed
There was a problem hiding this comment.
I have fixed locally, will push later
Description
A basic implementation of self-instruct pipeline
Motivation and Context
Why is this change required? What problem does it solve?
close #1214
Types of changes
What types of changes does your code introduce? Put an
xin all the boxes that apply:Checklist
Go over all the following points, and put an
xin all the boxes that apply.If you are unsure about any of these, don't hesitate to ask. We are here to help!