Ability to control the 'retry prompt' when using max_retries and the initial response fails to pass the pydantic validation.

[Mr-Ruben]

Ability to control/edit the additional prompt that is generated when using max_retries and the initial response fails to pass the pydantic validation.

In this example:

def uppercase_validator(v):
    if v.islower():
        raise ValueError("Name must be ALL CAPS")
    return v


class UserDetail(BaseModel):
    name: Annotated[str, AfterValidator(uppercase_validator)]
    age: int

The initial response from LLM

{'role': 'assistant', 'content': '{"arguments": "{\\"name\\":\\"jason\\",\\"age\\":12}", "name": "UserDetail"}'}

which is

"{\\"name\\":\\"jason\\",\\"age\\":12}"

fails the pydantic validation

1 validation error for UserDetail
name
  Value error, Name must be ALL CAPS [type=value_error, input_value='jason', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/value_error

And there is a retry (if max_retries > 0 ).
In that retry, the initial request is sent again, joining in the received response

{
  'role': 'assistant',
  'content': '{"arguments": "{\\"name\\":\\"jason\\",\\"age\\":12}", "name": "UserDetail"}'
},

With this additional "prompt"

{
  'role': 'user',
  'content': "Recall the function correctly, fix the errorsexceptions found\n1 validation error for UserDetail\nname\n  Value error, Name must be ALL CAPS [type=value_error, input_value='jason', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.6/v/value_error"
}

Note: The source code shows errors and exceptions but that is verbatim what I got from the openai logs. But that is irrelevant in this question.

---

Questions:

• Q1: Is there an easy way to edit/modify the 'retry prompt' ?
  I have seen that you have modified it a few times in the source code.

• Q2: This part 'For further information visit https://errors.pydantic.dev/2.6/v/value_error"' is rather useless (I think) and removing it would reduce the number of tokens, thus reducing the amount of useless input the LLM has to go through.

---

Apologies if it is responded somewhere. I searched in many places but couldn't find anything around it.

Note: instructor is an amazing library. It's simply and astonishingly beautiful because it is almost transparent.

[Mr-Ruben]

In case it helps someone/somehow ... [not really an answer I can use]

To modify the prompt that begins with "Recall the function correctly" as a user of the library without modifying the source code, you can utilize the library's functionality for customization, if available. In this specific case, since the source code is not to be altered directly, you can look for any configuration options or settings within the library that allow for prompt customization.

1. Check for Configuration Options: Look through the library's documentation or any accompanying resources to see if there are any configuration options related to modifying prompts or messages displayed by the library.

2. Utilize Customization Features: Some libraries provide hooks or callbacks that allow users to customize certain aspects without changing the source code directly. Explore if such features exist in the library you are using.

3. Override Default Behavior: If the library supports inheritance or overriding of certain functions, you may be able to subclass relevant components and override methods responsible for displaying prompts.

4. Dynamic Prompt Generation: Depending on the library's design, you might be able to dynamically generate prompts based on user input or other parameters, allowing for a more flexible approach to customizing prompts.

Without specific details from the search results provided, it is recommended to refer to the library's documentation, community forums, or reach out to the library maintainers for guidance on how to achieve prompt customization without modifying the source code directly.

Citations:
[1] https://github.com/jxnl/instructor/blob/ed2a70254a71f67785828a8bb057acb5d73d01ba/instructor/patch.py

[Mr-Ruben]

Fixed here
#662
0b3e28e

Documentation
https://disable-pydantic-error.instructor.pages.dev/concepts/reask_validation/?h=disable+url#optimizing-token-usage