Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove surrounding backticks before parsing output as JSON #118

Merged
merged 2 commits into from
Apr 17, 2023

Conversation

irgolic
Copy link
Contributor

@irgolic irgolic commented Apr 10, 2023

gpt-3.5-turbo consistently returns the output surrounded by backticks for me:

E.g.,

```json
{
    "action": "new_file",
    "new_file": {
        "filepath": "tic_tac_toe.py",
        "description": "Python file containing the implementation of the Tic Tac Toe game and AI"
    }
}
\```

(I added the \ on the closing backticks to avoid closing the codeblock)

This PR removes the backticks before JSON parsing, should they appear.

@irgolic irgolic force-pushed the post-process-backticks-output branch 2 times, most recently from 8aef6f1 to 2bb7180 Compare April 10, 2023 21:25
Copy link
Collaborator

@ShreyaR ShreyaR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

I can approve once tests are passing.

Btw, have you tried out the prompting technique where you most of the prompt in the system messages, and only pass in the low level task details in the user message? Curious if you're still getting backticks with that approach.

@irgolic irgolic force-pushed the post-process-backticks-output branch from 78c853b to d24b944 Compare April 11, 2023 22:44
@codecov
Copy link

codecov bot commented Apr 11, 2023

Codecov Report

Patch coverage: 60.00% and project coverage change: -0.08 ⚠️

Comparison is base (5210d41) 77.54% compared to head (74de373) 77.46%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
- Coverage   77.54%   77.46%   -0.08%     
==========================================
  Files          44       44              
  Lines        2329     2339      +10     
==========================================
+ Hits         1806     1812       +6     
- Misses        523      527       +4     
Flag Coverage Δ
unittests 77.46% <60.00%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
guardrails/run.py 90.90% <60.00%> (-3.48%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@irgolic irgolic force-pushed the post-process-backticks-output branch from 9893d98 to a060939 Compare April 11, 2023 22:49
@irgolic
Copy link
Contributor Author

irgolic commented Apr 11, 2023

Fixed tests, there seems to be some weird behavior that's different on 3.8 and 3.9 in the SQL test I fixed.

Btw, have you tried out the prompting technique where you most of the prompt in the system messages, and only pass in the low level task details in the user message? Curious if you're still getting backticks with that approach.

I've not! Is there an easy way to transition to this system? :D

@irgolic irgolic force-pushed the post-process-backticks-output branch from a060939 to 74de373 Compare April 12, 2023 21:30
@ShreyaR ShreyaR merged commit 68de138 into guardrails-ai:main Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants