feat: add spider example with 150 samples #165
Conversation
Co-authored-by: Saikat Mitra <saikatmitra91@gmail.com>
|
Empirical Run Summary
Total dataset samples: 2 |
| ## Usage | ||
|
|
||
| ``` | ||
| npx @empiricalrun/cli run |
There was a problem hiding this comment.
merge with spider example. known gaps
- iteration speed will get slower with the full dataset: do we want to fix that? (e.g. add a way to sample the test dataset through rc.json)
| "type": "model", | ||
| "name": "prompt-learner-gpt3.5", | ||
| "model": "gpt-3.5-turbo", | ||
| "prompt": "You are a helpful AI assistant.\nYou are helping a user with a SQLGeneration task.\nThe user gives you the following task description.\nPlease generate SQL query for the given texts to run on sqlite. I will use your output directly in sqlite so only give me the final executable SQL. Do not wrap it in backticks or quotes.\nHere are a few examples to help you understand the task better.\nschema: CREATE TABLE singer (\n singer_id NUMERIC PRIMARY KEY,\n name TEXT,\n country TEXT,\n song_name TEXT,\n song_release_year TEXT,\n age NUMERIC,\n is_male TIMESTAMP\n);\ntext: What is the average, minimum, and maximum age for all French singers?\nSQL: SELECT AVG(age), MIN(age), MAX(age) FROM singer WHERE country='France';\nThink step by step.\nGiven the text, you have to now generate a SQL query.Only output the SQL and nothing else.\nschema: {{schema}}\ntext: {{question}}\nSQL:", |
There was a problem hiding this comment.
move these to a script (blocked on how we run py-script; need prompt generation to be out of the generation loop)
| os.remove(sqlite_file_name) | ||
| con = sqlite3.connect(sqlite_file_name) | ||
| cur = con.cursor() | ||
| schema_script = load_json("/Users/arjun/notebooks/schema.json")[db_name] |
There was a problem hiding this comment.
- add preparation steps (from notebooks) file to this repo
- fix paths here
|
closing this since we've covered in #230 |
No description provided.