Sessionization and building an eval dataset #230
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eptance rate * Add spans in the completer to track token usage * Add spans for LogEvent * Ensure http clients for OpenAI and Anthropic use the OTEL transport.
…dataset from logs.
✅ Deploy Preview for foyle ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
* Add helper method to create a config object using a temporary directory for use in testing * Start defining constants for function names of logging messages; use reflection to ensure correctness in a unittest
… add it to the session.
* We want to incrementally checkpoint log processing when processing really long files. * Otherwise when processing really long files we could potentially run into problems with not making progress. * As part of this processing loop we need to periodically checking if the application has been shutdown and if so stop processing log entries.
* Define an EvalExample proto * Define code to generate the EvalExamples from the sessions
ALso start adding protos to ListSessions and GetSession
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use sqllite For Sessions
This PR introduces the use of sqlllite to store sessions
We use sqllite rather than a KV store (e.g. pebbledb) because we want to be able to index the sessions along various dimensions and search for those sessions.
We use the cgo-free port of SQLite (modernc.org/sqlite) to avoid introducing cgo
We use sqlcs to generate type safe code to run various queries
The generated SQL files live in the pkg/analyze/fsql package