feat: Stable system prompt and tool ordering for more cache hits#5192
feat: Stable system prompt and tool ordering for more cache hits#5192jsibbison-square merged 10 commits intomainfrom
Conversation
Updated comment to clarify prompt cache behavior.
DOsinga
left a comment
DOsinga
left a comment
There was a problem hiding this comment.
I think @tlongwell-block is also looking at some of this
| } | ||
|
|
||
| #[test] | ||
| fn test_build_system_prompt_orders_extensions_by_name() { |
There was a problem hiding this comment.
I thought it was useful to avoid regressing but have deleted it.
|
Yeah, #5027 removes the timestamp entirely from the system prompt |
|
@tlongwell-block So should I just delete the timestamp line in this PR so we get caching sooner or do you want me to not make that change and have your PR do that? I matched your change to delete the timestamp for system_prompt.md so shouldn't conflict when it merges Once you respond I'll be able to merge. |
…com:block/goose into jsibbison-20251015-no-dynamic-system-prompt * 'jsibbison-20251015-no-dynamic-system-prompt' of github.com:block/goose: Clarify comment on prompt cache filtering
Until #5027 is merged, this would mean goose won't know the time at all @DOsinga, what do you think about merging #5027 right after this one? If we're still not sure about MOIM as a concept, I'd say leave the timestamp in here @jsibbison-square. I will say that the MOIM build is my daily driver and works very well, including on the benchmarks. |
|
Ok I'll put the timestamp back at just the hourly granuality that way we get greater caching and you don't need to rush the other PR. |
| You are a general-purpose AI agent called goose, created by Block, the parent company of Square, CashApp, and Tidal. | ||
| goose is being developed as an open-source software project. | ||
|
|
||
| The current date is {{current_date_time}}. |
There was a problem hiding this comment.
we could consider adding this to the end so we get most of the system prompt cached
There was a problem hiding this comment.
Makes sense, will leave it unchanged here so tylers pr doesn't get a conflict when he removes the timestamp (we changed the timestamp to be at an hour granuality so we still get improved caching benefits).
jsibbison-square
deleted the
jsibbison-20251015-no-dynamic-system-prompt
branch
* main: Fix extension manager resource deadlock (#5066) add new prompt for memory extension (#4945) feat: Stable system prompt and tool ordering for more cache hits (#5192) Remove gtag analytics error (#5268) Feat/smart task organizer recipe (#5032) docs: `goose recipe open` command (#5264) docs: provide more clarity in tagging step of releases (#5269) docs: add GOOSE_DEBUG environment variable (#5265)
* main: (40 commits) Fix extension manager resource deadlock (#5066) add new prompt for memory extension (#4945) feat: Stable system prompt and tool ordering for more cache hits (#5192) Remove gtag analytics error (#5268) Feat/smart task organizer recipe (#5032) docs: `goose recipe open` command (#5264) docs: provide more clarity in tagging step of releases (#5269) docs: add GOOSE_DEBUG environment variable (#5265) Lifei/UI parameter input (#5222) turn off WAL (#5203) Skip subagents for gemini (#5257) Revert "Standardize Session Name Attribute" (#5250) improve provider request logging a bit (#5236) Fix OpenAI empty choices panic (#5248) Revert "Rewrite extension management tools" (#5243) Standardize Session Name Attribute (#5085) Docs: Include step-by-step model configuration instructions for first… (#5239) Delete message visibility filter (#5216) delete flaky pricing integration tests (#5207) chore: upgrade most npm packages to latest (#5185) ...
* main: live testing script (#5263) Fix extension manager resource deadlock (#5066) add new prompt for memory extension (#4945) feat: Stable system prompt and tool ordering for more cache hits (#5192) Remove gtag analytics error (#5268) Feat/smart task organizer recipe (#5032) docs: `goose recipe open` command (#5264) docs: provide more clarity in tagging step of releases (#5269) docs: add GOOSE_DEBUG environment variable (#5265)
* 'main' of github.com:block/goose: fix: include Windows GNU CLI artifact in releases (#5276) Fix bedrock tool call error response (#5067) fix: do not load active extensions when no extensions in the recipe (#5282) Fix pagination - fetch all open issues, not just first 100 Fix label case sensitivity - use lowercase labels Add GitHub Health Dashboard workflow (#5286) live testing script (#5263) escaping markdown fixing syntax Add GitHub Health Dashboard workflow Fix extension manager resource deadlock (#5066) add new prompt for memory extension (#4945) feat: Stable system prompt and tool ordering for more cache hits (#5192) Remove gtag analytics error (#5268)
…ck#5192) Signed-off-by: Blair Allan <Blairallan@icloud.com>
…ck#5192) Signed-off-by: Blair Allan <Blairallan@icloud.com>
4 participants
Summary
When running multiple goose instances for the same tasks (eg. recipes) we currently do not get any llm cache hits on the initial request. From my testing this happens for multiple reasons:
This PR makes the timestamp truncated at the hour to allow for decent time accuracy but greatly improved cache hit potential and sorts the extensions (for instructions) and tools by name so they always appear in the same order.
Type of Change
Testing
I've used an api key with openai and downloaded their usage csv to verify that caching on the same initial request run in subsequent sessions with the developer mcp results in almost 100% cached input token usage.