Enhancement: Artifacts Instruction Message Placement Affects OpenAI Prompt Caching Efficiency #4327
hongkai-neu
started this conversation in
Feature Requests & Suggestions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
When the Toggle Artifacts UI is enabled, the current version is moving the instruction message to the second-to-last position in the message array. This placement prevents OpenAI from effectively using its prefix matching for caching, as described in their documentation (https://platform.openai.com/docs/guides/prompt-caching).
We should consider modifying our message handling to keep the instruction message in a consistent position, preferably at the beginning of the array, to take full advantage of OpenAI's caching mechanism.
Beta Was this translation helpful? Give feedback.
All reactions