🗺 prompt playground

[mikeldking]

As a user of Phoenix I don't want to have to go back to my IDE to iterate on a prompt. I want to be able to use the data stored in Phoenix (spans, dataets) and run them through a prompt.

Use-cases

• Replay a template change on an LLM Span
• Run a template change on a dataset
• Construct an evaluation template on a single chosen production span or Dataset - Workflow is testing your Evals and be able to save as experiment
• Synthetic data Generation - Use to generate synthetic data, add columns to current rows of data in a dataset, to help create test data

Milestone 1 - LLM Replay

The most important thing to accomplish is the ability to take a specific LLM step (e.g. a span) and to be able to re-play that execution. In the process of re-executing that step, we should record the new response so that we can compare and evaluate.

Leads: @axiomofjoy @anticorrelator

Out of Scope

• focus on Chat Completions (things with roles) - not completions

Planning

• [playground] PRD #4658
• [playground][spike] API design for chat completions #4600

UI

• [playground][ui] UI skeleton and feature flag #4606
• [playground][ui] credential storage #4885
• [playground][ui] edit message role #4857
• [playground] llm span to playground parameter parsing utils #4886
• [playground][ui] edit model (e.g. model selector) #4887
• [playground][ui] mustache template editor for codemirror #4888
• [playground][ui] f-string vs mustache template formatting toggle #4889
• [playground][ui] f-string variable parsing utility function #4890
• [playground] Toggle to switch between f-string and mustache #4996
• [playground][ui] mustache template variable parsing utility function #5092
• [playground][ui] template variables (e.g. inputs) UI and storage #4891
• [playground][ui] message re-ordering via drag and drop #4892
• [playground][ui] add new tool UI #4902
• [playground][ui] UI for metrics for a chat completion (token count, latency) #5183
• [playground][ui] display playground instances in "alphabetic" order #4949
• [playground][ui] switch Input to be singular across all prompts / instances #4952
• [playground][ui] Provide a back button if coming from a span #4953
• [playground][ui] UI and playground store parsing of variables #4986
• [playground][ui] UI for editing input variables via resizing text area #4987
• [playground] copy prompt to clipboard button #4997
• [playground] parse and infer model config from span #4998
• [playground][ui] support function / tool call messages #5041
• [playground][ui] suppport function / tool call parsing from spans #5020
• [playground][ui] support azure params endpoint and api version #5023
• [playground] apply template variables when invoking the template #5034
• [playground][ui] invocation parameters ui #5040
• [playground][ui] create a non-streaming version of the playground #5181
• [playground] detect when phoenix is running in an environment that doesn't support streaming #5182
• [playground][ui] save off model config by provider and use as default in playground #5069
• [playground][ui] show token count, latency, after completing the invocation #5036
• [playground][ui] annotation slide-over #5035
• [playground] pull span information once the span is completed (e.g. latency, tokens) #5099
• [playground] tool JSON schema utils (anthropic -> open_ai -> ???) #5100
• [playground] option to view output as markdown #5112
• [playground][ui] support anthropic tool calls in playground #5209
• [playground][ui] support anthropic tools in playground #5210
• [playground][ui] add structured output support #5233
• [playground][ui] plumb through json invocation parameters #5252
• [playground][ui] create empty state for playground when no llm providers are installed on server #5253
• [playground][ui] support anthropic tool response messages #5267
• [playground][ui] use tooltip for dataset example slideover #5353

API

• [playground][api] prototype a subscription for chat completion #4773
• [playground][api] use mutations to submit a chat completion #4774
• [playground][gql] invocation parameters to chat completion #4858
• [playground][api] Models and vendors listing resolver #4859
• [playground][api] websocket authentication #4793
• [playground][gql] LLM span recording of chat completions (persistence) #4856
• [playground][gql] server side messages input and roles #4893
• [playground][backend] span recording in the DB in the playground project #4894
• [playground][gql] model input to chat completion #4900
• [playground][gql] tool definition input to chat completion #5093
• [playground][gql] support for output schema (e.g. JSON mode) #4901
• [playground][gql] support for Azure OpenAI #4895
• [playground][gql] support for Anthropic #4897
• [playground][gql] support for Gemini #5154
• [playground][gql] surface up metrics for a chat completion (token count, latency) #5091
• [playground][gql] support anthropic for chatCompletionSubscription #4960
• [playground][gql] take in model as a parameter to chat completion #4975
• [playground][gql] model provider and model name #4989
• [playground][gql] model tool calling parameters #4988
• [playground][gql] invocation parameter input #4990
• [playground] record llm.model_name #4995
• [playground][api] record token counts #5030
• [playground] use wss when https is used to initiate websockets connection #4930
• [playground][api] add background task for creating playground span #5031
• [playground][gql] return the chatcompletion resulting span node when available #5037
• [playground][gql] support chatCompletion as a mutation (non-streaming) #5038
• [playground][gql] wire up azure openai #5039
• [playground] anthropic tool calling support #5097
• [playground][gql] filter down the list of providers depending on whether or not client packages are intalled #5116
• [playground] add anthropic token counts #5160
• [playground] improve error handling #5187

Testing

• [playground] anthropic fixture data #4962
• [playground][tests] add api tests for chat completion subscription #4984

Fix

• [playground] resolve span caching issue #4983
• [playground] add openai as a container dependency #5075
• [playground][fix] ensure chat completion subscription unit tests run on all os and on postgres #5126
• [playground][ui] prompts accordion takes up the entire panel height when collapsed #5317

Instrumentation

• [playground][instrumentation] support for model system and provider in the python instrumentation #5101
• [playground][instrumentation] semantic convention for output schema definition (JSON mode) #4896
• [playground][instrumentation] openai support for output schema #4904
• [playground][instrumentation] tool call id semantic conventions #4908
• [playground][instrumentation] capture tool ids in openai python instrumentaiton #5194
• [playground][instrumentation] capture tool ids in openai js instrumentaiton #5195
• openai node sdk support for llm.system and llm.provider #5102
• [playground][instrumentation] tool use messages missing tool id for anthropic #5311
• [playground][instrumentation] tool result missing support for array content attribute for anthropic #5312
• [playground][instrumentation] tool calls in input messages are only properly instrumented for openai #5369
• [playground][instrumentation] message.tool.id is not instrumented messages on playground spans  #5370

OpenIneference

• [bug][anthropic] Tool result content is not added to spans openinference#1109

Milestone 2 - Datasets on Playgrounds

Add support to run a set of dataset examples through a prompt

API

• [playground][datasets] add rate limiter to llm clients #5238
• [playground] llm span context manager #5241
• [playground][api] dataset example chat completion subscription #4804
• [playground] record experiment run on chat completion over a dataset subscription #5261
• [playground][datasets] yield experiment, experiment runs, and spans on an ongoing basis from the subscription on an ongoing basis #5276
• [playground][datasets] ensure experiment runs are created for template formatting errors
• [playground][datasets] chat completion over datasets mutation (non-subscription) #5294
• [playground] display json for example input #5313

UI

• [playground][ui] dataset input mode #4803 - this contains a rough list of todos
• [playground][ui] add dataset selector and dataset examples table to playground #5281
• [playground][ui] map dataset chat completion results back to dataset examples in each instance #5282
• [playground][ui] add experiment button that links to experiment created by running playground on a dataset #5283
• [playground][ui] display "no variable" errors for dataset examples before and after running #5284
• [playground][ui] add link to span (slideover) for each playground experiment run on playground page #5285
• [playground][ui] add experiment run details slideover with reference input, reference output, and playground run output #5286
• [playground][datasets] ensure experiment runs are created for template formatting errors
• [playground][datasets] add button in input column to open dataset example slide over in playground #5300
• [playground][datasets][ui] wire up chat completion subscription to dataset examples table on playground #5304
• [playground][ui] wire up dataset mutation (non streaming) in playground #5328

Milestone 3 - Playground run annotations and evaluators

Add support for easily annotating playground runs via human annotations or evaluators.
Note: Annotations for playground spans is already supported as of the milestones above

Dogfooding

• [playground][dogfooding] initial playground dogfooding check list #5373
• [playground][BUG] playground span has too many quotes for tool call arguments in the message #5402
• [playground][ui] cannot unselect dataset #5443
• [playground][ui] cannot resize dataset example table cells #5444
• [playground] return errors for timeouts and unexpected errors for chat completion over dataset subscription #5448

UI

• [playground][instrumentation] message.tool.id is not instrumented messages on playground spans  #5370
• [playground][instrumentation] tool calls in input messages are only properly instrumented for openai #5369
• [playground][ui] add experiment run details slideover with reference input, reference output, and playground run output #5286
• [playground][ui] display "no variable" errors for dataset examples before and after running #5284
• [playground][ui] support anthropic tool response messages #5267
• [playground] llm span to playground parameter parsing utils #4886 - missing prompt template variable parsing
• [playground][ui] cannot paginate datasets table on playground page #5382
• [playground][ui] need to add errors into output display so they are persisted longer than just the notification for debugging #5383
• [playground][ui] bug- output schema is cleared every time you click the model config button #5384
• [playground][ui] output schema is available on anthropic but doesn't work #5385
• [playground][ui] bug - invocation params form makes a fetch on every key press when typing in an azure deployment name #5386
• azure invocation param serving needs investigation, do we call out to get them or just serve a default set #5387
• [playground][ui] bug - azure model config saving is broken #5388
• [playground][ENHANCEMENT][ui] allow messages containing python code that has too many curly braces #5411
• [playground][ENHANCEMENT][ui] users should be notified upfront about missing model parameters that are required so they don't get confused by errors #5431

Azure

• [playground][ENHANCEMENT][ui][azure] remember the deployment name #5413

Anthropic

• [playground][BUG][ui][anthropic] model name disappears even though config is not saved, resulting in error when run #5405
• [playground][BUG][anthropic] missing dropdown option for model alias with the latest suffix #5406
• [playground][BUG][ui][anthropic] stop_sequences invocation parameter is not populated immediately at playground initialization #5409

Server

• [playground][instrumentation] add azure as provider on playground spans for azure runs #5389
• [plagyround][server] invocation parameter defaults are missing for most models which leads to inconsistent ux #5390
• [playground][ui] surface that you don't have to set api keys in the browser can be added as environment variables for server #5391
• [playground][server] bug - not setting an api key in the browser for playground causes a swallowed error #5392
• [playground][server] investigate strategy for batching llm calls on dataset runs in playground #5394

Dogfooding issues round 2

• tool role message tool_call_id is not pulled into playground from span parsing #5459
• [playground][enhancement] add text to inform the user a playground run is still going even if some dataset examples have completed #5460
• [playground][enhancement] missing configuration settings for azure should raise intelligible error message #5461
• [playground][dataset] coroutine timeout message not displayed #5454
• [playground][ui] disable (+ Tool) button when invocation params do not include tool_choice #5462
• [playground][ui] improve experiment table column resizing logic #5455
• [playground][ui] unaligned hbars #5456
• [playground] throttler does not seem to be hit #5457
• [playground][ui] add scroll bar gutters #5458
• [playground][testing] final sanity tests #5463
• [playground][ui] streaming toggle should be remembered #5464
• [playground] limit number of concurrent async tasks in chat completion over dataset mutation #5465
• [playground][ui] update save config button text and icon #5466
• [playground] ensure that playground defaults are the provider defaults #5528
• [playground] improve default playground messages #5533
• [playground][ui] remove playground feature flag #5534

Presentations

• [playground] demo video #5474

Documentation

• [playground][docs] concept section on prompt engineering #5568
• [playground][docs] how to section on playground (e.g. compare)

Post-Launch

• [playground][ui] dataset example lines wrap mid-word #5483
• [playground][ui] provide a way to cancel an in-progress run over a dataset #5484
• [playground][ui] allow horizontal scroll on prompts #5485
• [playground][ui] copy output schema and tools when adding a new prompt #5530
• [playground][instrumentation] strip out internal graphql / playground param from input.value in playground spans #5393
• [playground] add global limit for in progress chat completion streams #5489
• [playground][bug] key off of provider value on span attributes to determine provider, fallback to using current model heuristic #5516
• [playground][instrumentation] strip out internal graphql / playground param from input.value in playground spans #5393

Punt

• [playground][instrumentation] Update OpenAI Instrumentor to include VertexAI as a provider #5303
• [playground][instrumentation] add llm provider and system attributes to vertexai python instrumentor #5350
• [playground][ui] persist message content when switching between tool calls and string content #5410
• [playground][BUG][anthropic] system prompt is ignored #5407
• [ENHANCEMENT][playground] Allow for passing in a string as a model name for providers with fixed model lists #5427

[heralight]

Hi!

Enhancement proposal

This feature should be similar to #2462 but with more depth. It would involve a simple button to replicate a query into an edit mode, allowing you to replay it. Additionally, it should offer the possibility to add notes on the result iterations, such as rating the quality, format output, etc., on a scale of 1 to 10.

Goal

The goal is to facilitate quick testing of prompts and inputs, enabling evaluation and visualization of progression.

Thank you,

Alexandre

[mikeldking]

Hi!

Enhancement proposal

This feature should be similar to #2462 but with more depth. It would involve a simple button to replicate a query into an edit mode, allowing you to replay it. Additionally, it should offer the possibility to add notes on the result iterations, such as rating the quality, format output, etc., on a scale of 1 to 10.

Goal

The goal is to facilitate quick testing of prompts and inputs, enabling evaluation and visualization of progression.

Thank you,

Alexandre

@heralight Hey! Thanks for the feedback! We have a ton of features coming out with regards to prompt iteration, notably prompt experiments. Stay tuned. It has evaluations built in

Noted on the replay and the annotations:) will give it some thought. We have a few ideas around replaying data at prompts, but haven't thought about human annotations on different prom versions a ton. Would love to hear more.

[heralight]

Very nice!
my ideal workflow, would be:

• trace some openai calls made from code
• transform some into a replayable prompt, where each modification can be tested and versioned and annoted and ranked.
• prompt can parametized and call from code

best,

[axiomofjoy]

🛝🎉