[playground][dogfooding] initial playground dogfooding check list

[Parker-Stafford]

Datasets

(Tony + Xander)

• test on large datasets
• streaming
• non streaming (ui side)
• [stretch] test behind proxy with some timeout

Tools

(Roger + Parker)

• tool switching for calls and schemas between providers

Tool calling

Does the playground allow you to simulate an actual tool call (user message with tool, assistant message with tool calls, tool message with results, proper response from llm)

• tool calling for anthropic
• tool calling for openai

Tool use

Does the playground allow you to add tools that can be appropriately picked out by an llm (add a tool with a relevant message)

• tool use for anthropic
• tool use for openai

Template application

• mustache (datasets / normal)
• fstring (datasets / normal)

Span replay

(Xander + Parker)

• test going from auto instrumentor to playground
• test going from playground to span to playground
• input
• output
• model
• invocation params
• tools
• tool calls

Structured output

Parker + Roger

Known issues / WIP

Playground spans

• [playground][instrumentation] message.tool.id is not instrumented messages on playground spans  #5370
• [playground][instrumentation] tool calls in input messages are only properly instrumented for openai #5369

UI

• [playground][ui] add experiment run details slideover with reference input, reference output, and playground run output #5286
• [playground][ui] display "no variable" errors for dataset examples before and after running #5284
• [playground][ui] support anthropic tool response messages #5267
• [playground] llm span to playground parameter parsing utils #4886 - missing prompt template variable parsing

Server

New Issues

• [playground][ui] cannot paginate datasets table on playground page #5382
• [playground][ui] need to add errors into output display so they are persisted longer than just the notification for debugging #5383
• [playground][ui] bug- output schema is cleared every time you click the model config button #5384
• [playground][ui] output schema is available on anthropic but doesn't work #5385
• [playground][ui] bug - invocation params form makes a fetch on every key press when typing in an azure deployment name #5386
• azure invocation param serving needs investigation, do we call out to get them or just serve a default set #5387
• [playground][ui] bug - azure model config saving is broken #5388
• [playground][instrumentation] add azure as provider on playground spans for azure runs #5389
• [plagyround][server] invocation parameter defaults are missing for most models which leads to inconsistent ux #5390
• [playground][ui] surface that you don't have to set api keys in the browser can be added as environment variables for server #5391
• [playground][server] bug - not setting an api key in the browser for playground causes a swallowed error #5392
• [playground][server] investigate strategy for batching llm calls on dataset runs in playground #5394

Lower prio

• [playground][instrumentation] strip out internal graphql / playground param from input.value in playground spans #5393

Dogfooding issues round 2

• tool role message tool_call_id is not pulled into playground from span parsing #5459
• [playground][enhancement] add text to inform the user a playground run is still going even if some dataset examples have completed #5460
• [playground][enhancement] missing configuration settings for azure should raise intelligible error message #5461
• [playground][dataset] coroutine timeout message not displayed #5454
• [playground][ui] disable (+ Tool) button when invocation params do not include tool_choice #5462
• [playground][ui] improve experiment table column resizing logic #5455
• [playground][ui] unaligned hbars #5456
• [playground] throttler does not seem to be hit #5457
• [playground][ui] add scroll bar gutters #5458
• [playground][testing] final sanity tests #5463
• [playground][ui] streaming toggle should be remembered #5464
• [playground] limit number of concurrent async tasks in chat completion over dataset mutation #5465
• [playground][ui] update save config button text and icon #5466
• [playground][ui] dataset example run spinner flickers more than expected #5472
• [playground][ui] display links to all experiments #5480
• [playground] add global limit for in progress chat completion streams #5489
• [playground][enhancement] add timestamp to playground experiment names #5491
• [playground][ui] provider selector is broken #5498
• [playground] dataset run failing with unexpected error #5499
• [playground][server] span latency numbers are affected by number of instances running / number of dataset examples #5510

Post-Launch

• [playground][ui] dataset example lines wrap mid-word #5483
• [playground][ui] provide a way to cancel an in-progress run over a dataset #5484
• [playground][ui] allow horizontal scroll on prompts #5485

Dogfooding round 3

• [playground][bug] make sure that default model config in playground works properly #5511
• [playground][bug] token counts are not added to spans for gemini #5512
• [playground][enhancement] truncate or scroll for playground output cells in datasets table #5513
• [playground][bug] gemini supported parameters are broken #5514
• [playground][bug] key off of provider value on span attributes to determine provider, fallback to using current model heuristic #5516

[Parker-Stafford]

-- moved issues up