feat(playground): parse span attribute `llm.input_messages` for playground span replay #4906

Parker-Stafford · 2024-10-07T19:22:01Z

resolves #4782

uses zod to parse span attributes to get llm.input_messages and put them into a playground instance

This does not account for any variance in span attributes. When dealing with the variants, it may turn out that zod doesn't get us as much as it seems here but is still nice for safely parsing. For example, we can use zod to parse and then have a series of fallbacks (try input_messages, then prompt template, then input.value) or we can create increasingly strict zod schemas to parse to determine the final values for the playground instance

Parker-Stafford · 2024-10-08T00:47:02Z

app/src/pages/playground/schemas.ts

+const toolCallSchema = z
+  .object({
+    function: z
+      .object({
+        name: z.string(),
+        arguments: z.string(),
+      })
+      .partial(),
+  })
+  .partial();
+
+/**
+ * The zod schema for llm message contents
+ * @see {@link https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md|Semantic Conventions}
+ */
+const messageContentSchema = z.object({


these are currently unused, but will be necessary in the future so i didn't want to remove them, can remove for now and add back later.

Parker-Stafford · 2024-10-08T01:14:02Z

app/src/pages/playground/utils.ts

+  if (parseError) {
+    throw new Error("Invalid attributes");
+  }
+  const { data, success } = llmAttributesSchema.safeParse(parsedAttributes);


the zod stuff is pretty nice, but i think we'll need to make the schemas purely optional and then check for variants, just input.value, input_messages with contents, prompt templates etc. etc. Given that i'm not sure how this will look but i think it makes sense to stick with for now

You might be right that it doesn't really provide a ton.

If you need to test variants of even wildly different schemas you can use a zod union. Each entry of the union will be tested against separately and the first one that passes wins.

Meaning you can make a schema for each variant and accept any of them via a zod union, and then further refine the type when you need to pass it somewhere specifically.

The union schema can be required meaning that at least one of the members must match in order to validate

Oh nice, yeah was going to take a look at unions and see what they had to offer

app/package.json

mikeldking · 2024-10-08T01:41:52Z

app/src/pages/playground/SpanPlaygroundPage.tsx


-  return <Playground />;
+  if (!span) {
+    throw new Error("Invalid span");


Can we avoid this with @required? If not, maybe throw a more user-friendly message

It's actually not nullable, but above this we return null in the event that the __typename is not Span, this is because of relay adding other to the type to be safe, will update the error message but should never happen

// This will never be '%other', but we need some // value in case none of the concrete values match. readonly __typename: "%other";

ah makes sense

app/src/store/playgroundStore.tsx

app/src/pages/playground/schemas.ts

app/src/pages/playground/types.ts

mikeldking · 2024-10-08T01:51:44Z

app/src/pages/playground/utils.ts

+  if (parseError) {
+    throw new Error("Invalid attributes");
+  }
+  const { data, success } = llmAttributesSchema.safeParse(parsedAttributes);


You might be right that it doesn't really provide a ton.

app/src/pages/playground/Playground.tsx

mikeldking · 2024-10-08T18:44:27Z

app/src/pages/playground/__tests__/playgroundUtils.test.ts

+    expect(getChatRole("assistant")).toEqual("ai");
+    expect(getChatRole("bot")).toEqual("ai");
+    expect(getChatRole("system")).toEqual("system");
+    expect(getChatRole("human:")).toEqual("user");


Suggested change

expect(getChatRole("human:")).toEqual("user");

expect(getChatRole("human")).toEqual("user");

ahh i had this intentionally to check the includes vs. strict matching

app/src/pages/playground/playgroundUtils.ts

mikeldking · 2024-10-08T18:58:36Z

app/src/pages/playground/SpanPlaygroundPage.tsx


-  return <Playground />;
+  if (!span) {
+    throw new Error("Invalid span");


ah makes sense

app/src/store/playgroundStore.tsx

…round span replay (#4906) * begin building zod schemas for llm attributes * feat(playground): parse span input messages * move to utils, update role * update initial instance id * update id generation, add todo, add test files * add tests * memoize * move jest-canvas-mock to dev dependencies * update span not found error message * parse roles as strings and corece to ChatMessageRole * update ChatRoleMap comment * fix typos * update prop type to be InitialPlaygroundState * update naming * fix naming conflict

Parker-Stafford force-pushed the parker/4782-transform-spans-for-playground branch from 182c053 to f01acd4 Compare October 7, 2024 19:24

Parker-Stafford commented Oct 8, 2024

View reviewed changes

Parker-Stafford marked this pull request as ready for review October 8, 2024 01:14

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Oct 8, 2024

mikeldking reviewed Oct 8, 2024

View reviewed changes

app/package.json Outdated Show resolved Hide resolved

mikeldking reviewed Oct 8, 2024

View reviewed changes

Parker-Stafford force-pushed the parker/4782-transform-spans-for-playground branch from 1ee44b7 to 458ee2d Compare October 8, 2024 16:41

mikeldking reviewed Oct 8, 2024

View reviewed changes

app/src/pages/playground/Playground.tsx Outdated Show resolved Hide resolved

Parker-Stafford added 13 commits October 8, 2024 15:15

begin building zod schemas for llm attributes

737f459

feat(playground): parse span input messages

8712740

move to utils, update role

3a8217b

update initial instance id

32363fd

update id generation, add todo, add test files

60cfda9

add tests

ab709ad

memoize

fd7b438

move jest-canvas-mock to dev dependencies

0f038c4

update span not found error message

a1535fa

parse roles as strings and corece to ChatMessageRole

bd3f6cc

update ChatRoleMap comment

7ffe592

fix typos

75c4389

update prop type to be InitialPlaygroundState

4b357be

Parker-Stafford force-pushed the parker/4782-transform-spans-for-playground branch from 80e35d7 to 4b357be Compare October 8, 2024 22:15

mikeldking approved these changes Oct 8, 2024

View reviewed changes

Parker-Stafford added 2 commits October 8, 2024 15:38

update naming

27e6101

fix naming conflict

467c5fa

Parker-Stafford merged commit 13e7312 into playground Oct 8, 2024
7 checks passed

Parker-Stafford deleted the parker/4782-transform-spans-for-playground branch October 8, 2024 22:51

Parker-Stafford mentioned this pull request Oct 9, 2024

[playground][ui] transform spans for playground #4782

Closed

mikeldking mentioned this pull request Oct 16, 2024

chore(main): release arize-phoenix 5.3.0 #4950

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(playground): parse span attribute `llm.input_messages` for playground span replay #4906

feat(playground): parse span attribute `llm.input_messages` for playground span replay #4906

Parker-Stafford commented Oct 7, 2024 •

edited

Loading

Parker-Stafford Oct 8, 2024

Parker-Stafford Oct 8, 2024

mikeldking Oct 8, 2024

cephalization Oct 8, 2024

Parker-Stafford Oct 8, 2024

mikeldking Oct 8, 2024

Parker-Stafford Oct 8, 2024

mikeldking Oct 8, 2024

mikeldking Oct 8, 2024

mikeldking Oct 8, 2024

Parker-Stafford Oct 8, 2024

mikeldking Oct 8, 2024

	expect(getChatRole("human:")).toEqual("user");
	expect(getChatRole("human")).toEqual("user");

feat(playground): parse span attribute llm.input_messages for playground span replay #4906

feat(playground): parse span attribute llm.input_messages for playground span replay #4906

Conversation

Parker-Stafford commented Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat(playground): parse span attribute `llm.input_messages` for playground span replay #4906

feat(playground): parse span attribute `llm.input_messages` for playground span replay #4906

Parker-Stafford commented Oct 7, 2024 •

edited

Loading