From 59c599fc9d97d28ed630b584947afad5f30f75fe Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 26 Feb 2024 22:32:10 -0500 Subject: [PATCH 001/239] Fix model routing (#161) From a92bdaeaffcb18936b66ff54dcfdc38fd8e86f9a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 27 Feb 2024 12:02:26 -0500 Subject: [PATCH 002/239] [UI] Add new span tree + viewer to Flow details page (#164) From 8fb914470a616882242264e0085cd1818be2240d Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 27 Feb 2024 23:33:43 -0500 Subject: [PATCH 003/239] Fetch models from API (#174) From 7bf826c7199e9cc4869c4d3cef29d89037d7f2ec Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Wed, 28 Feb 2024 11:04:07 -0500 Subject: [PATCH 004/239] Backend errors (#163) Display errors in the Prompt Playground component after receiving issues from backend From ca94c4507f76c7361521ae21b767c352c959a784 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 28 Feb 2024 13:11:57 -0500 Subject: [PATCH 005/239] [UI] Cleanup unimplemented pages from navbar (#180) From 9d90ca04205d4d497ae7f7419c19d2c2b1eda6ac Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 28 Feb 2024 13:13:28 -0500 Subject: [PATCH 006/239] [UI] Increase max-height of flow input/output (#179) Also update styles for running + error statues in output box. From ed6ab19b77a58e85c5dca0adbfc432c617d5e879 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Wed, 28 Feb 2024 10:57:00 -0800 Subject: [PATCH 007/239] Move flow runner to Actions page (#176) From 95b4c609a2fcd12385caa509ed460c21479a60e4 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 28 Feb 2024 14:38:00 -0500 Subject: [PATCH 008/239] [UI] Fix overflow of execution span tree (#183) From 8977bb7d2c2e051fe5b2acb770c38b834db16b9e Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Wed, 28 Feb 2024 16:31:39 -0500 Subject: [PATCH 009/239] Input validation disables prompt run button (#182) Input validation for prompt playground From 9f53ddefd1bd71a483f70b367a44c40911fe9a18 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 29 Feb 2024 09:10:12 -0500 Subject: [PATCH 010/239] Route playground from flows to action runners page (#191) From c145f96eeffb5de34be05b66ff93b1bcbe8e1224 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 29 Feb 2024 10:14:42 -0500 Subject: [PATCH 011/239] Switch temperature to the slider (#195) From f3c9593cc1cac67f96b11462e000cd2e291c5258 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 29 Feb 2024 11:07:11 -0500 Subject: [PATCH 012/239] Show validation errors on the playground (#196) From 597607874258bd937f454d6bcca295e0d5f49689 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 29 Feb 2024 11:20:54 -0500 Subject: [PATCH 013/239] [UI] Revamp flow details page layout (#197) From 0cd9569da8cc563f679b87f668f6411c6caf5d2b Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 29 Feb 2024 09:13:28 -0800 Subject: [PATCH 014/239] Fix validator issues (#194) From 0a35f0071dd7f5d78ab4ea9846df38adec505b6f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 29 Feb 2024 13:34:36 -0500 Subject: [PATCH 015/239] [UI] Initial design of span details view (#199) From c7dfaef1d50860223332cd4e0520181f52d5fa24 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 29 Feb 2024 11:09:14 -0800 Subject: [PATCH 016/239] Move flow runner to start from action-list instead of action-runner (#200) From f99d7066a7686c1a74841495b1d1de34c10462c6 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 29 Feb 2024 14:13:12 -0500 Subject: [PATCH 017/239] Add vertex-ai to the model playground (#201) Also add icons for all known action types From 500a3c716a18fc3ea98bf4e60bd9c8904bc4daeb Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 29 Feb 2024 15:54:09 -0500 Subject: [PATCH 018/239] [UI] Hide input/output pre if none available (#204) From d4083e4601d6049ff125f6b4ccf351e2c3c34f45 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 29 Feb 2024 16:36:30 -0500 Subject: [PATCH 019/239] [UI] Add "muted" helper class for secondary text (#206) From 88e9d357c297db3e6244d675633d7c065915051d Mon Sep 17 00:00:00 2001 From: Pavel Jbanov Date: Sun, 3 Mar 2024 18:19:10 -0500 Subject: [PATCH 020/239] Don't send blank stop sequences to the model, vertex gemini model doesn't like it (#217) From 0cf462119f53a949d9ceb378dcdb3c280c2a5b8e Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Tue, 5 Mar 2024 16:33:54 -0500 Subject: [PATCH 021/239] Provider specific model param restrictions on input (#224) From 515e66d91c26f370ec9a53fdd113590c08e982aa Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 5 Mar 2024 22:34:01 -0500 Subject: [PATCH 022/239] Use the minfied version of Monaco Editor in the angular app (#242) From 157fa804d2287bb4d5e538b67e6bd49eaf1dcd0c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 11:15:15 -0500 Subject: [PATCH 023/239] [UI] Update app name From 3a55b2b174b8348ba0fe2ed9c7f38e7140d1cf6f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 11:26:24 -0500 Subject: [PATCH 024/239] [UI] Update flow details layout (#246) Also adds new `` shared component which adds a button to show text in a larger pop-up dialog. From 3ea185964b31e579689c1895522cfd046c5a1882 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 11:59:58 -0500 Subject: [PATCH 025/239] [UI] Add callout component (#244) From d9535ddad95fe6e02f21a218c0d90bc5cdf9e6c8 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 14:57:39 -0500 Subject: [PATCH 026/239] [UI] Hide wrapper spans on details page (#254) From 7e8699f52164a18e87bd9e8596cb59456397a43a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 16:48:23 -0500 Subject: [PATCH 027/239] [UI] Update flow durations on details page (#256) From 64f4c629dc5919c598af242cf61b295f1b3f99d7 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 6 Mar 2024 19:44:59 -0500 Subject: [PATCH 028/239] [UI] Show error on flow details page (#258) From 6a48b65c9670289790ab9256ce1fb805d9204132 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 6 Mar 2024 22:02:53 -0500 Subject: [PATCH 029/239] Playground load trace (#262) * Code cleanup * Load playground from a trace From f4029a782cfda2fc7d596c49ef1847fa66ca5dce Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 7 Mar 2024 01:37:50 -0800 Subject: [PATCH 030/239] Add theme toggling for JSON editor and move schema to a tab next to the editor (#245) From 6e4519207b2ff264d4807329f0e5a729f181a08e Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 7 Mar 2024 10:30:39 -0500 Subject: [PATCH 031/239] Give topP the slider treatment (#264) It's only right, now that we've done temp. :-) From 9853760dddfdb2ccdf81046e4732a0da172c170c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 7 Mar 2024 13:40:05 -0500 Subject: [PATCH 032/239] [UI] Show flow name in tree (#266) From 52692ec874346a90a9a4136336f5f65ac235c334 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 7 Mar 2024 15:35:43 -0500 Subject: [PATCH 033/239] [UI] Show span state in details pane (#268) From 98b4a1cb72cf25dfc8a5f541de5f6b38d1dd7318 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 7 Mar 2024 16:26:26 -0500 Subject: [PATCH 034/239] [UI] Flows table style improvements (#269) From 656d3ccc6f03239a465525ade655f424197cecc5 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 7 Mar 2024 16:54:55 -0500 Subject: [PATCH 035/239] [UI] Small flow details page improvements for narrow screens (#273) From cf3991f11d6dc8ff559e3516fecd5b12b3a1251e Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 7 Mar 2024 21:44:31 -0500 Subject: [PATCH 036/239] Add CustomOptions (#276) Also, add stop sequences to the request. From d7a86f0e2766d30c50564b2e80c2641a077f9bcb Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 7 Mar 2024 20:53:27 -0800 Subject: [PATCH 037/239] [UI]Remove sample calls for unsupported actions. Small fixes in flow runner. (#275) From 172148ba9326e954ef9c8825af00730840d844c0 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Fri, 8 Mar 2024 13:11:14 -0500 Subject: [PATCH 038/239] Create Message sub component for ModelPlayground (#271) #148 From 29d353a8731d0b691bc50b589e56ced8259bc709 Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Fri, 8 Mar 2024 13:46:35 -0500 Subject: [PATCH 039/239] Fix error with model not accepting request_format (#279) From 877d3c7e400fe239a03137219a72605999d6d557 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Fri, 8 Mar 2024 14:19:57 -0500 Subject: [PATCH 040/239] Disable the minimap on the monaco editor (#286) From 6f5279ea5ed1aaa8c95fa17103f6ae37d249817f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sat, 9 Mar 2024 16:12:26 -0500 Subject: [PATCH 041/239] [UI] Add zero state for flows list page (#291) From 6a932adc07c103a32791e1edbdaafeae757efe8a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 11 Mar 2024 10:59:38 -0400 Subject: [PATCH 042/239] [UI] Fix ng error in flow runner (#297) From b4a2019afa64372af38f43e1a8556b372c59c664 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 11 Mar 2024 14:08:52 -0400 Subject: [PATCH 043/239] [UI] Hide stream response checkbox for durable flows (#299) From 56db2e2eacaab216c064dd6f188edd0e32b9eb88 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Mon, 11 Mar 2024 17:05:02 -0400 Subject: [PATCH 044/239] Integrating the Message component into the Prompt Playground From d527b5454c2d41ab1fb6beb56672327ee1c3c4d5 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 12 Mar 2024 09:18:31 -0400 Subject: [PATCH 045/239] Switch model select from native to mat-select (#306) From 7e99dfdeb5c6bd97377dc21a2d19f49926a57644 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 12 Mar 2024 09:30:15 -0400 Subject: [PATCH 046/239] Ability to show errors on actions page (#307) From 6c6633ab6898b8d914265d017158c386076414e0 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 12 Mar 2024 10:01:34 -0400 Subject: [PATCH 047/239] [UI] Revamp Actions list UI (#308) From 079029ed8251751c63d917c1ba498e064a91671b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 12 Mar 2024 10:11:41 -0400 Subject: [PATCH 048/239] [UI] Remove unnecessary return (#309) From a8f58c734fecbae2b1b35480494509b16a5ee636 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 12 Mar 2024 10:32:38 -0400 Subject: [PATCH 049/239] [UI] Prevent selecting action if no param is set (#310) From 6cba77e6277937fd036e07b679e28059effe72ae Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Tue, 12 Mar 2024 16:11:10 -0400 Subject: [PATCH 050/239] Enable support for multiple messages coming from traceId (#314) From 490e1739a988d9ea931606a52d28a1d465c928c9 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Tue, 12 Mar 2024 15:34:32 -0700 Subject: [PATCH 051/239] Avoid making flow runner editors read only (#321) From 8848671108f344cdd56c9e7d4b56f7aa5b196dfe Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 12 Mar 2024 20:02:43 -0400 Subject: [PATCH 052/239] [UI] Add filtering and expand/collapse all to actions list (#319) From 85b739d16d12e6c11990367645e4a5a510ce7c15 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 12 Mar 2024 21:42:39 -0400 Subject: [PATCH 053/239] Fix error where model selection does not update (#323) From 81ab1f9bf1c2499b46bb80fbb1852b2ecfb91880 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 13 Mar 2024 01:00:27 -0400 Subject: [PATCH 054/239] [UI] Fix action search input style (#325) From bfde2023a712250f7a82065e81c8c7cc4a797fe5 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 13 Mar 2024 10:52:49 -0400 Subject: [PATCH 055/239] [UI] Update action list name and key display (#328) From abf3a164aca610084ebcd1184d908e3c6aa9632d Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 13 Mar 2024 11:26:19 -0400 Subject: [PATCH 056/239] User error callout component on model playground (#330) From 8434a9c354d3c4118776c512bd5753ea69ebba38 Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Wed, 13 Mar 2024 11:58:47 -0400 Subject: [PATCH 057/239] refactor the code around checking for json output support (#304) From b62657a28a36b3c354326edfd8c83fe2da2600d6 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 13 Mar 2024 14:46:32 -0400 Subject: [PATCH 058/239] Render images in chat (#340) From d9c8978ecbaf9345a4f328a30e6c672c8d088add Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Wed, 13 Mar 2024 15:30:18 -0400 Subject: [PATCH 059/239] Functioning add and remove button (#335) From 7715a8cb2ec1b940a7d6a5ffab650ef23d6c0433 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Wed, 13 Mar 2024 15:45:28 -0400 Subject: [PATCH 060/239] Refactor criteria/validation logic out of playground component (#339) From 149ca45501440596ae9f5d69b3b595e3cec29dec Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 13 Mar 2024 17:14:14 -0400 Subject: [PATCH 061/239] [UI] Flow runner UI polish + improvements (#343) From 320062a67a81208228587ca2675a78243ed4bee1 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Wed, 13 Mar 2024 15:00:32 -0700 Subject: [PATCH 062/239] Move JSON editor to shared components since retriever playground also needs it (#344) From 1b1401ee5460a42b72c0d0eeb9e44089389e7d8b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 13 Mar 2024 21:02:48 -0400 Subject: [PATCH 063/239] [UI] Small handful of UI nit fixes (#345) From 317b79db9b55ed72c44f60a6e3a64623b350f948 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 13 Mar 2024 23:03:09 -0400 Subject: [PATCH 064/239] [UI] Add loading state to flows table (#349) From bf047aaf54c09929c0126362ee029f8349403241 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 14 Mar 2024 10:24:26 -0400 Subject: [PATCH 065/239] Do not load output from trace; typically we're interested in loading up the inputs, and re-running to get the output (#347) From bce99de26446dc41e3366f359cae526e9e0c801c Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 14 Mar 2024 11:13:26 -0400 Subject: [PATCH 066/239] Make response_format optional (#350) From bbd698c052bc353ea3176b5357546e1e2ec29f9f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 14 Mar 2024 13:20:02 -0400 Subject: [PATCH 067/239] [UI] Add Genkit icon (#371) From 75b87c2943ea138724327473f8d254faaa382f43 Mon Sep 17 00:00:00 2001 From: Pavel Jbanov Date: Thu, 14 Mar 2024 15:59:31 -0400 Subject: [PATCH 068/239] Reset streamed chunks when rerunning the streamed flow (#379) From 37e9934ac450359aaa2a232b42d0a196915a99e5 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 14 Mar 2024 16:57:16 -0400 Subject: [PATCH 069/239] [UI] Add tooltips to span state icons (#351) From adc4b5fff38cb566c52400506b69a732c5115996 Mon Sep 17 00:00:00 2001 From: tagboola Date: Thu, 14 Mar 2024 17:03:52 -0400 Subject: [PATCH 070/239] Prefer includes over contains (#376) Contains causes a `TypeError: _i.contains is not a function` when running evals. From 55802314c7e0ca156f0406837b18e6bb4ab57c11 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 14 Mar 2024 17:06:15 -0400 Subject: [PATCH 071/239] [UI] Add inspect flow state button if flow errors (#382) From dd1d92bdd71bdc04530787e6184ebc66ed910a5d Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Fri, 15 Mar 2024 11:13:30 -0400 Subject: [PATCH 072/239] Chat mode (#391) From 7280681423f6234696ca7564ae9244098ec8a2d8 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Fri, 15 Mar 2024 11:53:48 -0400 Subject: [PATCH 073/239] Ability to open Flow runner from the trace view (#394) From 09cc900f13bcc17a78acea6a7a1bdc862ca8a456 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 15 Mar 2024 16:29:07 +0000 Subject: [PATCH 074/239] Add basics of the eval runner page (#367) * initial ui changes * formatted * Add mocked evals page * Unnest runs * Remove evaluations tab from appbar From 9380cafe32e775e8513bde0c2087fa4fc52797bb Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 15 Mar 2024 13:06:35 -0400 Subject: [PATCH 075/239] [UI] Fix flow details sidebar colors in dark mode (#399) From af830fa8db13013a125a6b5dc681cff1451453ce Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 15 Mar 2024 13:11:37 -0400 Subject: [PATCH 076/239] [UI] Revamp model playground to chat-based layout (#397) From e0e0a4e785f678628b589cb2c077d179c38b6a7c Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Fri, 15 Mar 2024 12:59:35 -0700 Subject: [PATCH 077/239] [UI] Flow runner: Add a callout for no output so we dont show empty response boxes (#403) From 553817d6de06544e43ea3de7676a62dc05f5c4f5 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 15 Mar 2024 17:14:37 -0400 Subject: [PATCH 078/239] [UI] Add trace details view (#405) From a1d337f82bb6b910695fa8e7af77bcd4795aeef5 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Mon, 18 Mar 2024 10:04:24 -0400 Subject: [PATCH 079/239] role:system message allowed for models (#402) From a2b2572b02d1cb530c50c54709012aab7b8c4d40 Mon Sep 17 00:00:00 2001 From: Michael Bleigh Date: Mon, 18 Mar 2024 12:52:33 -0700 Subject: [PATCH 080/239] Adds support for image models. (#426) From 635fb79586cec7c33e02d5199ed845203a7c5e22 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Mon, 18 Mar 2024 15:55:53 -0400 Subject: [PATCH 081/239] fix playground runner after runAction change (#429) From e03d359114085327c603d3c3a30aca1f98f286a1 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Mon, 18 Mar 2024 16:38:12 -0400 Subject: [PATCH 082/239] Revert "fix playground runner after runAction change (#429)" (#431) This reverts commit 82264c0777dd47b0835dda01362a902298ec044b. From 549ffbea09bd52b39e7c571df9e876837c036b4f Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 18 Mar 2024 21:04:26 -0400 Subject: [PATCH 083/239] Small tweaks to model playground to reduce chat (#438) input clutter From 535040a744485b4c7cac30d35a3198085b6621aa Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 00:17:45 -0400 Subject: [PATCH 084/239] [UI] Update `stackTraceSpans` to filter out internal spans (#439) From 0d597c599414a1dd745fd90234ba87188dc20199 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 10:08:15 -0400 Subject: [PATCH 085/239] [UI] Add traces table to inspect index page (#448) From 9be4c2f41a6838cc51cbe2300db5c433163ce307 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Tue, 19 Mar 2024 10:29:59 -0400 Subject: [PATCH 086/239] Adding traces to Messages (#432) From f260f2321ad84e5e3866519ff508224573ea12c2 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 10:47:24 -0400 Subject: [PATCH 087/239] [UI] Update routing for inspect pages (#449) From 8ff5d6b59fead8fce579ecdb0982abbc37a989ca Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 10:55:05 -0400 Subject: [PATCH 088/239] [UI] Update routing for run pages (#450) From e0bd58275ba0516313e86e150356770276510f71 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 11:06:21 -0400 Subject: [PATCH 089/239] [UI] Fix trace display name in table (#451) From 1d63de4532d3aa5872e32a6306fe4b6ac0e51f4a Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 19 Mar 2024 11:13:39 -0400 Subject: [PATCH 090/239] Allow size to be optional (#452) Model returns error otherwise: 400 None is not of type 'string' - 'size' From 0d17f05ac90c815884cb0e57adceff22c291d561 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 12:00:48 -0400 Subject: [PATCH 091/239] [UI] Fix trace deep links in model playground (#453) From 111be8345098150e3a6c71c9761ba6e929c19dd2 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Tue, 19 Mar 2024 16:25:51 +0000 Subject: [PATCH 092/239] [UI] Add raw mat-table for evals view (#430) * initial ui changes * formatted * Add mocked evals page * Add mocked table prelim * tests * Use EvalResult for now * feedback changes From 4131f0373bac7e278256d71d8a73539a8e2670c9 Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Tue, 19 Mar 2024 12:27:45 -0400 Subject: [PATCH 093/239] Add embeddings models (#303) From a818995423963c81cfec13008c3c85a56040fcca Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 12:39:22 -0400 Subject: [PATCH 094/239] [UI] Update /evaluations route to /evaluate (#454) Matches other verb-based top-level routes. From 76f401d10107144c433abe3575b2fe1c9e82a291 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 13:25:45 -0400 Subject: [PATCH 095/239] [UI] Make all run buttons consistent in playgrounds (#455) From 1df184db57eaa68e4f72014348676c4830cdb098 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 13:34:04 -0400 Subject: [PATCH 096/239] [UI] Add cmd/ctrl + enter shortcut to playground editors (#456) From 7547bf74f0214be46028d0c22f986b720913786b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 19 Mar 2024 15:44:21 -0400 Subject: [PATCH 097/239] [UI] Add landing state for Run page (#465) From 1c544fb424a4d595d214b4ae8c7b7e83be53845e Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 08:08:43 -0400 Subject: [PATCH 098/239] [UI] Prevent mat-slider from shrinking (#473) From 40db8775049b7c2b840aae77cad3c7d89ed6b10e Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 08:21:42 -0400 Subject: [PATCH 099/239] [UI] Adjust element widths for narrow browsers (#474) From 696ddd7bf9118f2977633e1baaf06a5e9a08471a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 10:40:38 -0400 Subject: [PATCH 100/239] [UI] Prevent welcome page flicker on action refresh (#475) From d387807817115eb40dfceb877e6d4892b3ed9599 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Wed, 20 Mar 2024 11:00:42 -0400 Subject: [PATCH 101/239] Add tab for Auth input to Flow Runner action (#467) From 5a5142d389983641d080d5e8e4cd0c9667b86362 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 14:34:43 -0400 Subject: [PATCH 102/239] [UI] Add JSON sample to flow runner (#479) From 6fb2159460a57eea2ef7dda72527a9487d346d90 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 20 Mar 2024 15:19:10 -0400 Subject: [PATCH 103/239] Generic action runner (#484) From a73c32ff33ce21a6339fc617a1e6749e5eb3ac9a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 16:04:34 -0400 Subject: [PATCH 104/239] [UI] Add support for tool primitive on dev UI run page (#488) From e8c41a13e28c08f041569fa1cf8d6c3f287879a1 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 16:05:46 -0400 Subject: [PATCH 105/239] [UI] Tighten up spacing of actions list items (#489) From dbd97a257e1f6a206c07244f79ea6d4230efa32a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 17:12:51 -0400 Subject: [PATCH 106/239] [UI] Trigger change detection on flow runner response (#486) From 4f999f661d9e8079fbd49a06c221d83b357e868c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 17:45:31 -0400 Subject: [PATCH 107/239] [UI] Add cmd/ctrl + enter shortcut to model playground (#485) From cbe9fe1dae79941a5a62fc77e9df79b4ea61276d Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Wed, 20 Mar 2024 16:04:42 -0700 Subject: [PATCH 108/239] [UI] Update eval results UI to use expandable cards for results (#491) From 312370c5b02a30026cf1362ba1e13606db3c156d Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 20:15:25 -0400 Subject: [PATCH 109/239] [UI] Prevent scrolling past last line in monaco editor (#495) From c9b39d9451a31c3ef9fd2306f40a1d5e2b9c35c6 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 20 Mar 2024 23:08:50 -0400 Subject: [PATCH 110/239] [UI] Use helper class to style pre stacktrace in callout (#502) From 07a6e3312c46d0717cc9ab913a9ca57e7dfa9357 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Wed, 20 Mar 2024 22:37:27 -0700 Subject: [PATCH 111/239] [UI]Evals UI: Update inputs to use a table format (#496) From 93d4749c548b29fcf61e36505c3c7fa2f87c297c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 21 Mar 2024 15:11:07 -0400 Subject: [PATCH 112/239] [UI] Model playground message styling polish (#515) From f9b6eb73b416874be40c057ab0058a31884e7036 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 21 Mar 2024 16:01:48 -0400 Subject: [PATCH 113/239] [UI] Fix json editor to ignore initial value if no schema (#517) From b8da2070701c657eb60a3d6b6f26aad63615470b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 21 Mar 2024 16:03:44 -0400 Subject: [PATCH 114/239] [UI] Set retriever name in playground header (#518) From d3cc8ab84d38b9af10a105e2f609e230b820d695 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 21 Mar 2024 16:11:39 -0400 Subject: [PATCH 115/239] [UI] Prevent JSON sample pre-fill if unnecessary (#520) From b98183d8a66946abe07a6f0707d8aa101312f1c4 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 22 Mar 2024 15:56:52 +0000 Subject: [PATCH 116/239] Remove fdescribe in tests (#532) From 8f65d987a03b70f3a8fc2e73e029ea18ba884a6d Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 22 Mar 2024 19:03:53 +0000 Subject: [PATCH 117/239] Fix minor UI elements in eval page (#533) * WIP Eval UI changes * Clean scss * simplify name getter * trigger checks again * undo From 9d57c8bf6036ce13dbd61a542b3340eef45d849b Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 22 Mar 2024 19:56:50 +0000 Subject: [PATCH 118/239] Add inspect trace option (#540) * WIP Eval UI changes * Clean scss * WIP add inspect button * Add inspect button * Add inspect button * remove target * Use links instead of button * remove unused dep From 262521cf15a5acb97ad3e5ba74d8c86ca581a646 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 22 Mar 2024 20:59:36 +0000 Subject: [PATCH 119/239] Add inspect tab in the Dev UI (#546) * WIP Eval UI changes * Clean scss * WIP add inspect button * Add inspect button * Add inspect button * remove target * Use links instead of button * remove unused dep * Add evaluation tab * Update messaging From b2a417afe77558241e09ede35065e0bdaaeb4104 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 22 Mar 2024 22:18:52 +0000 Subject: [PATCH 120/239] hide inspect button if no traces (#548) From dd061982674b647d27651acd4e784e8c4995e0c9 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sat, 23 Mar 2024 18:23:21 -0400 Subject: [PATCH 121/239] [UI] Add typewriter effect to welcoem message (#554) - Also include missing Google Sans fonts From 390e6202cba0d0e59c8f58d30109b5ec60b3e56c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sat, 23 Mar 2024 23:07:00 -0400 Subject: [PATCH 122/239] [UI] Tweak logo kerning (#555) From ea1f2c2d48a583ca2f02e8516054cf88a2999898 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sun, 24 Mar 2024 13:50:40 -0400 Subject: [PATCH 123/239] [UI] UI polish for evaluate page (#553) From 107b0a8a0077318b92b3e03029a50294df0ab786 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sun, 24 Mar 2024 18:39:40 -0400 Subject: [PATCH 124/239] [UI] Fix issue in action runner JSON pre-fill (#559) From e2679561c425ec4c92e5cbfbc0f55f6753b8df04 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 11:17:11 -0400 Subject: [PATCH 125/239] [UI] Update typewriter animation to move left-to-right (#560) From ad1d4ecfd91a7efb51ea9f641292ed991185e630 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 12:22:38 -0400 Subject: [PATCH 126/239] [UI] Show custom metadata attributes last in span details (#563) - Also move span duration logic to shared util function and show seconds if > 1000ms. From a57a774a8257a2bfa0e8f51439aaca19dcbc98f6 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 12:44:32 -0400 Subject: [PATCH 127/239] [UI] Polish for eval result details pane (#564) From b0e1bfa48e48665fbceecac6b8e33ed951eeae26 Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Mon, 25 Mar 2024 13:09:17 -0400 Subject: [PATCH 128/239] Add support for text-embeddings (#538) From f2328a9ba229c8537d99a7fdbcea99b9adfa9c88 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 14:30:24 -0400 Subject: [PATCH 129/239] [UI] Update default font to Google Sans (#565) From f60139b178729acaba5e7e2621e50f2101ce8d0f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 16:21:49 -0400 Subject: [PATCH 130/239] [UI] Update span attributes styling (#568) From 00fa2c4239187655e2948a921788f9798fb53b4b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 17:17:30 -0400 Subject: [PATCH 131/239] [UI] Update border radius globally (#573) From 310ea7581f078b1bf9fc727dc58e2cf2e04fabe4 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 17:28:07 -0400 Subject: [PATCH 132/239] [UI] Clip model playground message loading bar to card radius (#576) From fbfff643f475e25ae62c1baccdbe46745fe13593 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 25 Mar 2024 17:54:34 -0400 Subject: [PATCH 133/239] [UI] Prevent shrinkage of breadcrumb chevron (#577) From 541ac5cadcaaa07f66d38fce356e5869d6d985f6 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 10:13:40 -0400 Subject: [PATCH 134/239] [UI] Upgrade angular deps to ^17.3.1 (#587) From 14ef01f740e3a2590f913586bcdfab4aae61961f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 10:50:38 -0400 Subject: [PATCH 135/239] [UI] Add logo lockup to app bar (#588) From 80dff699983099594f01fbfa41bd3f14947b8c5a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 16:43:20 -0400 Subject: [PATCH 136/239] [UI] Fix table not rendering for errored traces (#607) From ae1aaa0fbfec596b64b7a831c465180a5c08a4e2 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 16:53:18 -0400 Subject: [PATCH 137/239] [UI] Render base64-encoded images in span output (#606) From db62bf91f149f53b74d9253e319cd605c94751b4 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 16:53:35 -0400 Subject: [PATCH 138/239] [UI] Update label of expand text button (#608) From 6d16d98e8f01088660112003e094fd21cb79abc9 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 26 Mar 2024 22:31:08 -0400 Subject: [PATCH 139/239] [UI] Update lockup with new svg asset (#623) From c940ee765f03835eb5e057b65583a4769d6b46e5 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Wed, 27 Mar 2024 17:42:37 +0000 Subject: [PATCH 140/239] [Eval bugbash] Update tooltip to definitions, visible on entire chip (#624) * Update tooltip to definitions, visible on entire chip * typos From 091c88aa99be2c094d60181582a0546f31b86b34 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Wed, 27 Mar 2024 19:47:51 +0000 Subject: [PATCH 141/239] [Eval bugbash] Show errors as errors in eval UI (#626) * Update tooltip to definitions, visible on entire chip * typos * Mark errors as errors * use ngIf * Add TODO From db41f3e6fda02c80bd8f3848a9df7387794936aa Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Wed, 27 Mar 2024 20:30:52 +0000 Subject: [PATCH 142/239] [Eval bugbash] Only show icon if failed evaluator (#635) * Update tooltip to definitions, visible on entire chip * typos * WIP icons * Remove unused From 3721208529b65f27d4797174ced00102853a668e Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 27 Mar 2024 16:39:32 -0400 Subject: [PATCH 143/239] [UI] Fix trace timing display now that they are millis (#638) From 445c370933241ffbd899b4c576f3d07f428d77c8 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 28 Mar 2024 10:22:02 -0700 Subject: [PATCH 144/239] [UI] Fix JSON editor to show up for optional inputs as well (#613) From 177cb2aae96e661023ff149e60db33c2a055cfee Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Thu, 28 Mar 2024 13:46:55 -0400 Subject: [PATCH 145/239] Add trace id to model playground when error occurs (#631) From 802569039ee024f281818bdf57f65191f44fd38e Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 28 Mar 2024 11:44:58 -0700 Subject: [PATCH 146/239] Display context strings separately instead of a big array (#658) From c757cec0464c3ea13932fa016cde92f2594859c6 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 28 Mar 2024 12:16:41 -0700 Subject: [PATCH 147/239] [UI]: Update date format to medium (#659) From a4b7eaa85edf00bdf69f57f2b86b05147678e5e2 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Fri, 29 Mar 2024 00:56:41 +0000 Subject: [PATCH 148/239] Update error tooltip (#665) * Update error tooltip * typos * Show error message if available From 917e328a9c6dbd08a76b934982afc83a963e7573 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 28 Mar 2024 23:19:51 -0400 Subject: [PATCH 149/239] [UI] Tighten up kerning on mat tab labels (#680) From 4527f4b1eb7c7f157328ac5671b0603b188e5196 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 28 Mar 2024 23:55:19 -0400 Subject: [PATCH 150/239] [UI] Allow resizing of .pre-container and json editor (#682) From 494905d9648bea966d6a83e56066051cc2af4586 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 07:04:10 -0400 Subject: [PATCH 151/239] [UI] Add tooltips to temperature and top_p controls (#683) From caff28355fc4f722636ddec79d6e4ea000727a1d Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 08:53:03 -0400 Subject: [PATCH 152/239] [UI] Fix JSON sample autofill in retriever playground (#684) From d6116d1a80131dd12924a998c8ea7b0a193e1db8 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 11:07:43 -0400 Subject: [PATCH 153/239] [UI] Improve model playground param labels and add tooltips (#686) From 329b44cbc70089b666d61cf06946be3954d9534f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 11:30:21 -0400 Subject: [PATCH 154/239] [UI] Fix trace status in table (#687) From 3295717277f1aa5267479541be8dcda9fde7220c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 11:39:21 -0400 Subject: [PATCH 155/239] [UI] Update model icon to sparks (#688) From 40233d2c4eaf31482ffbfb818023bb0c304cefa0 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 12:14:19 -0400 Subject: [PATCH 156/239] [UI] Add action type to runner page title (#690) From a5b805df5781a5e236f8a42a6d3ea7ef6eae3943 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 12:37:02 -0400 Subject: [PATCH 157/239] [UI] Add title and close button to expand text dialog (#691) From 40004871b9998e2500b6e678cb16f7d91e508f20 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 29 Mar 2024 12:42:58 -0400 Subject: [PATCH 158/239] [UI] Remove redundant title from action runner (#692) From 465951cbb9b586e6322a33a26285c259c15ef51e Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Fri, 29 Mar 2024 14:29:34 -0400 Subject: [PATCH 159/239] Pass thru options to API (#695) From a20b4435b6a1896c3cf23b0c1781eeb943f3e527 Mon Sep 17 00:00:00 2001 From: Pavel Jbanov Date: Sat, 30 Mar 2024 14:32:37 -0400 Subject: [PATCH 160/239] Bump ragas to 0.0.6 (#719) From 5f86cb6d1be3b0df086e1baa6166d09bd9cf1137 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 1 Apr 2024 10:45:52 -0400 Subject: [PATCH 161/239] [UI] Cleanup system prompt styling in model playground (#725) From f471a7fa2bf9dc01692f2d4c849717aa6831826d Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 1 Apr 2024 12:54:17 -0400 Subject: [PATCH 162/239] Update system/message placeholders (#727) * Update placeholders * Update message.component.ts From 317289ff43b2454b04a6f512f66e1a5753bacda4 Mon Sep 17 00:00:00 2001 From: MaesterChestnut <40321652+MaesterChestnut@users.noreply.github.com> Date: Mon, 1 Apr 2024 15:46:21 -0400 Subject: [PATCH 163/239] Update Eval Error handling (#685) From 08eaae6f0ce094f9641a3d75d7c4a32fe7935dfe Mon Sep 17 00:00:00 2001 From: Max Lord Date: Tue, 2 Apr 2024 11:15:12 -0400 Subject: [PATCH 164/239] Clarifying label on button formerly known as "Open in Playground" (#636) - Label now says 'Open in flow runner', 'Open in model runner', etc. to make it more clear which step will be run. - Changing to secondary style button to make it look less like the action will be run immediately. From 82b31486dc37747ed10d00cb5f1e9d41a95db280 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 2 Apr 2024 12:35:49 -0400 Subject: [PATCH 165/239] [UI] Fix callout content not stretching to fit width (#757) From 44daaafc8bfb369f81ff201c9528c6e8875990f6 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Tue, 2 Apr 2024 12:13:04 -0700 Subject: [PATCH 166/239] [UI]: Add metrics table in evals results card (#747) From 9729e660fb14cc45b395170e9ea9e8ef63029904 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 2 Apr 2024 16:07:01 -0400 Subject: [PATCH 167/239] [UI] Add support for specifying model version in playground (#760) From 8bd8066d24351b7b5488d8d1197cd2a2179893be Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 2 Apr 2024 16:56:55 -0400 Subject: [PATCH 168/239] [UI] Remove Evaluate tab in top nav bar (#765) From 2f075f65f56e47a6e517f8e85093a229e69ef22f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 2 Apr 2024 17:33:56 -0400 Subject: [PATCH 169/239] [UI] Use flask icon for Evaluate tab (#772) From df14113fc10af80aee25e0f6c38898913079397e Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 11:46:56 -0400 Subject: [PATCH 170/239] [UI] Style updates to eval result details (#790) From 56389956e37a77b10be96161d92bf23570e9190f Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 14:05:30 -0400 Subject: [PATCH 171/239] [UI] Render eval metric name in error callout consistently (#792) From bd0e8bbd9dc2ed110d0e9ccf209ae1214659e0c1 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 14:22:08 -0400 Subject: [PATCH 172/239] [UI] Fix span duration display (#797) From 2b3b3829ce4b232b8104d004c2c541935e45df80 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 3 Apr 2024 15:44:46 -0400 Subject: [PATCH 173/239] Show safety errors in the model runner (#800) From fe788cf8b427c80cc45bc689b2ad9fc8b55ea845 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 3 Apr 2024 16:57:55 -0400 Subject: [PATCH 174/239] Rename model playground => runner (#803) From 0832e4feed0a004b0bd7cd5863d411fa9912e943 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 3 Apr 2024 18:22:28 -0400 Subject: [PATCH 175/239] Rename retriever playground => runner (#805) From 03c5d56a0364b01b3274af6485b1a1fb5dd76d60 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 20:53:43 -0400 Subject: [PATCH 176/239] [UI] Adjust metrics table to be full-width (#810) From cbe3eb3a1747acc7d4e293459f8b4562d21b7cb9 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 21:02:26 -0400 Subject: [PATCH 177/239] [UI] Only show eval zero state when loaded (#811) Prevents a quick distracting flash of the zero state when the page loads. From 4535531964160b538b9ca72bc243403b0b4c40ba Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 3 Apr 2024 21:16:48 -0400 Subject: [PATCH 178/239] [UI] Set All traces as default in Inspect view (#812) From 8759b3d95771fecbd7e720174c0c4febb525e997 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Thu, 4 Apr 2024 08:46:31 -0700 Subject: [PATCH 179/239] [UI] ThemeToggleService unit tests (#816) From 7b48c85bd53bd2ebe6f5b7ddbafc6c3209cc148b Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 4 Apr 2024 12:29:11 -0400 Subject: [PATCH 180/239] [UI] Make spans deep-linkable in trace + flow details views (#819) From 0605d3183b2c5e8e9ecadfb80f214568e018f3f0 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 4 Apr 2024 12:29:27 -0400 Subject: [PATCH 181/239] [UI] Update model runner title to use selected model in config (#822) From 09b818e00706d827afb4a9b942f30ed79cf05794 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 4 Apr 2024 17:32:40 -0400 Subject: [PATCH 182/239] [UI] Clear out images from data-rendered upon receiving new input (#840) From 4038e4c01afbe371bcd37edb3ff8331f9c6a195d Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 4 Apr 2024 23:22:35 -0400 Subject: [PATCH 183/239] [UI] Hide append mode for models that do not support multiturn (#847) From ccb036032b98188f6eef4e98b791b0bd806bb840 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 5 Apr 2024 00:27:05 -0400 Subject: [PATCH 184/239] [UI] Show banner for unsupported models (#848) From ce0f6c98a216e07b366968763ee94531468dc41c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 5 Apr 2024 10:14:43 -0400 Subject: [PATCH 185/239] [UI] Reset scroll position of input/output when switching spans (#852) From 79c4bc3840a27a172a87d4a96b53d91732068cba Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 5 Apr 2024 10:18:16 -0400 Subject: [PATCH 186/239] [UI] Hide "Add message" if model does not support multiturn (#853) From 7d937d4ed9fc879f1593773f1012a4ef861ad4e7 Mon Sep 17 00:00:00 2001 From: Pavel Jbanov Date: Fri, 5 Apr 2024 11:14:40 -0400 Subject: [PATCH 187/239] Fix missed version 0.5.0-rc.1 (#858) From 5a165f1b64b7ee3305892d54ce411f0dd2511bd0 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 5 Apr 2024 12:00:40 -0400 Subject: [PATCH 188/239] [UI] Fix display of system prompt (#860) From ddb6c0af8422f052dc92d36fc704342285dcdc6d Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 5 Apr 2024 12:36:26 -0400 Subject: [PATCH 189/239] [UI] Fix tools icon (#862) From 5c8a33e6f1113e2c54d2992babd78ed51a3f3f87 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sun, 7 Apr 2024 17:08:21 -0400 Subject: [PATCH 190/239] [UI] Prevent stuck browser back when redirecting to first evaluation run (#13) From 703fc06a73992ed3a44a280995c7bdf9f3550f22 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 8 Apr 2024 11:54:50 -0400 Subject: [PATCH 191/239] [UI] Add missing app text color style (#16) From 716bf1c4512bc2c715399169983553eb3e6299cf Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 8 Apr 2024 20:30:32 -0400 Subject: [PATCH 192/239] [UI] Apply theme to scrollbars (#20) From 998216f911f4ac692e896a34245acf9781edf1ca Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 10 Apr 2024 10:00:05 -0400 Subject: [PATCH 193/239] [UI] Clarify ID in flows/traces tables (#23) From 118ce13ac04ea1d810b1a969c7690fc1872311cc Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 10 Apr 2024 16:01:44 -0400 Subject: [PATCH 194/239] [UI] Show flow error in trace details view, if applicable (#28) From 54dce1b3ca3b2b19197ec66a318e594ac0f0cf37 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 10 Apr 2024 16:40:37 -0400 Subject: [PATCH 195/239] [UI] Fix eval zero state callout spacing (#24) From 45e208f43291437b46de6253251ec7f451d5c2d3 Mon Sep 17 00:00:00 2001 From: huangjeff5 <64040981+huangjeff5@users.noreply.github.com> Date: Thu, 11 Apr 2024 11:33:00 -0400 Subject: [PATCH 196/239] Export textEmbedding (#36) From 948f042db7a108c50904362859fcafcc7f9e44d2 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 11 Apr 2024 17:41:40 -0400 Subject: [PATCH 197/239] [UI] Update README doc with up-to-date instructions (#50) From 3600beba01ab99ae532368e2485d516bbf56225d Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 11 Apr 2024 21:00:47 -0400 Subject: [PATCH 198/239] [UI] Create skeleton prompt runner component (#54) Will serve as a base for prompt-specific runner features that we will add. From b153ed409d4b134b5993a6c6c70a78a6b35d6257 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 12 Apr 2024 05:01:04 -0400 Subject: [PATCH 199/239] [UI] Add icon to all view trace buttons (#57) From f1f5137eedc8eb673ee19d79a50fce5eca0b2045 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 12 Apr 2024 13:10:49 -0400 Subject: [PATCH 200/239] [UI] Show template in prompt runner next to input (#58) From fad6b0ca1bdf0d55b44de908608c2211be88bb0a Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 12 Apr 2024 13:23:52 -0400 Subject: [PATCH 201/239] [UI] Use button toggle group for inspect table filter (#56) From 4a59e04874f90494ec569af230df0672fb931607 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 12 Apr 2024 19:13:04 -0400 Subject: [PATCH 202/239] [UI] Update play icon for run/dispatch span states (#60) From 26bff0885c74ea26011d4fbd3a80f6671eac0968 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 15 Apr 2024 22:05:02 -0400 Subject: [PATCH 203/239] More sensible default model params (#65) From 768b09430132e37c6b87d0e68f9a591af53beb8f Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 15 Apr 2024 22:54:25 -0400 Subject: [PATCH 204/239] Always clear message when not in chat mode - otherwise if an error is shown, we'll still see the previous message. (#67) From 2125ce71c2d39184d64b8ab029f83ac2bd8f6f65 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Tue, 16 Apr 2024 11:03:03 -0400 Subject: [PATCH 205/239] [UI] Show raw prompt template in modal (#70) From cc5e0f75408cfba4cae6fd752abf1f0d4285886c Mon Sep 17 00:00:00 2001 From: Max Lord Date: Tue, 16 Apr 2024 12:28:19 -0400 Subject: [PATCH 206/239] Nesting user input in prompt runner (#72) From 547f028a016f2dafef20c36cd4f50e827d880968 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 17 Apr 2024 11:29:17 -0400 Subject: [PATCH 207/239] [UI] Add support for prompt variants (#74) From a4a0931948f56ccbd7da1fa993fe9fab69f60958 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 17 Apr 2024 12:11:03 -0400 Subject: [PATCH 208/239] Allow system role for Gemini 1.5 Pro (#85) Also removes references to OpenAI from UI. From 20cdbce9d717771ae689b727d5236535173a58bf Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 17 Apr 2024 13:59:18 -0400 Subject: [PATCH 209/239] Create modular component for a multi-modal message (#83) From 6a761f30f3241e03dab9e9d3176b3d1b2558dd35 Mon Sep 17 00:00:00 2001 From: ssbushi <66321939+ssbushi@users.noreply.github.com> Date: Wed, 17 Apr 2024 15:58:48 -0400 Subject: [PATCH 210/239] Update faithfulness to v0.1.7 (#87) * Update faithfulness to v0.1.7 * Update METADATA From d006dfe21c690a1eb4721f3f394842e8fc1174d0 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 17 Apr 2024 23:43:31 -0400 Subject: [PATCH 211/239] [UI] Add prompt variant to query params to support deep-linking (#88) From afe7feaa1eb4de47b0ef85f06a0eacd6d6f69c91 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 18 Apr 2024 10:29:53 -0400 Subject: [PATCH 212/239] [UI] Fix race condition when setting content in monaco (#96) From dab39387b634c1d693c31855ed91f68e7a658da3 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 18 Apr 2024 10:52:08 -0400 Subject: [PATCH 213/239] [UI] Small visual fix in app nav bar (#98) From d2994b42965431c708e2a2514e822f45d107c070 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 18 Apr 2024 12:37:43 -0400 Subject: [PATCH 214/239] [UI] Fix incorrect height for modal runner header (#101) From 4fc9db4668a712d5d77c961e7454f6bf12d0ffa4 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Thu, 18 Apr 2024 12:38:08 -0400 Subject: [PATCH 215/239] [UI] Update placeholder label for model version select (#100) From b99e447a32291410c9129f6e276335a3e6fb335a Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Thu, 18 Apr 2024 15:09:08 -0400 Subject: [PATCH 216/239] Message list component (#84) Co-authored-by: Chris Chestnut Co-authored-by: Michael Doyle From ac9203853893a774ff559ce7d75b6c2bd7cc1018 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 19 Apr 2024 11:48:25 -0400 Subject: [PATCH 217/239] [UI] Fix view evaluation report button to read correct metdata (#119) From 590eca7c14af5002093558d0f27b4af31b176491 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 19 Apr 2024 13:21:47 -0400 Subject: [PATCH 218/239] [UI] Save action sidebar expansion state to `localStorage` (#120) From 5d8711154452f779e86e4ec4a3fb73833eaeefab Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Fri, 19 Apr 2024 11:11:44 -0700 Subject: [PATCH 219/239] [UI]: Move model config params to a separate component (#103) From 618d107bb9cb7ea70d5ad2447f7daf2bff42afc5 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Fri, 19 Apr 2024 12:25:41 -0700 Subject: [PATCH 220/239] [UI] Update model runner to use the new model config component (#124) From f45e6565247e306b0effd9771a87835ba755660c Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Fri, 19 Apr 2024 13:01:42 -0700 Subject: [PATCH 221/239] [UI] Pull the new defaults for model config into the new config component (#125) From 08eb2d6d9c6402e4803b7ab09b0ef8f50241a611 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Fri, 19 Apr 2024 17:06:00 -0400 Subject: [PATCH 222/239] [UI] Add ability to export prompt file from model runner (#115) From c49cbc0632640b0b1c18167cfdb52d1495750bee Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sat, 20 Apr 2024 23:30:29 -0400 Subject: [PATCH 223/239] [UI] Fix model versions not being loaded on initial render (#131) Fixes https://github.com/google/genkit/issues/130. This is more of a stop-gap fix, going to explore refactoring these components to utilize Angular signals to eliminate this class of error entirely. From e9c2ac3569a2f0228c124fdd7ec3dc36ff48dc2a Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Sun, 21 Apr 2024 15:15:53 -0400 Subject: [PATCH 224/239] Integrate the new MessageList component into the ModelRunner (#114) From 5185143d1229b183cc9aff8843f03943192c091c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 22 Apr 2024 09:33:50 -0400 Subject: [PATCH 225/239] [UI] Refactor model-config to use signals (#133) From 2d5730b38edfc27c431f5611c4490a0e28fefbd6 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 22 Apr 2024 14:45:03 -0400 Subject: [PATCH 226/239] Create placeholder for system prompt and first user message (#144) From 94c4306ed58945ac3d185752a98461c9e87daf3c Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 22 Apr 2024 14:50:23 -0400 Subject: [PATCH 227/239] [UI] Remove oops from model config template (#143) From dbfed9e31c14601a72c2cdace9a3e106f259e7b5 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 22 Apr 2024 16:04:59 -0400 Subject: [PATCH 228/239] Ensure selected model is set when using left nav (#148) From 67df20ed96768b514cecd722b16ccf9401aa8495 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Mon, 22 Apr 2024 16:14:11 -0400 Subject: [PATCH 229/239] [UI] Prevent button icons from flex-shrinking (#151) From b6753b4b1be909e3c4375127f54156195e3d3966 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Mon, 22 Apr 2024 23:29:59 -0400 Subject: [PATCH 230/239] Show large multimedia in a modal (#156) From cf01937d4fa05274a039d74ef3a492324c1b8609 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 23 Apr 2024 13:25:00 -0400 Subject: [PATCH 231/239] Enable all image types in model runner (#160) From 1a2ea200c55f843b88aac744bc5c71e14a281054 Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Tue, 23 Apr 2024 15:13:42 -0400 Subject: [PATCH 232/239] Re-enable gemini vision models (#168) From ef1bfb344dfaaefa092040fc851e087802550398 Mon Sep 17 00:00:00 2001 From: shrutip90 Date: Tue, 23 Apr 2024 13:57:46 -0700 Subject: [PATCH 233/239] [UI] Remove system prompt for single-turn models (#169) From c464752b3a9d10f3c4adfb1905f410c3e617d0ef Mon Sep 17 00:00:00 2001 From: Michael Doyle Date: Wed, 24 Apr 2024 10:36:39 -0400 Subject: [PATCH 234/239] Set a reasonable (but arbitrary) number of media files per message (#172) From 381425b7c31cc819ee5d660b1b2ede376f1f2459 Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Wed, 24 Apr 2024 17:35:25 -0400 Subject: [PATCH 235/239] [UI] Remove obsolete MONACO_PATH provider (unused) (#182) From 69416cff266e1c858676c7dbeb7c6d414e915edf Mon Sep 17 00:00:00 2001 From: Anthony Barone Date: Sun, 28 Apr 2024 18:44:06 -0400 Subject: [PATCH 236/239] [UI] Sort eval metrics for consistent/comparable viewing (#209) Fixes #207. From 32618470bcb0f22517f5ccba75600057c3cfb9f9 Mon Sep 17 00:00:00 2001 From: Jonathan Amsterdam Date: Wed, 1 May 2024 09:24:24 -0400 Subject: [PATCH 237/239] change action latency name (#200) Change the name of the action latency histogram from "genkit.action.action_latency" to "genkit.action.latency" to avoid stutter. From 2a1f31ff467f454a5ded6a746117a34f3043e2e6 Mon Sep 17 00:00:00 2001 From: Tunde Agboola Date: Wed, 1 May 2024 17:00:35 -0400 Subject: [PATCH 238/239] Add vertex evaluator information to docs --- docs/plugins/vertex-ai.md | 44 ++++++++++++++++++++++++++- js/plugins/vertexai/src/evaluation.ts | 1 + 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/plugins/vertex-ai.md b/docs/plugins/vertex-ai.md index b5b7de8185..dd984b5d25 100644 --- a/docs/plugins/vertex-ai.md +++ b/docs/plugins/vertex-ai.md @@ -7,6 +7,13 @@ through the [Vertex AI API](https://cloud.google.com/vertex-ai/generative-ai/doc - Imagen2 image generation - Gecko text embedding generation +It also provides access to subset of evaluation metrics through the Vertex AI [Rapid Evaluation API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation). + +- [Safety](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#safetyinput) +- [Groundeness](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#groundednessinput) +- [ROUGE](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#rougeinput) +- [BLEU](https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations/evaluateInstances#bleuinput) + ## Installation ```posix-terminal @@ -66,6 +73,8 @@ credentials. ## Usage +### Generative AI Models + This plugin statically exports references to its supported generative AI models: ```js @@ -115,7 +124,7 @@ const embedding = await embed({ }); ``` -### Anthropic Claude 3 on Vertex AI Model Garden +#### Anthropic Claude 3 on Vertex AI Model Garden If you have access to Claude 3 models ([haiku](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-haiku), [sonnet](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-sonnet) or [opus](https://console.cloud.google.com/vertex-ai/publishers/anthropic/model-garden/claude-3-opus)) in Vertex AI Model Garden you can use them with Genkit. @@ -147,3 +156,36 @@ const llmResponse = await generate({ prompt: 'What should I do when I visit Melbourne?', }); ``` + +### Evaluators + +To use the evaluators from Vertex AI Rapid Evaluation, add an `evaluation` block to your `vertexAI` plugin configuration. + +```js +import { vertexAI, VertexAIEvaluationMetricType } from '@genkit-ai/vertexai'; + +export default configureGenkit({ + plugins: [ + vertexAI({ + projectId: 'your-cloud-project', + location: 'us-central1', + evaluation: { + metrics: [ + VertexAIEvaluationMetricType.SAFETY, + { + type: VertexAIEvaluationMetricType.ROUGE, + metricSpec: { + rougeType: 'rougeLsum', + }, + }, + ], + }, + }), + ], + // ... +}); +``` + +The configuration above adds evaluators for the `Safety` and `ROUGE` metrics. The example shows two approaches- the `Safety` metric uses the default specification, whereas the `ROUGE` metric provides a customized specification that sets the rouge type to `rougeLsum`. + +Both evaluators can be run using the `genkit eval:run` with a compatible dataset i.e. a datset with `output` and `reference` fields. The `Safety` evaluator can also be run using the `genkit eval:flow -e vertexai/safety` command since it only requires an `output`. diff --git a/js/plugins/vertexai/src/evaluation.ts b/js/plugins/vertexai/src/evaluation.ts index c8fa1937e0..f5a9195e8c 100644 --- a/js/plugins/vertexai/src/evaluation.ts +++ b/js/plugins/vertexai/src/evaluation.ts @@ -25,6 +25,7 @@ import { EvaluatorFactory } from './evaluator_factory'; * https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation#parameter-list */ export enum VertexAIEvaluationMetricType { + // Update genkit/docs/plugins/vertex-ai.md when modifying the list of enums SAFETY = 'SAFETY', GROUNDEDNESS = 'GROUNDEDNESS', BLEU = 'BLEU', From a171f0bdcdfa1af1ef905b94bc2737165c5d1e81 Mon Sep 17 00:00:00 2001 From: tagboola Date: Thu, 2 May 2024 09:25:19 -0400 Subject: [PATCH 239/239] Address PR comments Co-authored-by: Kevin Cheung --- docs/plugins/vertex-ai.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plugins/vertex-ai.md b/docs/plugins/vertex-ai.md index dd984b5d25..464eb43026 100644 --- a/docs/plugins/vertex-ai.md +++ b/docs/plugins/vertex-ai.md @@ -188,4 +188,4 @@ export default configureGenkit({ The configuration above adds evaluators for the `Safety` and `ROUGE` metrics. The example shows two approaches- the `Safety` metric uses the default specification, whereas the `ROUGE` metric provides a customized specification that sets the rouge type to `rougeLsum`. -Both evaluators can be run using the `genkit eval:run` with a compatible dataset i.e. a datset with `output` and `reference` fields. The `Safety` evaluator can also be run using the `genkit eval:flow -e vertexai/safety` command since it only requires an `output`. +Both evaluators can be run using the `genkit eval:run` command with a compatible dataset: that is, a dataset with `output` and `reference` fields. The `Safety` evaluator can also be run using the `genkit eval:flow -e vertexai/safety` command since it only requires an `output`.