Add Chat Assistant Service as a New Micro-service

[Xiao-zhen-Liu]

Overview

We are in the process of merging a chat assistant service into main. The chat assistant is powered by a multi-agent system based on OpenAI Agents SDK, and can help users construct Texera workflows through natural language conversations. This issue tracks the design and PR plan of this project.

Design

The chat assistant service will be a new micro-service in Texera:

• The backend is a Python web-server and connects to an external LLM API (currently it is OpenAI API, non-OpenAI models are also supported by Agents SDK)
• Frontend communicates with the service through a single WebSocket session. Currently a chat session is bundled with a WebSocket session, and the chat session will be lost when the chat interface is closed. We plan to add session persistence in the future.

The following diagram shows the lifecycle of a user's frontend establishing a WebSocket connection to the chat assistant service, using the connection to send user chat messages to OpenAI's service, and receiving streamed response messages from OpenAI's service.

The agents are equipped with function-calling capabilities and can interact with user's interface (e.g., add an operator on the canvas). Functions are defined in the Python server and need WebSocket communications with the frontend. The following diagram shows the lifecycle of a function call.

Relationship with Existing AI Components

There are existing lightweight AI components in the current Texera codebase. As these components are written in TexeraWebApplication (Scala), the new service will not reuse code from these components. The following diagram shows their relationships.

Python ChatAssistantService Architecture

The following diagram shows the architecture of the Python ChatAssistantService server.

PR Plan

The PRs for this project will be created and reviewed incrementally. Each PR is designed to be testable.

PR1: Add basic Python server skeleton

• Add a basic Python web server with only a health-check endpoint.
• Set up project files and CI
• How to test: Verify if web-server & health-check works.

PR2: Add chat interface with WebSocket handling logics

• Add the chat interface on the UI
• Add the framework of WebSocket handling logic in frontend and backend
• To keep this PR manageable, the chat experience will be a simple echo of the user's message. No OpenAI dependency in this PR yet.
• How to test: Verify if the frontend and backend can work together with simple echo'ed responses.

PR3: Add LLM-agent-based chat experience

• Add a simple single-agent based chat experience based on OpenAI Agents SDK
• Add in-memory session management
• Introduce dependencies on OpenAI and API keys, prompt loading, model configurations, etc.
• How to test: Verify if inside Texera's frontend a user can chat with a OpenAI model with in-session conversation history.

PR4: Add multi-agent system for Texera workflow generation

• Enables the complete Texera Chat Assistant experience
• Add a 3-agent orchestration system (Manager, Planner, Builder)
• Add function calling capabilities
• How to test: Verify if a workflow can be created by chatting with the assistant.

PR5: Add deployment files

• All previous PRs are to be tested locally.
• This PR adds deployment-related files, including Dockerfile, K8s templates, etc.
• How to test: Verify if the new microservice can be deployed using the added files.

PR6: Add persisted chat session

• This is a feature to be implemented.
• Currently chat sessions are bundled with a WebSocket connection, and creating the chat interface destroys the chat session.
• We want the chat session to be bundled with a workflow and persist the chat sessions to a database, so a chat session can be reconnected if the workflow is re-opened.