🗺️ Session Tracking

[mikeldking]

As a user of phoenix, I want to be able to associate multiple traces (e.g. conversational flows) under a single session_id. This way I can track the back and forth between a user and a chat completion endpoint (e.g. visualize the back and forth).

In addition to tracking a session.id, we might also want to track session metadata such as user_id etc.

As a developer building a chat or agent application, I want to be able to track user interactions with my application. Notably if my application keeps track of user interactions under a single “session”, I want to be able to view the user interactions at a higher level rather than a trace.

OpenInference Semantic Convention changes

• session
  • .id the application provided correlation id

OpenInference instrumentation changes

New “openinference-instrumentation” package
- contains utilities around correlating spans to sessions

from openinference.instrumentation.session import using_session, set_session, clear_session

# Using a context manager
With using_session(id: “my-unique-id”):
       // Invoke application code here
       
# implicit
set_session(id: "my-session-id")
// run code here
clear_session(id: "my-session-id")       

Proposed solution is to leverage trace context to set “inheritable” attributes that would be added to each span that is nested below a session context. This means that the setting of the session

Open questions:

• Threading @axiomofjoy
• What if a person starts another session (session collision) @axiomofjoy

Custom Instrumentation


Custom instrumentation would also have to inherit the attributes in a simple way. We need to provide convenience to pull these attributes and attach them to the span.

from openinference.instrumentation import get_context_attributes

def do_work():
    with tracer.start_as_current_span("span-name", get_context_attributes()) as span:
        # do some work that 'span' will track
        print("doing some work...")
        # When the 'with' block goes out of scope, 'span' is closed for you

Architecture

Under the above, session.id would correlate to a trace (not a span) to avoid the danger of spans that get captured in the absence of a session.id. This must be the assumption when querying for sessions.

Milestone 1 - Instrumentation

• [sessions] Tech spec on sessions #2680

Session Tracking

• [sessions] sessions ERD #2755
• [session] tech spec #2693
• [sessions] semantic conventions for session #2678
• [sessions] openinference-instrumentation package #2689
• refactor phoenix.trace.suppress_tracing to openinference-instrumentation #2701
• create conda feedstock for openinference-instrumentation #2697
• [sessions] use_prompt_template #3022
• [sessions] apis for starting and stopping a session #2764
• [sessions] using_sessions context manager #2759
• [sessions] using_user context_manager #2773
• [sessions] using_metadata context manager #2774
• [sessions] using_tags context manager #2775
• [sessions] using_attributes context manager #2760
• [sessions] define a base instrumentor class #2756
• [sessions] refactor llama-index to use base instrumentor #2765
• [sessions] refactor langchain to use base instrumentor #2766
• [sessions] prompt template and variables propagation #3074
• [sessions] refactor mistralai to use base instrumentor #3076
• [sessions] refactor dspy to use base instrumentor #3077
• [sessions] refactor bedrock to use base instrumentor #3075
• [sessions] update instrumentor to be session-aware
• [sessions] update instrumentor to be user-aware
• [sessions] update instrumentor to be metadata-aware
• [sessions] update instrumentor to be tag-aware
• [sessions][langchain] capture session and user ids passed into langchain metadata #2761

Instrumentation Documentation

• [sessions][docs] api reference for starting and stopping a session #2767
• [sessions][docs] api reference for using_sessions context manager #2768
• [sessions][docs] api reference for using_user context_manager #2769
• [sessions][docs] api reference for using_metadata context manager #2770
• [sessions][docs] api reference for using_tags context manager #2772
• [sessions][docs] api reference for using_attributes context manager #2771
• [sessions][docs] how-to #2758
• research and document patterns for propagating context #2754
• [sessions][docs] e2e custom instrumentation example #2706

Milestone 2 - Sessions in the UI

Engineering Leads: @RogerHYang @Parker-Stafford

Testing and Fixtures

• [sessions] fixture data for sessions #5585

Server Support for Sessions

• [sessions] track session ID for a given trace in ingestion
• [sessions] DB ERD #4854
• [sessions] [spike] session user tracking #5066
• [sessions] at ingestion, only recognize session and user IDs on the root span (where parent is null), and ignore them on all other spans #5347
• [sessions][db] remove end_time from db #5355
• [sessions][db] alembic data migration query to populate project_sessions table #5537

Sessions User-Interface

• [sessions] mocks for sessions UI #4884
• [sessions] sessions feature flag #4958
• [sessions] sessions table on the projects page #4959
• [sessions][ui] session slideover with trace previews #5008
• [sessions][ui] from a trace, give a way to navigate to the "parent" session #5009
• [sessions][ui] search field #5007
• [sessions][ui] enable sorting on the sessions table #5278
• [sessions][ui] remove end_time from sessions table columns #5354
• [sessions][ui] add last_start_time to sessions table columns #5356
• [sessions][ui] add p50_latency to sessions table columns #5357
• [sessions][ui] add p99_latency to sessions table columns #5358
• [sessions] add python sessions quickstart in the empty state #5583
• [sessions] add typescript sessions quickstart in the empty state #5584

Sessions API

• [sessions][gql] project sessions connection #5010
• [sessions][gql] session node with first input, last output, num traces, duration #5011
• [sessions][gql] resolve a trace's session on a trace node #5058
• [sessions][gql] session search parameter #5013
• [sessions][gql] resolve feedback at the session traces level for viz #5014
• [sessions][gql] compute tokens at a session level (stretch) #5059
• [sessions][gql] sessions computed fields for analytics #5012
• [sessions][gql] enable server side sorting of sessions #5279

Milestone 2.5 Sessions JavaScript Support

JavaScript

• [sessions] initialize openinfernce-core #3811
• [sessions] session.id and user.id semantic conventions #3042
• [sessions][js]switch openai-node to use OITracer #5084
• [sessions][js] startSession and endSession in openinference Core #3810
• [session]respect langchain thread / conversation / session ids in LangChain js instrumentation #5085

Documentation and Tutorials

• [sessions] how to section on how to add sessions (python) #5588
• [sessions] how to section on how to add sessions (typescript) #5587
• [playground][docs] update llamaindex fullstack example to use sessions #5594
• [sessions][docs] sessions demo video #5651

Milestone 3 - Evaluations / Annotations

As a user I might want to evaluate how a session went.

• [sessions][evals] session evals user-stories
• [sessions] session evals scoping (data model, ingestion) #4899

[axiomofjoy]

With respect to threading and async, OpenTelemetry by default uses contextvars to ensure the context is unique to each thread/ task out of the box. So it's safe for a session context manager to interact with these APIs.

https://github.com/open-telemetry/opentelemetry-python/blob/721beb8b530e7a830c1e27b70c2fb9af6465baf1/opentelemetry-api/src/opentelemetry/context/contextvars_context.py#L19

[DixitAdh]

@mikeldking This looks like a great feature and looking forward to try it out. Let me know if you need any support on testing this. I have a phoenix server deployed in azure containers and logging the traces via databricks notebook. This all is done for RAG chain which is built using langchain.

[mikeldking]

This will ship as part of Phoenix 7.0. Closing.