🗺️ Persistence

[mikeldking]

As a user of phoenix, I would like a persistent backend - notably a way to

• Resume phoenix on previous collection of data
• Keep track of evaluation results

Spikes

• [persistence] Investigate EFS, GFS, S3 file mounts #2158
• [persistence] scope out how to scale to projects #2335
• ERD for sql lite #2704
• [persistence][spike] alembic migrations #2742
• [persistence][spike] Target Postgres as a persistence db #2793

Server

• [persistence] Host a simple file explorer #2336
• [persistence] define the persistence interface as a persistence #2512
• [persistence] Support resuming on traces and evals persisted to the mounted directory #2546
• [persistence] sql lite span_store #2687
• [persistence]Write traces to files #2547
• [persistence] Simple append-based file storage implementation for saving and loading traces #2548
• [persistence] enum for storage strategies #2559
• [persistence] allow storage options for launch_app #2560
• [persistence] EvaluationResult Store #2628
• [container] sqlite install #2721
• [persistence] use sql alchemy #2743
• [persistence] sqlAlchemy session mounting in graphql #2753
• [persistence] annotation tables #2776
• [gql] evals API changes to read from tables #2777
• [persistence] pyarrow to sqlite ingestion of evals #2778
• [persistence] initialize span fixtures for read-only deployment #2806
• [persistence] dataloader for children spans #2781
• [persistence] get or delete projects using sql #2838
• [persistence] populate p50 and p99 statistics for latency_ms from sql queries #2807
• [persistence] spans connection sorting #2779
• [persistence] filters on spans #2780
• [persistence] explode and concat #2782
• [persistence] deal with insertion batching on spans #2786
• [persistence] evaluation filtering on spans and traces #2787
• [persistence] make queries work with postgres #2810
• [projects][grapql] clear traces #2539
• [persistence] Eval sorting on traces / spans #2877
• [persistence] Eval aggregation metric #2878
• [persistence] Document eval aggregation metric #2879
• [persistence] update GraphQL document_retrieval_metrics at the span level #2957
• [persistence] sql dataloader for document_retrieval_metrics for spans #2961
• [persistence] fetch evaluation names #2963
• [persistence] delete in-memory for evals #2930
• [persistence] delete in-memory for projects #2931
• [persistence] delete in-memory for spans #2932
• [persistence] remove defaulting to memory
• [persistence] promote db config out of experimental
• [persistence] wire evaluations GET endpoint to db #2894
• [persistence] add time range to get evaluations endpoint #2901
• [persistence] standardize project name query parameter #2909
• [persistence] convert bulk inserter to use a single queue #2906
• [persistence] rename document_index column to document_position #2913
• [persistence] Update trace- and span- eval summaries with filters #2925
• [persistence] add project color and plumb through to the UI #2926
• Perform a sqlite version and compilation flag check  #2922
• [persistence] fetch streaming_last_updated_at from sql #2813
• launch_app using temp directory #2843
• add filter condition to document evaluation summary #2889
• [persistence] make DB_URL non-dangerous flag #2929
• [persistence] ensure joins are not cross-joins #2969
• [persistence] verify automatic naming of indexes and constraints #2975
• [persistence] investigate postgres rowid #3057

UI

• [persistence] restrict the project page metrics to 7D #2882
• [persistence] time range selector for traces / spans #2883
• [ui][projects] project dropdown to clear traces #2540

Metrics / Observability

• [persistence] add python prometheus exporting for container deployments #2615
• Add Prometheus environment variable #2712
• [persistence] ui disable client side sort #2927
• [persistence] SQLAlchemy and Starlette tracing instrumentation for query performance analysis #2970

Infra

• get prometheus server running in phoenix GCP cluster #2676
• [testing] integration tests #2783
• [persistence] make alembic runnable manually via command line #2794
• [persistence][infra] Make it super easy to purge the sql lite instance when desired #2795
• [persistence] experimental persistence builds #2811
• [persistence] deprecate span_store #2815
• bug: migration failures don't block the startup of the application #2921

Remote Session management

• [remote] Create RemoteSession class that is available from a notebook setting #2136
• [remote] Support SpanQuery execution on RemoteSession #2137
• [remote] Support get_spans_datataframe execution on RemoteSession #2091
• [remote] get_trace_dataset on RemoteSession #2138

Performance

• [persistence] deal with hard cardinality of spans in a time window #3003
• [persistence][performance] strip out embeddings from graphql queries #3016
• [persistence][performance] put limits on span_dataframe #3017
• [persistence][performance] put limits on get_evaluations #3018
• [persistence] Consider adding a max concurrent requests on the web server #3019
• [persistence] client pulls of dataframes errors out #3021
• [persistence] throughput numbers for sqlite / postgres #3027
• [persistence] wire up prometheus #3028
• [ENHANCEMENT] Support otlp-grpc #2123
• [persistence][performance] compute summaries in db (if possible) instead of fetching individual entities #3033
• [persistence][performance] cache for summary dataloaders (sqlite only) #3049
• [persistence] deleting a project while it's receiving spans errors #3068
• [persistence] time range graphql pull from min / max from time range #3071
• [persistence] clear all TODO(persistence) #3108
• [persistence] index on spans start time, and others #3072
• [persistence] indexes on annotation tables #3073

Notebook-Side Persistence

• [persistence] launch_app should check for the existence of a persistence directory #2814

Docs

• [persistence] documentation #2935
• [persistence] examples #2936

Breaking Changes

• remove experimental #2933
• remove datasets #2934

Testing

• get a postgres instance up and running #2880
• get a datagenerator working in dev #2881
• [persistence] add tests for dataloaders #3040
• Make sure delete cascades work for data retention
• Make sure delete cascades work for data retention on evals @axiomofjoy
• Test run some migrations on named constraints
• Test pandas and parquet endpoints and interactions with the client
• Filters and Sorting on the UI
• SpanQuery compatibility
• Metadata filtering
• Data volume
• Data size (large context windows, large IO)
• Embedding querying

Open Questions

• Storage of embeddings
• Controlling sqlite version explicitly
• trace retention / management needed?

[aazizisoufiane]

I think at least for the second need you can see here: https://docs.arize.com/phoenix/integrations/llamaindex#traces

[axiomofjoy]

I am investigating using Parquet as the file format. Here's a snippet to add custom metadata to a Parquet file:

"""
Snippet to write custom metadata to a single parquet file.

NB: "Pyarrow maps the file-wide metadata to a field in the table's schema
named metadata. Regrettably there is not (yet) documentation on this."

From https://stackoverflow.com/questions/52122674/how-to-write-parquet-metadata-with-pyarrow
"""

import json

import pandas as pd
import pyarrow
from pyarrow import parquet

dataframe = pd.DataFrame(
    {
        "field0": [1, 2, 3],
        "eval0": ["a", "b", "c"],
    }
)
OPENINFERENCE_METADATA_KEY = b"openinference"
openinference_metadata = {
    "version": "v0",
    "evaluation_ids": ["eval0", "eval1"],
}

original_table = pyarrow.Table.from_pandas(dataframe)
print("Metadata:")
print("=========")
print(original_table.schema.metadata)
print()

updated_write_table = original_table.replace_schema_metadata(
    {
        OPENINFERENCE_METADATA_KEY: json.dumps(openinference_metadata),
        **original_table.schema.metadata,
    }
)
parquet.write_table(updated_write_table, "test.parquet")
updated_read_table = parquet.read_table("test.parquet")
print("Metadata:")
print("=========")
print(updated_read_table.schema.metadata)
print()

updated_metadata = updated_read_table.schema.metadata
updated_metadata.pop(OPENINFERENCE_METADATA_KEY)
assert updated_metadata == original_table.schema.metadata

[axiomofjoy]

Notes on Parquet and PyArrow:

• Large "row groups" are suggested for quicker analytical queries (on the order of a GB). Many of our datasets will be far smaller than this. There are potentially performance consequences at query time for writing small Parquet files frequently.
• Parquet files are immutable. As far as I can tell, there is no notion of updating just the file metadata.
• It's possible to augment the metadata of individual Parquet files (see above). Another pattern used by Spark and Dask actually is to write a separate metadata file at _common_metadata to describe all the Parquet files in an Arrow dataset (a single metadata file describing multiple Parquet files).
• Arrow supports directory partitioning. It looks straightforward to partition, for example, on date.
• Arrow also provides nice file system interfaces to the various cloud storage providers.

[stdweird]

@axiomofjoy what kind of backends will you target? i see some code related to file backends, but why not sql databases (given the .to_sql for dataframes, and probably other/better methods); or native json support in most solutions. given phoenix coupling with RAG, most people will already have a vectordb that should work.

[axiomofjoy]

@mikeldking Can you provide an update for @stdweird?

[mikeldking]

@axiomofjoy what kind of backends will you target? i see some code related to file backends, but why not sql databases (given the .to_sql for dataframes, and probably other/better methods); or native json support in most solutions. given phoenix coupling with RAG, most people will already have a vectordb that should work.

@stdweird - good point - I think we see some limitations with sql backends and so we are currently benchmarking different backends. In general we will probably have a storage interface and you will be able to choose your storage mechanism but for now we are working on keeping the backend pretty lean and figuring out the interface as we go

[mikeldking]

🥳