Run evals on experiment spans

[omrihar]

I think it would be very useful to be able to run span and document evals on spans generated during experiment runs, to be able to better analyze reasons for failure in specific experiment examples.
Is there a supported way to do that? I've seen that the spans go into a hidden experiment project, can I simply run evals on that somehow?

Thanks, Omri

[dosubot[bot]]

Yes, you can run span and document evaluations on spans generated during experiment runs. You can use the SpanEvaluations and DocumentEvaluations classes to construct evaluations from dataframes, save them, and load them back while preserving the data.

Here's a brief example:

evals = SpanEvaluations(
    eval_name="eval-name",
    dataframe=dataframe,
)
eval_id = evals.save(tmp_path)
read_evals = Evaluations.load(eval_id, tmp_path)

Similarly, for document evaluations:

evals = DocumentEvaluations(
    eval_name="eval-name",
    dataframe=dataframe,
)
eval_id = evals.save(tmp_path)
read_evals = Evaluations.load(eval_id, tmp_path)

These classes allow you to evaluate spans programmatically, making it easier to analyze specific experiment examples ^[1][2].

To continue talking to Dosu, mention @dosu.

[axiomofjoy]

Hey @omrihar, this is definitely possible! After you've run your experiment, you can pull your traces and spans from the experiments project using the Phoenix client. Methods like get_qa_with_reference have a project_name parameter you can use. After you've run the experiment, the process of running evals should be similar to this notebook.

[omrihar]

Thanks @axiomofjoy , so a quick question about that - is the proejct name for experiments following a fixed template I can rely on in the future? Because I'd like to automate as much as possible in this scenario - so fetching the experiment proejct name automatically after the experiment run and then running evals.

[omrihar]

I saw now that the experiment object holds a project_name attribute that can be used.

[axiomofjoy]

Yep, that should work.