TruLens 1.3.2

Bug Fixes

• Handle pydantic upgrade that now handles model_fields as a property that can resolve to a dict when there's nothing. by @sfc-gh-dkurokawa in #1726
• Don't create event tables unless TRULENS_OTEL_TRACING env variable is set. by @sfc-gh-dkurokawa in #1724

Full Changelog: trulens-1.3.1...trulens-1.3.2

TruLens 1.3.1

What's Changed

• Update credit consumption table for Cortex LLM by @sfc-gh-dhuang : #1721

Bug Fixes

• Fix Snowflake SQL alchemy breaking behavior and ensure AUTOCOMMIT is enabled to fix trulens ingestion by @sfc-gh-dhuang #1719
• Fix and update Snowflake quickstart notebook by @sfc-gh-dhuang #1722

Full Changelog: trulens-1.3.0...trulens-1.3.1

TruLens 1.3.0

Optimizing Feedback Functions

In this release, we add important changes for improving the alignment of their LLM-Judge evals to human evaluations.

Global Improvement of Groundedness Feedback

The first is the global improvement of the groundedness feedback function (benchmarks and methods forthcoming). We invite any users to submit feedback (positive or negative) on the effectiveness of the new groundedness function using GitHub Issues or Discussions.

You can view the addition of new groundedness criteria in the GitHub diff below.

New levers for aligning feedback functions

The second change is that we add new easy-to-use levers for you to change the behavior of feedback functions using few-shot examples and custom criteria. Early customers have seen useful benefit in aligning their feedback functions to their collected expert evaluations using these levers.

Adding custom criteria to a feedback function

custom_criteria = """
A positive sentiment should be expressed with an extremely encouraging and enthusiastic tone.
"""

provider.sentiment(
    "When you're ready to start your business, you'll be amazed at how much you can achieve!",
    criteria=custom_criteria,
)

Adding few-shot examples to guide feedback functions

from trulens.feedback.v2 import feedback

fewshot_relevance_examples_list = [
    (
        {
            "query": "What are the key considerations when starting a small business?",
            "response": "You should focus on building relationships with mentors and industry leaders. Networking can provide insights, open doors to opportunities, and help you avoid common pitfalls.",
        },
        3,
    ),
]

provider.relevance(
    "What are the key considerations when starting a small business?",
    "Find a mentor who can guide you through the early stages and help you navigate common challenges.",
    examples=fewshot_relevance_examples_list,
)

What's Changed

• Feedback customization (including few-shot examples) by @sfc-gh-jreini in #1674
• Custom criteria for feedback by @sfc-gh-jreini in #1705
• Update groundedness criteria (with more optimized prompt) by @sfc-gh-dhuang in #1710
• Allow existing tables to be used in ground truth datasets by @sfc-gh-dhuang in #1698

Bug Fixes

• Allow passthrough of feedback parameters including temperature, groundedness configs in the Feedback class by @sfc-gh-jreini in #1674
• Remove / retire sql instrumentation in Cortex Endpoint by @sfc-gh-dhuang in #1715
• Poetry < 2.0.0 by @sfc-gh-jreini in #1709
• Update docs to use postgres + psycopg in order to avoid known issues with psycopg2 by @sfc-gh-gtokernliang in #1701
• Update prpr example notebook to reflect latest Cortex provider API by @sfc-gh-dhuang in #1712

Preparations for Open Telemetry compatibility

• Introduce Event table for ORM to prepare for OTEL traces by @sfc-gh-gtokernliang in #1692
• Prototype OTEL exporter by @sfc-gh-gtokernliang in #1694
• Prototype @Instrument with OTEL by @sfc-gh-gtokernliang in #1693
• Move main_input, main_output, and _extract_content out of app.py by @sfc-gh-gtokernliang in #1706
• Move span-related validation + setting logic out of instrument.py by @sfc-gh-gtokernliang in #1707

Full Changelog: trulens-1.2.11...trulens-1.3.0

TruLens 1.2.11

What's Changed

• Add snowflake PrPr notebook by @sfc-gh-dhuang in #1683
• Support types for Python < 3.9. by @sfc-gh-dkurokawa in #1675
• Change issue assignee to Prudhvi to triage by @sfc-gh-jreini in #1686
• TREC DL and LLM AggreFact experiments for relevance benchmark + prompts comparisons and groundedness vs Bespoke Minicheck 7B by @sfc-gh-dhuang in #1660
• Agents: Example of observability for CrewAI by @sfc-gh-jreini in #1621
• Set langchain version for crewai example by @sfc-gh-chu in #1695

Full Changelog: trulens-1.2.10...trulens-1.2.11

TruLens 1.2.10

What's Changed

• decode app and record json in get_df_and_cols by @sfc-gh-chu in #1672
• reset_database only drops trulens tables by @sfc-gh-chu in #1676
• trulens-dashboard: handle selected_rows is None and fix fallback to st.dataframe in SiS by @sfc-gh-chu in #1677
• Fix Cortex complete SDK response parsing. by @sfc-gh-dkurokawa in #1679
• Have Pace create an event loop if it doesn't exist. by @sfc-gh-dkurokawa in #1680
• Rename trulens-semconv to trulens-otel-semconv. by @sfc-gh-dkurokawa in #1681
• trulens-semvar conda build files by @sfc-gh-chu in #1678

Full Changelog: trulens-1.2.9...trulens-1.2.10

TruLens 1.2.9

What's Changed

• adding jenkins file for e2e tests by @sfc-gh-srudenko in #1661
• Memoize base endpoint creation for cost tracking by @sfc-gh-chu in #1659
• add importlib resources to conda build by @sfc-gh-chu in #1662
• Switch from SQL function to REST API backend for Cortex Complete - cost tracking of both feedback computations and app generation by @sfc-gh-dhuang in #1650
• fix meta.yaml spacing by @sfc-gh-chu in #1663
• Fix Cortex provider for tests. by @sfc-gh-dkurokawa in #1666
• relax snowflake-ml-python version by @sfc-gh-chu in #1664
• Fix encoding issues in dashboard by @sfc-gh-chu in #1668
• fix async pace by @sfc-gh-pmardziel in #1654
• Fix poetry environment issues. by @sfc-gh-dkurokawa in #1670
• Create trulens-semconv package. by @sfc-gh-dkurokawa in #1669
• strip quotes from connection params by @sfc-gh-chu in #1673

Full Changelog: trulens-1.2.6...trulens-1.2.9

TruLens v1.2.6

What's Changed

• Allow Cortex provider to only take a connection object. by @sfc-gh-pdharmana in #1617
• Fix code example formatting in docs by @sfc-gh-jreini in #1610
• Fix old references to mae in GroundTruthAgreement feedback function by @sfc-gh-dhuang in #1622
• Add try on tag creation since its an enterprise feature by @sfc-gh-pdharmana in #1623
• Stop using snowflake.snowpark.session.Session::sql and use snowflake.connector.cursor.SnowflakeCursor::execute instead as its thread-safe. by @sfc-gh-dkurokawa in #1620
• logos section on homepage by @sfc-gh-jreini in #1602
• Don't check if Cortex providers can be deserialized for deferred feedback functions. by @sfc-gh-dkurokawa in #1626
• Ensure make clean does indeed clean or it will fail out. by @sfc-gh-dkurokawa in #1627
• Create smoke test for Snowflake notebooks. by @sfc-gh-dkurokawa in #1619
• For snowflake dialects, when inserting a feedback result with a NULL result, first insert a -1, then update it. by @sfc-gh-dkurokawa in #1628
• update link for migration page by @sfc-gh-jreini in #1630
• human feedback with metadata by @sfc-gh-pmardziel in #1629
• Move print message up to not invoke error when we can't create a tag. by @sfc-gh-dkurokawa in #1634

Full Changelog: trulens-1.2.4...trulens-1.2.6

TruLens v1.2.4

What's Changed

• try catch on tag creation (#1623) by @sfc-gh-pdharmana in #1624

Full Changelog: trulens-1.2.2...trulens-1.2.4

TruLens v1.2.2

What's Changed

• Use snowflake connector over snowpark session in trulens Snowflake DB connector as snowpark session isn't thread-safe. by @sfc-gh-dkurokawa in #1604
• Don't open extra Snowflake connections and don't recycle connections as quickly. by @sfc-gh-dkurokawa in #1609
• Remove unnecessary deps from trulens-connectors-snowflake. by @sfc-gh-dkurokawa in #1611

Full Changelog: trulens-1.2.1...trulens-1.2.2

TruLens v1.2.1

Bug Fixes

• Don't check for user and account in snowpark sessions because Streamlit apps might hide them. by @sfc-gh-dkurokawa in #1600
• catch source code not available in code_line by @sfc-gh-pmardziel in #1592
• use float nan in place of numpy for skipped evals by @sfc-gh-chu in #1595
• Fix the misspelled trulens-providers-openai package in examples by @SSK-14 in #1601
• fix assertion to nan by @sfc-gh-jreini in #1605

New Contributors

• @SSK-14 made their first contribution in #1601

Full Changelog: trulens-1.2.0...trulens-1.2.1