python[minor]: Release 0.3.0 #1439
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com> Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Log pytest tests to LangSmith. Useful for:
- Evaluations where the eval logic is different for each datapoint,
making it difficult to use generic evaluators on a whole dataset
- Unit tests where you want both the local pytest experience and the
ability to trace and share results
### Install
```bash
pip install "langsmith[pytest]==0.2.11rc14"
```
### Simple usage
```python
# tests/test_simple.py
import pytest
from langsmith import testing as t
@pytest.mark.langsmith
def test_addition_single():
x = 3
y = 4
# directly log example inputs if you don't want to use fixtures
t.log_inputs({"x": x, "y": y})
expected = 7
# directly log example outputs if you don't want to use fixtures
t.log_reference_outputs({"sum": expected})
actual = x + y
# directly log run outputs
t.log_outputs({"sum": actual})
# test pass/fail automatically logged to langsmith
assert actual == expected
```
Run
```bash
pytest --outputs='ls' tests/test_foo.py
```
Results
<img width="952" alt="Screenshot 2025-01-08 at 2 53 04 AM"
src="https://github.com/user-attachments/assets/d695747b-cfdf-4248-b5fd-c5c420aa92ec"
/>
### Advanced usage
```python
#tests/test_advanced.py
import openai
import pytest
from langsmith import wrappers
from langsmith import testing as t
oai_client = wrappers.wrap_openai(openai.Client())
@pytest.mark.langsmith
def test_openai_says_hello():
# Traced code will be included in the test case
text = "Say hello!"
response = oai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": text},
],
)
t.log_inputs({"text": text})
t.log_outputs({"response": response.choices[0].message.content})
t.log_reference_outputs({"response": "hello!"})
# Use this context manager to trace any steps used for generating evaluation
# feedback separately from the main application logic
with t.trace_feedback():
grade = oai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": "Return 1 if 'hello' is in the user message and 0 otherwise.",
},
{"role": "user", "content": response.choices[0].message.content},
],
)
t.log_feedback(
key="llm_judge", score=float(grade.choices[0].message.content)
)
assert "hello" in response.choices[0].message.content.lower()
@pytest.mark.langsmith(output_keys=["expected"])
@pytest.mark.parametrize(
"a, b, expected",
[(1, 2, 3), (3, 4, 7)],
)
def test_addition_parametrized(a: int, b: int, expected: int):
t.log_outputs({"sum": a + b})
assert a + b == expected
```
Run using pytest-xdist to parallelize (`pip install pytest-xdist first`)
```bash
LANGSMITH_TEST_SUITE="Test suite name" LANGSMITH_EXPERIMENT="Experiment name" pytest --outputs='ls' tests
```
Results:
https://dev.smith.langchain.com/public/cea0e7fd-2d27-47d1-8ada-141069acdf0d/d
<img width="1030" alt="Screenshot 2025-01-08 at 3 07 51 AM"
src="https://github.com/user-attachments/assets/db770d00-67b3-4b53-8e33-289c4f0edfeb"
/>
---------
Co-authored-by: isaac hershenson <ihershenson@hmc.edu>
Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: isaac hershenson <ihershenson@hmc.edu>
baskaryan
commented
Jan 21, 2025
Comment on lines
+323
to
+326
| if isinstance(data, BufferedReader): | ||
| encoded_data = data.read() | ||
| else: | ||
| encoded_data = str(data).encode() |
There was a problem hiding this comment.
Looks good
I think you can optimize this more if you create a case that streams the reader's content into the compressor_writer, but I didn't do the compressor_writer work
nfcampos
approved these changes
Jan 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.