Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(api): completely redesigned API #30

Merged
merged 98 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
9f223a5
chore(tests): remove old tests
mharrisb1 May 1, 2024
4846db5
chore(endpoints): remove old endpoints
mharrisb1 May 1, 2024
94d94c7
chore(utils): move utils to separate files
mharrisb1 May 1, 2024
c70ad14
feat(types): add partial types for chat
mharrisb1 May 1, 2024
f2316e4
feat(types): add protocols
mharrisb1 May 1, 2024
7258aab
feat(types): add generics
mharrisb1 May 1, 2024
30d163d
feat(routes): add create chat completion route
mharrisb1 May 1, 2024
ea1f458
chore(state): remove old state store
mharrisb1 May 1, 2024
2662c85
feat(mock): refactor to use new single mock api
mharrisb1 May 1, 2024
6ae6938
chore(examples): remove old examples
mharrisb1 May 1, 2024
84b2775
chore(git): ignore .plan file
mharrisb1 May 1, 2024
dfb3cf7
feat(examples): add example for chat routes
mharrisb1 May 1, 2024
a5ba75b
feat(examples): add example for passing raw response
mharrisb1 May 1, 2024
ab1b4a7
feat(route): add create embeddings
mharrisb1 May 1, 2024
2d91e85
chore(examples): rename const
mharrisb1 May 2, 2024
c0c73ee
chore(api): move to init
mharrisb1 May 2, 2024
732ae8b
chore(routes): distinguish between stateful and stateless routes
mharrisb1 May 2, 2024
3b931c0
feat(state): add state store
mharrisb1 May 2, 2024
a2534ec
feat(examples): add example for custom response handler
mharrisb1 May 2, 2024
4065b93
feat(examples): add example for passing explicit model
mharrisb1 May 2, 2024
40a3787
feat(routes): add default handler implementation for all routes
mharrisb1 May 2, 2024
8ce7d84
feat(helpers): add builders
mharrisb1 May 2, 2024
d434af1
feat(types): add partial files
mharrisb1 May 2, 2024
22151f2
feat(routes): add create file route
mharrisb1 May 2, 2024
d13b2fb
chore(examples): move async examples to separate file
mharrisb1 May 3, 2024
c90dd64
fix(utils): handle validation error in parse
mharrisb1 May 3, 2024
2cdfdcf
feat(routes): add file list route
mharrisb1 May 3, 2024
05a355c
feat(types): add partial for file list
mharrisb1 May 3, 2024
76cf779
feat(api): create a new mock router per test
mharrisb1 May 3, 2024
4a3ce2f
feat(examples): add test file route examples
mharrisb1 May 3, 2024
a678c45
fix(route): use default handler for side effect
mharrisb1 May 3, 2024
69d7fca
chore(route): pull regex out to const
mharrisb1 May 3, 2024
98cc95c
feat(routes): add file retrieve route
mharrisb1 May 3, 2024
264a531
feat(routes): add file delete route
mharrisb1 May 3, 2024
bee8987
feat(helpers): add create embedding builder
mharrisb1 May 3, 2024
b553f8e
feat(routes): add create assistant route
mharrisb1 May 3, 2024
b72ff41
feat(helpers): add assistant builder
mharrisb1 May 3, 2024
6d9b02e
feat(examples): add create assistant example
mharrisb1 May 3, 2024
123faba
chore(examples): add docstring
mharrisb1 May 3, 2024
3e96794
chore(api): add comments
mharrisb1 May 3, 2024
b2d41bd
chore(routes): update import path
mharrisb1 May 3, 2024
740e231
chore(routes): add docstring for response setter
mharrisb1 May 3, 2024
ed69c68
chore(tox): format file
mharrisb1 May 3, 2024
0f2fefc
fix(utils): allow model parse to fail
mharrisb1 May 3, 2024
6d68416
feat(routes): add remaining assistants routes
mharrisb1 May 3, 2024
7b52021
chore(types): update partial name to reflect oai model name
mharrisb1 May 3, 2024
d44bd99
chore(poetry): bump openai minor
mharrisb1 May 3, 2024
16cdfbb
chore(git): ignore doc site
mharrisb1 May 3, 2024
20cb99c
feat(routes): add thread create route
mharrisb1 May 3, 2024
f03266e
fix(routes): simplify assistants create builder
mharrisb1 May 3, 2024
124e01b
feat(routes): add thread retrieve route
mharrisb1 May 3, 2024
cc8bf7f
feat(routes): add thread update route
mharrisb1 May 3, 2024
6dcac57
feat(routes): add thread delete route
mharrisb1 May 3, 2024
7aeb8a0
feat(helpers): add thread builder
mharrisb1 May 3, 2024
82ace1e
feat(routes): add create message route
mharrisb1 May 9, 2024
f94a1a3
feat(routes): add list messages route
mharrisb1 May 9, 2024
f9d839f
fix(routes): dict merge precedent
mharrisb1 May 9, 2024
7549a58
feat(routes): add retrieve message route
mharrisb1 May 9, 2024
34ffbf5
feat(routes): add update message route
mharrisb1 May 10, 2024
ccedb4a
feat(routes): add delete message route
mharrisb1 May 10, 2024
48e1e6f
chore(api): add docstring for args
mharrisb1 May 10, 2024
9da9d08
feat(routes): add create run route
mharrisb1 May 10, 2024
6e4e8f4
fix(routes): create additional message if supplied in create thread r…
mharrisb1 May 10, 2024
c65590b
chore(builders): move to separate files to avoid circular imports
mharrisb1 May 10, 2024
6d9a17d
chore(builders): move to separate files to avoid circular imports
mharrisb1 May 10, 2024
a261fb7
chore(builders): move to separate files to avoid circular imports
mharrisb1 May 10, 2024
2aeac4b
feat(api): add state property
mharrisb1 May 10, 2024
fbb503f
chore(builders): move to separate files to avoid circular imports
mharrisb1 May 10, 2024
a098d7d
feat(examples): add example of creating thread with messages
mharrisb1 May 10, 2024
ffc5e92
fix(api): remove state property
mharrisb1 May 10, 2024
4d14eb7
feat(routes): add create thread and run route
mharrisb1 May 10, 2024
ad5b414
feat(routes): add list runs route
mharrisb1 May 10, 2024
bbeee0f
feat(routes): add retrieve runs route
mharrisb1 May 10, 2024
7ab8e0b
feat(routes): add update runs route
mharrisb1 May 10, 2024
928356f
feat(routes): add submit tool outputs route
mharrisb1 May 10, 2024
26d420b
feat(routes): add cancel run route
mharrisb1 May 10, 2024
0fb9087
fix(routes): store as cancelled but respond as cancelling
mharrisb1 May 10, 2024
5778e4a
fix(routes): use copy of model
mharrisb1 May 10, 2024
4e83154
feaT(builders): add build run from request helper
mharrisb1 May 13, 2024
7949be9
feat(routes): add run step routes
mharrisb1 May 13, 2024
1c53959
feat(examples): add run step route examples
mharrisb1 May 13, 2024
68262b4
chore(api): add run step routes
mharrisb1 May 13, 2024
cfd3817
chore(state): export resource type
mharrisb1 May 13, 2024
e1d17bd
fix(types): make some run step fields required
mharrisb1 May 13, 2024
874d2dd
feat(helpers): add helper for putting resource in state store
mharrisb1 May 13, 2024
303d559
feat(builders): add build message helper
mharrisb1 May 13, 2024
a38a4e8
feat(builders): add build run step helper
mharrisb1 May 13, 2024
c1582da
chore(tokens): remove token count estimates
mharrisb1 May 13, 2024
c794fb0
chore(docs): add breaking change warning about v0.3.0
mharrisb1 May 13, 2024
99ed3ed
chore(docs): remove design since its outdated
mharrisb1 May 13, 2024
d780c9d
fix(docs): change example to use new api
mharrisb1 May 13, 2024
c7dc593
chore(poetry): bump minor
mharrisb1 May 13, 2024
0fc05ed
chore(utils): remove token utils
mharrisb1 May 14, 2024
8d23634
feat(docs): update for new API
mharrisb1 May 14, 2024
79612f2
chore(api): remove unused const
mharrisb1 May 14, 2024
668be60
fix(docs): new link to coverage table
mharrisb1 May 14, 2024
14da26c
chore(docs): add link to examples dir
mharrisb1 May 14, 2024
fb167d0
chore(utils): remove token utils
mharrisb1 May 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
__pycache__/
.mypy_cache/
.pytest_cache/
.tox/
.venv/
.vscode/
__pycache__/
dist/
nbs/
site/

*.lock
.plan
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,55 @@
# Changelog

## 0.3.0

> [!IMPORTANT]
> **Breaking change**: Completely redesigned API

Introducing an all-new API that is both simpler to use and much more flexible. See [docs](https://mharrisb1.github.io/openai-responses-python) for more.

In addition to a new API, this release closed these issues:

- [#1: feat: ability to raise exceptions](https://github.com/mharrisb1/openai-responses-python/issues/1)
- [#9: feat: base url override](https://github.com/mharrisb1/openai-responses-python/issues/9)
- [#28: feat: automatically share state between chained mocks](https://github.com/mharrisb1/openai-responses-python/issues/28)

Additional notes:

- Removes token estimation. This is now the responsibility of the user to provided mock token count
- Adds more example files
- Still not completely happy with current state of mocking run steps. Will likely change in the near future.

## 0.2.1

> [!WARNING]
> Deprecated

Fixes issue where messages included in run create params (using `additional_messages`) was ignored.

- [#10: fix: create run ignores additional_messages](https://github.com/mharrisb1/openai-responses-python/issues/10)

## 0.2.0

> [!CAUTION]
> Yanked

Migrates assistant endpoints to Assistants V2

- [#8: feat: assistants v2](https://github.com/mharrisb1/openai-responses-python/issues/8)
- [#13: feat(endpoints): add token usage estimates to chat endpoint](https://github.com/mharrisb1/openai-responses-python/issues/13)

## 0.1.1

> [!CAUTION]
> Yanked

Fixes some issues with chat completions and other stateless mocks

- [#7: fix(endpoints): fix issues with chat completions endpoint](https://github.com/mharrisb1/openai-responses-python/issues/7)

## 0.1.0

> [!CAUTION]
> Yanked

Initial release with minimally useful support for what I needed.
46 changes: 8 additions & 38 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ I (Michael) am the BDFL of the project and can and will arbitrarily rank issues
- _Effort_ is a rough estimate of whether something will take a lot of work, some work, or little work to implement
- _Impact_ is a rough estimate of how important something is

Both are unscientific flawed but they allow me to try to focus on maximizing impact while minimizing effort which is important for an open source project.
Both are unscientific and flawed but they allow me to try to focus on what to work on when.

If you look at the [project's issues](https://github.com/mharrisb1/openai-responses-python/issues), you'll see they are labeled with some non-standard labels.
If you look at the [project's issues](https://github.com/mharrisb1/openai-responses-python/issues), you'll see they are labeled with some non-standard issue labels.

| Label | Description |
| -------------------------------------------------------------------------------------- | ------------- |
| <span style='background-color: green; padding: 2px; border-radius: 5px'>e0 🌵</span> | Low effort |
| Label | Description |
| ------------------------------------------------------------------------------------ | ------------- |
| <span style='background-color: green; padding: 2px; border-radius: 5px'>e0 🌵</span> | Low effort |
| <span style='background-color: yellow; padding: 2px; border-radius: 5px'>e1 ⚡️</span> | Medium effort |
| <span style='background-color: red; padding: 2px; border-radius: 5px'>e2 🔥</span> | High effort |
| <span style='background-color: green; padding: 2px; border-radius: 5px'>i0 🌵</span> | Low impact |
| <span style='background-color: red; padding: 2px; border-radius: 5px'>e2 🔥</span> | High effort |
| <span style='background-color: green; padding: 2px; border-radius: 5px'>i0 🌵</span> | Low impact |
| <span style='background-color: yellow; padding: 2px; border-radius: 5px'>i1 ⚡️</span> | Medium impact |
| <span style='background-color: red; padding: 2px; border-radius: 5px'>i2 🔥</span> | High impact |
| <span style='background-color: red; padding: 2px; border-radius: 5px'>i2 🔥</span> | High impact |

Ranking is evaluated according to this quadrant where the priority is ordered from top to bottom and left to right.

Expand Down Expand Up @@ -75,33 +75,3 @@ poetry install --with dev # install deps including development
poetry shell # activate venv
tox run # run lint, static analysis, unit tests, and examples
```

## Design Overview

### Mocks

Mocks are classes that are responsible for encapsulating all request patching of a given endpoint.

Endpoints are classified as either _stateless_ or _stateful_ mocks. Right now, the only difference between `StatelessMock` and `StatefulMock` is the injection of `used_state` (see [state store](#state-store) below for more).

```mermaid
classDiagram
Mock <-- StatelessMock
Mock <-- StatefulMock

StatelessMock <-- ChatCompletionMock
StatelessMock <-- EmbeddingsMock

StatefulMock <-- FilesMock
StatefulMock <-- AssistantsMock
StatefulMock <-- ThreadsMock
StatefulMock <-- MessagesMock
StatefulMock <-- RunsMock
StatefulMock <-- RunStepsMock
```

### State store

The state store is responsible for managing the state of all mocked objects throughout the lifetime of the test scope (function, module, session).

The current implementation of the state store is just a naive dictionary-based KV stores with support for common CRUD operations.
48 changes: 22 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,44 @@
# 🧪🤖 openai-responses

Pytest plugin for automatically mocking OpenAI requests. Built on top of [RESPX](https://github.com/lundberg/respx).
Pytest plugin for automatically mocking OpenAI requests. Powered by [RESPX](https://github.com/lundberg/respx).

## Supported endpoints
[![sdk support](https://img.shields.io/badge/SDK_Support-v1.25+-white?logo=openai&logoColor=black&labelColor=white)](https://github.com/openai/openai-python)

## Supported Routes

- `/v1/chat/completions`
- `/v1/embeddings`
- `/v1/files`
- `/v1/assistants`
- `/v1/threads` (+ messages, runs, and steps)

View full support coverage [here](https://mharrisb1.github.io/openai-responses-python/endpoints/).
View full support coverage [here](https://mharrisb1.github.io/openai-responses-python/routes).

## Usage

Simply decorate any test function that contains code that makes a call to an OpenAI endpoint. See [docs](https://mharrisb1.github.io/openai-responses-python) for more.
Just decorate any test function that makes a call to the OpenAI API (either using the [official Python SDK](https://github.com/openai/openai-python) or with [HTTPX](https://www.python-httpx.org/)).

```python
import openai

import openai_responses
from openai import OpenAI


@openai_responses.mock.chat.completions(
choices=[
{"message": {"content": "Hello, how can I help?"}},
{"message": {"content": "Hi! I'm here to help!"}},
{"message": {"content": "How can I help?"}},
],
)
def test_create_completion_with_multiple_choices():
client = OpenAI(api_key="fakeKey")
completion = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"},
],
n=3,


@openai_responses.mock()
def test_create_assistant():
client = openai.Client(api_key="sk-fake123")

assistant = client.beta.assistants.create(
instructions="You are a personal math tutor.",
name="Math Tutor",
tools=[{"type": "code_interpreter"}],
model="gpt-4-turbo",
)
assert len(completion.choices) == 3

assert assistant.name == "Math Tutor"
```

> [!IMPORTANT]
> This project does not try to generate fake responses from the models. Any part of a response that would be generated by a model will need to be defined by the user or will fallback to a default value.
See [examples](https://github.com/mharrisb1/openai-responses-python/tree/main/examples) or [docs](https://mharrisb1.github.io/openai-responses-python) for more.

## Installation

Expand Down
49 changes: 0 additions & 49 deletions docs/endpoints/assistants/assistants.md

This file was deleted.

44 changes: 0 additions & 44 deletions docs/endpoints/assistants/messages.md

This file was deleted.

30 changes: 0 additions & 30 deletions docs/endpoints/assistants/run_steps.md

This file was deleted.

58 changes: 0 additions & 58 deletions docs/endpoints/assistants/runs.md

This file was deleted.

Loading
Loading