diff --git a/.github/workflows/test-genai-function-calling.yml b/.github/workflows/test-genai-function-calling.yml new file mode 100644 index 0000000..b6d31e8 --- /dev/null +++ b/.github/workflows/test-genai-function-calling.yml @@ -0,0 +1,28 @@ +name: test-genai-function-calling + +on: + pull_request: + branches: + - main + paths: + - 'genai-function-calling/openai-agents/**' + - '!**/*.md' + - '!**/*.png' + +jobs: + test: + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.12 + uses: actions/setup-python@v5 + with: + python-version: 3.12 + + - name: openai-agents + run: | + pip install -r requirements.txt + pip install -r requirements-dev.txt + pytest --vcr-record=none + working-directory: genai-function-calling/openai-agents diff --git a/genai-function-calling/README.md b/genai-function-calling/README.md index 30095f2..a8776cc 100644 --- a/genai-function-calling/README.md +++ b/genai-function-calling/README.md @@ -15,6 +15,7 @@ and Kibana. Here are the examples: +* [OpenAI Agents SDK (Python)](openai-agents) * [Semantic Kernel .NET](semantic-kernel-dotnet) * [Spring AI (Java)](spring-ai) * [Vercel AI (Node.js)](vercel-ai) @@ -60,9 +61,10 @@ flexibility in defining and testing functions. ## Observability with EDOT -Each example uses a framework with built-in OpenTelemetry instrumentation. -While features vary, each of these produces at least traces, and some also logs -and metrics. +The OpenTelemetry instrumentation approach varies per GenAI framework. Some are +[native][native] (their codebase includes OpenTelemetry code), while others +rely on external instrumentation libraries. Signals vary as well. While all +produce traces, only some produce logs or metrics. We use Elastic Distributions of OpenTelemetry (EDOT) SDKs to enable these features and fill in other data, such as HTTP requests underlying the LLM and @@ -70,6 +72,11 @@ tool calls. In doing so, this implements the "zero code instrumentation" pattern of OpenTelemetry. Here's an example Kibana screenshot of one of the examples, looked up from a -query like: http://localhost:5601/app/apm/traces?rangeFrom=now-15m&rangeTo=now +query like: + +http://localhost:5601/app/apm/traces?rangeFrom=now-15m&rangeTo=now ![Kibana screenshot](./kibana-trace.png) + +--- +[native]: https://opentelemetry.io/docs/languages/java/instrumentation/#native-instrumentation \ No newline at end of file diff --git a/genai-function-calling/kibana-trace.png b/genai-function-calling/kibana-trace.png index b2e724f..1957293 100644 Binary files a/genai-function-calling/kibana-trace.png and b/genai-function-calling/kibana-trace.png differ diff --git a/genai-function-calling/openai-agents/Dockerfile b/genai-function-calling/openai-agents/Dockerfile new file mode 100644 index 0000000..4b534df --- /dev/null +++ b/genai-function-calling/openai-agents/Dockerfile @@ -0,0 +1,18 @@ +# Use glibc-based image with pre-compiled wheels for psutil +FROM python:3.12-slim + +# TODO: temporary until openai-agents 0.0.5 +RUN apt-get update \ + && apt-get install -y --no-install-recommends git \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +RUN --mount=type=cache,target=/root/.cache/pip python -m pip install --upgrade pip + +COPY requirements.txt /tmp +RUN --mount=type=cache,target=/root/.cache/pip pip install -r /tmp/requirements.txt +RUN --mount=type=cache,target=/root/.cache/pip edot-bootstrap --action=install + +COPY main.py / + +CMD [ "python", "main.py" ] diff --git a/genai-function-calling/openai-agents/README.md b/genai-function-calling/openai-agents/README.md new file mode 100644 index 0000000..3e8e3d7 --- /dev/null +++ b/genai-function-calling/openai-agents/README.md @@ -0,0 +1,90 @@ +# Function Calling with OpenAI Agents SDK (Python) + +[main.py](main.py) implements the [example application flow][flow] using +[OpenAI Agents SDK (Python)][openai-agents-python]. + +[Dockerfile](Dockerfile) starts the application with Elastic Distribution +of OpenTelemetry (EDOT) Python, via `opentelemetry-instrument`. + +Notably, this shows how to add extra instrumentation to EDOT, as the OpenAI +Agents support is via [OpenInference][openinference]. + +## Configure + +Copy [env.example](env.example) to `.env` and update its `OPENAI_API_KEY`. + +An OTLP compatible endpoint should be listening for traces, metrics and logs on +`http://localhost:4317`. If not, update `OTEL_EXPORTER_OTLP_ENDPOINT` as well. + +For example, if Elastic APM server is running locally, edit `.env` like this: +``` +OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200 +``` + +## Run with Docker + +```bash +docker compose run --build --rm genai-function-calling +``` + +## Run with Python + +First, set up a Python virtual environment like this: +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install --upgrade pip +pip install 'python-dotenv[cli]' +``` + +Next, install required packages: +```bash +pip install -r requirements.txt +``` + +Now, use EDOT to bootstrap instrumentation (this only needs to happen once): +```bash +edot-bootstrap --action=install +``` + +Finally, run `main.py` (notice the prefix of `opentelemetry-instrument): +```bash +dotenv run --no-override -- opentelemetry-instrument python main.py +``` + +## Tests + +Tests use [pytest-vcr][pytest-vcr] to capture HTTP traffic for offline unit +testing. Recorded responses keeps test passing considering LLMs are +non-deterministic and the Elasticsearch version list changes frequently. + +Run like this: +```bash +pip install -r requirements-dev.txt +pytest +``` + +OpenAI responses routinely change as they add features, and some may cause +failures. To re-record, delete [cassettes/test_main.yaml][test_main.yaml], and +run pytest with dotenv, so that ENV variables are present: + +```bash +rm cassettes/test_main.yaml +dotenv -f ../.env run -- pytest +``` + +## Notes + +The LLM should generate something like "The latest stable version of +Elasticsearch is 8.17.3", unless it hallucinates. Just run it again, if you +see something else. + +OpenAI Agents SDK's OpenTelemetry instrumentation is via +[OpenInference][openinference] and only produces traces (not logs or metrics). + +--- +[flow]: ../README.md#example-application-flow +[openai-agents-python]: https://github.com/openai/openai-agents-python +[pytest-vcr]: https://pytest-vcr.readthedocs.io/ +[test_main.yaml]: cassettes/test_main.yaml +[openinference]: https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai-agents diff --git a/genai-function-calling/openai-agents/cassettes/test_main.yaml b/genai-function-calling/openai-agents/cassettes/test_main.yaml new file mode 100644 index 0000000..e8f3d7a --- /dev/null +++ b/genai-function-calling/openai-agents/cassettes/test_main.yaml @@ -0,0 +1,1158 @@ +interactions: +- request: + body: |- + { + "messages": [ + { + "role": "user", + "content": "What is the latest version of Elasticsearch 8?" + } + ], + "model": "gpt-4o-mini", + "stream": false, + "temperature": 0, + "tools": [ + { + "type": "function", + "function": { + "name": "get_latest_elasticsearch_version", + "description": "Returns the latest GA version of Elasticsearch in \"X.Y.Z\" format.", + "parameters": { + "properties": { + "major_version": { + "default": 0, + "description": "Major version to filter by (e.g. 7, 8). Defaults to latest", + "title": "Major Version", + "type": "integer" + } + }, + "title": "get_latest_elasticsearch_version_args", + "type": "object" + } + } + } + ] + } + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '551' + content-type: + - application/json + host: + - api.openai.com + user-agent: + - Agents/Python 0.0.0 + x-stainless-arch: + - arm64 + x-stainless-async: + - async:asyncio + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.66.3 + x-stainless-read-timeout: + - '600' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.9 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + body: + string: |- + { + "id": "chatcmpl-BCezuQjRIx8MotUzQJcjJSYe9TyHv", + "object": "chat.completion", + "created": 1742357230, + "model": "gpt-4o-mini-2024-07-18", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_pT4CJ0D2kmnTP5WoVhv2edrt", + "type": "function", + "function": { + "name": "get_latest_elasticsearch_version", + "arguments": "{\"major_version\":8}" + } + } + ], + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 104, + "completion_tokens": 19, + "total_tokens": 123, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_b8bc95a0ac" + } + headers: + CF-RAY: + - 922a19f41e4a071d-ATL + Connection: + - keep-alive + Content-Type: + - application/json + Date: + - Wed, 19 Mar 2025 04:07:11 GMT + Server: + - cloudflare + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + cf-cache-status: + - DYNAMIC + content-length: + - '1108' + openai-processing-ms: + - '502' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '10000' + x-ratelimit-limit-tokens: + - '200000' + x-ratelimit-remaining-requests: + - '9999' + x-ratelimit-remaining-tokens: + - '199971' + x-ratelimit-reset-requests: + - 8.64s + x-ratelimit-reset-tokens: + - 8ms + x-request-id: + - req_2f55cdcb162fb892f0050536bbd987b5 + status: + code: 200 + message: OK +- request: + body: '' + headers: + accept: + - '*/*' + accept-encoding: + - gzip, deflate + connection: + - keep-alive + host: + - artifacts.elastic.co + user-agent: + - python-httpx/0.28.1 + method: GET + uri: https://artifacts.elastic.co/releases/stack.json + response: + body: + string: |- + { + "releases": [ + { + "version": "6.7.0", + "public_release_date": "2019-03-26" + }, + { + "version": "6.7.1", + "public_release_date": "2019-04-04" + }, + { + "version": "6.7.2", + "public_release_date": "2019-05-02" + }, + { + "version": "6.8.0", + "public_release_date": "2019-05-20" + }, + { + "version": "6.8.1", + "public_release_date": "2019-06-20" + }, + { + "version": "6.8.10", + "public_release_date": "2020-06-03" + }, + { + "version": "6.8.11", + "public_release_date": "2020-07-27" + }, + { + "version": "6.8.12", + "public_release_date": "2020-08-18" + }, + { + "version": "6.8.13", + "public_release_date": "2020-10-22" + }, + { + "version": "6.8.14", + "public_release_date": "2021-02-10" + }, + { + "version": "6.8.15", + "public_release_date": "2021-03-23" + }, + { + "version": "6.8.16", + "public_release_date": "2021-05-25" + }, + { + "version": "6.8.17", + "public_release_date": "2021-07-07" + }, + { + "version": "6.8.18", + "public_release_date": "2021-08-03" + }, + { + "version": "6.8.19", + "public_release_date": "2021-09-21" + }, + { + "version": "6.8.2", + "public_release_date": "2019-07-30" + }, + { + "version": "6.8.20", + "public_release_date": "2021-10-14" + }, + { + "version": "6.8.21", + "public_release_date": "2021-12-13" + }, + { + "version": "6.8.22", + "public_release_date": "2021-12-19" + }, + { + "version": "6.8.23", + "public_release_date": "2022-01-13" + }, + { + "version": "6.8.3", + "public_release_date": "2019-09-05" + }, + { + "version": "6.8.4", + "public_release_date": "2019-10-23" + }, + { + "version": "6.8.5", + "public_release_date": "2019-11-20" + }, + { + "version": "6.8.6", + "public_release_date": "2019-12-18" + }, + { + "version": "6.8.7", + "public_release_date": "2020-03-03" + }, + { + "version": "6.8.8", + "public_release_date": "2020-03-31" + }, + { + "version": "6.8.9", + "public_release_date": "2020-05-13" + }, + { + "version": "7.0.0 GA", + "public_release_date": "2019-04-10" + }, + { + "version": "7.0.0-alpha1", + "public_release_date": "2019-04-12" + }, + { + "version": "7.0.0-alpha2", + "public_release_date": "2019-04-12" + }, + { + "version": "7.0.0-beta1", + "public_release_date": "2019-04-10" + }, + { + "version": "7.0.0-rc1", + "public_release_date": "2019-04-10" + }, + { + "version": "7.0.0-rc2", + "public_release_date": "2019-04-10" + }, + { + "version": "7.0.1", + "public_release_date": "2019-05-01" + }, + { + "version": "7.1.0", + "public_release_date": "2019-05-20" + }, + { + "version": "7.1.1", + "public_release_date": "2019-05-28" + }, + { + "version": "7.10.0", + "public_release_date": "2020-11-11" + }, + { + "version": "7.10.1", + "public_release_date": "2020-12-09" + }, + { + "version": "7.10.2", + "public_release_date": "2021-01-14" + }, + { + "version": "7.11.0", + "public_release_date": "2021-02-10" + }, + { + "version": "7.11.1", + "public_release_date": "2021-02-17" + }, + { + "version": "7.11.2", + "public_release_date": "2021-03-10" + }, + { + "version": "7.12.0", + "public_release_date": "2021-03-23" + }, + { + "version": "7.12.1", + "public_release_date": "2021-04-27" + }, + { + "version": "7.13.0", + "public_release_date": "2021-05-25" + }, + { + "version": "7.13.1", + "public_release_date": "2021-06-02" + }, + { + "version": "7.13.2", + "public_release_date": "2021-06-14" + }, + { + "version": "7.13.3", + "public_release_date": "2021-07-07" + }, + { + "version": "7.13.4", + "public_release_date": "2021-07-20" + }, + { + "version": "7.14.0", + "public_release_date": "2021-08-03" + }, + { + "version": "7.14.1", + "public_release_date": "2021-09-01" + }, + { + "version": "7.14.2", + "public_release_date": "2021-09-21" + }, + { + "version": "7.15.0", + "public_release_date": "2021-09-22" + }, + { + "version": "7.15.1", + "public_release_date": "2021-10-14" + }, + { + "version": "7.15.2", + "public_release_date": "2021-11-10" + }, + { + "version": "7.16.0", + "public_release_date": "2021-12-07" + }, + { + "version": "7.16.1", + "public_release_date": "2021-12-13" + }, + { + "version": "7.16.2", + "public_release_date": "2021-12-19" + }, + { + "version": "7.16.3", + "public_release_date": "2022-01-13" + }, + { + "version": "7.17.0", + "public_release_date": "2022-02-01" + }, + { + "version": "7.17.1", + "public_release_date": "2022-02-28" + }, + { + "version": "7.17.10", + "public_release_date": "2023-05-02" + }, + { + "version": "7.17.11", + "public_release_date": "2023-06-29" + }, + { + "version": "7.17.12", + "public_release_date": "2023-07-25" + }, + { + "version": "7.17.13", + "public_release_date": "2023-09-06" + }, + { + "version": "7.17.14", + "public_release_date": "2023-10-10" + }, + { + "version": "7.17.15", + "public_release_date": "2023-11-13" + }, + { + "version": "7.17.16", + "public_release_date": "2023-12-12" + }, + { + "version": "7.17.17", + "public_release_date": "2024-01-23" + }, + { + "version": "7.17.18", + "public_release_date": "2024-02-06" + }, + { + "version": "7.17.19", + "public_release_date": "2024-03-26" + }, + { + "version": "7.17.2", + "public_release_date": "2022-03-31" + }, + { + "version": "7.17.20", + "public_release_date": "2024-04-08" + }, + { + "version": "7.17.21", + "public_release_date": "2024-05-02" + }, + { + "version": "7.17.22", + "public_release_date": "2024-06-13" + }, + { + "version": "7.17.23", + "public_release_date": "2024-07-30", + "manifest": "https://artifacts.elastic.co/downloads/7.17.23.json" + }, + { + "version": "7.17.24", + "public_release_date": "2024-09-10", + "manifest": "https://artifacts.elastic.co/downloads/7.17.24.json" + }, + { + "version": "7.17.25", + "public_release_date": "2024-10-22", + "manifest": "https://artifacts.elastic.co/downloads/7.17.25.json" + }, + { + "version": "7.17.26", + "public_release_date": "2024-12-03", + "manifest": "https://artifacts.elastic.co/downloads/7.17.26.json" + }, + { + "version": "7.17.28", + "public_release_date": "2025-02-25", + "manifest": "https://artifacts.elastic.co/downloads/7.17.28.json" + }, + { + "version": "7.17.3", + "public_release_date": "2022-04-20" + }, + { + "version": "7.17.4", + "public_release_date": "2022-05-24" + }, + { + "version": "7.17.5", + "public_release_date": "2022-06-28" + }, + { + "version": "7.17.6", + "public_release_date": "2022-08-24" + }, + { + "version": "7.17.7", + "public_release_date": "2022-10-25" + }, + { + "version": "7.17.8", + "public_release_date": "2022-12-08" + }, + { + "version": "7.17.9", + "public_release_date": "2023-02-02" + }, + { + "version": "7.2.0", + "public_release_date": "2019-06-25" + }, + { + "version": "7.2.1", + "public_release_date": "2019-07-30" + }, + { + "version": "7.3.0", + "public_release_date": "2019-07-31" + }, + { + "version": "7.3.1", + "public_release_date": "2019-08-22" + }, + { + "version": "7.3.2", + "public_release_date": "2019-09-12" + }, + { + "version": "7.4.0", + "public_release_date": "2019-10-01" + }, + { + "version": "7.4.1", + "public_release_date": "2019-10-23" + }, + { + "version": "7.4.2", + "public_release_date": "2019-10-31" + }, + { + "version": "7.5.0", + "public_release_date": "2019-12-02" + }, + { + "version": "7.5.1", + "public_release_date": "2019-12-18" + }, + { + "version": "7.5.2", + "public_release_date": "2020-01-21" + }, + { + "version": "7.6.0", + "public_release_date": "2020-02-11" + }, + { + "version": "7.6.1", + "public_release_date": "2020-03-03" + }, + { + "version": "7.6.2", + "public_release_date": "2020-03-31" + }, + { + "version": "7.7.0", + "public_release_date": "2020-05-13" + }, + { + "version": "7.7.1", + "public_release_date": "2020-06-03" + }, + { + "version": "7.8.0", + "public_release_date": "2020-06-18" + }, + { + "version": "7.8.1", + "public_release_date": "2020-07-27" + }, + { + "version": "7.9.0", + "public_release_date": "2020-08-18" + }, + { + "version": "7.9.1", + "public_release_date": "2020-09-03" + }, + { + "version": "7.9.2", + "public_release_date": "2020-09-24" + }, + { + "version": "7.9.3", + "public_release_date": "2020-10-22" + }, + { + "version": "8.0.0 GA", + "public_release_date": "2022-02-10" + }, + { + "version": "8.0.0-alpha1", + "public_release_date": "2021-08-10" + }, + { + "version": "8.0.0-alpha2", + "public_release_date": "2021-09-16" + }, + { + "version": "8.0.0-beta1", + "public_release_date": "2021-11-09" + }, + { + "version": "8.0.0-rc1", + "public_release_date": "2022-01-12" + }, + { + "version": "8.0.0-rc2", + "public_release_date": "2022-02-03" + }, + { + "version": "8.0.1", + "public_release_date": "2022-03-01" + }, + { + "version": "8.1.0", + "public_release_date": "2022-03-08" + }, + { + "version": "8.1.1", + "public_release_date": "2022-03-22" + }, + { + "version": "8.1.2", + "public_release_date": "2022-03-31" + }, + { + "version": "8.1.3", + "public_release_date": "2022-04-20" + }, + { + "version": "8.10.0", + "public_release_date": "2023-09-12" + }, + { + "version": "8.10.1", + "public_release_date": "2023-09-18" + }, + { + "version": "8.10.2", + "public_release_date": "2023-09-21" + }, + { + "version": "8.10.3", + "public_release_date": "2023-10-10" + }, + { + "version": "8.10.4", + "public_release_date": "2023-10-17" + }, + { + "version": "8.11.0", + "public_release_date": "2023-11-07" + }, + { + "version": "8.11.1", + "public_release_date": "2023-11-13" + }, + { + "version": "8.11.2", + "public_release_date": "2023-12-07" + }, + { + "version": "8.11.3", + "public_release_date": "2023-12-12" + }, + { + "version": "8.11.4", + "public_release_date": "2024-01-11" + }, + { + "version": "8.12.0", + "public_release_date": "2024-01-17" + }, + { + "version": "8.12.1", + "public_release_date": "2024-02-06" + }, + { + "version": "8.12.2", + "public_release_date": "2024-02-22" + }, + { + "version": "8.13.0", + "public_release_date": "2024-03-26" + }, + { + "version": "8.13.1", + "public_release_date": "2024-04-02" + }, + { + "version": "8.13.2", + "public_release_date": "2024-04-08" + }, + { + "version": "8.13.3", + "public_release_date": "2024-05-02" + }, + { + "version": "8.13.4", + "public_release_date": "2024-05-08" + }, + { + "version": "8.14.0", + "public_release_date": "2024-06-05" + }, + { + "version": "8.14.1", + "public_release_date": "2024-06-12" + }, + { + "version": "8.14.2", + "public_release_date": "2024-07-04", + "manifest": "https://artifacts.elastic.co/downloads/8.14.2.json" + }, + { + "version": "8.14.3", + "public_release_date": "2024-07-11", + "manifest": "https://artifacts.elastic.co/downloads/8.14.3.json" + }, + { + "version": "8.15.0", + "public_release_date": "2024-08-08", + "manifest": "https://artifacts.elastic.co/downloads/8.15.0.json" + }, + { + "version": "8.15.1", + "public_release_date": "2024-09-05", + "manifest": "https://artifacts.elastic.co/downloads/8.15.1.json" + }, + { + "version": "8.15.2", + "public_release_date": "2024-09-26", + "manifest": "https://artifacts.elastic.co/downloads/8.15.2.json" + }, + { + "version": "8.15.3", + "public_release_date": "2024-10-17", + "manifest": "https://artifacts.elastic.co/downloads/8.15.3.json" + }, + { + "version": "8.15.4", + "public_release_date": "2024-11-12", + "manifest": "https://artifacts.elastic.co/downloads/8.15.4.json" + }, + { + "version": "8.15.5", + "public_release_date": "2024-11-26", + "manifest": "https://artifacts.elastic.co/downloads/8.15.5.json" + }, + { + "version": "8.16.0", + "public_release_date": "2024-11-12", + "manifest": "https://artifacts.elastic.co/downloads/8.16.0.json" + }, + { + "version": "8.16.1", + "public_release_date": "2024-11-21", + "manifest": "https://artifacts.elastic.co/downloads/8.16.1.json" + }, + { + "version": "8.16.2", + "public_release_date": "2024-12-17", + "manifest": "https://artifacts.elastic.co/downloads/8.16.2.json" + }, + { + "version": "8.16.3", + "public_release_date": "2025-01-21", + "manifest": "https://artifacts.elastic.co/downloads/8.16.3.json" + }, + { + "version": "8.16.4", + "public_release_date": "2025-02-11", + "manifest": "https://artifacts.elastic.co/downloads/8.16.4.json" + }, + { + "version": "8.16.5", + "public_release_date": "2025-03-04", + "manifest": "https://artifacts.elastic.co/downloads/8.16.5.json" + }, + { + "version": "8.17.0", + "public_release_date": "2024-12-12", + "manifest": "https://artifacts.elastic.co/downloads/8.17.0.json" + }, + { + "version": "8.17.1", + "public_release_date": "2025-01-21", + "manifest": "https://artifacts.elastic.co/downloads/8.17.1.json" + }, + { + "version": "8.17.2", + "public_release_date": "2025-02-11", + "manifest": "https://artifacts.elastic.co/downloads/8.17.2.json" + }, + { + "version": "8.17.3", + "public_release_date": "2025-03-04", + "manifest": "https://artifacts.elastic.co/downloads/8.17.3.json" + }, + { + "version": "8.2.0", + "public_release_date": "2022-05-03" + }, + { + "version": "8.2.1", + "public_release_date": "2022-05-24" + }, + { + "version": "8.2.2", + "public_release_date": "2022-05-26" + }, + { + "version": "8.2.3", + "public_release_date": "2022-06-14" + }, + { + "version": "8.3.0", + "public_release_date": "2022-06-28" + }, + { + "version": "8.3.1", + "public_release_date": "2022-06-30" + }, + { + "version": "8.3.2", + "public_release_date": "2022-07-07" + }, + { + "version": "8.3.3", + "public_release_date": "2022-07-28" + }, + { + "version": "8.4.0", + "public_release_date": "2022-08-24" + }, + { + "version": "8.4.1", + "public_release_date": "2022-08-30" + }, + { + "version": "8.4.2", + "public_release_date": "2022-09-20" + }, + { + "version": "8.4.3", + "public_release_date": "2022-10-05" + }, + { + "version": "8.5.0", + "public_release_date": "2022-11-01" + }, + { + "version": "8.5.1", + "public_release_date": "2022-11-15" + }, + { + "version": "8.5.2", + "public_release_date": "2022-11-22" + }, + { + "version": "8.5.3", + "public_release_date": "2022-12-08" + }, + { + "version": "8.6.0", + "public_release_date": "2023-01-10" + }, + { + "version": "8.6.1", + "public_release_date": "2023-01-26" + }, + { + "version": "8.6.2", + "public_release_date": "2023-02-16" + }, + { + "version": "8.7.0", + "public_release_date": "2023-03-30" + }, + { + "version": "8.7.1", + "public_release_date": "2023-05-02" + }, + { + "version": "8.7.2", + "public_release_date": "2023-05-23" + }, + { + "version": "8.8.0", + "public_release_date": "2023-05-25" + }, + { + "version": "8.8.1", + "public_release_date": "2023-06-08" + }, + { + "version": "8.8.2", + "public_release_date": "2023-06-29" + }, + { + "version": "8.9.0", + "public_release_date": "2023-07-25" + }, + { + "version": "8.9.1", + "public_release_date": "2023-08-17" + }, + { + "version": "8.9.2", + "public_release_date": "2023-09-06" + }, + { + "version": "9.0.0-beta1", + "public_release_date": "2025-02-18", + "manifest": "https://artifacts.elastic.co/downloads/9.0.0-beta1.json" + } + ] + } + headers: + Age: + - '306' + Alt-Svc: + - h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 + Cache-Control: + - public, max-age=600 + Content-Type: + - application/json + Date: + - Wed, 19 Mar 2025 04:02:05 GMT + ETag: + - W/"d008556073cae98499043fe65c244543" + Last-Modified: + - Fri, 07 Mar 2025 09:30:54 GMT + Strict-Transport-Security: + - max-age=31536000; includeSubDomains + Vary: + - Accept-Encoding + Via: + - 1.1 google + content-length: + - '20984' + status: + code: 200 + message: OK +- request: + body: |- + { + "messages": [ + { + "role": "user", + "content": "What is the latest version of Elasticsearch 8?" + }, + { + "role": "assistant", + "tool_calls": [ + { + "id": "call_pT4CJ0D2kmnTP5WoVhv2edrt", + "type": "function", + "function": { + "name": "get_latest_elasticsearch_version", + "arguments": "{\"major_version\":8}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_pT4CJ0D2kmnTP5WoVhv2edrt", + "content": "8.17.3" + } + ], + "model": "gpt-4o-mini", + "stream": false, + "temperature": 0, + "tools": [ + { + "type": "function", + "function": { + "name": "get_latest_elasticsearch_version", + "description": "Returns the latest GA version of Elasticsearch in \"X.Y.Z\" format.", + "parameters": { + "properties": { + "major_version": { + "default": 0, + "description": "Major version to filter by (e.g. 7, 8). Defaults to latest", + "title": "Major Version", + "type": "integer" + } + }, + "title": "get_latest_elasticsearch_version_args", + "type": "object" + } + } + } + ] + } + headers: + accept: + - application/json + accept-encoding: + - gzip, deflate + connection: + - keep-alive + content-length: + - '817' + content-type: + - application/json + host: + - api.openai.com + user-agent: + - Agents/Python 0.0.0 + x-stainless-arch: + - arm64 + x-stainless-async: + - async:asyncio + x-stainless-lang: + - python + x-stainless-os: + - MacOS + x-stainless-package-version: + - 1.66.3 + x-stainless-read-timeout: + - '600' + x-stainless-retry-count: + - '0' + x-stainless-runtime: + - CPython + x-stainless-runtime-version: + - 3.12.9 + method: POST + uri: https://api.openai.com/v1/chat/completions + response: + body: + string: |- + { + "id": "chatcmpl-BCezvs9dfzeyJNyWqWMTNrHfQlfHh", + "object": "chat.completion", + "created": 1742357231, + "model": "gpt-4o-mini-2024-07-18", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "The latest version of Elasticsearch 8 is 8.17.3.", + "refusal": null, + "annotations": [] + }, + "logprobs": null, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 138, + "completion_tokens": 17, + "total_tokens": 155, + "prompt_tokens_details": { + "cached_tokens": 0, + "audio_tokens": 0 + }, + "completion_tokens_details": { + "reasoning_tokens": 0, + "audio_tokens": 0, + "accepted_prediction_tokens": 0, + "rejected_prediction_tokens": 0 + } + }, + "service_tier": "default", + "system_fingerprint": "fp_b8bc95a0ac" + } + headers: + CF-RAY: + - 922a19f98d39071d-ATL + Connection: + - keep-alive + Content-Type: + - application/json + Date: + - Wed, 19 Mar 2025 04:07:12 GMT + Server: + - cloudflare + Transfer-Encoding: + - chunked + X-Content-Type-Options: + - nosniff + access-control-expose-headers: + - X-Request-ID + alt-svc: + - h3=":443"; ma=86400 + cf-cache-status: + - DYNAMIC + content-length: + - '857' + openai-processing-ms: + - '481' + openai-version: + - '2020-10-01' + strict-transport-security: + - max-age=31536000; includeSubDomains; preload + x-ratelimit-limit-requests: + - '10000' + x-ratelimit-limit-tokens: + - '200000' + x-ratelimit-remaining-requests: + - '9998' + x-ratelimit-remaining-tokens: + - '199967' + x-ratelimit-reset-requests: + - 16.52s + x-ratelimit-reset-tokens: + - 9ms + x-request-id: + - req_11be6b5836b6a31e8c1d9e64eec1314e + status: + code: 200 + message: OK +version: 1 diff --git a/genai-function-calling/openai-agents/conftest.py b/genai-function-calling/openai-agents/conftest.py new file mode 100644 index 0000000..72518b6 --- /dev/null +++ b/genai-function-calling/openai-agents/conftest.py @@ -0,0 +1,107 @@ +# +# Copyright Elasticsearch B.V. and contributors +# SPDX-License-Identifier: Apache-2.0 +# +import json +import os + +import pytest +import yaml + + +@pytest.fixture +def default_openai_env(monkeypatch): + """Prevent offline tests from failing due to requiring the ENV OPENAI_API_KEY.""" + + if "OPENAI_API_KEY" not in os.environ: + monkeypatch.setenv("OPENAI_API_KEY", "test_openai_api_key") + + +@pytest.fixture(scope="module") +def vcr_config(): + """Scrub sensitive headers and gunzip responses so they are readable""" + sensitive_request_headers = [ + "authorization", + "cookie", + "openai-organization", + "openai-project", + ] + sensitive_response_headers = {"openai-organization", "set-cookie"} + return { + "decode_compressed_response": True, + "filter_headers": sensitive_request_headers, + "before_record_response": lambda r: { + **r, + "headers": {k: v for k, v in r["headers"].items() if k.lower() not in sensitive_response_headers}, + }, + } + + +# Below this line ensures that cassettes are readable and pretty-printed. The +# default yaml serialization isn't easy to read or understand. Elastic also +# contributed this code to opentelemetry-python-contrib for the same reason. + + +class LiteralBlockScalar(str): + """Formats the string as a literal block scalar, preserving whitespace and + without interpreting escape characters""" + + +def literal_block_scalar_presenter(dumper, data): + """Represents a scalar string as a literal block, via '|' syntax""" + return dumper.represent_scalar("tag:yaml.org,2002:str", data, style="|") + + +yaml.add_representer(LiteralBlockScalar, literal_block_scalar_presenter) + + +def process_string_value(string_value): + """Pretty-prints JSON or returns long strings as a LiteralBlockScalar""" + try: + json_data = json.loads(string_value) + return LiteralBlockScalar(json.dumps(json_data, indent=2)) + except (ValueError, TypeError): + if len(string_value) > 80: + return LiteralBlockScalar(string_value) + return string_value + + +def convert_body_to_literal(data): + """Searches the data for body strings, attempting to pretty-print JSON""" + if isinstance(data, dict): + for key, value in data.items(): + # Handle response body case (e.g., response.body.string) + if key == "body" and isinstance(value, dict) and "string" in value: + value["string"] = process_string_value(value["string"]) + + # Handle request body case (e.g., request.body) + elif key == "body" and isinstance(value, str): + data[key] = process_string_value(value) + + else: + convert_body_to_literal(value) + + elif isinstance(data, list): + for idx, choice in enumerate(data): + data[idx] = convert_body_to_literal(choice) + + return data + + +class PrettyPrintJSONBody: + """This makes request and response body recordings more readable.""" + + @staticmethod + def serialize(cassette_dict): + cassette_dict = convert_body_to_literal(cassette_dict) + return yaml.dump(cassette_dict, default_flow_style=False, allow_unicode=True) + + @staticmethod + def deserialize(cassette_string): + return yaml.load(cassette_string, Loader=yaml.Loader) + + +@pytest.fixture(scope="module", autouse=True) +def fixture_vcr(vcr): + vcr.register_serializer("yaml", PrettyPrintJSONBody) + return vcr diff --git a/genai-function-calling/openai-agents/docker-compose.yml b/genai-function-calling/openai-agents/docker-compose.yml new file mode 100644 index 0000000..3c4e195 --- /dev/null +++ b/genai-function-calling/openai-agents/docker-compose.yml @@ -0,0 +1,9 @@ +services: + genai-function-calling: + container_name: genai-function-calling + build: + context: . + env_file: + - .env + extra_hosts: # send localhost traffic to the docker host, e.g. your laptop + - "localhost:host-gateway" diff --git a/genai-function-calling/openai-agents/env.example b/genai-function-calling/openai-agents/env.example new file mode 100644 index 0000000..4e8b1f0 --- /dev/null +++ b/genai-function-calling/openai-agents/env.example @@ -0,0 +1,37 @@ +# Update this with your real OpenAI API key +OPENAI_API_KEY= + +# Uncomment to use Ollama instead of OpenAI +# OPENAI_BASE_URL=http://localhost:11434/v1 +# OPENAI_API_KEY=unused +# # This works when you supply a major_version parameter in your prompt. If you +# # leave it out, you need to update this to qwen2.5:3b to proceed the tool call. +# CHAT_MODEL=qwen2.5:0.5b + +# Uncomment and complete if you want to use Azure OpenAI Service +## "Azure OpenAI Endpoint" in https://oai.azure.com/resource/overview +# AZURE_OPENAI_ENDPOINT=https://YOUR_RESOURCE_NAME.openai.azure.com/ +## "API key 1 (or 2)" in https://oai.azure.com/resource/overview +# AZURE_OPENAI_API_KEY= +## "Inference version" from https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation +# OPENAI_API_VERSION=2024-10-01-preview +## "Name" from https://oai.azure.com/resource/deployments +# CHAT_MODEL=YOUR_DEPLOYMENT_NAME + +OTEL_SERVICE_NAME=genai-function-calling + +# OTEL_EXPORTER_* variables are not required. If you would like to change your +# OTLP endpoint to Elastic APM server using HTTP, uncomment the following: +# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200 +# OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf + +# Change to 'false' to hide prompt and completion content +OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true + +# Export metrics every 3 seconds instead of every minute +OTEL_METRIC_EXPORT_INTERVAL=3000 +# Export traces every 3 seconds instead of every 5 seconds +OTEL_BSP_SCHEDULE_DELAY=3000 +# Change to affect behavior of which resources are detected. Note: these +# choices are specific to the language, in this case Python. +OTEL_EXPERIMENTAL_RESOURCE_DETECTORS=process_runtime,os,otel,telemetry_distro diff --git a/genai-function-calling/openai-agents/main.py b/genai-function-calling/openai-agents/main.py new file mode 100644 index 0000000..32c2877 --- /dev/null +++ b/genai-function-calling/openai-agents/main.py @@ -0,0 +1,72 @@ +import asyncio +import os + +from httpx import AsyncClient +from agents import ( + Agent, + ModelSettings, + OpenAIProvider, + RunConfig, + Runner, + function_tool, +) +from agents.tracing import GLOBAL_TRACE_PROVIDER +from openai import AsyncAzureOpenAI + +# Shut down the global tracer as it sends to the OpenAI "/traces/ingest" +# endpoint, which we aren't using and doesn't exist on alternative backends +# like Ollama. +GLOBAL_TRACE_PROVIDER.shutdown() + + +@function_tool(strict_mode=False) +async def get_latest_elasticsearch_version(major_version: int = 0) -> str: + """Returns the latest GA version of Elasticsearch in "X.Y.Z" format. + + Args: + major_version: Major version to filter by (e.g. 7, 8). Defaults to latest + """ + async with AsyncClient() as client: + response = await client.get("https://artifacts.elastic.co/releases/stack.json") + response.raise_for_status() + releases = response.json()["releases"] + + # Fetch releases and filter out non-release versions (e.g., -rc1) or + # those not matching major_version. In any case, remove " GA" suffix. + versions = [] + for r in releases: + v = r["version"].removesuffix(" GA") + if "-" in r["version"]: + continue + if major_version and int(v.split(".")[0]) != major_version: + continue + versions.append(v) + + if not versions: + raise ValueError("No valid versions found") + + # "8.9.1" > "8.10.0", so coerce to a numeric tuple: (8,9,1) < (8,10,0) + return max(versions, key=lambda v: tuple(map(int, v.split(".")))) + + +async def main(): + model_name = os.getenv("CHAT_MODEL", "gpt-4o-mini") + openai_client = AsyncAzureOpenAI() if os.getenv("AZURE_OPENAI_API_KEY") else None + model = OpenAIProvider(openai_client=openai_client, use_responses=False).get_model(model_name) + agent = Agent( + name="version_assistant", + tools=[get_latest_elasticsearch_version], + model=model, + model_settings=ModelSettings(temperature=0), + ) + + result = await Runner.run( + starting_agent=agent, + input="What is the latest version of Elasticsearch 8?", + run_config=RunConfig(workflow_name="GetLatestElasticsearchVersion"), + ) + print(result.final_output) + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/genai-function-calling/openai-agents/main_test.py b/genai-function-calling/openai-agents/main_test.py new file mode 100644 index 0000000..98cf843 --- /dev/null +++ b/genai-function-calling/openai-agents/main_test.py @@ -0,0 +1,17 @@ +# +# Copyright Elasticsearch B.V. and contributors +# SPDX-License-Identifier: Apache-2.0 +# +import pytest + +from main import main + + +@pytest.mark.vcr +@pytest.mark.asyncio +async def test_main(default_openai_env, capsys): + await main() + + reply = capsys.readouterr().out.strip() + + assert reply == "The latest version of Elasticsearch 8 is 8.17.3." diff --git a/genai-function-calling/openai-agents/pytest.ini b/genai-function-calling/openai-agents/pytest.ini new file mode 100644 index 0000000..0102b0a --- /dev/null +++ b/genai-function-calling/openai-agents/pytest.ini @@ -0,0 +1,2 @@ +[pytest] +asyncio_default_fixture_loop_scope = function diff --git a/genai-function-calling/openai-agents/requirements-dev.txt b/genai-function-calling/openai-agents/requirements-dev.txt new file mode 100644 index 0000000..57a97db --- /dev/null +++ b/genai-function-calling/openai-agents/requirements-dev.txt @@ -0,0 +1,3 @@ +pytest +pytest-asyncio +pytest-vcr diff --git a/genai-function-calling/openai-agents/requirements.txt b/genai-function-calling/openai-agents/requirements.txt new file mode 100644 index 0000000..6bdeb3c --- /dev/null +++ b/genai-function-calling/openai-agents/requirements.txt @@ -0,0 +1,7 @@ +# TODO: temporary until openai-agents 0.0.5 +openai-agents @ git+https://github.com/openai/openai-agents-python.git@main +httpx~=0.28.1 + +elastic-opentelemetry~=0.8.0 +# Use openai-agents instrumentation from OpenInference +openinference-instrumentation-openai-agents~=0.1.1 diff --git a/genai-function-calling/semantic-kernel-dotnet/env.example b/genai-function-calling/semantic-kernel-dotnet/env.example index 6d11108..c617aa3 100644 --- a/genai-function-calling/semantic-kernel-dotnet/env.example +++ b/genai-function-calling/semantic-kernel-dotnet/env.example @@ -4,7 +4,9 @@ OPENAI_API_KEY= # Uncomment to use Ollama instead of OpenAI # OPENAI_BASE_URL=http://localhost:11434/v1 # OPENAI_API_KEY=unused -# CHAT_MODEL=qwen2.5:3b +# # This works when you supply a major_version parameter in your prompt. If you +# # leave it out, you need to update this to qwen2.5:3b to proceed the tool call. +# CHAT_MODEL=qwen2.5:0.5b # Uncomment and complete if you want to use Azure OpenAI Service ## "Azure OpenAI Endpoint" in https://oai.azure.com/resource/overview diff --git a/genai-function-calling/spring-ai/env.example b/genai-function-calling/spring-ai/env.example index 7f3eb0b..2689c41 100644 --- a/genai-function-calling/spring-ai/env.example +++ b/genai-function-calling/spring-ai/env.example @@ -4,7 +4,9 @@ OPENAI_API_KEY= # Uncomment to use Ollama instead of OpenAI # OPENAI_BASE_URL=http://localhost:11434/v1 # OPENAI_API_KEY=unused -# CHAT_MODEL=qwen2.5:3b +# # This works when you supply a major_version parameter in your prompt. If you +# # leave it out, you need to update this to qwen2.5:3b to proceed the tool call. +# CHAT_MODEL=qwen2.5:0.5b # Uncomment and complete if you want to use Azure OpenAI Service ## "Azure OpenAI Endpoint" in https://oai.azure.com/resource/overview diff --git a/genai-function-calling/vercel-ai/env.example b/genai-function-calling/vercel-ai/env.example index 3b99c13..d61250a 100644 --- a/genai-function-calling/vercel-ai/env.example +++ b/genai-function-calling/vercel-ai/env.example @@ -4,6 +4,7 @@ OPENAI_API_KEY= # Uncomment to use Ollama instead of OpenAI # OPENAI_BASE_URL=http://localhost:11434/v1 # OPENAI_API_KEY=unused +# # This needs qwen2.5:3b, as qwen2.5:0.5b doesn't process the tool call # CHAT_MODEL=qwen2.5:3b # Uncomment and complete if you want to use Azure OpenAI Service