Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: basic Support for OpenTelemetry Metrics and Token Usage Metrics in OpenAI V1 #369

Merged
merged 42 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
1d120c5
add metrics basic support and add metrics collect in OpenAIV1's opena…
Humbertzhang Jan 27, 2024
ea736fa
refine comments
Humbertzhang Jan 27, 2024
3ae9024
align with 'Semantic Conventions for OpenAI Metrics'
Humbertzhang Jan 31, 2024
e19ae5e
fix lint errors
Humbertzhang Feb 1, 2024
c8ee9c7
Merge branch 'main' into metrics
Humbertzhang Feb 1, 2024
f8897ea
Add test for openai metrics
Humbertzhang Feb 6, 2024
3fabd7a
fix lint
Humbertzhang Feb 6, 2024
3ce0840
complete chat completions metrics and add tests for it
Humbertzhang Feb 8, 2024
c1c4649
Merge branch 'main' into metrics
Humbertzhang Feb 8, 2024
b021421
Add embedding and image_gen metrics and corresponding tests
Humbertzhang Feb 10, 2024
db0b3e6
fix
Humbertzhang Feb 10, 2024
e045e50
Merge branch 'main' into metrics
Humbertzhang Feb 18, 2024
713d721
using vcr for tests in metrics
Humbertzhang Feb 18, 2024
787b026
No need for updating project.json
Humbertzhang Feb 18, 2024
0c97f61
Merge branch 'main' into metrics
nirga Feb 21, 2024
4d091c1
Merge branch 'main' into metrics
nirga Feb 21, 2024
0b3377c
Merge branch 'main' into metrics
Humbertzhang Feb 22, 2024
3f806da
fix stream test failed
Humbertzhang Feb 22, 2024
ee9e89d
fix: disable metrics endpoint for traceloop
nirga Feb 22, 2024
24098c0
chore: moved openai tests to openai instrumentation
nirga Feb 22, 2024
c690b0d
chore: added missing openai dep
nirga Feb 22, 2024
223ca4f
fix: remove unneeded import
nirga Feb 22, 2024
8fc4b51
merge metrics record with trace, fix issues
Humbertzhang Feb 25, 2024
0cfb092
Merge branch 'main' into metrics
Humbertzhang Feb 25, 2024
72ba1ac
fix opneai poetry.lock
Humbertzhang Feb 26, 2024
90e4a42
fix import issues
Humbertzhang Feb 26, 2024
7b9ee12
Merge branch 'main' into metrics
Humbertzhang Feb 26, 2024
8b6a097
Merge branch 'main' into metrics
nirga Feb 26, 2024
e41308a
Merge branch 'main' into metrics
nirga Feb 26, 2024
0be525d
fix merge conflict
Humbertzhang Feb 27, 2024
668c24f
Merge branch 'main' of https://github.com/traceloop/openllmetry into …
Humbertzhang Feb 27, 2024
9f807ae
Merge branch 'main' into metrics
nirga Feb 27, 2024
66aa8fc
chore(openai): regenerated cassettes
nirga Feb 27, 2024
a8cb641
Merge branch 'main' into metrics
nirga Feb 27, 2024
91a4883
chore: conftest
nirga Feb 27, 2024
e7062fc
chore(sdk): regenerated cassettes
nirga Feb 27, 2024
4e81ca7
chore: more cassettes
nirga Feb 27, 2024
bb9648c
chore: lock
nirga Feb 27, 2024
47f623c
chore(openai): missing test dep
nirga Feb 27, 2024
01c99a4
fix(sdk): reverted haystack comment out
nirga Feb 27, 2024
20dd07d
Merge branch 'main' into metrics
nirga Feb 27, 2024
2fb9ef6
fix: _get_openai_base_url now can really get openai base_url. Added t…
Humbertzhang Feb 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packages/opentelemetry-instrumentation-anthropic/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion packages/opentelemetry-instrumentation-chromadb/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion packages/opentelemetry-instrumentation-cohere/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion packages/opentelemetry-instrumentation-haystack/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions packages/opentelemetry-instrumentation-langchain/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -40,27 +40,23 @@ interactions:
response:
body:
string: !!binary |
H4sIAAAAAAAAA1RUTW8cNwy9+1cQe14v4rj116Vwg6B1arS3pkFTGByJM8NGI01IajaLIP89oGbj
j8tiVuQjxfee+PUEYMNxcwObMM3p9Gq5pnf7+NvV9Mc/f344XPzKb36/P7ztX53dfZDN1rNL9z8F
c4TRF3sIZZoTGZe8hoMQGnnFs8tXVxfnr3+6vG6BqURKDhtmOz3f/XxqVbpyyllNarAjeiwcSDc3
8O8JAMDX9gtrLwd/zB/zXQYbCSaOalD69udvDlaEMQMJbqFkQFAKqUaK0BGGEdBAa1ayLSAMUurs
4D1hsvEAKKxWgqApDGgjCfRFAGFP9IlyBCHz0XZwO5U8eNMJWFvzjrAa9zUB5ggjYVQTT7rHeIA7
xY5Swi3sx+IQygMOFMFKQ/+4QVcNQknxdCQUowj3RSK8jXuUuIPbtZXWDEqmrVND40IKQVBHwAGd
zjVxLEJbMMrKJSsIKzWQUpBWQAiEFsJEcQfO67FF5mE0mKUMQqqk23a6EsYKc8KhOqmHRrEwaRNB
cOAAtFA23cEtTAc1Ei5VIRLa6KyrlUQZRmJJpUzbdiEEDZgjJk/FvkcWwJQglImco+T32UIijOzE
FxdlZEl7ztFbYwhV0dqYXrAjEzxg2r3kH3rOUWEkUUo9BKw+Jz+aKSbyamzefQtWJHupPbUbC6Sy
UPPEM12OgovTs/fb9USJ86At0QuHEWXyiGdS5mFC4wBvcDbkDO9wIt05++/ZxoZAEV4wtckgkpHj
KbbPYLzQKohJtRE6GvnoBFeA4gE6Gjir81Sz4ELphXncB+rv42jcMK7lSg4EM2Eg9/HR7GBVsgLn
RnqHZol6phQB9XE6DEaiMFV1/+ZeSm4WZIEZ1drgPQZaASUrfa6Uw9E2LQ9DU28H7zmllehQ8udK
qxOKwN4Dz6d8fM0vZnGXLAS5gFZZeCmiv8BfOR3AeKJjFUpplZ0VBuF5dnmeFkgUnNwsKVGEvasy
o/oj2kKkQGyrbWsyntAoHdzZP+yf465tsra0OEf6srmBV48nqQyzlM4XXK4pPZ73nFnHByHUkn3P
qZV506LfTgD+a8uxKg60uTkuxc0sZZrtwconyl7w4mwtt3layU/B1+eXx6gVw/QscH114k2+nXwH
AAD//wMA7P6fXwwGAAA=
H4sIAAAAAAAAA2xTTW8TMRC951eM9sJlE6UtaUtuIEACpPJVqCqCqok9u2vijLeecZpQ9b8je7ft
hYtle+Y9v3meuZ8AVM5WS6jMtvfT87vb62/XF7dvr67+Xnzpvm4+Hf94f7U+Pz24j5+rOmeH9R8y
mhFKe70xYdt7Uhd4CJtIqJQZj87mr+Yni7OTRQlsgyWfYW2v05PZYqoprsPUsWhMRkd0F5whqZbw
awIAcF9WGN7KYFjxij8waEewdVYUQlMOotEZBWQLfQw9RZBgHOkhJ/x0RkN0yPCOW49sa0BoY0h9
jjbREVuBFrWjCGFg7wkNNcmDdCGS5EQEIeOTJQtrQtOBBrhDzZusILGQzuC1lCPtiB23kLgJ3koN
SiwusEB0QkWokImkAhgJIu0IPdkaEkfckfcZPAiJDRmFBg1aGut1EYRo67j1h6cM73YkTwIkMUih
ZwsW44ZJBCyJKdWmfiy0lFKDRmzJHoqRG5IaPOHuUcPgVROyygimQ27JzuCyiyG1XTEmusGkxqN0
azSb4WHHShFNbpDsQUeAyTpiQ3DnNEuiZ780b1HBk4XUZ3u1cwINavmJR0szb4aYwEK3KZONwCZ4
H+5mcOW0fMp20OTDjmpYk0Y8oK8H80t7oAfa92QUR4Wr6nI0AhW+ly997Ig32ahVBZb8jiRXFkYZ
eQT2TkcHurRFBkZNkf6vNSc5m0sJTC8ENCYCId/MoMxBaXnHlvbVEuZPNz60fQzrPB6cvH+6bxw7
6W4ioQTOUyIa+qpEHyYAv8toJcGWquU4UlUfw7bXGw0b4kx4ejTQVc8D/Rw8erkYoxoU/XPgeH46
yY88TP4BAAD//wMARyqwekoEAAA=
headers:
CF-Cache-Status:
- DYNAMIC
CF-RAY:
- 8599dbc1aa720e8b-MXP
- 85c050e1ba1a0e19-MXP
Cache-Control:
- no-cache, must-revalidate
Connection:
Expand All @@ -70,14 +66,14 @@ interactions:
Content-Type:
- application/json
Date:
- Thu, 22 Feb 2024 20:08:01 GMT
- Tue, 27 Feb 2024 12:08:57 GMT
Server:
- cloudflare
Set-Cookie:
- __cf_bm=9NuHyDZIMakT15own8nk.Fw5RkIE8yaH5UAlOjN_hNM-1708632481-1.0-Ab+1XDwre0yOAmQjsP0MyZ0lpk7ST6THx7v7+XVL+hyO011XDjZx0W5VYGqY4W0KDvElkYrGNV/a+4osduAzDXA=;
path=/; expires=Thu, 22-Feb-24 20:38:01 GMT; domain=.api.openai.com; HttpOnly;
- __cf_bm=tweHeZeMtVPJLUV7SZB26Lipwpw.BQ1.4X_Y6CSKRRc-1709035737-1.0-AT33S2bZA8bffZwwKiVZne+rlqct/seEe76WNfaaj+/EMZBFJwAE9pYDb1NwytfKNW3NYdRhWae+H0Ezuy6Pgf0=;
path=/; expires=Tue, 27-Feb-24 12:38:57 GMT; domain=.api.openai.com; HttpOnly;
Secure; SameSite=None
- _cfuvid=Zy_cZ2ergrwNiP.Jz4CzeosEr7eQDa8qETn0ExU6DLQ-1708632481546-0.0-604800000;
- _cfuvid=oa6mlrcQaWGFRnmczT1FQHoA3dVLo.ySSS881dJvSvM-1709035737088-0.0-604800000;
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
Transfer-Encoding:
- chunked
Expand All @@ -90,7 +86,7 @@ interactions:
openai-organization:
- traceloop
openai-processing-ms:
- '2355'
- '1553'
openai-version:
- '2020-10-01'
strict-transport-security:
Expand All @@ -102,38 +98,32 @@ interactions:
x-ratelimit-remaining-requests:
- '2999'
x-ratelimit-remaining-tokens:
- '249683'
- '249682'
x-ratelimit-reset-requests:
- 20ms
x-ratelimit-reset-tokens:
- 76ms
x-request-id:
- req_b9fd55c360b7ca6ee87ff7a98241b549
- req_c0e8c1f38e972bfbc4998f99c2421b3a
status:
code: 200
message: OK
- request:
body: '{"model": "gpt-3.5-turbo-instruct", "prompt": ["You are a play critic from
the New York Times. Given the synopsis of play, it is your job to write a review
for that play.\n\n Play Synopsis:\n \n\nIn the midst of the Victorian
era, on a secluded beach at sunset, a group of wealthy aristocrats gather for
a weekend retreat. Among them is the beautiful and headstrong Lady Isabella,
who is engaged to the wealthy but cold-hearted Lord Edward. As the sun sets
and the waves crash against the shore, tensions rise and secrets are revealed.
\n\nAs the night progresses, the group is plagued by a series of tragic events.
A mysterious death, a stolen heirloom, and a scandalous affair all come to light,
leading to a whirlwind of accusations and betrayal. Lady Isabella finds herself
caught in the middle of it all, torn between her love for Lord Edward and her
growing feelings for the charming and enigmatic Captain James.\n\nWith the arrival
of a determined detective, the truth behind the tragedy begins to unravel. As
the sun rises on the beach, the once peaceful retreat turns into a battlefield
as the characters must confront their past and face the consequences of their
actions. Will love conquer all or will the tragedy at sunset on the beach leave
no survivors? Only time will tell in this gripping Victorian drama filled with
passion, deceit, and ultimately, a tragic end.\n Review from a New York Times
play critic of the above play:"], "frequency_penalty": 0, "logit_bias": {},
"max_tokens": 256, "n": 1, "presence_penalty": 0, "temperature": 0.7, "top_p":
1}'
for that play.\n\n Play Synopsis:\n \n\nIn the midst of the strict and
proper society of Victorian England, a group of friends gather on the peaceful
shores of a secluded beach to watch the sunset. As the evening unfolds, tensions
rise and secrets are revealed, unravelling the perfect facade of their seemingly
perfect lives. As the sun sets and darkness descends upon the beach, tragedy
strikes, leaving the group forever changed. Through a series of flashbacks and
interactions, the audience witnesses the events that led up to this fateful
evening and the consequences that follow. With themes of love, betrayal, and
societal expectations, \"Tragedy at Sunset on the Beach\" delves into the complexities
of human nature and the consequences of hiding one''s true self. \n Review
from a New York Times play critic of the above play:"], "frequency_penalty":
0, "logit_bias": {}, "max_tokens": 256, "n": 1, "presence_penalty": 0, "temperature":
0.7, "top_p": 1}'
headers:
accept:
- application/json
Expand All @@ -142,12 +132,12 @@ interactions:
connection:
- keep-alive
content-length:
- '1577'
- '1127'
content-type:
- application/json
cookie:
- __cf_bm=9NuHyDZIMakT15own8nk.Fw5RkIE8yaH5UAlOjN_hNM-1708632481-1.0-Ab+1XDwre0yOAmQjsP0MyZ0lpk7ST6THx7v7+XVL+hyO011XDjZx0W5VYGqY4W0KDvElkYrGNV/a+4osduAzDXA=;
_cfuvid=Zy_cZ2ergrwNiP.Jz4CzeosEr7eQDa8qETn0ExU6DLQ-1708632481546-0.0-604800000
- __cf_bm=tweHeZeMtVPJLUV7SZB26Lipwpw.BQ1.4X_Y6CSKRRc-1709035737-1.0-AT33S2bZA8bffZwwKiVZne+rlqct/seEe76WNfaaj+/EMZBFJwAE9pYDb1NwytfKNW3NYdRhWae+H0Ezuy6Pgf0=;
_cfuvid=oa6mlrcQaWGFRnmczT1FQHoA3dVLo.ySSS881dJvSvM-1709035737088-0.0-604800000
host:
- api.openai.com
user-agent:
Expand All @@ -171,28 +161,28 @@ interactions:
response:
body:
string: !!binary |
H4sIAAAAAAAAA3RUTW8jNwy951cQPjvBfrWb5pYtWrRFuofuogu0KRa0RM8w0UgKSdkeLPrfC2rs
OJdeBhqRFB/fI/ntAmDFcXUDqzDVdHm9+4Hu3v8xp+2Xvz4+fdzePbwbPvxe2/bNJ75drd27bB4o
mEcYHexrKFNNZFzyYg5CaOQvvn7/6vr7t2/eXb/uhqlESh42VLt8e/XdpTXZlEvOatKCHaPHwoF0
dQN/XwAAfOtfWHJ5MNzn+3y/up04qoGNBJWDNSF9agRKZpwHKFtAUAqpRYqwIQwjoIG2rGRruF99
6icouT/xwR3uV2D4SArYIlMOpG5GsDa1ZK00hYfSJNMMNkppw9hjNWCOmNy8L5Ki5/6TgxVhzIDC
aiUIml7BF7bRH8RE2ShCQDXAHAFhEK7VkasVmRNnWoONrFATzhAp8Y5EAaGWPcm2pR4nvKNeLx0q
SQd95fz8LGVawBmKrfvxVBWwQhTcZ+BspZtKbY4IPIk6fr/cEyYbZwgjCgYj0TUkJ3Pu5oDVeIc9
+x3GGX5V3FBK2IF1j5Li5UgoXupdkQg/xT1KvIJbdW2ETKFlwR0t1Rhl5ZIVhJUW0L36R6LqpRsP
o8FYnOMMbGelHBnuHIqNNJ1UpTjQsRoWUHINnJ3PDm6kidVkhg3ZnmiJONfqLFVMFTeJ1kBaKTCm
dHb/n5pHlMlx/IjVkDP8hhPpFXzuELZFNhwjZUhlR4AxKmAGOpggJJxJHC5nEx4awVEdTEKebBm0
A9RUrL8IKMI7TCfFIhl5dor9GIzPSYqNJGB79qEpx94oMnfmYmeuvGzDkHjCA9iIBntOqRNML0ZD
reVM8ZnQKiW24GsAIikP2Rmkw1NjZaM1bE8tmXDHOkIoam0iPaGJZMjeX31ATTBrLWJHTc/du8Hw
CJzBeHpm6DxuJLhQ05SOS2AZ9OMK0O7vb0Qp9UgOKOXF3dM+NU5sc5d0Q9hsPqXpHO5ZySXqIe4T
BSc0DkA7yqa+Qxx0TXgcxl/KnnYka9g7mSYtzV6kvphvrCjmhPWm3iwArHjPTolUve8S5QiUaOpZ
yhakTNh7f5rVyKXsXSg4UJwXFvzHkeWu8FnHvmv7WuUc6bC6gVfPN6kMVcrGV3BuKT3fbzmzjl+F
UEv2TZwoDzauuv3fC4B/+gJviv8pPVXJClpwKxUU5ecWlMSX5Gen5oGMNLIwgZiohKg3kGRNzaCy
JfkliTkICVMTAy6QLbVcAAAAAP//AwBR+2ScsQYAAA==
H4sIAAAAAAAAA2xUS28kRQy+51dYc55EWcIqkCPSckFiEYmQlg1aearc3Sbucsfl6sloxX9Hrp6E
PXAZTdfD/vw96usFwI7z7g52aV7k8ofj86eHj8/r7zd4P2XkX75797Pd/zl+/ITf024fp/XwNyWP
G04v/iXpvAg5a9m2kxE6RcV3t9c/Xt+8v7257RuzZpK4Ni5+eXP1/tKbHfSSS3Vryc+3J+VEdXcH
ny8AAL72X9h6xeXH8lgedw+GI+UToMN9K5UctIBPBD8RpulxB1wBwSdt4+SXi+mqT1xGwJJhNF6W
+FgET+ATOjg+UY3vRMBbIeORcz+/mC5kUDUx+Ql0gD84uRpjgQ9lFCz5Ch4m2uolKk5WAU1byYAw
mrYlbg3GVHKF46Qwok9kARqhUpKWKcMhsIMrHNHjz0RQ+3B7ODQHrH2JViodveloVCvVPTiVyloq
GFfqoCslIw8YBEYroVDeQxPnGZ3kBEKYo4xrEGU4cuqlfWMkTVhG6h3ZQHilCoNGJbsKCX4TPB2N
x8nh8684019Qn1hkaCInOBLGedeR+pgI1dU6daIr7eFAbnhC2W9YO7MoQC8LJUfvo3QduELCxXlF
P8+cW4r9jfE0oWE6801wJJHLTCuJLrSJt+E3kq3qxEvt67NuRaNTXKWXRdQoR9tMiwf9Fu7pAhlT
7RIK1umA6WkrwiE1djz1Cj6Eem+IwoGDWqIcFCctg2knN/DosQCXQgaZ5g5hgxrnKj03KmlrOPEm
Ur/l1ggqSXDL3TlnS3bFVpRGtXdSm9lPgAddCVAESCp12YK06ji+ZiGSk6nyWKKbv5o4GKkcuYYT
OdAwUHIO5VBEj2dIgC1zQI0JB02tvqZwexNeegsKqrWgQGYUHRuF/Eei89E3CTdJW6VXKFseMJIc
nGfTBTDnMFbf15UspkOftS4TGX07xB6O7FuK3LA8Nxbe4tuDRd4dVcnWTkbtLkV7CgbdsPprG282
K8vGciuDSu6DBo//seookfwMVCrNByFIUSNTZMcqLHokG5p0TkrDEs5YyEKs+Kh7OBiX8ZxJ4YF6
9zkyO2CiqP2t38+O4eLGCdPZof/j984rG6SJZq4eOTyDD48uKAsG3CjY2T3bX/uD3N9eLpledndw
/bYiOi6mh3inSxN5Wx+4cJ2+GGHVEs+1UBl92vX9fy7+lYJCLLiULy1OTE9VsoKW7koFRfm5BSXx
JfnZqXkgIw0tjSAmKiEqF4SskakZVLYkvyQxByFhYmLBBbKllgsAAAD//wMAcIMQMtYGAAA=
headers:
CF-Cache-Status:
- DYNAMIC
CF-RAY:
- 8599dbd1efba0e8b-MXP
- 85c050ed49520e19-MXP
Cache-Control:
- no-cache, must-revalidate
Connection:
Expand All @@ -202,7 +192,7 @@ interactions:
Content-Type:
- application/json
Date:
- Thu, 22 Feb 2024 20:08:04 GMT
- Tue, 27 Feb 2024 12:09:00 GMT
Server:
- cloudflare
Transfer-Encoding:
Expand All @@ -216,7 +206,7 @@ interactions:
openai-organization:
- traceloop
openai-processing-ms:
- '2295'
- '3284'
openai-version:
- '2020-10-01'
strict-transport-security:
Expand All @@ -228,13 +218,13 @@ interactions:
x-ratelimit-remaining-requests:
- '2999'
x-ratelimit-remaining-tokens:
- '249396'
- '249507'
x-ratelimit-reset-requests:
- 20ms
x-ratelimit-reset-tokens:
- 144ms
- 118ms
x-request-id:
- req_046213cfe6291d983ed25863287aad7b
- req_2c7e25c90aa81889ede319a3fea4ca53
status:
code: 200
message: OK
Expand Down
Loading
Loading