feat(api): oasdiff OpenAI openAPI spec against ours #3529
Conversation
meta-cla
bot
added
the
CLA Signed
cdoern
force-pushed
the
oasdiff-for-openai
branch
10 times, most recently
from
67b7c6e to
b509e85
Compare
cdoern
requested review from
ashwinb,
bbrowning,
ehhuang,
hardikjshah,
leseb,
mattf,
raghotham,
reluctantfuturist,
slekkala1,
terrytangyuan and
yanxi0830
as code owners
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
b509e85 to
8a2bb5f
Compare
| key: openai-openapi.yml | ||
|
|
||
| # Cache oasdiff to avoid checksum failures and speed up builds | ||
| - name: Cache oasdiff |
There was a problem hiding this comment.
we should probably also support a force no-cache conformance check as well - we should probably run it at the time of release.
There was a problem hiding this comment.
I agree, but should I make this a separate action so it can be triggered by maintainers manually?
cdoern
force-pushed
the
oasdiff-for-openai
branch
4 times, most recently
from
da078af to
c5bd55e
Compare
leseb
left a comment
leseb
left a comment
There was a problem hiding this comment.
I think this action could result in a "badge" on the repo once we feel confident, having the compatibility ratio again openai is very important :)
cdoern
force-pushed
the
oasdiff-for-openai
branch
3 times, most recently
from
d7e8b77 to
5e68691
Compare
|
this now checks all of our APIs against all of the openAI apis. The removal of |
cdoern
force-pushed
the
oasdiff-for-openai
branch
2 times, most recently
from
f423104 to
d9abb2e
Compare
|
@cdoern do you plan to pick this up? |
|
yes @ashwinb I can get back on this today/tomorrow. thanks for the reminder! |
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
d9abb2e to
867e0a4
Compare
|
@ashwinb this should be set now for review After my last commit, I realized that while the oasdiff command was working, our CI was still vulnerable to upstream changes in the OpenAI spec since we were fetching it live on every run. My initial thought was to find a permanent, version-locked URL to pin against, but after some investigation, it turns out there isn't a stable, versioned URL available for the exact spec we need, openai only published the most recent one to a stainless URL. To guarantee build stability and make our conformance test fully hermetic, I've changed my approach to vendor the OpenAI spec directly into our repository at docs/static/openai-spec.yml. |
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
867e0a4 to
a8928b6
Compare
| info: | ||
| title: OpenAI API | ||
| description: The OpenAI REST API. Please see https://platform.openai.com/docs/api-reference for more details. | ||
| version: 2.3.0 |
There was a problem hiding this comment.
I'm wondering if we should either:
- put this into a 2.3.0 directory
- or add -2.3.0 to the file name
Or perhaps, are we always replacing / overwritting with a new spec?
There was a problem hiding this comment.
I think naming -2.3.0 is good! I only think we should overwrite this file when we choose to bump to a different version of the OpenAI spec, it'll likely be a losing battle to constantly be targeting the newest version of the spec.
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
a8928b6 to
061422c
Compare
Looks like it is erroring? |
diff the `/v1/` routes that are OpenAI compatible against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed. This will catch errors with new handwritten additions to our openAI compat routes. Instead of fetching the OpenAPI spec from a dynamic URL, which could cause non-deterministic build failures, this change uses a local copy stored at `docs/static/openai-spec-2.3.0.yml`. This makes the conformance check fully reproducible and prevents CI failures caused by uncontrolled upstream changes. Signed-off-by: Charlie Doern <cdoern@redhat.com>
cdoern
force-pushed
the
oasdiff-for-openai
branch
from
061422c to
2b187cb
Compare
|
the spec was incomplete (I must have not copied it all the first time) which is why the test was erroring, updated it! |
5 participants
What does this PR do?
diff the
/v1/routes that are OpenAI compatible against the OpenAI openAPI spec. This will of course only trigger on PRs where the spec is changed.This will catch errors with new handwritten additions to our openAI compat routes.
Instead of fetching the OpenAPI spec from a dynamic URL, which could cause non-deterministic build failures,
this change uses a local copy stored at
docs/static/openai-spec.yml.This makes the conformance check fully reproducible and prevents CI failures caused by uncontrolled upstream changes.
I am marking this test with
continue-on-error: true, until we get rid of all of the errors. Nevertheless, this is a nice utility to have so folks know if their spec changes introduce more breaking changes or fix breakages when comparing to the OpenAI openapi spec.Test Plan
test should pass.