OpenAPI 3.1.0 Support

[Stratus3D]

I'd like to use Open API Spex to generate OpenAPI 3.1 specifications that can be consumed by another system that uses JSON Schema. OpenAPI 3.1 has been out several years now and I'm wondering if support will ever be added to this library. Changes between OpenAPI 3.0 and 3.1 are minimal, but there are changes that would need to make to this library to support it. The biggest one for me right now is using multiple types to indicate a field may be null. Open API Spex does not allow a list for schema type.

Invalid schema.type. Got: [:string, :null] at /my/0/field/location

There are a few other changes, but I don't think they'd be a ton of work to implement. The bigger challenging would be supporting both 3.0 and 3.1 at the same time if that is desired. I'm happy to contribute if there is a clear path to getting this library to OpenAPI 3.1. Thanks!

[zorbash]

Hi @Stratus3D 👋
Thanks for offering to help with this.

To support OpenAPI 3.1 we'd have to support JSON Schemas (see: oas-3.1-data-types). I think we can finally move ahead with #47. The main blocker is ex_json_schema not supporting the more recent versions of the JSON schema spec, see: jonasschmidt/ex_json_schema#73

There are some 3.1 changes which can be implemented in a backwards compatible way, one example is #654

[Stratus3D]

Thanks for the update @zorbash ! I'd be happy to contribute but my open source has been taken up by asdf recently. It'll probably take me another couple weeks to wrap up my initiative on it.

[iStefo]

To weigh in on this: I've recently started using the (very new) https://hexdocs.pm/jsv/JSV.html library which supports 2020-12 (and takes the whole topic of meta-schema/vocabulary very seriously) for JSON Schema validation. It also brings customizable resolvers and more, which I like.

With OpenApiSpex (which we have used successfully in smaller projects in the past), I've now encountered the need to reference "external" JSON Schema documents from my OpenAPI spec, which currently doesn't really seem to be possible, right?

Maybe fully integrating JSV (and letting it do the heavy lifting of resolving schemas...) could be an option? Please let me know what you think. I could image being able to spend some time on this very soon.

[gmile]

@iStefo how does JSV compares to exonerate? 🤔

My impression about Exonerate so far has been it compiles JSON Schema into native Elixir code, that's subsequently used to validate input JSONs. According to benchmarks included in the Exonerate repo, it's both quite fast and complete when it comes to validating arbitrary JSONs using the Schema. I wonder how would JSV compare to it 🤔

[iStefo]

@gmile I'm sure that exonerate excels in validation performance if the primary source of schemas is strings or files.

However, in our use case, I didn't pursue it further because I feel like it's scope too narrow for the things we need from such a library. Our application has distributed schemas across dozens of files (one for each "component") where every component is authoritative about its own schema and uses it to validate its configuration.

Schemas are composed into larger schemas and those schemas are also served through a /schemas endpoint s.t. frontend JSON Editors can use it for validation and autocompletion.

Additionally, we didn't want to define schemas using strings or maps, but instead make use of JSV.Schema's "DSL" which was super easy to extend to our needs. For example, we can define schemas like this:

def MyModule do
  def schema do
    use MyProject.Schemas.DSL

    object()
    |> props(
      var_key: string(description: "The name of the context variable to assign."),
      value:
        oneOf([
          string(),
          number(),
          null(),
          jsonLogic()
        ])
    )
    |> required([:var_key])
  end
end

where our DSL module provides commonly used helpers (and additions that are just not (yet) part of JSV.Schema) such as

defmodule MyProject.Schemas.DSL do
  defmacro __using__(_opts) do
    quote do
      import JSV.Schema
      import MyProject.Schemas.DSL
    end
  end

  def oneOf(base \\ nil, options) do
    JSV.Schema.override(base, oneOf: options)
  end

  def jsonLogic do
    JSV.Schema.ref("/schemas/json-logic.json")
  end

  # more
end

And still, using plain maps for special cases (e.g. when choosing the OAS 3.1 dialect) is also possible (even though those special fields are probably simply disregarded by JSV until the vocabulary is implemented?)

  def schema_components do
    use MyProject.Schemas.DSL

    id("components")
    |> override("$schema": "https://spec.openapis.org/oas/3.1/dialect/2024-10-25")
    |> items(%{
      type: :object,
      discriminator: %{propertyName: "type"},
      oneOf: MyProject.Components.schemas()
    })
  end

To alleviate the need for building at runtime, I've already experimented with a defschema macro (presented at the most recent Munich Elixir Meetup but not yet polished enough to publicize it). Building a schema in JSV means converting it to a "validation root" which is not executable elixir code, but something the validation engine can work with more directly.

As an upside for us, the built JSV validation root also contains a JSON-encodable version of the schema which we can easily post-process in Phoenix Controllers (e.g. to inject the Endpoint's hostname to create fully-functional references to other schemas) and then serve to clients.

Long story short: I can't 100% say which library better fulfills the need of either OpenApiSpex or the general public. For us, I currently greatly value the extensibility of JSV and the general design decisions made in that project more than validation performance.

[gmile]

@iStefo thank you so much for a detailed breakdown of the differences in design choices between the two libraries!

I'm always on the lookout for the new JSON schema implementations in Elixir ecosystem, will keep an eye on your lib now too

[iStefo]

Glad to shed some light on that topic. Just to avoid any misunderstanding though, I did not create JSV, @lud did. I only found out about it earlier this year :)

[lud]

Hello, thank you for bringing up JSV and your kind words. I'll give my two cents here if it can help to define the path to 3.1 support with JSV or any other tool.

About JSV

I'm sorry by advance because this is gonna be 90% off-topic but I wanted to answer to some points raised here. Feel free to discuss use cases or possible improvements on the Elixir forum or on the JSV Github repo.

Jump to "About OpenAPI" below for an on-topic answer!

I wonder how would JSV compare to Exonerate

Indeed, Exonerate is very much focused on perfomance by using compiled schemas. The main difference is that it does not (yet?) implement the full spec. For instance you cannot use $dynamicRef to implement composition (say you have a generic schema PageOfItems that describes pagination and a list of items but uses a $dynamicRef to describe the schema of the paginated items.)

Also Exonerate cannot validate that a JSON schema is valid against a meta-schema.

This may have change, it's been long since I have checked.

And still, using plain maps for special cases (e.g. when choosing the OAS 3.1 dialect) is also possible (even though those special fields are probably simply disregarded by JSV until the vocabulary is implemented?)

Glad you like the "DSL". I am not planning on implementing the keywords from other specs, as it could lead people to believe that those keywords will be used when using a JSV.Schema struct.

I don't really use that "DSL" like you do, I intended it to be used as an easy way of defining schemas that we see very often like just {"type": "integer"}.

  def schema_components do
    use MyProject.Schemas.DSL

    id("components")
    |> override("$schema": "https://spec.openapis.org/oas/3.1/dialect/2024-10-25")
    |> items(%{
      type: :object,
      discriminator: %{propertyName: "type"},
      oneOf: MyProject.Components.schemas()
    })
  end

  def schema_components do
    %{
      "$id": "components",
      "$schema": "https://spec.openapis.org/oas/3.1/dialect/2024-10-25",
      items: %{
        type: :object,
        discriminator: %{propertyName: "type"},
        oneOf: MyProject.Components.schemas()   
      }
    }
  end

Between the two example above I still think the second form is more straight forward and easier to grasp.

Anyway, it's nice if you find value in those functions. I don't plan to add one function for all possible keywords but the oneOf and friends are indeed good candidates.

Maybe the JSV.Schema.override/2 function should not enforce the return value to be a %JSV.Schema{} struct. It could just merge the given keyword or map overrides into it's base, as long as the base is a map (including struct). When giving a struct base, the user would have the responsibility to make sure that the target struct defines the overriden keys. This would let anyone use those helpers on their own structs.

The %JSV.Schema{} struct is there mostly for autocompletion. JSV does not care at all about what is given as a schema as long as it's a map or a boolean. There is actually a cost to using the struct, because we need to remove all the nil values from the struct, whereas %{type: :integer} just has one key.

Same for the schema normalization, I guess instead of calling Map.from_struct for any struct that is not a %JSV.Schema{}, a protocol should be available.

To alleviate the need for building at runtime, I've already experimented with a defschema macro (presented at the most recent Munich Elixir Meetup but not yet polished enough to publicize it). Building a schema in JSV means converting it to a "validation root" which is not executable elixir code, but something the validation engine can work with more directly.

I don't know if you are aware that JSV has a defschema macro that is building a schema and a struct at the same time, and validating with that schema returns the target struct. I'm also building a defschema_for macro that builds a schema that deserializes into another's module struct. I'all also build a special case for Ecto that calls the changeset but I've not looked into it for now.

Anyway, if you are building schemas at runtime though, I guess Exonerate will be slower because it's going to do some compilation.

Also I think it would not be hard to implement "extending" a root: You would build the "base root" at compile-time, and then add more definitions ($refs and $dynamicRefs I guess) and change the validation entrypoint at runtime. I am not sure how to patch the "raw" schema though.

Unless you are always building those schemas from scratch.

About OpenAPI

So yeah, sorry again for the digression.

I am actually planning to implement an OpenAPI tool much like OpenApiSpex that would just start on version 3.1 and skip the 3.0 specification entirely. The current modifications I am doing on JSV are to support that purpose.

I believe this would be much more easy to do than supporting both at the same time. Plus, v3.2 and then Moonwalk are coming.

But I know that usage of OpenApiSpex is much more widespread and established than a libray that does not even exist :D It would be also much more easier for people to update OpenApiSpex's version than to migrate to another library.

I'd suggest OpenApiSpex should support 3.1 with a breaking change, have two concurrent versions that support either 3.0 or 3.1 but not both. Though it's going to be a burden for you to have multiple branches.

Anyway, if you are considering using a third-party library to implement 3.1 I'd be glad to help as I'm working on it these weeks as the support I'm adding is probably going to be serving the same needs that you have.

[lud]

@iStefo is there some place I can contact you for further discussion on your usage of JSV ? I'm planning some changes and I'm seeking advice on design.

[iStefo]

@lud I've sent you an email :)

[mbuhot]

I am actually planning to implement an OpenAPI tool much like OpenApiSpex that would just start on version 3.1 and skip the 3.0 specification entirely. The current modifications I am doing on JSV are to support that purpose.

@lud That's great to hear we'll have an OpenAPI 3.1 option in the ecosystem! Is there a repo I can follow to see the progress?

The other major change I'd make to OpenAPISpex in hindsight is to avoid compile-time dependencies between schema modules as much as possible. On larger projects we've had reports of long compile times being due to schema definitions referencing each other through the module names (#382).

One of the use cases for OpenApiSpex is with AshJsonAPI where the schemas are derived from Ash resources. Will your new library support that kind of programmatic generation of the API spec and schemas?

[lud]

Hey @mbuhot !

I have not started working on an openai 3.1 implementation for phoenix yet. I thinks it's a large scope and I am not sure anyone besides me will use it since OpenApiSpex is well established and Ash JSON API has its own schema system. Plus I have other libraries that I need to take care of before as they are kinda broken with the new Elixir versions.

Regarding compile-time dependencies, JSV too supports referencing another schema by using the module name and I guess the same problem arises with lots of them. I am not sure how to avoid that:

• Even if you do not build the JSV validation root, and so JSV does not dereference the modules to check if they are actual modules, it seems to be a compile-time dependency just because the module name is used
• In a real app, you do want to build the validation roots at compile time to do it just once, so JSV will call Code.ensure_loaded and stuff.

There is a way to use a string instead of a module name but it's more error prone and more manual. But yes if you want to avoid compile-time dependencies, one can use raw schemas that reference each other by $id.

Will your new library support that kind of programmatic generation of the API spec and schemas?

Schemas would be raw schemas so yes, not a big deal to generate those from Ash as it's currently done in AshJsonApi. For other stuff like routes, I was planning on doing like OpenApiSpex: take a plug router and extract the routes/paths, and let the user manually define the top spec fields like info.

But since I am not working on it I don't know the complications that can arise. From my point of view, users declare elements (like operation on controller functions that reference schemas, and routes in the router that reference a controller), and the openapi spec tool role is just to collect all that and build the spec. Right? So Ash should generate all the data (of course using code provided by the openapi spec library).