Add built-in support for generating OpenAPI document from APIs

[captainsafia]

This issue outlines background content, proposed design, and proposed API to add support for built-in OpenAPI document generation to minimal APIs and MVC in the Microsoft.AspNetCore.OpenApi package. It also captures some open questions and future plans for work in this space. Some of the implementation details are subject to change as work evolves.

Background

The OpenAPI specification is a standard for describing HTTP APIs. The standard allows developers to define the shape of APIs that can be plugged into client generators, server generators, testing tools, documentation and more. Despite the universality and ubiquity of this standard, ASP.NET Core does not provide support for OpenAPI by default within the framework.

ASP.NET Core does not provide first-class, built-in support for OpenAPI. Instead, ASP.NET Core has shipped with support for ApiExplorer (not to be confused with Visual Studio's API Explorer) for quite some time. The ApiExplorer is a helpful abstraction that provides metadata about the routes that are registered in an application. This metadata is accessible via the DI container and is used by tools in the ecosystem like Asp.Api.Versioning, NSwag, and Swashbuckle to introspect and query the metadata aggregated by ApiExplorer.

In .NET 6, minimal APIs was introduced and support for minimal APIs was added to ApiExplorer via the EndpointMetadataApiDescriptionProvider which allowed querying the ApiExplorer metadata to introspect registered minimal endpoints in an application.

In .NET 7, the Microsoft.AspNetCore.OpenApi package was introduced (note: this package ships via NuGet and is not part of the shared framework). It exposed the WithOpenApi extension method for modifying the OpenApiOperation associated with a single endpoint in minimal APIs. The package takes a dependency on the Microsoft.OpenApi package which provides an object model and deserializers/serializers for interacting with various versions of the OpenAPI specification.

The evolution of our OpenAPI "support" has resulted in a large quantity of bugs and feature gaps (ref). To resolve this and provide a more seamless experience for users, we're incorporating OpenAPI document generation as a first-class feature in ASP.NET Core.

Future Implementation

Implementation Overview

The flow diagram outlines the proposed implementation. New components are in a bordered box. The OpenApiComponentService is responsible for managing state that will be serialized to the top-level components field in the OpenAPI document. At the moment, it is largely responsible for generating and managing JSON schemas associated with application types. The OpenApiDocumentService exposes a GetOpenApiDocument method for resolving the OpenApiDocument associated with an application. These new components build on top of the metadata that is produced by the ApiExplorer, allowing us to take advantage of a time-tested and well-established component.

flowchart LR
	mvc[Controller-based API]
	minapi[Minimal API]
	defdescprov[DefaultApiDescriptionProvider]
	endpdescprov[EndpointMetadataDescriptionProvider]
	desccoll[IApiDescriptionCollection]
	compservice[[OpenApiComponentService]]
	docsvc[[OpenApiDocumentService]]
	idocprovider[[IDocumentProvider]]
	meas[Microsoft.Extensions.ApiDescription.Server]
	mvc --> defdescprov 
	mvc --> endpdescprov
	minapi --> endpdescprov
	defdescprov --> desccoll
	endpdescprov --> desccoll
	desccoll --> docsvc
	idocprovider --> docsvc
	meas --> idocprovider
	compservice --> docsvc

Loading

Document Generation

The OpenAPI document contains a well-defined set of fields to be populated with established meanings. To make it easier to understand why documents are generated the way they are, we intend to document and implement the following semantics are used when generating the OpenAPI document.

• info.name: Derived from the entry-point assembly name.
• info.version: Defaults to 1.0.
  • Future: integrate with versioning information in Asp.Versioning.
• servers: Derived from the host info registered in the IServer implementation.
• paths: Aggregated from ApiDescriptions defined in the ApiExplorer
  • paths.operations.parameters: captures all routes understood by the model binding systems of MVC and minimal except inert parameters like those coming from the DI container
    • Information in route constraints and validation attributes will be added onto the schemas associated with each parameter type
• components: Aggregates JSON schemas from the OpenApiComponentService (see below)
• security: No autogeneration by default.
• tags: Aggregates all tags discovered during the construction of the document.

JSON Schema Generation

OpenAPI definitions rely on a derivative implementation of JSON Schema to describe the shapes of types that are used in request parameters and responses. .NET does not contain a built-in solution for generating or validating JSON schemas from .NET types (although this work is outlined here). To fill this gap, this implementation will ship with an OpenApiComponentService that uses System.Text.Json's type resolver APIs to generate the underlying OpenAPI schemas. This gives us the opportunity to address some of the gaps that exist with how certain types are currently implemented as seen in the following issues:

• Correct handling duplicate FromForm parameters in a given API (ref)
• Correct handling nullability annotations for response types (ref)
• Correctly handling IFormFile and IFormFileCollection inputs
• Correctly handling polymorphic serialization metadata from STJ in the generated schema using oneOf and type discriminaotrs
• Correctly handling inheritance hierarchies in types using allOf
• Applying all supported route constraints to generated schemas
• Applying all supported ComponentModel annotations to generated schemas

Note: The version of OpenAPI.NET that we intend to target uses the JsonSchema.NET to handle JSON schema representation in the OpenAPI document.

Question: As part of this work, we'll build a test bed to validate schema generation behavior. Please share examples of APIs you'd like to make sure appropriate schemas are generated for so they can be included in the test set.

Generating Operation IDs

The OpenAPI specification consists of operations that uniquely identify an endpoint by it's operation type (HTTP method) and path. Each of these operations can optionally include an operation ID, a unique string that identifies the operation. Operation IDs play an important role in OpenAPI integrations with client generators, OpenAI plugins, and more. Users can define operation IDs for each endpoint in their application themselves, but ideally we should be able to generate high-quality operation IDs by default to make the process more seamless for the user. Operation IDs should be deterministic so it's not sufficient to generate an arbitrary GUID for each operation in an application. The proposed semantics for generated operation IDs are as follows:

• If a name is provided on an action via the route method, use that name.
• If a name is provided on an action or endpoint using the EndpointName metadata, use that name.
• If neither is available, attempt to create an operation ID using information available in the route template and endpoint metadata in sequential order.
  • The HTTP method associated with the operation stringified (GET, POST, etc.)
  • If route segments exist on the application, the route segment values concatenated by _.
  • If the route segments contain parameters, use the parameter name sans any constraints.
• If duplicate operation IDs are generated with these semantics, disambiguate them with a monotonically increasing integer (Get_1, Get_2, etc.)

Swagger UI (Or Lack Thereof)

The web templates currently expose two endpoints by default in relation to OpenAPI: one that serves the OpenAPI document as a JSON file and another that serves an Swagger UI against the backing JSON file. At the moment, we don't intend to ship with Swagger UI as a supported UI in ASP.NET Core. Although it provides the benefit of an accessible UI for ad-hoc testing, it introduces engineering overhead around shipping (need to bundle web assets), has some security implications (it's easy to accidently leak client secrets for certain authentication configurations), and introduces maintenance overhead (need to make sure that we upgrade swagger-ui as needed).

Since swagger-ui is independent of the OpenAPI document, users can independently incorporate into their applications if needed via third-party packages or their own code (swagger-ui just needs a pointer to the served OpenAPI JSON file). Users can also take advantage of other ad-hoc testing tools that plug in to OpenAPI, like ThunderClient.

Customizing document generation

The automatic document generation will make use of metadata exposed via ApiExplorer to generate the OpenAPI document. There are currently several avenues that exist in the framework for influencing the generation of this document:

• Accepts and Produces metadata allow limited support for customizing the content-types and object types associated with an endpoints parameters and responses
• EndpointName, EndpointTags, EndpointSummary, and EndpointDescription metadata and their associated attributes/extension methods allow customization of the tags, summary, and description fields associated with a request
• WithOpenApi extension method supports overriding the OpenApiOperation associated with an endpoint in its entirety

The customization story is disparate at the moment, and it's largely a result of the way the system evolved. As we move to support generating entire documents, there are certain aspects we don't provide APIs for customizing, like the version number specified in the info of the OpenAPI document or the supported security schemes. This effort provides a nice avenue for unifying the various strategies that have proliferated in the codebase for customizing these aspects.

The current customization options that we provide are largely endpoint-focused, mostly because we've never had the need to manage document-level settings like properties in the info property of the OpenAPI document.

XML Documentation Support

One of the most upvoted issues with regards to OpenAPI/ApiExplorer in our repo is around supporting XML code comments (ref). Being able to generate the OpenAPI document as standard functionality pushes us towards support for this feature. Work here will requiring reading the generated XML documentation at runtime, mapping members to the appropriate operations, and merging information from the XML comment into the target operation.

Ecosystem Integration

OpenAPI plays an important role into several existing technologies in the space. At the center of this effort is the goal to produce a high-quality OpenAPI document that provides strong integrations with existing tools in the ecosystem including:

• Kiota client generation
• NSwag generators
• Asp.Api.Versioning
• OpenAI plugins
• Swagger UI/Redoc and other ad-hoc testing tools

Question: Are there other components we should validate integration with?

Build Time OpenAPI Document Generation

As far as build time generation goes, we'll plug-in to the existing integration provided by the dotnet-getdocument command line tool. dotnet-getdocument enforces a loose, string-based contract that requires an implementation of Microsoft.Extensions.ApiDescription.IDocumentProvider to be registered in the DI container. This means that users will be able to generate OpenAPI files at build-time using the same strategy they currently do by enabling the following MSBuild flags.

<OpenApiGenerateDocumentsOnBuild>true</OpenApiGenerateDocumentsOnBuild>
<OpenApiDocumentsDirectory>.\</OpenApiDocumentsDirectory>

Behind the scenes, this functionality works by booting up the entry point assembly behind the scenes with an inert IServer implementation, querying the DI container on the entry point's host for the IDocumentProvider interface, then using reflection to invoke the appropriate methods on that interface.

Note: although I've explored strategies for generating OpenAPI documents at build-time using static analysis, this is out-of-scope for the time being.

Runtime OpenAPI Document Generation

Templates will be updated to include the following code in Program.cs when OpenAPI is enabled. The AddOpenApi method registers the appriopriate OpenAPI-related services and MapOpenApi exposes an endpoint that serves the serialized JSON document.

var builder = WebApplication.CreateBuilder();

builder.Services.AddOpenApi();

var app = builder.Build();

app.MapOpenApi();

app.MapGet("/", () => "Hello world!");

app.Run();

Underlying API

See #54600 for full API review.

// Assembly: Microsoft.AspNetCore.OpenApi
namespace Microsoft.AspNetCore.Builder;

public static class IEndpointRouteBuilderExtensions
{
  public static IEndpointRouteBuilder MapOpenApi(this IEndpointRouteBuilder builder);
}

// Assembly: Microsoft.AspNetCore.OpenApi
namespace Microsoft.Extensions.DependencyInjection;

public static class IServiceCollectionExtensions
{
  public static IServiceCollection AddOpenApi(this IServiceCollection serviceCollection);
  public static IServiceCollection AddOpenApi(this IServiceCollection serviceCollection, Action<OpenApiOptions> configureOptions);
}

// Assembly: Microsoft.AspNetCore.OpenApi
namespace Microsoft.AspNetCore.OpenApi;

public class OpenApiOptions
{
  public string JsonFilePath { get; set; }
  public OpenApiSpecVersion OpenApiVersion { get; set; }
}

Tasks

preview4

• Support generating OpenAPI document using ApiExplorer metadata
• Support generating JSON schemas using System.Text.Json schema generation support and Microsoft.OpenAPI
• Support serving JSON OpenAPI document from server
• Support generating OpenAPI document at build with Microsoft.Extensions.ApiDescription.Server infrastructure
• Support customization of OpenAPI document (document and operations)

preview5

• Replat WithOpenApi on top of operation transformers
• Update web API templates to use new APIs
• Verify native AoT comapt for minimal APIs scenarios

preview6

• Add support for schema transformers
• Add support for reference IDs and schema reference

preview7

• Support integrating XML comments into generated code
• Support configuring discriminator mappings on polymorphic types
• Update gRPC JSON transcoding implementation

P1: Ecosystem Integration and Enhancements

• Support incorporating API versions from Asp.Versioning
• Validate generation experience with Kiota, NSwag

[cphorton]

Apologies if this is the wrong place to ask these questions, but will Yaml file generation be supported out of the box?

It would also be great to have the ability to perform a schema validation if you are doing schema first I.e. load a schema from a file and validate that the generated OpenAPI spec matches the loaded schema.

[captainsafia]

Apologies if this is the wrong place to ask these questions, but will Yaml file generation be supported out of the box?

I haven't thought about YAML generation as something we'd ship as part of preview4. However, it should be trivial to support and something that we've discussed in the API review of the first batch of APIs (ref).

If folks will get a lot of value from supporting YAML generation by default, I'd be happy to add support for it.

[cphorton]

If folks will get a lot of value from supporting YAML generation by default, I'd be happy to add support for it.

Thanks for your swift response.

I understand that the objective here is not necessarily a like for like replacement for Swashbuckle, but Yaml was one of the formats that was supported by that library, you just needed to request the swagger.json spec from the endpoint with a .yaml extension

[captainsafia]

I understand that the objective here is not necessarily a like for like replacement for Swashbuckle, but Yaml was one of the formats that was supported by that library, you just needed to request the swagger.json spec from the endpoint with a .yaml extension

Yep, the serialization functionality in both places is actually supported by the underlying Microsoft.OpenAPI package so support for YAML is largely a manner of wiring it up and calling the right APIs. That being said, I want to be careful about introducing too many configurability toggles early on hence the ask for user feedback here. Let's use the thumbs up reaction on this comment to signal that having YAML serialization before GA would be valuable.

[RicoSuter]

.NET does not contain a built-in solution for generating or validating JSON schemas from .NET types (although this work is outlined here)

The link in “here” seems to be wrong, can you provide the right one?

Note: The version of OpenAPI.NET that we intend to target uses the JsonSchema.NET to handle JSON schema representation in the OpenAPI document.

Can you explain this? What is OpenAPI.NET? I thought you will use your Microsoft.OpenApi package?

Will you use/reference JsonSchema.NET from the asp package to generate schemas? Why not eg NJsonSchema (disclaimer: I’m the author)?

Best would be to split up Microsoft.OpenApi into Microsoft.OpenApi and System.Text.Json.Schema and then let other libs extend generators with filters/processors. For example OSS could provide an extension for xml docs (I did that already for NSwag/NJsonSchema with Namotion.Reflection)

[captainsafia]

The link in “here” seems to be wrong, can you provide the right one?

Ooops. That should've been a link to dotnet/runtime#29887. 😅

Can you explain this? What is OpenAPI.NET? I thought you will use your Microsoft.OpenApi package?

Yes, I use the terms OpenAPI.NET and Microsoft.OpenApi interchangeably. They refer to the same package though.

Will you use/reference JsonSchema.NET from the asp package to generate schemas? Why not eg NJsonSchema (disclaimer: I’m the author)?

Yes, we're planning on taking a dependency on the v2 version of the Microsoft.OpenApi package for GA which takes a dependency on JsonSchema.NET's APIs for representing the schemas so the two fit together well.

Best would be to split up MS.OpenApi into OAI and System.TextJson.Schema and then let other libs extend generators with filters/processors.

I'm understanding this question is related to the relationship between JSON Schema generation and the OpenAPI document.

Assuming I understood the question correctly, the challenge with this as I see it there's two aspects to the JSON schema component. You need the object model to represent the JSON schema within your OpenAPI document and then you need the reflection-based APIs to support generating JSON schemas from .NET types. At the moment, the two are deeply intertwined as there's no common JSON schema object model that the reflection-based APIs can write to.

[RicoSuter]

Yes, we're planning on taking a dependency on the v2 version of the Microsoft.OpenApi package for GA which takes a dependency on JsonSchema.NET's APIs for representing the schemas so the two fit together well.

Ok, I see the refs in Microsoft.OpenApi (you should probably not use another repo name than the package name :-)):

<ItemGroup>
  <PackageReference Include="JsonSchema.Net" Version="4.1.5" />
  <PackageReference Include="JsonSchema.Net.OpenApi" Version="1.1.0" />
</ItemGroup>

I think it's important that MS eventually also even owns the JSON Schema model and generate it as part of System.Text.Json, eventually even the JsonConverter should have a "ConvertSchema" method so the serializer metadata can fully and correctly describe its schemas... (/cc #29887)

Other tools should also be able to hook into the full schema generation process (schema, operation and document filters/processors).

[captainsafia]

I think it's important that MS eventually also even owns the JSON Schema model and generate it as part of System.Text.Json, eventually even the JsonConverter should have a "ConvertSchema" method so the serializer metadata can fully and correctly describe its schemas...

Yep! Totally agree with this. I think JSON schema is foundational enough that coverage in STJ to resolve this would be much valued. But, for now, we have to work with the constraints of the ecosystem. 😅

Would definitely recommend chiming in to the issue in the dotnet/runtime repo with your thoughts around filters/processors since I haven't seen mention of that in that issue thread yet.

[swythan]

Since swagger-ui is independent of the OpenAPI document, users can independently incorproate into their applications if needed via third-party packages or their own code (swagger-ui just needs a pointer to the served OpenAPI JSON file).

Will this be a tested and documented deliverable of the work you are doing here?

Exposing the Swagger UI on internal APIs is a deeply embedded workflow at my employer. It's often the lowest friction way to get information out of (or into) a system that doesn't really justify the expense of maintaining a dedicated UI.

[captainsafia]

@swythan Yes, Swagger UI operates fairly independently of the OpenAPI document generation pattern. Most of the magic is in bundling the static web assets needed by Swagger UI and giving it a pointer to the OpenAPI document.

Here's a quick sample doing this with a Minimal API to showcase what the packages are doing under the hood.

app.MapGet("/swagger", () => Results.Content("""
    <html>
    <head>
        <meta charset="UTF-8">
        <title>OpenAPI</title>
        <link rel="stylesheet" type="text/css" href="https://unpkg.com/swagger-ui-dist/swagger-ui.css">
    </head>
    <body>
        <div id="swagger-ui"></div>

        <script src="https://unpkg.com/swagger-ui-dist/swagger-ui-standalone-preset.js"></script>
        <script src="https://unpkg.com/swagger-ui-dist/swagger-ui-bundle.js"></script>

        <script>
            window.onload = function() {
                const ui = SwaggerUIBundle({
                url: "/openapi.json", // or the URL where your OpenAPI document is located
                    dom_id: '#swagger-ui',
                    deepLinking: true,
                    presets: [
                        SwaggerUIBundle.presets.apis,
                        SwaggerUIStandalonePreset
                    ],
                    plugins: [
                        SwaggerUIBundle.plugins.DownloadUrl
                    ],
                    layout: "StandaloneLayout",
                })
                window.ui = ui
            }
        </script>
    </body>
    </html>
    """, "text/html"));

With regard to testing, a generated OpenAPI document that can plug into Postman/Redoc/Kioata/etc. should also be renderable in Swagger UI with no issues.

[desjoerd]

I think it's important that MS eventually also even owns the JSON Schema model and generate it as part of System.Text.Json, eventually even the JsonConverter should have a "ConvertSchema" method so the serializer metadata can fully and correctly describe its schemas...

Yep! Totally agree with this. I think JSON schema is foundational enough that coverage in STJ to resolve this would be much valued. But, for now, we have to work with the constraints of the ecosystem. 😅

Would definitely recommend chiming in to the issue in the dotnet/runtime repo with your thoughts around filters/processors since I haven't seen mention of that in that issue thread yet.

We use filters and customizations a lot on the openapi generation. Mainly around conventions like schema naming, but also to add the correct authentication schemes based on the authorization policies (including scopes). I am happy to share them, but will require some cleaning.

Another important thing is being able to generate multiple documents (also on build) based on custom logic, mainly around versioning or different types of apis (apis facing the front end vs backend, operations apis etc.)

[captainsafia]

We use filters and customizations a lot on the openapi generation. Mainly around conventions like schema naming, but also to add the correct authentication schemes based on the authorization policies (including scopes). I am happy to share them, but will require some cleaning.

Yep! I see the use of filters for authentication schemes a lot. I actually did some experimentation in the space about two years ago and shared some of my notes on the experience in the comments of this issue. It's definitely not trivial to do and I'll have to rethink how we approach this if document generation is a built-in feature. I'm also a little bit cautious about being too automagical with anything authentication-related. 😅 But yes, asset built-in support for auto-generating auth schemes in the future, this is something we'll definitely need document-wide configurations for.

Another important thing is being able to generate multiple documents (also on build) based on custom logic, mainly around versioning or different types of apis (apis facing the front end vs backend, operations apis etc.)

I've been reasoning through support for multiple API versions using the Asp.Api.Versioning library. @desjoerd Are you using this in your codebase or a different tool? If you have a custom setup, I'd be curious to see how it works.

[desjoerd]

I do versioning the explict/KISS way, that is, checking the namespace of the endpoint whether a part contains a version. Split on '.' and looking for V{0}. And then duplicating (with the required modifications for a version bump) the supported Endpoints. During all past assignments which I've done it was rarely needed to do a v2, and when it was, it was of major changes. In almost all cases it was enough to just add fields for get, or add optional fields for post/put. Because changing the api (version bump) costs a lot, and that cost is mostly on the consumer side ^^.

In the current situation we have an sync api used by an app, and a management api to control assign tasks to users which will be synced. So a "public" api and an "internal" api. Splitting is done with endpoint tags.

The hardest part of splitting up the api with Swashbuckle (which we're currently using) are the schemas, and having one big schema repository which is shared between the generation of multiple documents.

[desjoerd]

I also want to add, custom json schema support is, I would say, a must. To support custom STJ converters, for example for GeoJSON.

[Lanayx]

As an author of one of F# web frameworks around asp.net I'd like to ask for an extension method to explicitly specify inbound/return types of minimal API handler. And the reflection option to process those types, since source generators are unavailable in F#. This will be helpful for the frameworks where ReqeuestDelegate oveload is used rather than just Delegate.

[jimcward]

Just to add that we make fairly extensive usage of the currently available extension points to enhance/tweak our swagger output, things like:

• IDocumentFilter
• IOperationFilter
• ISchemaFilter

It would leave us (and I am guessing others) in a difficult place if there weren't equivalent extensible points provided, thanks.

[captainsafia]

Just to add that we make fairly extensive usage of the currently available extension points to enhance/tweak our swagger output, things like:

@jimcward I would be curious to learn about what kinds of modifications you're making with each filter type. Are the changes you are making tweaks to fix poorly generated schemas or modifications to enhance the generated documents (e.g. adding auth schemes).

[captainsafia]

As an author of one of F# web frameworks around asp.net I'd like to ask for an extension method to explicitly specify inbound/return types of minimal API handler. And the reflection option to process those types, since source generators are unavailable in F#. This will be helpful for the frameworks where ReqeuestDelegate oveload is used rather than just Delegate.

@Lanayx I'm curious to learn more about this -- perhaps it's worthwhile to file another issue with a repro to discuss this. My understanding is that there should be metadata in the ApiExplorer types to help you with the above.

[jimcward]

Just to add that we make fairly extensive usage of the currently available extension points to enhance/tweak our swagger output, things like:

@jimcward I would be curious to learn about what kinds of modifications you're making with each filter type. Are the changes you are making tweaks to fix poorly generated schemas or modifications to enhance the generated documents (e.g. adding auth schemes).

Sure! So:

• IDocumentFilter - so we use some IApplicationPartTypeProvider / ApplicationPart implementation to auto-generate some controllers for repetitive reference data via reflecting over DTOs - we've found that some of the output for auto generated controllers is less than ideal and so we can use an IDocumentFilter at the end of the process to tidy up certain aspects to make the output more sensible

• IOperationFilter - similar to the above - when generating operations based on reflectively generated controllers the output is not ideal so we use an IOperationFilter to properly name parameters and operation summaries in a more readable format - this allows us to generate easier to understand documentation when reviewing available APIs via something like Swagger UI

• ISchemaFilter - we use these for a few different reasons - one example is when we want to more clearly differentiate between the OpenAPI "date" and "date-time" schema types - by default DateTime will be output as "date-time" but we tag certain properties to highlight these expect only dates - we also use it to provide nicer output for schema elements that might have generic arguments as well as to apply "vendor extensions" where we may want endpoints to support things like power automate (for example, to apply dynamic values extensions: https://learn.microsoft.com/en-us/connectors/custom-connectors/openapi-extensions#use-dynamic-values )

Please let me know if you would like any more details on any of the above!

[martincostello]

I must have misunderstood what you mean by this then 😄

If its not in the framework, why not put it

[JohnGalt1717]

The biggest issue here, and I'd like to see @davidfowl chime in is:

With Aspire, the dependencies that are injected for any given micro service are spun up by host on dev. (and NEVER on CI/CD where this is already a massive problem) As a result, your micro service CANNOT independently run successfully because it's dependencies won't be there (i.e. Redis cache has to ping, RabbitMQ won't be there, DataProtection will execute and blow up because the database isn't there.

Hence, the strategy, which was never the right approach and just forced because of how the documenter worked of running the project, actually spinning it up, and then reflecting on the running solution, is no longer viable in the day and age of Aspire. (it never was, but now it's really bad)

This needs a source generator and the effect of that generator is the swagger.

It should be looking for the minimal apis, controllers, or anything else that is documented (i.e. see FastEndpoints/FastEndpoints#677 for Fast-Endpoints) appropriately and then walking the stack of dependent DTOs etc. from the endpoint implementations without having to execute the code.

Anything else is just going to have to be rebuilt anyhow because of the Aspire problem IMHO. Add in that actually executing your web service during CI/CD and making sure that it has connection strings, and can actually access dependencies like listed above, was always a PITA that needs to go away.

Same goes for KOTA clients.

[captainsafia]

@JohnGalt1717 Some thoughts on this based on prototyping and thinking I've done in the past.

An OpenAPI source generator is an appealing option but there's a few challenges with it:

• Given our current capabilities, we'd only be able to support OpenAPI generation for minimal APIs, at minimum. There's prior art for being able to statically analyze a minimal API and create a in-memory model that we can use to generate the OpenAPI document at build-time. We don't have prior art for static analysis on controller-based APIs so there's an unknown to resolve here.
• There are aspects of the application that influence the OpenAPI model that are really hard to statically analyze at runtime. Input/output formatters in MVC, implementers of the IEndpointMetadataProvider interface, etc. are really hard to statically analyzer and introspect.
• At the moment, source generators don't support producing anything but C# source files. There's hacky workarounds you can use to produce non-C# files but they're not viable strategies for something we'd ship in framework. There's an open proposal to add capabilities to emit non-C# files from generators but until that is done you get a host of wacky emission strategies.
• With a runtime-based implementation, we can build on the capabilities of the IApiDescriptionProvider to allow any web framework built on-top of ASP.NET to light up OpenAPI support. There's overhead to consider what kinds of abstractions a source generator based approach would provide to allow web frameworks in the ecosystem to opt-in. Is it up to each framework to roll its own generator? Do the frameworks implement a generator that writes some intermediary state that ASP.NET Core's OpenAPI generator would consume? There's design thinking to do here.

So, those are the challenges that make a purely source-generator based implementation for OpenAPI difficult to implement.

Now, you've mentioned a totally valid problem with the current approach (for both Aspirified and non-Aspirief implementations), our current approach always requires you to run the application in order to resolve the OpenAPI file. For local development scenarios, this is tangible. But it gets complicated in CI when you have dependencies that might not be configured. People end up having to configure their application startup with special configuration that stubs out uninitialized dependencies when called from the build-time generation problem. Source generators are one approach to solve this problem, but there are likely alternatives that are less expensive for solving this problem.

[julealgon]

People end up having to configure their application startup with special configuration that stubs out uninitialized dependencies when called from the build-time generation problem. Source generators are one approach to solve this problem, but there are likely alternatives that are less expensive for solving this problem.

@captainsafia isn't this similar to the problem that the EFCore team faces with the need to run migrations (and then have to load assemblies with potential configuration) to do that?

I recall at one point a strategy was devised where people could implement a custom "factory" method that would only be called by the migration mechanism to load the needed dbcontext without the rest of the dependencies configured. They called this "design-time factory".

This later gave rise to the idea of "migration bundles" which are an even more decoupled approach to apply migrations.

Perhaps something similar could be devised for OpenAPI generation, that would only look at API-related aspects of the application without loading all of its external dependencies and then produce the OpenAPI spec document.

I can say it is an absolute PITA to have to run your full API app just to grab the openapi document from it at build-time... we need that for updating our Azure API Management schemas during CI/CD for each of our APIs hosted there. A mechanism similar to EFCore bundles (or even the "design-time factory" approach) would be very welcome here.

[Eneuman]

I can only agree, it's such a pita having to start a the app just to create the json. We have lots of code just to disable different things like Application Insight Profiler because it can't run when creating the json

[JohnGalt1717]

People end up having to configure their application startup with special configuration that stubs out uninitialized dependencies when called from the build-time generation problem. Source generators are one approach to solve this problem, but there are likely alternatives that are less expensive for solving this problem.

@captainsafia isn't this similar to the problem that the EFCore team faces with the need to run migrations (and then have to load assemblies with potential configuration) to do that?

I recall at one point a strategy was devised where people could implement a custom "factory" method that would only be called by the migration mechanism to load the needed dbcontext without the rest of the dependencies configured. They called this "design-time factory".

This later gave rise to the idea of "migration bundles" which are an even more decoupled approach to apply migrations.

Perhaps something similar could be devised for OpenAPI generation, that would only look at API-related aspects of the application without loading all of its external dependencies and then produce the OpenAPI spec document.

I can say it is an absolute PITA to have to run your full API app just to grab the openapi document from it at build-time... we need that for updating our Azure API Management schemas during CI/CD for each of our APIs hosted there. A mechanism similar to EFCore bundles (or even the "design-time factory" approach) would be very welcome here.

Swashbuckle has this. It is great in theory but in practice, you end up with it breaking all of the time on top of the fact it's a black box that can't be debugged.

PS: EF Core is a little different in that they need the connection created so that they can interogate the state of the database to create the migration. That isn't required here.

[JohnGalt1717]

@JohnGalt1717 Some thoughts on this based on prototyping and thinking I've done in the past.

An OpenAPI source generator is an appealing option but there's a few challenges with it:

• Given our current capabilities, we'd only be able to support OpenAPI generation for minimal APIs, at minimum. There's prior art for being able to statically analyze a minimal API and create a in-memory model that we can use to generate the OpenAPI document at build-time. We don't have prior art for static analysis on controller-based APIs so there's an unknown to resolve here.
• There are aspects of the application that influence the OpenAPI model that are really hard to statically analyze at runtime. Input/output formatters in MVC, implementers of the IEndpointMetadataProvider interface, etc. are really hard to statically analyzer and introspect.
• At the moment, source generators don't support producing anything but C# source files. There's hacky workarounds you can use to produce non-C# files but they're not viable strategies for something we'd ship in framework. There's an open proposal to add capabilities to emit non-C# files from generators but until that is done you get a host of wacky emission strategies.
• With a runtime-based implementation, we can build on the capabilities of the IApiDescriptionProvider to allow any web framework built on-top of ASP.NET to light up OpenAPI support. There's overhead to consider what kinds of abstractions a source generator based approach would provide to allow web frameworks in the ecosystem to opt-in. Is it up to each framework to roll its own generator? Do the frameworks implement a generator that writes some intermediary state that ASP.NET Core's OpenAPI generator would consume? There's design thinking to do here.

So, those are the challenges that make a purely source-generator based implementation for OpenAPI difficult to implement.

Now, you've mentioned a totally valid problem with the current approach (for both Aspirified and non-Aspirief implementations), our current approach always requires you to run the application in order to resolve the OpenAPI file. For local development scenarios, this is tangible. But it gets complicated in CI when you have dependencies that might not be configured. People end up having to configure their application startup with special configuration that stubs out uninitialized dependencies when called from the build-time generation problem. Source generators are one approach to solve this problem, but there are likely alternatives that are less expensive for solving this problem.

All valid.

So what I've done, because I got so fed up with the state of swagger in the past, was I wrote a CLI that you pointed at the result of the compile stage. It would load the assembly graph, reflect on that finding all endpoints from that reflection, and then generate clients directly that were done the way I wanted them.

Could this approach be done while we're waiting to convert that to a source generator? A dotnet tool that takes as the input the dll/exe and reflects on it and generates the openAPI like the source generator would create?

That way you could easily just create a tasks.json execution for it, and you could easily CI/CD it with the tool that would generate an artifact in the destination folder for the openapi.json file and it wouldn't require that it be run.

One would think that using the attributes on the stuff being reflected, that that would also be straight forward for controllers since a controller that takes more than JSON should be adorned with the output types that it generates. Just make that a requirement at at least the class level to define anything more than JSON as output and input types, and you solve that issue in the reflective model.

If built carefully, the KOTA team could also make use of the same work to generate clients in other languages directly instead of requiring the same steps one would think.

Only thing I can think of is that AOT might defeat this, but perhaps there's an easy why to leave the unshaken version around that still has the metadata needed? Is this even an issue really?

As an aside, please, whatever you generate, please generate OpenAPI 3.1 with proper enums with key/values and generics correctly too.

[Eneuman]

It would be great if we could get a easy way to check if the app is running because it want to generate a OpenAI json. I have seen that people are using different way and they all look kinda messy :(

[gregsdennis]

At the moment, source generators don't support producing anything but C# source files.

Sounds like a reason to go API-first, to me. Design the API with your OpenAPI doc and let a source generator build the code for it.

Thanks for coming to my TED talk.

[captainsafia]

@vipwan Thanks for trying this out! And good bug find. Do you mind filing a separate issue for this so it can be tracked there?

For some context into why this is happening, the OpenAPI implementation is taking advantage of a prototype of the new JSON schema generation support in System.Text.Json. The APIs used there JsonSchemaMapper.GetJsonSchema(JsonSerializerOptions, Type) return a JsonObject that we use a custom JSON converter implementation to map to the OpenAPI schema superset. The bug you're running into is likely happening in this layer.

[bulat30]

As an author of one of F# web frameworks around asp.net I'd like to ask for an extension method to explicitly specify inbound/return types of minimal API handler. And the reflection option to process those types, since source generators are unavailable in F#. This will be helpful for the frameworks where ReqeuestDelegate oveload is used rather than just Delegate.

@Lanayx I'm curious to learn more about this -- perhaps it's worthwhile to file another issue with a repro to discuss this. My understanding is that there should be metadata in the ApiExplorer types to help you with the above.

Are there ways to get metadata in runtime in ApiExplorer? For example, i want to provide swagger for YARP based reverse-proxy using yarp configs and proxies services swaggers

[julealgon]

Are there ways to get metadata in runtime in ApiExplorer? For example, i want to provide swagger for YARP based reverse-proxy using yarp configs and proxies services swaggers

@bulat30 that's what the ApiExplorer abstraction is for. I'd suggest raising a question in the "Discussions" area or on StackOverflow for further discussion as this doesn't seem to be related to this issue here.

[captainsafia]

Hi everyone -- I've been a bit quiet on this issue on account of all the code slinging that's been happening 😅 but as we approach .NET 9 RC1, I wanted to take some time to share some updates on the work and issue a CTA for folks to try out the release.

I've been updating the original issue with completed items as dev work has been going but here's a bird's eye view of what will land in RC1:

• Support for generating an OpenAPI document at runtime via the AddOpenApi/MapOpenApi APIs
• Support for customizing the OpenAPI document via the transformers (at both the document and operation level)
• Support for customizing generated schemas via schema transformers
• End-to-end native AoT support for OpenAPI + minimal APIs
• Support for build-time document generation using the pre-existing Microsoft.Extensions.ApiDescription.Server package

Here are the things that aren't landing in RC1:

• Support for XML comments. I'm planning on shipping this support via an experimental package for .NET 9. For more info on this, check out the comment I recently added to the original issue.
• Support for generating OpenAPI documents for gRPC. This is blocked on some new APIs we'd need in the underlying JsonSchemaExporter APIs. See Provide API to allow JSON converters/types to describe their associated JSON schema runtime#105769 for more info.

As for the CTA, now is the best time to try out the nightly builds of the package and share your feedback. We're past the point of being able to introduce new APIs (unless it's absolutely a blocker), but this is the time to do some serious bug-squashing and polishing ahead of RC1.

You can follow the instructions in the SDK repo for how to install the latest nightly SDK builds and setting up NuGet feeds to access the nightly packages. You can find the current draft of the docs on this page.

[martincostello]

We're past the point of being able to introduce new APIs

This part of the comment reminded me that I'd been meaning to check out this comment regarding YAML support: #54598 (comment)

I just had a quick look at trying to generate the OpenAPI document as YAML with user code, and it's possible but it needs to use reflection and knowledge of the implementation details to do so as the API to generate it relies on an internal type (OpenApiDocumentService).

Here's a Gist containing the (proof-of-concept) code to do it: https://gist.github.com/martincostello/c6d6167783b811d842163cdaf39a0987

Other than the reflection, the only downside is that it requires a copy of the ScrubbingOpenApiJsonWriter to remove the internal book-keeping IDs from the document if it's not wanted in the output.

It does make me think that maybe there's a missing extensibility hook here, as even without explicit support for "YAML instead of JSON", there's no way to get the "finalised" document to choose to do something else with it.

[captainsafia]

Here's a Gist containing the (proof-of-concept) code to do it: gist.github.com/martincostello/c6d6167783b811d842163cdaf39a0987

Thanks for sharing this workaround! Agree that some of the private reflection is blegh but it'll hopefully be a good resource for folks who want to work around this.

My original intent for solving this was to expose a new option in OpenApiOptions although I admittedly dropped the ball on this.

Assuming that what's important for folks is generating YAML at build-time, we can probably get this supported in the document generation CLI tool without needing API review.

Other than the reflection, the only downside is that it requires a copy of the ScrubbingOpenApiJsonWriter to remove the internal book-keeping IDs from the document if it's not wanted in the output.

Getting microsoft/OpenAPI.NET#1719 into OpenAPI.NET should hopefully alleviate the need for this.

It does make me think that maybe there's a missing extensibility hook here, as even without explicit support for "YAML instead of JSON", there's no way to get the "finalised" document to choose to do something else with it.

Yep, this is something I was intentionally conservative about this go around. In the long-term, I'd eventually like to see us replace our pubternal IDocumentProvider API with a public one that allows access to the OpenApiDocument directly. I think it makes sense for OpenApiDocumentService to continue to be a private generator service.

[xC0dex]

Hey,

I'm really excited about the upcoming feature 🥳! Currently, I'm working on a library that enhances the built-in generator by providing useful OpenApiOperation and OpenApi transformers along with a custom Swagger UI.

I’m curious if there’s a way to retrieve the registered document names via the ServiceProvider. The motivation behind this question is that I provide an endpoint called, MapSwaggerUi, that can dynamically gather all registered OpenApi documents, so users don’t need to manually specify the routes or document names. Unfortunately, the interface IDocumentProvider and the classes OpenApiDocumentProvider and NamedService are internal.

As a workaround, I’ve wrapped the AddOpenApi() method (here’s the implementation: AddOpenApiDocument() to store the document names in internal options that MapSwaggerUi can then access.

I’d appreciate any kind of feedback! Hopefully, I haven’t missed an existing issue that already addresses this.

Thanks in advance.

Edit:

I spotted an issue #58353 which addresses this feature request.

[xwellingtonx]

I would like to suggedt that example generation for different content types are taken in consideration, I was using Swashbuckle and it doesn't not work properly to generate XML response/request examples when using XML Attributes, details below.
https://stackoverflow.com/questions/78317993/swagger-xml-example-value-doesnt-obey-xml-annotations-on-class

My suggestion in terms of architecture sould maybe have some IResponseFilter and IRequestBodyFilter, which would be couple to the operation. Swashbuckle had the IRequestBodyFilter in place so that why I'm mentioning here.

Also, just wanna bring to your knowledge this other library with implement few custom filters might be useful for testing or ideas.

https://github.com/mattfrear/Swashbuckle.AspNetCore.Filters?tab=readme-ov-file

[captainsafia]

With the release of .NET 9 right around the corner, I'm gonna close out this issue to mark completion of work in this area. However, it's by no means the end of our efforts to make OpenAPI in ASP.NET a world-class experience. We've got a few follow-ups on deck including:

• Support for enhancing OpenAPI documents with XML doc comments which is tracked in Support XML-based OpenAPI docs for minimal and controller-based APIs with Microsoft.AspNetCore.OpenApi #39927. Experimental builds of this support are available in Support XML-based OpenAPI docs for minimal and controller-based APIs with Microsoft.AspNetCore.OpenApi #39927 (comment).
• Support for rendering OpenAPI documents in YAML format is tracked in Support emitting OpenAPI documents in YAML format #58516 and currently in PR.
• Improvements to build-time OpenAPI document generation which are tracked in Improve build-time OpenAPI document generation #58353.
• Plans to add support for OpenAPI v3.1 are tracked in Upgrade Microsoft.AspNetCore.OpenAPI and add support for OpenAPI v3.1 #58619.
• Improvements to documentation for this area, both via conceptual docs on docs.microsoft.com and improved API documentation.

And, of course, we'll continue to address bugs, improve performance, and enhance the feature set/API surface for built-in OpenAPI support. Thanks for your patience as we delivered this big feature for .NET 9 and we look forward to hearing your feedback! 😄 🚀