The future of JSON in .NET Core 3.0

[terrajobst]

JSON has become an essential part of virtually all modern .NET applications and in many cases even surpassed the usage of XML. However, .NET hasn't had a (great) built-in way to deal with JSON. Instead we've relied on Json.NET which continues to serve the .NET ecosystem well.

Moving forward, we plan on making some changes to our JSON support:

• We need high-performance JSON APIs. We need a new set of JSON APIs that are highly tuned for performance by using Span<T> and allows for processing UTF-8 directly without having to transcode to UTF-16 string instances. Both aspects are critical for our web server Kestrel, where throughput is a key requirement.

• Remove dependency from ASP.NET Core to Json.NET. Today, ASP.NET Core has a dependency on Json.NET. While this provides a tight integration between ASP.NET Core and Json.NET, it also means that application developers cannot freely choose which JSON library they are using. This is also problematic for customers of Json.NET as the version is dictated by the underlying platform. However, Json.NET is frequently updated and application developers often want to -- or even have to -- use a specific version. Thus, we want to remove the dependency from ASP.NET Core 3.0 to Json.NET so that customers can choose which version to use, without fearing they might accidentally break the underlying platform. In addition, this makes it also possible to plug-in an entirely different JSON library.

• Provide an ASP.NET Core integration package for Json.NET. Json.NET has basically become the Swiss Army knife of JSON processing in .NET. It provides many options and facilities that allow customers to handle their JSON needs with ease. We don't want to compromise on the Json.NET support customers are getting today, for example, the ability to configure the JSON serialization via the AddJsonOptions extension method. Thus, we want to provide the Json.NET integration as a NuGet package that developers can optionally install so they get all the bells and whistles they get from Json.NET today. The other part of this work item is to ensure we have the right extension points so that other parties can provide similar integration packages for their JSON library of choice.

Below are more details around this plan.

The need for high-performance JSON APIs

The requirements for the .NET stack have changed a bit since the arrival of .NET Core. Historically, .NET has valued usability and convenience. With .NET Core, we've added a focus on performance, and we've made significant investments to serve high performance needs. And the improvements we made in the popular TechEmpower benchmark are a testament to that.

With .NET Core 2.1, we've added a brand new primitive called Span<T> that allows us to represent native memory and arrays in a uniform way. With this type, we've also added a set of parsing and encoding APIs that are much more memory efficient without having to resort to unsafe code.

Part of the work of minimizing allocations is to avoid having to transcode UTF-8 payloads into UTF-16 strings, purely for parsing reasons. Currently, Json.NET is implemented by reading UTF-16. We need the ability to read (and write) JSON documents directly in UTF-8 because most network protocols (including HTTP) use UTF-8.

During .NET Core 2.1 we've learned that updating our existing APIs to leverage Span<T> has limits. While we did add a bunch of overloads that accept spans, we also had to produce brand new APIs that are designed around minimizing allocations and dealing with buffers, which we exposed in System.Buffers namespaces. And with System.IO.Pipelines we've also added a programming model that enables developers to share buffers without having to deal with lifetime issues.

Based on these experiences we believe in order to support JSON parsing, we'll need to expose a new set of JSON APIs that are specifically geared for high-performance scenarios.

You might wonder why we can't just update Json.NET to include support for parsing JSON using Span<T>? Well, James Newton-King -- the author of Json.NET -- has the following to say about that:

Json.NET was created over 10 years ago, and since then it has added a wide range of features aimed to help developers work with JSON in .NET. In that time Json.NET has also become far and away NuGet's most depended on and downloaded package, and is the go-to library for JSON support in .NET. Unfortunately, Json.NET's wealth of features and popularity works against making major changes to it. Supporting new technologies like Span<T> would require fundamental breaking changes to the library and would disrupt existing applications and libraries that depend on it.

Going forward Json.NET will continue to be worked on and invested in, both addressing known issues today and supporting new platforms in the future. Json.NET has always existed alongside other JSON libraries for .NET, and there will be nothing to prevent you using one or more together, depending on whether you need the performance of the new JSON APIs or the large feature set of Json.NET.

Move Json.NET integration into a separate NuGet package

Today, you cannot use ASP.NET Core without Json.NET because it is a dependency of ASP.NET Core itself. Over the years, we've received feedback that the dependency can conflict with other libraries that have their own dependency on a different version of Json.NET. In the past, we've considered addressing this issue by using a private copy of Json.NET in ASP.NET. However, this would create problems when developers want to configure Json.NET (for instance, in order to control how the serializer behaves when formatting JSON objects).

Moving forward we'd like to:

1. Replace the internal usage of Json.NET in ASP.NET Core by the new platform-provided JSON APIs.

2. Factor the public facing usage of Json.NET into an optional integration package that can be acquired from NuGet.

So the existing integration between ASP.NET Core and Json.NET will continue to be supported, but will be moving out of the platform and into a separate package. However, since the integration is then designed to sit on top of the platform, it will also allow customers to update Json.NET to later versions.

Furthermore, customers who need more performance can also choose to use the new JSON APIs, at the expense of the rich feature set that Json.NET offers.

[steveoh]

This is great. I am all for faster and less allocating json parsing.

Will there be a discussion about the features from json.net that the new json apis will support? If there is,I think the two major features that come to mind would be renaming/casing properties and ignoring null properties.

[terrajobst]

Will there be a discussion about the features from json.net that the new json apis will support?

Yes. We've done some early thinking that we will migrate to CoreFx. It will be a feature that is designed & built in the open as usual. In addition, I've reached to authors of many of the popular JSON libraries and invited them to review early drafts of this announcement. My hope is that we can work together to create a solid JSON component for the platform while also keeping the ecosystem on top of it (such as ASP.NET Core) pluggable to allow for others. In the end, different consumers will have different goals and being able to plug-in a different library means you can get maximum flexibility in choosing the component that has the best cost/benefit for your app.

[garfbradaz]

Hey @terrajobst. Will the new JSON appear as a netstandard API surface, or just integrated into Core for now?

[terrajobst]

Hey @terrajobst. Will the new JSON appear as a netstandard API surface, or just integrated into Core for now?

Yes, the question is just which release train it can catch. 2.1 might be too early.

[seangwright]

So the JSON parsing bits baked into the framework are planned to be available when v3.0 goes to RTM or will only the integration Apis in ASP.NET Core be complete (with just one implementation - JSON.NET) that will be swappable at a later date?

[terrajobst]

The plan for 3.0 is as follows:

1. Built-in high-performance JSON APIs. Low level reader/writer, a Stream based reader/writer, and a serializer.
2. ASP.NET Core is pluggable w.r.t. to the JSON component.

There is an open ended question what the templates for ASP.NET in 3.0 will use. Depending on fidelity we can provide by 3.0 we might have them pull in the Json.NET integration package. However, the goal is to deliver enough fidelity & parity to only depend on the built-in ones by default.

[seangwright]

Thanks - that helps clear things up. 👍

And some additional questions!

If an integration package is used, will it be used throughout the entire ASP.NET Core pipeline or only in some places?
I'm assuming Kestrel will always use the internal readers/writers.

Would the Api ergonomics be:

• Supply an integration only when you want to enhance the built-ins feature set.
• Supply an integration package always but one would be built-in, integrating the low-level reader/writers into higher level functionality.

[ThatRendle]

Assuming that this new parser will be used for all the built-in JSON stuff like appSettings.json, could I put in an early request for comments to be supported?

Thanks.

[thefringeninja]

This is awesome news! Quick question: What packages will this library depend on?

[Thorium]

Why to reinvent a wheel that is tested by production customers? If there is a problem with Json.Net, just send a PR as it's open source.

I suppose the problem with Json.NET is that it's not owned by Microsoft, so it has to be replaced. Oh but there is the one already in System.Runtime.Serialization, called DataContractJsonSerializer. Can you use that, or is it just so fun to code new APIs, DIY, that it cannot be avoided?

The reason I'm not very happy with this is that Json.Net supports already edge cases like e.g. F# Discriminated Unions. Not particularly well, but on a level that developers can live with that. Instead any new API's usually forget anything else than the use case of an ASP.NET-website.

[JamesNK]

@markrendle There is a opt-in setting on JsonReader (work in progress) to allow comments. The configuration system will likely enable that setting by default.

[ThatRendle]

@Thorium Did you actually read the OP? It explains why not JSON.NET, and that JSON.NET will continue to be officially supported with an add-in package.

[ThatRendle]

@JamesNK 😄

[JamesNK]

@Thorium Json.NET isn't going away. You aren't losing anything. This is another option for simple and high performance scenarios.

[Thorium]

@Thorium Json.NET isn't going away. You aren't losing anything. This is another option for simple and high performance scenarios.

How will the Json generated to be backward compatible?

For example, I'm using SignalR which is using Json.NET in the background. Now, will my F# discriminated unions serialize to similar structures so that I will not be fighting issues with the new Azure Signalr Service (backplane) throwing runtime exceptions because of serializating the structures differently than my server's current SignalR library?

[jkonecki]

I hope others will pick up the new APIs quickly. Looking at you, @AzureCosmosDB ;-)

[Eirenarch]

Are you planning on including a class like JObject and support for dynamic or this is out of scope for this feature?

[rafaelsc]

I recommend a look in one of this libs:

• https://github.com/kevin-montrose/Jil or
• https://github.com/neuecc/Utf8Json

this could be a really good way to get inspirations.

[dotMorten]

Will DataContractJsonSerializer have this new reader/writer used internally ?

[mythz]

I've reached to authors of many of the popular JSON libraries and invited them to review early drafts of this announcement. My hope is that we can work together to create a solid JSON component for the platform while also keeping the ecosystem on top of it (such as ASP.NET Core) pluggable to allow for others.

Is there a reason why the 2nd most popular JSON library after JSON.NET - ServiceStack.Text which has already been refactored to be built on Span APIs has been excluded? ServiceStack.Text serializers are what's used to power ServiceStack which is one of the most popular "Alternative .NET Core compatible Web Framework" that supports running on more platforms in .NET. I'm curious about which "ecosystem" you're referring to and hoping to "work together" with here? I'd obviously be interested in how compatible these "pluggable" APIs end up being or whether this ends up being another area where the adoption and integration of new MS libraries ends up killing the ecosystem that it's replacing.

[ctaggart]

It may be worth reviewing the MIT licensed high-performance https://github.com/neuecc/Utf8Json

[rizamarhaban]

This is definitely what we need... my suggestion for the main class name, just use "Json".

[galvesribeiro]

@terrajobst I was wondering when this would happen...

I was always wondering why JSON.Net was added as a direct dependency rather than an abstraction (even considering it is the de-facto JSON package for .Net ecosystem).

However, I think add an abstraction for JSON-only is somehow a shoot on your feet. I think a serializer abstraction like we have in Orleans IExternalSerializer in a shape of Microsoft.Extensions.Serialization or something would be more effective...

Is there any particular reason why make is JSON-only? I see other cases where people can plug other types of serializers...

[yaakov-h]

@galvesribeiro Something like IOutputFormatter/IInputFormatter?

[galvesribeiro]

@yaakov-h wasn't aware of those... Were are they?

[yaakov-h]

@galvesribeiro AspNetCore: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.formatters.iinputformatter?view=aspnetcore-2.1

[galvesribeiro]

Okey... makes sense now. So where this new JSON-only abstractions comes to play?

[jges42]

The decision to start this undertaking is also a testament about the inefficiency of System.String (UTF-16 String).
I think that the new JSON hooks that will abstract all json handling between asp.net and a json library would look significantly better if you tackle the task to create a utf-8 string BaseType first.
--> Maybe create a System.Utf8String

[galvesribeiro]

Yeah... I remember @migueldeicaza saying a while ago that someday, he will make .Net use utf8 strings 😄

[svick]

@jges42 @galvesribeiro The proposal to add Utf8String is https://github.com/dotnet/corefx/issues/30503. It seems it's also planned for .Net Core 3.0.

[TsengSR]

@JonPSmith: Imho both use cases are invalid from both best practice and DDD point of view.

1. I have never seen a best practice which recommends deserializing entities directly (except in the most simple tutorial examples). Circular references always come at a cost. It requires tracking of already processed objects, this means: memory allocations and additional CPU cycles. But one of the mail goal of the new JSON library is to exactly avoid these memory allocations
2. Invalid too, since you never serialize into an Domain model, especially not when you get the data via a web request such as an WebApi call. In DDD you should always work with events/commands, send the command to your web application, get (and dehydrate) the entity from the repository (via ORM mapping or EventSourcing) , apply the command, persist it.

On top of that, the new JSON API is for high performance scenarios. For everything else where you need rich feature set, you (and should) still use JSON.NET or whatever fulfills your needs.

[ThatRendle]

@suncodefactory This is the opposite of a breaking change. Right now, in ASP.NET Core 2.2, JSON.NET is used by the framework as well as by user code. This has the potential to cause conflicts with your own use of Newtonsoft.Json; if ASP.NET Core 3.0 moved to JSON.NET 12.x and there was some kind of issue in there that broke your application, you'd have a problem.

For example, look at Microsoft.Extensions.Configuration.Json 2.2.0 - it has a dependency on Newtonsoft.Json 11.0.2. That's a configuration package; nothing to do with HTTP request handling or ASP.NET Core MVC. Or look at Microsoft.IdentityModel.Protocols.OpenIdConnect, which uses it for handling JSON Web Tokens; that's a hot path which needs as much performance as possible. JSON.NET is not a slow library by any standards, but it strikes a balance between performance, feature-richness and support for a massive range of user scenarios. Microsoft's new JSON library doesn't need to do that, because JSON.NET exists. So it can focus on handling the absolute basics with maximum performance.

.NET has always had its own JSON serialization solution in System.Runtime.Serialization.Json, but in the high-performance world of .NET Core it's not a very good one. I certainly wouldn't want it being invoked to check credentials on every incoming request. A new JSON library, with modern UTF-8 data handling and minimal allocations, is very welcome.

You will still be able to reference Newtonsoft.Json in your application, and continue to use it as the deserialization/serialization pipeline for request/response data as before. And from now on, you'll be able to do so without worrying about which version the Core framework depends on. That's a win for everybody.

[JonPSmith]

Thanks @phillip-haydon and @TsengSR for your thoughts. I was asking if these features would be supported and you says that aren't, which understand and accept. I will continue to use Json.NET for the case where I need to serializing/deserializing EF Core classes.

BTW. I do have a valid reason for serializing/deserializing DDD-styles EF Core entity classes. I have a library that contains a feature I call Seed from Production which allows developers to take a snapshot of data from a production database, anonymise any private data, and then seed a new database with the snapshot.

I needed this feature for one on my clients and instead of writing just for them I built it into my open-source library EfCore.TestSupport so others can use it (and my client didn't have to pay me for it).

[bruno-garcia]

Is there a plan to support [DataContract], [DataMember] and friends?

Today this is a way to define how types should serialize/deserialize (e.g. the field name) in a way that doesn't bring a dependency to any serialization library to the project using it.

The current JsonNamingPolicy takes a string only so there's no way to inspect the member's attributes.

[agjini]

Hi.
We just tried to switch our micro services to DotNet core 3 preview 6 and we are unable to deserialize our immutable reference types : class with immutable properties (no setters) and only one constructor to set all properties. Json.net correctly handle these classes.
Is this an issue of the need System.Text.Json API or is this a plan to support it ?
Thanks for your responses

[khellang]

@agjini See https://github.com/dotnet/corefx/issues/38569

[agjini]

Thanks @khellang.
Support is planned indeed but not for the 3.0 release.
It seems to be possible to continue using Json.net with DotNet core 3 but I don't know how to do it (adding package reference is not enough). Is there a way to do that?

[davidfowl]

@agjini :

services.AddControllers()
           .AddNewtonsoftJson()

[agjini]

Thanks for your help guys.
It works !
I've missed the migration guide where all is explained :

https://docs.microsoft.com/fr-fr/aspnet/core/migration/22-to-30?view=aspnetcore-2.2&tabs=visual-studio

[kofifus]

IMO json.net is half baked and making it the default (ie for signalr) which breaks existing code was premature.

[Kralizek]

On the other hand, migrating from .NET Core 2.2 to 3.0 is a major version upgrade and even if the .NET Core team isn't strictly following semantic versioning, I would expect things to break while upgrading from one version to another without explicit changes (like adding explicitly Newtonsoft's library in the pipeline)

[joperezr]

Closing given this is an announcement and not an issue

[sxlvalue]

Although the community has a lot of voices against improvement, as a new high-performance framework, the bad speed is unacceptable.

[alsami]

I know it has been said before, but I'd like to add my wish as well.

It would be really awesome if we could have immutable objects. I know it's possible by adding Json.NET to the MVC-Pipeline but in my case my tests are all failing since I am using ReadAsAsync<> which is now implemented somewhere in a peer-dependency of Microsoft.AspNet.WebApi.Client and that relies on System.Text.Json

[mwoo-o]

We provide .NET Standard Class library to customers so that they can use our library to work on any platform that supports .NET Standard. We need to use System.Text.Json in our class library. What will be the plan to support System.Text.Json in .NET Standard?

[terrajobst]

@alsami

It would be really awesome if we could have immutable objects.

Do you only need the ability to prevent others from mutating it or do you also need the ability to create new instances with parts being smartly replaced (like immutable collections and Roslyn)? If you need the former, we've got you covered with the upcoming JsonDocument DOM APIs.

[terrajobst]

@mwoo-o

What will be the plan to support System.Text.Json in .NET Standard?

It's available as a NuGet package for .NET Standard 2.0: System.Text.Json.

[mwoo-o]

@terrajobst

Thanks. When will this System.Text.Json be included in the .NET Standard SDK?
Will the .NET standard 3.0 (or some other later release versions) include System.Text.Json package? Will it happen in .NET Core 3.0 SDK production release?

[sleemer]

@terrajobst

Are there any plans to make Deserialize method work with PipeReader? Or add Patch method that can be used in streaming scenarios where we don't have all the data when we start deserialization.

Here is a simplified version of the proposed API:

private async ValueTask<T> Deserialize<T>(PipeReader reader, CancellationToken cancellationToken) 
	where T: new()
{
    T model = new T();
    while (!cancellationToken.IsCancellationRequested)
    {
        ReadResult readResult = await reader.ReadAsync(cancellationToken);
        ReadOnlySequence<byte> buffer = readResult.Buffer;
        
        if (readResult.IsCanceled) break;
        if (buffer.IsEmpty && readResult.IsCompleted) break;
        
        SequencePosition consumed = JsonSerializer.Patch(model, buffer, readResult.IsCompleted);
        reader.AdvanceTo(consumed, buffer.End);               
    }

    return model;
}

public SequencePosition Patch<T>(T model, ReadOnlySequence<byte> jsonData, bool isFinalBlock, JsonSerializerOptions options = null)
{
      ...            
}

[alsami]

@terrajobst

ability to prevent others from mutating it

Only this currently. Is really just for 'data-transfer-objects'. Great news!

[TsengSR]

@mwoo-o

Thanks. When will this System.Text.Json be included in the .NET Standard SDK?
Will the .NET standard 3.0 (or some other later release versions) include System.Text.Json package? Will it happen in .NET Core 3.0 SDK production release?

There is no .NET Standard SDK. .NET Standard is an API surface, available on all supported Platforms. You can ship System.Text.Json in any application which runs targets the supported platforms supported by the .NET Standard, see .NET implementation support.

[terrajobst]

@TsengSR

There is no .NET Standard SDK. .NET Standard is an API surface, available on all supported Platforms.

Well, there is a project type that allows you to use the APIs. I think @mwoo-o is asking whether we have plans to add System.Text.Json to .NET Standard. The answer is no. Right, now we're planning on leaving this to be a NuGet package.

[samchenws]

It's terrible.Too few functions to be applied in the project.