Will JsonSerializer support parsing to existing object?

[gongdo]

When I need to call deserializer for the same type inside a long running loop, I prefer to overwrite to existing instance rather than create a new instance every time.

Json.NET has a feature for it: JsonConvert.PopulateObject(string, object)
(I think the target should be the first argument of it, but it doesn't matter.)

For System.Text.Json, its usage might look like this:

using System.Text.Json
var tmp = new MyClass();
JsonSerializer.ParseTo(ref tmp, "{\"some\":\"json\"}");

I found an option ClassMaterializerStrategy that might do similar thing, but it was an internal option. Even if it was not an internal, I believe a direct method like ParseTo would be better.

[ahsonkhan]

cc @steveharter

[ahsonkhan]

From @tig in https://github.com/dotnet/corefx/issues/42633

I hadn't actually used Json.NET's ObjectCreationHandling support and may have mis-interpreted what it did. I thought it gave an option for me to deserialize into an existing object instance. E.g. (psuedocode):

class Doc{ 
    public string Title;
    public string Type;
}
...
// jsonString holds instance where doc.Type = "Foo";
Document doc = new Document() { Title = "Title" };
Deserialize<Document>(doc, jsonString);

With this, doc would have doc.Title == "Title" and 'doc.Type == "Foo"`.

Forgive me if I misunderstood. But this is what I'm looking for. I don't want Deserialization to create a new instance, but update an existing instance.

[tig]

FWIW, here's my scenario just so y'all have an understanding of why I think this feature is important:

I'm building an app that uses JSON as a config-file format, ala VS Code, Windows Terminal, etc... All the cool kids are doing this. Like those apps, I have a filesystem watcher that watches the .config file for changes. Like those apps, I'd like all my settings to refresh when this happens.

The classes I'm serializing with System.Text.Json are actually some of my Models (in MVVM) which implement INotifyPropertyChanged. My upper layers thus subscribe to property changes.

This all works groovy, except that without System.Text.Json supporting the equivalent of PopulateObject I have to jump through hoops with temporary instances and deep copies. I have it working for now, but would prefer the deserialization to just allow me to pass an existing instance that it would update.

Thanks for listening.

[reazhaq]

I vote for this one too. If objects' hierarchies are deep with lots of fields/properties; this shall come in handy. Support polymorphic deserialization example looks simple because derived classes only have one property each.

I have a custom converter that writes generic object type name along with bounding types as part of the json. During deserialization, it creates correct object using those two values, and rest of the reading is done by serializer.Populate(.... call. Thinking about converting it into System.Text.Json flavor; looks like I may have to hand roll bunch of code

[mwadams]

@reazhaq - this is exactly the scenario we have - I have a custom JsonConverter<object> which examines a type discriminator to instantiate the relevant type.

It then wants to populate the contents of the type it has instantiated, using the default behaviour, otherwise you get into an infinite recursion.

It would be nice to be able to do a JsonDocument.ParseValue() over the reader so I could inspect the document and pull out the discriminator without having to handle the reading ahead myself [we find it generally unsafe to assume that the discriminator is first property on the object], create a Utf8JsonReader over that document to avoid allocating extra buffers, and pass it on to the default ObjectConverter<T> (now we know what T is) to be populated.

[BrunoBlanes]

With C#8 nullable types, we get a warning for not initializing a non-nullable list, but System.Text.Json doesn't make use of initialized lists and so we can't and therefore can't dismiss the warning. Am I doing something wrong or is that just the way it is for now?

[rwkarg]

With C#8 nullable types, we get a warning for not initializing a non-nullable list, but System.Text.Json doesn't make use of initialized lists and so we can't and therefore can't dismiss the warning. Am I doing something wrong or is that just the way it is for now?

Our workaround for DTOs is to create an empty constructor (if there isn't one already) and wrap it in:

#nullable disable
public MyClass() {}
#nullable restore

This will stop the analysis for uninitialized properties. We only use this for DTOs where we have knowledge that something in the system stack has already ensured that the property won't be null when deserialized.

The other option is to place the #nullable directives around each property that is marked as non-nullable but isn't initialized. This is more specific, but also more verbose.

[BrunoBlanes]

Yes, I figured suppressing the warning would be a workaround, but was kind of hopping that it wouldn't be necessary. I wish System.Text.Json would make use of initialized lists, that way I didn't have to deal with it in this way.

[layomia]

Linking #30258 which is concerned with an option to reuse rather than replace object properties on deserialization. They should be considered together during design and implementation.

[eiriktsarpalis]

Triage: we want to eventually add this feature but unfortunately it won't fit our schedule for .NET 6, moving to future.

[eiriktsarpalis]

Closely related to #30258. For .NET 7 we will be prioritizing deserialization on existing collections but not objects. Moving to future.

[olivermue]

By the way, even Microsoft itself needs a Populate() method in some places: https://github.com/microsoftgraph/msgraph-sdk-dotnet-core/blob/57861dc4aea6c33908838915c97fc02105b6e788/src/Microsoft.Graph.Core/Serialization/DerivedTypeConverter.cs#L112-L114

And just another IMHO:
I think the reason why this is still not implemented and will be postponed every time back to Future milestone is the signature of JsonConverter<T>.Read(). This method is one of the core basics used in this context and there are already dozens of own implementations out in the wild. To really support Populate(), this method must either be changed (what is impossible, cause it is already shipped) or another base class / interface must be designed. But this means, that all customers self-written JsonConverter<T> have to be re-written too to support Deserialize AND Populate or some awkward exceptions will come up, depending on how you mix and match old and new converters in conjunction with an arbitrary object tree and a desired JSON string. This makes the feature too costly and it will be postponed to another Future. So I think this feature will never come up in System.Text.Json.JsonSerializer, maybe some shiny day in System.Text.Json.BetterJsonSerializer, but never in the current used namespace/class.

[kierenj]

This would be fantastic for reducing memory allocations in my app; I want to re-use target objects to avoid allocaitons / GC pauses. Unless there's another route to achieving this?

[BreyerW]

@kierenj Once public ContractResolver is available and u port this snippet to STJ

using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading.Tasks;

namespace Sharp.Serializer
{
	public class ReferenceConverter : JsonConverter
	{
		public override bool CanWrite => false;
		public override bool CanConvert(Type typeToConvert)
		{
			return (typeToConvert.IsClass || typeToConvert.IsInterface) && typeToConvert != typeof(string);
		}
		public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
		{
			reader.Read();
			if (reader.TokenType is JsonToken.StartObject)
				reader.Read();
			var id = string.Empty;
			var valueId = string.Empty;
			if (reader.Value is string and "$id" or "$ref")
			{
				id = reader.Value as string;
				reader.Read();
				valueId = reader.Value as string;
				reader.Read();
			}
			if (reader.Value is string and "$type")
			{
				reader.Read();
				reader.Read();
			}
			var contract = serializer.ContractResolver.ResolveContract(objectType) as JsonObjectContract;
			var target = serializer.ReferenceResolver.ResolveReference(null, valueId) ?? RuntimeHelpers.GetUninitializedObject(objectType);
			if (id is "$ref")
			{
				reader.Read();
				return target;
			}
			else
				serializer.ReferenceResolver.AddReference(null, valueId, target);

			while (reader.TokenType is not JsonToken.EndObject)
			{
				var propName = (string)reader.Value;
				reader.Read();
				var p = contract.Properties.GetClosestMatchProperty(propName);
				if (p.Ignored)
				{
					reader.Skip();
					reader.Read();
				}
				else
				{
					p.ValueProvider.SetValue(target, serializer.Deserialize(reader, p.PropertyType));
					//while (depth != reader.Depth && reader.TokenType is not JsonToken.PropertyName)
					if (reader.TokenType is not JsonToken.PropertyName)
						reader.Read();
				}
			}
			return target;
		}

		public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
		{
			throw new NotImplementedException();
		}
	}
}

it should be possible to achieve deep PopulateObject effect using standard Deserialize methods. Few notes: this snippet is minimal it doesnt support any knobs nor polymorphism (should be kinda easy to add tho if you change if (reader.Value is string and "$type") and below). Keep this converter as last in your chain if you have any custom. Also to have deep PopulateObject all the way you have to make sure custom converters for reference objects also have customized ReferenceHandler behaviour like in this snippet (calling ResolveReference before AddReference on deserialize which is different from default)

EDIT: also i dont think it will work with "special" classes like List or Dictionary, for those you will need custom converter but only if you care about deep PopulateObject

EDIT2: forgot to add (was in a hurry when writing this) that for this to work custom ReferenceHandler is needed since standard Deserialize doesnt accept populated object and this have to be worked around via custom ref handler. If you dont need deep Populate then instead of custom converter+custom ref handler i would write just JsonPopulator.Populate based on construct shown in above custom converter. But that will be possible only in next vs 2022 preview release where custom contract should be avalable unlees ur willing to build bcl yourself

[TonyValenti]

Is there a .NET6 / .NET7 way to deserialize JSON onto an existing object? I'm hoping to be able to provide a JSON snippet that overrides properties on an existing object.

[BreyerW]

@TonyValenti built-in no neither on 6 or 7 but with public contract in NET 7 (should be available in next preview or right now if ur willing to build BCL yourself from this repo) you can build it almost trivially yourself (unless you want to support all the knobs in which case it becomes cumbersome and time-consuming but not particularly difficult).

[RicoSuter]

In .NET 7 you can do that quite easily (however this implementation only overwrites the root object and does not merge children [unlike Newtonsoft.Json]):

    public static class JsonUtilities
    {
        internal class PopulateTypeInfoResolver : IJsonTypeInfoResolver
        {
            private bool _isRootResolved = false;

            private readonly object _rootObject;
            private readonly Type _rootObjectType;
            private readonly IJsonTypeInfoResolver _jsonTypeInfoResolver;

            public PopulateTypeInfoResolver(object rootObject, IJsonTypeInfoResolver jsonTypeInfoResolver)
            {
                _rootObject = rootObject;
                _rootObjectType = rootObject.GetType();
                _jsonTypeInfoResolver = jsonTypeInfoResolver;
            }

            public JsonTypeInfo? GetTypeInfo(Type type, JsonSerializerOptions options)
            {
                var typeInfo = _jsonTypeInfoResolver.GetTypeInfo(type, options);
                if (typeInfo != null && type == _rootObjectType)
                {
                    typeInfo.CreateObject = () =>
                    {
                        if (!_isRootResolved)
                        {
                            _isRootResolved = true;
                            return _rootObject;
                        }
                        else
                        {
                            return Activator.CreateInstance(type)!;
                        }
                    };
                }

                return typeInfo;
            }
        }

        public static void PopulateObject(string json, object obj, JsonSerializerOptions? options = null)
        {
            var modifiedOptions = options != null ?
                new JsonSerializerOptions(options)
                {
                    TypeInfoResolver = new PopulateTypeInfoResolver(obj, options.TypeInfoResolver ?? new DefaultJsonTypeInfoResolver()),
                } :
                new JsonSerializerOptions
                {
                    TypeInfoResolver = new PopulateTypeInfoResolver(obj, new DefaultJsonTypeInfoResolver()),
                };

            JsonSerializer.Deserialize(json, obj.GetType(), modifiedOptions);
        }
    }

Tests:

    public class JsonUtilitiesTests
    {
        [Fact]
        public void SystemTextJson()
        {
            var options = new JsonSerializerOptions
            {
                PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
            };

            var person = new Person
            {
                FirstName = "Abc",
                Child = new Person
                {
                    FirstName = "Foo"
                },
                Children =
                {
                    new Person { FirstName = "Xyz" }
                }
            };

            var json = @"{ ""lastName"": ""Def"", ""child"": { ""lastName"": ""Bar"" }, ""children"": [ { ""lastName"": ""mno"" } ] }";
            JsonUtilities.PopulateObject(json, person, options);

            Assert.Equal("Abc", person.FirstName);
            Assert.Equal("Def", person.LastName);

            Assert.Null(person.Child.FirstName);
            Assert.Equal("Bar", person.Child.LastName);

            Assert.Single(person.Children);
            Assert.Null(person.Children.First().FirstName);
            Assert.Equal("mno", person.Children.First().LastName);
        }
    }

    public class Person
    {
        public string? FirstName { get; set; }

        public string? LastName { get; set; }

        public Person? Child { get; set; }

        public List<Person> Children { get; set; } = new List<Person>();
    }

Personally this is what I'd expect from PopulateObject(), if existing children should also be updated then I'd call this method MergeInto() - however I dont know how you'd do that with .NET 7 STJ...

[eiriktsarpalis]

@RicoSuter that's a cool idea, however one issue I'm seeing is that you're creating a fresh JsonSerializerOptions instance on every serialization. This would result in metadata needing to be recomputed every time, killing serialization performance. I've adapted your example somewhat so that the same metadata can be used across serializations:

using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Text.Json;
using System.Text.Json.Serialization.Metadata;

var value = new MyPoco();
JsonSerializerExt.PopulateObject("""{"Value":42}""", typeof(MyPoco), value);
Console.Write(value.Value); // 42

public class MyPoco
{
    public int Value { get; set; }
}

public static class JsonSerializerExt
{
    // Dynamically attach a JsonSerializerOptions copy that is configured using PopulateTypeInfoResolver
    private readonly static ConditionalWeakTable<JsonSerializerOptions, JsonSerializerOptions> s_populateMap = new();

    public static void PopulateObject(string json, Type returnType, object destination, JsonSerializerOptions? options = null)
    {
        options = GetOptionsWithPopulateResolver(options);
        PopulateTypeInfoResolver.t_populateObject = destination;
        try
        {
            object? result = JsonSerializer.Deserialize(json, returnType, options);
            Debug.Assert(ReferenceEquals(result, destination));
        }
        finally
        {
            PopulateTypeInfoResolver.t_populateObject = null;
        }
    }

    private static JsonSerializerOptions GetOptionsWithPopulateResolver(JsonSerializerOptions? options)
    {
        options ??= JsonSerializerOptions.Default;

        if (!s_populateMap.TryGetValue(options, out JsonSerializerOptions? populateResolverOptions))
        {
            JsonSerializer.Serialize(value: 0, options); // Force a serialization to mark options as read-only
            Debug.Assert(options.TypeInfoResolver != null);

            populateResolverOptions = new JsonSerializerOptions(options)
            {
                TypeInfoResolver = new PopulateTypeInfoResolver(options.TypeInfoResolver)
            };

            s_populateMap.TryAdd(options, populateResolverOptions);
        }

        Debug.Assert(options.TypeInfoResolver is PopulateTypeInfoResolver);
        return populateResolverOptions;
    }

    private class PopulateTypeInfoResolver : IJsonTypeInfoResolver
    {
        private readonly IJsonTypeInfoResolver _jsonTypeInfoResolver;
        [ThreadStatic]
        internal static object? t_populateObject;

        public PopulateTypeInfoResolver(IJsonTypeInfoResolver jsonTypeInfoResolver)
        {
            _jsonTypeInfoResolver = jsonTypeInfoResolver;
        }

        public JsonTypeInfo? GetTypeInfo(Type type, JsonSerializerOptions options)
        {
            var typeInfo = _jsonTypeInfoResolver.GetTypeInfo(type, options);
            if (typeInfo != null && typeInfo.Kind != JsonTypeInfoKind.None)
            {
                Func<object>? defaultCreateObjectDelegate = typeInfo.CreateObject;
                typeInfo.CreateObject = () =>
                {
                    object? result = t_populateObject;
                    if (result != null)
                    {
                        // clean up to prevent reuse in recursive scenaria
                        t_populateObject = null;
                    }
                    else
                    {
                        // fall back to the default delegate
                        result = defaultCreateObjectDelegate?.Invoke();
                    }

                    return result!;
                };
            }

            return typeInfo;
        }
    }
}

[maxreb]

Very nice! I created a Benchmark for this to compare it with Newtonsoft. Its faster and uses far less memory. I also created a generic method PopulateObject<T> to have exact the same overload as Newtonsoft has (and its even a little bit faster).

The Benchmark Results:

Method	Mean	Error	StdDev	Gen0	Allocated
PopulateClass_FromInt_WithNewtonsoft	458.7 ns	9.16 ns	16.52 ns	0.3152	2640 B
PopulateClass_FromInt_WithSystemTextJson	318.4 ns	1.58 ns	1.40 ns	0.0219	184 B
PopulateClass_FromInt_WithSystemTextJson_AsObject	326.4 ns	1.03 ns	0.91 ns	0.0219	184 B

The code gist: https://gist.github.com/maxreb/947dd57b159d82aa75ac6943d66679e5

[eiriktsarpalis]

Nice! I should point out that this technique would only work with the synchronous serialization methods, thread statics stop working when async methods are being used and you might need to use something like AsyncLocal<T> instead.

[RicoSuter]

My main gripe with this is that STJ should expose this natively - it must have a internal populate method where they actually populate the object created with CreateObject, no?

[eiriktsarpalis]

There is work happening in this space currently tracked by #78556. I think it might make sense to close this issue in favor of the other one since its proposal already includes Populate methods.