Add snake_case support for System.Text.Json

[hez2010]

Contents below were taken from @khellang, thanks!

API Proposal

Add additional casing support for System.Text.Json, based on this comment.

namespace System.Text.Json;

public partial class JsonNamingPolicy
{
    public static JsonNamingPolicy CamelCase { get; }
+    public static JsonNamingPolicy SnakeLowerCase { get; }
+    public static JsonNamingPolicy SnakeUpperCase { get; }
+    public static JsonNamingPolicy KebabLowerCase { get; }
+    public static JsonNamingPolicy KebabUpperCase { get; }
}

public enum JsonKnownNamingPolicy
{
    Unspecified = 0,
    CamelCase = 1,
+    SnakeLowerCase = 2,
+    SnakeUpperCase = 3,
+    KebabLowerCase = 4,
+    KebabUpperCase = 5,
}

Behavior Proposal

I propose the same behavior as Newtonsoft.Json, just like the existing camel case behavior. The implementation is here and the tests are here.

Other Comments

I think snake case naming is pretty common, especially in the Ruby world, and should be supported out of the box. GitHub's API is probably the most popular example using this naming convention off the top of my head.

Implements

See: dotnet/corefx#40003

[Gnbrkm41]

If you have any designs in mind, could you update the top post with what the proposed APIs would look like / how commonly used snake case is / etc? Here's what I think is a good example: https://github.com/dotnet/corefx/issues/28933#issue-312415055. This should help people better understand the problem/importance of this issue, and may come up with improvements.

[khellang]

API Proposal

namespace System.Text.Json
{
    public abstract class JsonNamingPolicy
    {
        protected JsonNamingPolicy() { }
        public static JsonNamingPolicy CamelCase { get { throw null; } }
+       public static JsonNamingPolicy SnakeCase { get { throw null; } }
        internal static JsonNamingPolicy Default { get { throw null; } }
        public abstract string ConvertName(string name);
    }

+   internal sealed class JsonSnakeCaseNamingPolicy : JsonNamingPolicy
+   {
+       public override string ConvertName(string name) { throw null; }
+   }
}

Behavior Proposal

I propose the same behavior as Newtonsoft.Json, just like the existing camel case behavior. The implementation is here and the tests are here.

Other Comments

I think snake case naming is pretty common, especially in the Ruby world, and should be supported out of the box. GitHub's API is probably the most popular example using this naming convention off the top of my head.

UPDATE: Looks like dotnet/corefx#40003 implemented more or less this exact proposal.

[hez2010]

@khellang I've pasted your proposal contents to the top post, thanks!

[xsoheilalizadeh]

I've prepared a benchmark that compares other implementation with dotnet/corefx#40003 PR.

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.18362
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview8-013656
  [Host]     : .NET Core 3.0.0-preview8-28405-07 (CoreCLR 4.700.19.37902, CoreFX 4.700.19.40503), 64bit RyuJIT
  DefaultJob : .NET Core 3.0.0-preview8-28405-07 (CoreCLR 4.700.19.37902, CoreFX 4.700.19.40503), 64bit RyuJIT

|                          Method |        Mean |      Error |     StdDev | Rank |  Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------------------------- |------------:|-----------:|-----------:|-----:|-------:|------:|------:|----------:|
|  ToSnakeCaseStringBuilderBySpan |    85.97 ns |  1.0500 ns |  0.9821 ns |    1 | 0.0637 |     - |     - |     200 B |
| ToSnakeCaseNewtonsoftJsonBySpan |    86.35 ns |  1.7816 ns |  1.4877 ns |    1 | 0.0484 |     - |     - |     152 B |
|       ToSnakeCaseNewtonsoftJson |    88.89 ns |  0.8417 ns |  0.7461 ns |    2 | 0.0484 |     - |     - |     152 B |
|               ToSnakeCaseBySpan |   122.01 ns |  1.9066 ns |  1.6902 ns |    3 | 0.0253 |     - |     - |      80 B |
|                 ToSnakeCaseLinq |   365.75 ns |  4.4839 ns |  3.5008 ns |    4 | 0.1450 |     - |     - |     456 B |
|                ToSnakeCaseRegex | 2,108.71 ns | 40.6829 ns | 31.7625 ns |    5 | 

[xsoheilalizadeh]

I investigated a couple of weeks about this case and tried other approaches + dotnet/corefx#40003 that have better performance to having snake case naming policy. Finally, I got the best result and it have less allocation and fast execution. My last implementation is ToSnakeCaseBySpan . Shoud I open a new PR for this?

|                          Method |        Mean |      Error |     StdDev | Rank |  Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------------------------- |------------:|-----------:|-----------:|-----:|-------:|------:|------:|----------:|
|               ToSnakeCaseBySpan |    27.47 ns |  0.4629 ns |  0.4330 ns |    1 | 0.0153 |     - |     - |      48 B |
|  ToSnakeCaseStringBuilderBySpan |    85.23 ns |  1.6495 ns |  1.3774 ns |    2 | 0.0637 |     - |     - |     200 B |
|       ToSnakeCaseNewtonsoftJson |    85.72 ns |  1.6418 ns |  1.4554 ns |    2 | 0.0484 |     - |     - |     152 B |
| ToSnakeCaseNewtonsoftJsonBySpan |    86.96 ns |  1.7060 ns |  1.5958 ns |    2 | 0.0484 |     - |     - |     152 B |
|                 ToSnakeCaseLinq |   353.42 ns |  3.9670 ns |  3.7108 ns |    3 | 0.1450 |     - |     - |     456 B |
|                ToSnakeCaseRegex | 2,056.69 ns | 29.5694 ns | 26.2125 ns |    4 | 0.1526 |     - |     - |     496 B |

\cc @karelz

[hez2010]

@xsoheilalizadeh I think the name of a property is usually not too long, so maybe you can use stackalloc to reduce memory allocation on heap to reduce GC pressure? (It may achieve zero allocation)

public unsafe ReadOnlySpan<char> ToSnakeCaseBySpan()
{
    ...
    Span<char> buffer = stackalloc char[bufferSize];
    ...
}

[xsoheilalizadeh]

Nice suggestion, but here is a question how I should return buffer, I've got following:

Cannot use local 'buffer' in this context because it may expose referenced variables outside of their declaration scope

[Gnbrkm41]

Should I open a new PR for this?

Nope, not yet 😅 not until the API is approved.

[hez2010]

Because buffer now is allocated on the stack, you cannot return it directly since it will be disposed after return. To solve it, you can simply change `return buffer` to `return buffer.ToString()`; Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: Soheil Alizadeh <notifications@github.com> Sent: Friday, September 13, 2019 3:48:38 PM To: dotnet/corefx <corefx@noreply.github.com> Cc: Steve <hez2010@outlook.com>; Author <author@noreply.github.com> Subject: Re: [dotnet/corefx] Add UnderScoreCase support for System.Text.Json (#39564) Nice suggestion, but here is a question how I should return buffer, I've got following: Cannot use local 'buffer' in this context because it may expose referenced variables outside of their declaration scope — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcorefx%2Fissues%2F39564%3Femail_source%3Dnotifications%26email_token%3DADSENWIDJVGBMFT3T4V3U5LQJNAVNA5CNFSM4IEL7W4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6UIBGI%23issuecomment-531136665&data=02%7C01%7C%7C68574ff9a79b43fbaf2a08d7381ecaaf%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637039577204910801&sdata=PhAc8%2BvEwr9AHgXI1vNdlLENvuKw1jDC6x%2F%2FNuc7PDk%3D&reserved=0>, or mute the thread<https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADSENWIHQJ7DC64RZH2RQFDQJNAVNANCNFSM4IEL7W4A&data=02%7C01%7C%7C68574ff9a79b43fbaf2a08d7381ecaaf%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637039577204920805&sdata=M9kuqi6W4P3Msx76loMm8q2laC%2FrNgb4ZwIozAwcDZ4%3D&reserved=0>.

[karelz]

@ahsonkhan are we ready to review the API? @xsoheilalizadeh is eager to implement it ...

[xsoheilalizadeh]

I've figured out that the ToSnakeCaseBySpan can't pass these tests, it's just compatible with pascal case properties, I've tried to make it compatible with other formats but unfortunately, I've failed and it'd be unreadable. I was hurrying to contribute and got a bad result, sorry.

[ahsonkhan]

This looks like a reasonable API addition. I was considering how common such formats are to warrant adding support built-in (rather than folks creating their own naming policy to support it), and looks like quite a few places use snake_case including Facebook, AWS, and Twitter APIs (aside from the fact that Json.NET supports it):
https://stackoverflow.com/questions/5543490/json-naming-convention

And of course, what @khellang said:

I think snake case naming is pretty common, especially in the Ruby world, and should be supported out of the box. GitHub's API is probably the most popular example using this naming convention off the top of my head.

cc @steveharter

[terrajobst]

Video

Makes sense. We also considered others (such as kebap casing) but it seems we should defer that until someone actually needs that.

[Gnbrkm41]

@xsoheilalizadeh, are you still working on it? :D you should be able to open PRs for it.

[xsoheilalizadeh]

Unfortunately no till I pass tests, I tried to pass tests but my implementation only support pascal-case naming for example FirstName will be first_name but in other cases like IsCIA it will be Is_c_i_a that isn't expected.

[hez2010]

@Gnbrkm41 How about the existing PR dotnet/corefx#40003

[Gnbrkm41]

I mean... sure, why not, as long as it has all the tests and it passes :D

[YohDeadfall]

@ahsonkhan Did the same thing already, so taking this.

[VassilAtanasov]

Makes sense. We also considered others (such as kebap casing) but it seems we should defer that until someone actually needs that.

The lack of PascalCase option is a show stopper for our migration.
I believe there should be a "preserve case" option which will also work for us.

[khellang]

The lack of PascalCase option is a show stopper for our migration.
I believe there should be a "preserve case" option which will also work for us.

Isn't that the default? 🤔

https://github.com/dotnet/corefx/blob/296c0e71ddcc4885d30af18dea829c1923cfdc5c/src/System.Text.Json/src/System/Text/Json/Serialization/JsonDefaultNamingPolicy.cs#L9

[VassilAtanasov]

The lack of PascalCase option is a show stopper for our migration.
I believe there should be a "preserve case" option which will also work for us.

Isn't that the default? 🤔

https://github.com/dotnet/corefx/blob/296c0e71ddcc4885d30af18dea829c1923cfdc5c/src/System.Text.Json/src/System/Text/Json/Serialization/JsonDefaultNamingPolicy.cs#L9

Yes - that worked ... but not easily!
You can not select the Default !!!
JsonSerializerOptions.PropertyNamingPolicy = JsonNamingPolicy.Default
and also PropertyNamingPolicy gets initialised in constructor to JsonCamelCaseNamingPolicy
new JsonOptions().JsonSerializerOptions.PropertyNamingPolicy

However you can trick it with :-)
JsonSerializerOptions.PropertyNamingPolicy = null;
and PascalCase goes through unobstructed!

[Gnbrkm41]

the default is to use exact match IIRC. setting it to null also does that as well.

[niemyjski]

I'd prefer if we could just define our own naming strategies. We use our own naming method for snake_case because it occurred before Json.net came out with it and it's slightly different. So perhaps just allow us to set a naming policy or use the default one.

[khellang]

@niemyjski Of course you can. Just implement JsonNamingPolicy like the built-in ones and set that. Almost everything about the new serializer is configurable and/or extensible, these issues are usually just about including more stuff in the box.

[sveinfid]

Please could KebabCase also be added?

It should be a very minor addition - basically the same as SnakeCase but with - in place of _

E.g. requests sent to Apple Push Notification Service require kebab-case JSON fields: https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/generating_a_remote_notification#2943360

[ahsonkhan]

@sveinfid - There was precedence for snake casing (e.g. in newtonsoft json) and enough user need to provide built-in support for it. I am not sure that's the case for KebabCase. Please file a separate issue (with the API proposal) and we can consider it, but my hunch is, this is something that the caller would have to implement (like any other custom casing/naming strategy) and doesn't seem common enough to require built-in support.

Like you said, the impl could be relatively straight forward (take existing snake case impl and replace the delimiter), which means there is also little value in providing one built-in.

[layomia]

From @YohDeadfall in #673:

Originally proposed by @hez2010 in dotnet/corefx#39564.

@ahsonkhan, could you add the original issue here and important comments from dotnet/corefx#41354?

@karelz We need your opinion here, since @khellang asked a question about altering the existing behavior of the camel case policy too follow common translator rules for the JS world as splitting a string by words and concatenating it back using some separator and applying a policy to change casing for each word. It's a breaking change and @stephentoub likes it. Me too. See dotnet/corefx#41354 (comment).

I still would like to drive this feature.

@steveharter @layomia

[terrajobst]

@dotnet/area-system-text-json based on the comments from @teo-tsirpanis and the latest change request from @YohDeadfall I have pushed the API back to api-ready-for-review.

[teo-tsirpanis]

Any particular questions you're trying to answer?

My question was what was the approved API shape was Well, forget it, I found #782 (comment) (😳), but it might still be useful to re-review it for the additional casings.

JsonKnownNamingPolicy might also need to be updated.

[terrajobst]

Well, forget it, I found #782 (comment) (😳),

Ah yes, that's why didn't find the notes -- we approved it but didn't apply the label. Oops. But yeah, we'll take another quick look tomorrow.

[OnamChilwan]

Is there any update on this, when can we expect this to be added? I am targeting .NET 6 and can only see CamelCase as an option..

[YohDeadfall]

@OnamChilwan, you can currently use a package mentioned in #782 (comment). The pull request I made uses it's code with style changes only.

@terrajobst, till what date pull request can be merged and appear in .NET 7? If it's too close and no time left for an approval of public API changes we can go with option 2. It allows to trim the PR only to SnakeCase and add other options later, but with Shouting (less preferred naming in my opinion).

[J0rgeSerran0]

@OnamChilwan you can use this code too
I did 3 years ago, is in some threads about this issue, and I shared it for the people that is looking for a solution with this

https://github.com/J0rgeSerran0/JsonNamingPolicy

[wardboumans]

Incredible this is still not in .NET7. This is why people still stick to Newtonsoft.

[GiorgioG]

This is pathetic. I can't advocate for System.Text.Json anymore if it's going to remain a worse option than Newtonsoft.

[georgiosd]

Really @wardboumans @GiorgioG? Using a custom naming policy that already exists out there is that despicable that you'd drop the library? Wow.

[bartonjs]

Video

• We discussed some other options, like PascalCase (forcing a field or non-conforming name from camelCase to PascalCase)
• We felt that SnakeCaseUpper and friends were better than SnakeUpperCase and friends.

namespace System.Text.Json;

public partial class JsonNamingPolicy
{
    public static JsonNamingPolicy SnakeCaseLower { get; }
    public static JsonNamingPolicy SnakeCaseUpper { get; }
    public static JsonNamingPolicy KebabCaseLower { get; }
    public static JsonNamingPolicy KebabCaseUpper { get; }
}

public enum JsonKnownNamingPolicy
{
    SnakeCaseLower = 2,
    SnakeCaseUpper = 3,
    KebabCaseLower = 4,
    KebabCaseUpper = 5,
}

[YohDeadfall]

Cool, thanks! Then I will align the pull request according to it and address requests.

[GiorgioG]

@georgiosd Yes really. I can't tell you how many times I've advocated to use System.Text.Json and then have to defend these basic kind of omissions. JSON.NET has had this since at least 2016: https://github.com/JamesNK/Newtonsoft.Json/commits/master/Src/Newtonsoft.Json/Serialization/SnakeCaseNamingStrategy.cs

[georgiosd]

Well maybe they should not have optimized the framework and include a snake case naming strategy.

Really, if you're deciding on the basis of having to write a naming policy implementation (that is what, 5 lines?) and not the overall improvements perhaps you have bigger problems in the organization.

[J0rgeSerran0]

Totally agree with @georgiosd

I thought that my implementation would be up and running for 6 months (more or less)?
3 years later... I see the end of this discussion is near

Speaking of naming and in my case, the classes JsonKebabCaseNamingPolicy and JsonSnakeCaseNamingPolicy (KebabCasexxx) (SnakeCasexxx)

As you can see, what Georgios says is a fact