revisit: Support JSON serialisation of BigInt values

[kaizhu256]

#24 mainly discussed literal JSON-serialization (and how it would break JSON-spec).

what about serializing to string-from, like Date? yes its one-way, but it provides needed guidance, and i'm skeptical we'll come up with a better solution 5 years from now.

JSON.stringify({aa:1234n}) // '{"aa":"1234"}'
// or
JSON.stringify({aa:1234n}) // '{"aa":"1234n"}'

[littledan]

This proposal exposed a hook specifically for this purpose: BigInt.prototype.toJSON. Just set that to BigInt.prototype.toString to turn on this behavior.

I think this option should be used with caution since it doesn't round-trip: JSON.parse of the serialized form will give you a Number. It's an important goal of this proposal to not implicitly round BigInt to Number, but permit explicit conversion.

When you want to include BigInt in JSON, I would recommend using a library to do the serialization and deserialization, such as granola.

[kaizhu256]

no, JSON.parse would just return a string (copying behavior of Date):

JSON.parse(JSON.stringify(new Date())) // '2018-07-28T09:41:47.519Z'
JSON.parse(JSON.stringify(1234n)) // '1234' or '1234n'

this is good-enough guidance imo (and let user figure out how to revive BigInt, just like Date). i don't see a simpler, easier-to-use solution than Date-like behavior in the forseeable future.

i just played with granola. seems to have bugs stringifying Set and Map. and not sure how to run it in the browser (where i can test BigInt).

[kaizhu256]

oh, figured out problem with granola (top-level must be object). still, i think having guidance to copy Date-like JSON-serialization behavior for BigInt makes better sense.

[littledan]

JSON.parse(JSON.stringify(1234n)) // '1234' or '1234n'

Not sure what you mean by this. Under what conditions would the answer be 1234n?

It seems like we're coming down to a difference in goals: I have been trying to avoid implicit precision-loss, while permitting JSON serialization, and it sounds like you're OK with some implicit conversions but are more bothered by the JSON serialization behavior not being present by default. Is that accurate?

[kaizhu256]

Not sure what you mean by this. Under what conditions would the answer be 1234n?

i'll remove the confusing n-suffix proposal (don't care too much about it). maybe this will clarify?

var aa;

// we copy JSON-behavior of Date
aa = 12345678901234567890n // <bigint primitive>
aa = JSON.stringify(aa) // '"12345678901234567890"'
aa = JSON.parse(aa) // '12345678901234567890'
aa = BigInt(aa) // <bigint primitive>

aa = new Date() // <Date object>
aa = JSON.stringify(aa) // '"2018-07-28T09:41:47.519Z"'
aa = JSON.parse(aa) // '2018-07-28T09:41:47.519Z'
aa = new Date(aa) // <Date object>

[littledan]

OK, I see what you mean. I still would prefer not to go with those semantics by default, for the reason explained in #162 (comment) : Date will "round-trip" through that process accurately, whereas BigInt will be rounded, based on the implicit conversion to a number and back.

[kaizhu256]

i'm confused now by what you mean by rounded? there's no implicit conversion to number (or implicit precision-loss) for JSON.parse. it remains a string (with full-precision). and JSON-spec remains unchanged.

[kaizhu256]

just to clarify further:

var aa;

// we copy JSON-behavior of Date
aa = 12345678901234567890n // <bigint primitive>
aa = JSON.stringify(aa) // '"12345678901234567890"' (escaped string)
aa = JSON.parse(aa) // '12345678901234567890' (un-escaped string)
aa = BigInt(aa) // <bigint primitive> (no precision-loss)

aa = new Date() // <Date object>
aa = JSON.stringify(aa) // '"2018-07-28T09:41:47.519Z"'' (escaped string)
aa = JSON.parse(aa) // '2018-07-28T09:41:47.519Z' (un-escaped string)
aa = new Date(aa) // <Date object>

[littledan]

Ah, I see, you are proposing that it be a string that includes the quotes. Interesting idea. How about we consider this as a follow-on proposal, and experiment in userspace for now by overriding BigInt.prototype.toJSON?

[kaizhu256]

no. i dislike the current behavior (from web-integration perspective), and believe it should be a last-resort option, only if no palatable default-serialization solution is found. and i think Date's JSON-behavior is a palatable solution that is least offensive to everyone, unless someone can think of serious technical-faults for it.

people are already familiar with Date's JSON-behavior, so it won't surprise anyone (or at least less-surprising than throwing an error).

i agree that for the forseeable future, ecma-404 and JSON.parse should not change for stability-reasons. with that constraint, this is the best possible outcome i can think of for both BigInt and possible future primitives like BigDecimal.

[cyberphone]

To put it differently:
Would anybody (freed from legacy etc) seriously design a typed general purpose format for data interchange utilizing a single type for everything from a byte to BigNumber? I guess not, which is why using the JSON Number type outside of its already firmly established (JavaScript) scope seems like an unnecessary difficult path although the JSON specification admits it.

There is also a bunch of IETF standards using JSON structures like JOSE. As far as I know, none of them are incompatible with JSON.parse() or JSON.stringify().

[tjcrowder]

I think better to get this in and standarized now, rather than as a follow-on proposal. But not with a raw number string, with a number string with n suffix. Quoting myself from the es-discuss thread.

I agree with you and Anders that this should be sorted out now, not as a
follow-on proposal. If it's left to later, people will supply their own
BigInt.prototype.toJSON and cause themselves compatibility problems
down-the-line.

Don't like the raw number in quotes ("conservative" option) at all. There
should be some indication that this is a bigint, just as pattern matching
tells us the default Date serialization is a date. This could be achieved
with just the lower-case n at the end, as in some early examples in the
github issue. And to support easy two-way, BigInt(string) should support
the lower-case n at the end.

aa = 12345678901234567890n; // BigInt primitive
aa = JSON.stringify(aa);    // '"12345678901234567890n"'
aa = JSON.parse(aa);        // '12345678901234567890n'
aa = BigInt(aa);            // BigInt primitive

^\d+n$ isn't much of a pattern, but it's a pattern. (Earlier in this
thread I suggested /BitInt(123)/ to match Microsoft's /Date(123)/ which
is unambiguous and offers a path to extendibility, but Date already goes a
different way...)

This parallels Date handling and seems a reasonable stop-gap until the next
thing after JSON.

I don't like to say things like "X should" without offering to help where possible, so if I can help, ping me to tell me how.

[cyberphone]

@tjcrowder In this particular case tc39 is faced with a bunch of already established solutions so the choice(s) boils down to "best practices" rather than innovation or standardization.

[kaizhu256]

i also originally wanted @tjcrowder's slightly-modified form, but there was pushback from the champion on issue 160.

[kaizhu256]

i would be happy with either - anything is better than the current awful-behavior

[cyberphone]

If we stick to compatibility with existing practices (why shouldn't we?), a JSONNumber wrapper object as suggested by Michael Theriot on the es-discuss@mozilla.org list could be a way to reduce pushback to a minimum.

That is, the default/standard BigInt serialization method could be the "" model like for Date, while those who insist that JSON needs to be used to its fullest, can get their thing as well. The latter requires some changes to the ES6 JSON object but they seem fairly reasonable.

[cyberphone]

Replaced by: https://github.com/cyberphone/es6-bigint-json-support#summary-of-changes

Something along these lines could probably work:

Serializing:

// Standard mode
JSON.stringify({big: 555555555555555555555555555555n}); 

Expected result: '{"big":"555555555555555555555555555555"}'
 
 

// RFC mode
// Note: this could be achieved by an option forcing mandatory JSON Number notation
JSON.stringify({big: 555555555555555555555555555555n}, 
  (k, v) => typeof v === 'bigint' ? JSONNumber(v) : v
);

Expected result: '{"big":555555555555555555555555555555}'
 

Parsing:

// Standard mode
JSON.parse('{"big": "555555555555555555555555555555"}',
  (k, v) => k == 'big' ? BigInt(v) : v  
);

Expected result: {"big":555555555555555555555555555555n}
 
 

// RFC mode
// Requires a new method (or option) which returns all numbers as JSONNumber
JSON.parse2('{"big": 555555555555555555555555555555}',
  (k, v) => typeof v === 'jsonnumber' ? k == 'big' ? v.getBigInt() : v.getNumber() : v
);

Expected result: {"big":555555555555555555555555555555n}
 
 

The JSONNumber object would only hold a string in proper JSON Number notation.

[robjtede]

Looks like the conversation is sort of moving here so I'd like to chip in my latest reply in the es-discuss thread. https://esdiscuss.org/topic/json-support-for-bigint-in-chrome-v8#content-44

+1 on getting this sorted before stage 4

As people have said before, JSON already supports BigInts. We of course need to preserve backwards compatibility with JS and try not to break how other languages use JSON.

I think the best way to satisfy everyone is to give ourselves some flexibility going forwards and JSONifying BigInts as strings doesn’t allow JSON to do what it can do. That and it breaks the idea that JSON.parse(JSON.stringify(x)) is a deep clone.

So, flexibility: there are two options in my head for this.

1. An object passed to JSON.parse with options for how to deal with integers larger than Number can support. The current behaviour of misrepresenting them is the default. But other options could include converting to BigInts (or strings) if needed and always converting to BigInts (since they are non-interoperable currently). This approach allows the programmer to decide the behavior and plan for it in their code.

2. It seems like the already suggested JSONNumber object could offer the same flexibility and guarantees though. A valueOf that returns the misrepresented Number for interop with existing numbers. A toString that gives a well represented string conversion. (Im not 100% certain of the order these two functions are used in auto-casting. On mobile or else would check spec.) A toBigInt / unwrap (monad-ish terminology) to get the true value as and when the programmer wants.

Hope these ideas seem reasonable. Scrutiny welcome.

[MichaelTheriot]

I'll continue to push for a more compliant JSON.parse alternative being introduced (perhaps its own proposal?). Parsing a JSON number as a JS number has always been a convenient lie and now we are paying the toll. JSONNumber would also pave the way for parsing BigDecimal as well (userland or not). Anyone currently dealing with big numbers serialized in environments with arbitrary precision (Python 3) and without the flexibility to receive them as strings has ditched JSON.parse for a smarter userland solution.

As for serialization, BigInt should serialize just how a JSON number would (at least in a strict RFC mode). The fact that JSON.parse does not support it should be irrelevant, and a bigger sign JSONNumber parsing is needed. Not serializing at all is just avoiding the issue which is still present.

[cyberphone]

Parsing a JSON number as a JS number has always been a convenient lie and now we are paying the toll.

That's a bit of a simplification. There is no other typed information exchange format that uses a single notation for everything that smells like a number, be it a bit or BigDecimal. That's simply put a pretty broken idea. By constraining JSON Number = JS Number = IEEE Double Precision it works quite satisfactory anyway since it effectively boils down to a single data type.

There are (AFAIK) no IETF standards using JSON structures exploiting JSON Number outside of JS Number.

JSON.parse(JSON.stringify(x)) does not perform deep cloning for widely used data types like Date and "blobs" either.

I suggested introducing a JSON.parse2() for dealing with JSON in the way it was "intended", making the default behavior the only true issue.

[robjtede]

I should clarify, JSON.parse(JSON.stringify(x)) performs a deep clone for primitive types (including plain objects and arrays). Of course Date is more complicated than that. BigInt should not be treated in the same way as Date. JSON does not support Date objects; it does support BigInts though.

I don't think having both JSON.parse and JSON.parse2 is a good idea. An options object is the JS way for when different behavior is wanted (see something like the Intl API for example.) or else we could end up with JSON.parse3 for other data types (eg. BigDecimal) down the road.

[cyberphone]

The JSON.parse2() proposal is compatible with any other JSON Number addition, but an optional boolean to JSON.parse() can surely do the same job.

I believe developers will have to explicitly deal with non JS Numbers no matter what scheme we come up with. My "problem" with strict JSON is that on top of that, you introduce backward incompatibility with at least JS and with pretty limited gain. Neither Date or "blobs" seem to have been particularly hampered by not having a direct JSON counterpart.

There are some "unwanted side effects" as well: Oracle's JSON support in Java depends on BigDecimal as the core number type in order to deal with dynamic parsing. That doesn't sound like a great solution from a performance point of view. My RFC mode parse proposal doesn't actually convert JSON Number data at all; it puts that burden on the developer. How would your scheme work?

[cyberphone]

Until proven wrong (I'm not infallible 😏), I believe the only point of disagreement is what the default serialization of a BigInt should be, the rest are details of less importance.

[robjtede]

For stringifying:

const numbers = {
  a: 2,
  b: 40000000000000000000000n
}

// returns -> '{ "a": 2, "b": 40000000000000000000000 }'
// NOTE: no "n" suffix
JSON.stringify(numbers, null, null, {
  bigint: 'raw' // default setting
})

// returns -> '{ "a": 2, "b": '40000000000000000000000' }'
// NOTE: no "n" suffix
JSON.stringify(numbers, null, null, {
  bigint: 'string'
})

For parsing:

const numbers = '{ "a": 2, "b": 40000000000000000000000 }'

// returns -> {a: 2, b: 4e+22}
JSON.parse(numbers, null, {
  bigint: 'never' // default setting
})

// returns -> { a: 2, b: 40000000000000000000000n }
JSON.parse(numbers, null, {
  bigint: 'whenNeeded'
})

// returns -> { a: 2n, b: 40000000000000000000000n }
JSON.parse(numbers, null, {
  bigint: 'always'
})

The extra nulls are due to existing optional params I only just found out about.
This kinda makes me prefer the JSONNumber idea but it's not too unsightly.

[robjtede]

Granted this does mean the deep clone idea needs to be changed to:

JSON.parse(JSON.stringify(x), null, { bigint: 'whenNeeded' })

Another reason to support JSONNumber, for me.

[cyberphone]

Does 'whenNeeded' mean that JSON.parse() would return different JS data types depending on numeric value?

I wouldn't take this path. 'always' seems quite awkward as well, why would anybody want all numbers to be BigInt?

[robjtede]

@cyberphone basically, yes. It is a flag for the programmer that any numbers they expect could be large (think twitter IDs) might need a typeof x.id check.

I can't think of many use cases for 'always' either, it's just an option for cross property compatibility. Eg. (x.small + x.large) being guaranteed to be correct and not throw.

[cyberphone]

@robjtede There seems to be multiple issues here. I don't see how you can use a future a BigNumber type at the same time as BigInt since JSON (unlike every other typed data interchange format in existence...) doesn't tell you what's behind a number.

Therefore I stick to my assertion that JSONNumber is the "missing link" in order to support arbitrary JSON Numbers. Then people can certainly argue to death which mode that should be the default although that's not really paramount for success.

[kaizhu256]

@robjtede your suggestion for modifying JSON.parse is too open-ended and high-maintennance. nobody benefits from the tooling-instability caused by a JSON.parse thats under constant maintennance and revisions by tc39 (which your suggestion would open the floodgates to).

[robjtede]

@kaizhu256 Which part is open ended?

Whichever way it works, I'm highly in favor of a literal string-ification (no quotes or "n" suffix) by default with an option to have quotes.

I believe that it should be as transparent as possible to start serializing and parsing BigInts in JS but flexible enough to fit most needs without having to resort to userland libraries. This is a primitive after all, not a class (like Date).

[MichaelTheriot]

In 4/5 it should correctly serialize to a valid JSON number (not a string). I also agree it should be default rather than throwing, but ultimately some native solution should exist.

[cyberphone]

I strongly support keeping the default behavior (keeping the exception) which is what @bakkot suggested

[cyberphone]

Since 2 already is fully supported "as is" (including all known and unknown variations), it doesn't appear to qualify as an action item anymore

Since 3 is currently not known to be used anybody anywhere it can also safely be removed

What's left is support for RFC compatible serialization and parsing

[cyberphone]

Pardon for bringing up trivial stuff that probably already is somewhere in the documentation.

Anyway, I tried creating a write-up for using quoted Base64Url-encoded BigInt serialization (like used by some IETF standards), but couldn't find any BigInt method for getting the bytes.

There is apparently a NumberToRawBytes but where?

[bakkot]

@kaizhu256, I am reasonably confident this is the best solution, yes. I do not agree that silently doing something unexpected would be easier to debug than loudly throwing an error.

@cyberphone, NumberToRawBytes is an abstract operation which isn't directly exposed anywhere. If you look through the spec, you see it's involved in writing BigInts to TypedArrays. You can maybe build something on top of that, but it's going to be easier just to pass a radix to toString: bigInt.toString(16) gives you the number in hexadecimal, which you can then easily parse to a byte array (or, for your application, just directly reencode).

[cyberphone]

@bakkot Thanx! Since Base64/Base64Url is a very common format, would it be too much asking for a BigInt counterpart to https://docs.oracle.com/javase/8/docs/api/java/math/BigInteger.html#toByteArray-- and then of course something in the opposite direction as well?

[cyberphone]

Patching already posted issue comments began to look awkward so I created a separate proposal:
https://github.com/cyberphone/es6-bigint-json-support#summary-of-changes
Enjoy!

[d3x0r]

Minor note; unless the recipient can deal with bigints, it's not going to be able to run a script expecting bigints. If a script can expect to receive bigints, the the related JSON stringify can just generate bigints.

Nothing now (before now) can run scripts the have bigints to require operating with bigints, so it shouldn't be receiving any JSON content that has bigints.

if ( !bigint ) receive (bigint = fail) ; this can easily just be the parsing exception on the 1234n if just written as a number.
if( bigint ) receive ( bigint (decoded however, but if just fit as a number) ) ; wouldn't fail.
if( bigint ) receive ( non bigint encoded number ) then it's a compatibility program, and would probably expect to have custom code to translate.

if( bigint ) and not using bigint, JSON.stinrigfy that outputs like 1234n would never generate incompatible output.

if( !bigint ) and not using bigint , JSON.parse would never expect to receive bad numbers.

[kaizhu256]

Plenty of frontend engineers, myself included, consider correctness to be paramount, and are not ok with incorrectness for any reason.

again, fundamental philosophical disagreement. javascript is not a language that prioritizes on writing correct-programs. no successful web-product, including facebook, amazon, etc would have shipped if their creators had focused on javascript-correctness ... they would have never launched. pretty much all successful javascript-products start off as steaming-piles of c***, that had to cut many painful corners to ship. their real-world focus has never been correctness, but trade-offs for speed-of-delivery and "good-enough" UX to execute their business model.

and that is javascript's value to industy: a simple, high-productivity glue-language where one can quickly create good-enough product-demos to pitch to the market and fail-fast and re-iterate until something sticks. why waste 5x time-and-effort trying to architect "correct" javascript-programs, when you know very well its more than likely going to fail in the market, ending with you having to rewrite everything again?

most real-world javascript projects are not high-quality projects. they are throwaway demos/mvps. and if javascript-developers primarily write disposable web-projects (statistically more likely to fail than succeed), how can they justify the cost of complicated, extra tooling/libraries to get JSON-support for bigint, when a default, zero-tooling quoted-string default-behavior is good-enough?

[cyberphone]

@kaizhu256 AFAICT, the core of what has been requested is support for JSON Number formatting for BigInt. Several people including myself (https://github.com/cyberphone/es6-bigint-json-support#21-the-jsonnumber-primitive) have suggested introducing a new datatype to cope with this requirement.

The default mode of JSON.stringify() is somewhat less significant, particularly given that no solution (except "breaking JSON" through the n-notation), will anyway provide functioning roundtrip support.

[littledan]

OK, it sounds like this thread has converged on the conclusion that the current JSON behavior is the right default, and we should consider possible follow-on proposals to make it easier to convert BigInts to JSON. Is this an accurate summary?

[cyberphone]

Is this an accurate summary?

Probably, although I can only speak for myself. Slight clarification: Follow-on proposals should deal with both JSON.stringify() and JSON.parse() for the RFC mode (raw number). JSON.parse() is the tricky one.

Using the "quoted" mode (IMO) works sufficiently OK right out of the box using the toJSON scheme. Well, using the quite popular Base64Url notation is somewhat awkward but that is another issue and thread.

[kaizhu256]

i still strongly prefer option 2, quoted-string-mode (without n-suffix) as default. javascript is not a low-level general-purpose language like java/c++, and most web-developer are not expected/prioritized to waste their time, writing the kinds of low-level code expected from java/c++ developers.

having tc39 force js-developers to implement their own JSON-solution for native primitives like BigInt, is the kind of low-level nuisance/tech-debt most web-developers would prefer not having to deal with, when they are already overwhelmed with so many high-level (and more business-critical) UX-issues requiring their attention in order to get their products shipped and sold, over their competitors.

[cyberphone]

@kaizhu256 Just for my poor understanding: If the idea is that BigInt should behave like the core JS primitives Number, String, null, true and false, i.e. "automatic", the (AFAICT...) only working solution is bringing the JS-specific BigInt literal scheme to JSON. Or is your only concern Serialization and rather leave Deserialization to a future standardization effort?

[kaizhu256]

i believe we should resolve the Serialization matter now, because i'm skeptical tc39 will make good on its promise of a future standardization thats better/easier-to-use than whats currently available. 2-5 years from now, quoted-string-serialization will likely still be the least controversial/risky option for web-developers who want something both reliable and doesn't slowdown product-development with undue integration/tooling issues. its less harmful to industry to bless it now, then to have web-developers constantly bicker over custom toJSON implementations (especially when integrating products with 3rd parties).

Deserialization could be left to a future standard, but again, i'm skeptical tc39 could deliver something actually user-friendly for web-developers. but thats ok, me and most ppl will likely be happy just using Date's well-known deserialization model (an extra-step constructor-call, requiring additional, contextual knowledge of serialized data).

[d3x0r]

I decided to leave JSON alone since it is declared immutable. The RFC under parsers says A JSON parser MAY accept non-JSON forms or extensions. Although there is no such note for Generators.
I decided to fork JSON6 and create JSOX. This is a parser that accepts all valid JSON, plus encodes Dates as a subtype of Number, BigInt with an 'n' suffix, TypedArray and ArrayBuffers as typed-arrays containing a base64-encoding of the byte data in the backing ArrayBuffer. It keeps the same simple API (JSOX.parse( JSOX.stringify( object [, replacer [, spaces]] ) [, replacer] ).

It also adds a feature to define typed-objects, that is define the object and its fields, so later, that object
type can be referenced, and just the values encoded later. Revived objects that are typed all share the same prototype, and an API extension allows pre-definition of prototypes to use for typed-objects on a parser object.

It also adds an internal typed-array called 'ref' which defines the path to some previously existing object or array. The path consists of object property names and/or array indexes. This allows encoding cyclic data structures.

Usage of either of the typed features generates output which is beyond the syntax handling of JSON. (as does Date and BigInt encoding, or any of the extended keywords). The typed object/array syntax also is not JS that can be interpreted with eval(), same with encoded Date()s; otherwise the input to JSOX parser may be valid JS (minus expressions).

Github Repo ( RFC-like documentation )
NPM

---

So for compatibility with legacy JSON, BigInt serialization should probably just be a string, revived similarly to dates.

[littledan]

I think @kaizhu256 is right that there's a good chance that we won't actually follow up in TC39 and make a built-in solution like JSONNumber. I'd like to spend some more time experimenting with solutions like @d3x0r 's JSOX and @jseijas 's vilanova for now, and we can see over time whether there is something we should standardize.

At this point, we've looked into this topic thoroughly and have reaffirmed the current proposal's semantics, so closing this issue.

[cyberphone]

@bakkot @littledan @kaizhu256 @ljharb The comparison with Date is a bit flawed since Date doesn't have a specific JSON representation. BigInt, OTOH is just a number having full JSON RFC support which some pretty popular JSON implementations including Microsoft's .NET and Jackson use as their default for BigInt. Fortunately, these systems permit adapting the serializer/deserializer to compensate for the ES6/JSON object's crippled functionality.

If you are considering other JSON solutions, many developers have found schema-driven designs like https://github.com/OAI (aka "Swagger") useful since they address messaging in a structured way while removíng the need for dealing with low-level operations of the kind discussed in this thread. These solutions use JSON "as is" and meet stringent requirements for passing data between applications.

[MicahZoltu]

it's going to be easier just to pass a radix to toString: bigInt.toString(16) gives you the number in hexadecimal, which you can then easily parse to a byte array (or, for your application, just directly reencode).

@bakkot This isn't correct as it doesn't work with negative numbers. BigInt(-32).toString(16) results in -20, which is not a value that can be serialized as-is as a number to a byte stream. In order to serialize this, you need to strip off the leading - and then convert the 0x20 to negative 0x20 using twos complement which requires knowing the target number of bits in order to correctly serialize.

At the moment, this spec does not provide any mechanism for serializing negative values to a byte stream.