Should 'integer' type validate bigint ?

[MaximeGagnebin]

This is more of a question than a proposal.

What problem do you want to solve?
I want to validate objects containing a bigint.

What do you think is the correct solution to problem?
Bigint instance could pass the validation against 'integer' type. This may sound weird as "regular" integer would also pass this validation, and in this way the "type" isn't really validated. But the JSON specifications do not impose a limit on the range of integers.

The other solution would be to extend the types to include BigInt.
Thanks for your answer.

[epoberezkin]

it's ok with option "bigint"

[MaximeGagnebin]

Could you clarify ? option is not a keyword in ajv ? Currently there is no "type" that will validate against a BigInt.

[epoberezkin]

I mean that I am ok with the change to support it with a new option “bigint” that would treat them as “integer” from validation point of view. I recently did the similar change in json-source-maps to support bigint.

You can also use “typeof” custom keyword defined in ajv-keywords to specifically establish that the type (from JS point of view) is bigint rather than number.

Adding bigint to the list of types (from JSON Schema point of view) is definitely not ok - bigint type is purely JS thing and JSON Schema is supposed to be cross platform. I draw the line at allowing custom keywords - custom types are off limits :)

[epoberezkin]

Treating bigint as integer seems reasonable to me, as “integer” is not a type in JS and indeed JSON has no limit on integer size. But as this is a non-backwards compatible change it requires an option, it cannot be the default behaviour (although this can become default behaviour in the next major version)

[MaximeGagnebin]

Thanks for the detailled reply ! You make good points. Especially concerning custom types.

I think I will try to work with the "typeof" keyword for now.
I will also look at how much work would be needed to accept bigint as "integer". This would be nice as it would allow to make the schema less JS specific. In that regard, it is helpful that expressions like "3 > 2n" returns true in JS. This make validation for "maximum" (and similar) a bit nicer.

Anyways, thanks again for the reply! :)

[max-preuschen]

I'm fairly new to AJV, so I'm a bit stuck at the moment.
I have a given JSON-schema with

"type": "integer",
"format": "int64"

Currently the schema doesn't pass validation, so I want to add int64 as a custom format to the validator.

Unfortunately my tests are failing, since type: integer is converted into a JS-number instead of bigint.
I read the conversation above, but still don't quite understand how I can change the type to bigint instead of number.

Thanks for any help!

[epoberezkin]

You could use typeof: bigint (with Ajv-keywords). But format won’t be applied - Ajv currently does not see bigint as integer. The issue is about supporting it, but it is not implemented.

[max-preuschen]

Thanks for the clarification.
So, if format doesn't work when using typeof, is there any other workaround how one could validate a 64-bit integer?

[MaximeGagnebin]

You can always create a custom keyword. This is a keyword I used was called dataType with values uint64 and sint64. And falls back to trying to interpret the input as type otherwise (this is probably unnecessary).

I don't know if this is the recommanded way, but it worked for me :)

ajv.addKeyword('dataType', {
  validate: function (schema, data) {
    switch(schema) {
      case 'uint64':
        return typeof data === 'bigint' && data < 2n ** 64n && 0n <= data;
      
      case 'sint64':
        return typeof data === 'bigint' && data < 2n ** 63n && -(2n ** 63n)< data;

      default:
        return ajv.compile({"type" : schema})(data)

    } 
  },
    errors: true
});

[max-preuschen]

Thanks everybody for your help!
I may be able to work around that, if it's possible to change the schema and replace type / format with custom keyword typeof or dataType.

If changing schema is not an option: the only other workaround, I see at the moment: Adding format int64 to unknownFormats.

[zdm]

If type is set to bigint - value should be coerced to BigInt, if cortcion is enabled.
So, it would be nice to implement this things:

1. type "integer" should recognize BigInt javascript primitives;
2. new type: "bigint" - coerce data to BigInt, if validation passed and coercion is enabled. Of course, this is javascript-only specific, but users can decide, use it or not;

Of course we can achieve this using custom keywords, but having "bigint" type will make schemas more clear and simple.
And as I understand, it is not possible to implement coercion, using custom keyword, at least for case, when validated data is not a object, but raw string or number.

validate("string");

[epoberezkin]

@zdm I am ok with supporting “bigint” as an option that would make Ajv accept bigints as number/integer, apply all number-specific keywords to them, and also coerce long strings to bigint when they are “integer” but out of 32 bit integer range. It’s quite a substantial feature already - if you’d like to implement, happy to help.

“bigint” as a type I am not so happy about, “type” is not just a keyword, it’s quite hard-wired; also I’m not sure why would you require (rather than allow) bigints?

Also, requiring bigints can already be done with “typeof” keyword from ajv-keywords package.

[zdm]

For coercion - i think, if type is specified as "integer" or "number" - it shouldn't coerced to BigInt. Coercion should happen only when user explicitly specified, that he expected bigint. Because BigInt is not compatible with Number - we shouldn't mix them, If user waiting for integer - string coerced to Number, if bigint - to BigInt.

This can be solved elegantly by adding new "bigint" type, all other solutions will be ugly, i think.

[zdm]

So, ideally:

• integer, number - recognize and validate BigInt, but only if BigInt is not exceeded Number.MAX_SAFE_NUMBER, coerce BigInt to Number;
• new type "bigint" - coerce strings and numbers to BigInt, if data is valid;

[epoberezkin]

We can split these to two changes then and exclude coercion from the scope - only add option bigint: true to accept bigint and correctly apply number specific keywords to it.

I’m not happy with extending the list of types beyond what’s specified in JSON Schema (they are effectively JSON types), it may change in the future though.

[zdm]

How do you want to implement it? Add new keyword bigint, which can be used together with the integer or number types?

type: "integer",
bigint: true,

With bigint type we can write simple and clear rule:

type: ["bigint", "string"]

without we will need to use more complex syntax:

anyOf: [
    {type: "integer", bigint: true}, 
    {type: "string"}
]

[epoberezkin]

No, there should be no new keywords or schema changes at all. There should be an instance level option that you pass to Ajv constructor that would change generated code to allow bigint where integer/number is required and also to apply the number-specific keywords to bigint values (maximum etc.)

[zdm]

Ah, ok. This setting will allow numbers / integers to accept bigint type. Why not to do this silently, without additional options?

But how coercion can be implemented in the future? With additional keywords?

Seems that I understood your point of view, you don;t want to add changes to JSON-SCHEMA standard.
I am sorry, but without introducing new type this will look like as one global hack. BigInt - new data type and logically it should be implemented as type.

[epoberezkin]

Why not to do this silently, without additional options?

It will have a small negative performance impact even if bigints are not used, just because their support is enabled, so having it as an option will make it opt-in...

But how coercion can be implemented in the future? With additional keywords?

I don’t know yet. There should be some design decision how “type” keyword will evolve to support JSON Type Definition RFC, so it may indeed become extensible by users, in which case it can be possible to treat bigint as an additional value of the type keyword that is also enabled by the same option. There is a simpler design approach for JTD though. On its own, coercion doesn’t justify making type extensible, but together with other things it may be justified (also it was requested as a separate feature long time ago, but it’s rarely asked for).

What I am definitely not supportive of is to just add one more hard-coded value for type - while the list is hard-coded it should stay aligned with JSON types. It’s actually a TS type level list at the moment, and there are benefits of having it locked, as well as there are benefits to allow extending it. Something to think about...

But supporting bigint and supporting coercion to bigint are two separate problems, not something that must happen at the same time.

User defined keyword is the option available today - it doesn’t require any changes in Ajv. It may actually be possible to update typeof keyword in ajv-keywords to perform this coercion (although there may be issues with when this coercion would happen and without supporting other number keywords first it is kind of pointless...

BigInt - new data type and logically it should be implemented as type.

The boundary that JSON schema draws between what is type, what is format and what is a referenced schema is quite arbitrary - many languages (including typescript) would treat all these things as types. So it’s definitely a consideration to extend “type” to allow any value to mean any arbitrarily defined hook, but saying that bigint is a type but, for example, a specific collection of properties, Map, Set or anything else really, is not a type is a very JavaScript-y approach.

Currently types are tied to JSON itself, not even JSON Schema, which is a stronger reason not to change it. On another hand, JTD has type “datetime” that is represented as a string but should be validated/parsed as datetime - so making type extensible is one of possible approaches to support JTD...

[zdm]

We shouldn't mix Number and BigInt types, because they are not compatible internally. User should have possibility to specify explicitly what data type he is expected (number or bigint).

With schema option bigint: true it will be impossible to specify that i want integer for some data and bigint for other in the same schema.

Next, validation should not be separated from coercion.
This is important, that when user specified, that he want integer - he will receive Number primitive or validation error, when bigint - he explicitl should get BigInt (not Number).

Other possibility - add new keyword, for example dataType: "bigint". In this case we shouldn't specify type and we were unable to use additional keywords for numeric data types (minimum, maximum, etc...).
Schema example:

{
    type: "array",
    items: {
        dataType: "bigint"
    }
}

But this will be a ugly hack, not a solution. ;-))

I don't know, how to implement this clearly and fully functional without introducing new type. You are better know your code and maybe you will have ideas. If not - better to leave it as is.

[mazz-seven]

Hey,
I encountered this issue today, and I'm not sure how to solve it without waiting for a fix.
I tried

very_big_number: {
 typeof: "bigint" 
 type: "integer" 
 format: "int64" 
}

{ very_big_number: 3726n }

I get a validation error must be integer.

[epoberezkin]

There is no support for bigint now, it's more than a fix. Given a limited interest I do not see it yet as a valuable feature... There is an RFC that advises to send integers larger than 2^32 as strings, and most public APIs are designed to comply with it, so even 2^64 is not needed in most cases - JavaScript support for 2^53 is usually sufficient...

If you do need to support bigint, you can use typeof: bigint and create custom keywords, similar to standard ones.

[epoberezkin]

... but you really should just use {type: "string", pattern: "^\-?[0-9]*$"} and convert to bigint in the application code, if you need to do any calculations with it.

[mazz-seven]

Thank you for the quick response.
Yeah, I will go with string and convert it to bigint on the application side.

[slavafomin]

Hello! Thank you for your hard work, ajv is a very cool project.

Please pardon my interruption.

JavaScript support for 2^53 is usually sufficient...

Usually yes, however, I'm working with Web 3.0 ecosystem and BigInt's a very prominent there — all cryptocurrency values are often BigInt, because general int's are just not enough (e.g. 1 ETH = 1000000000000000000 WEI). So with Web 3.0 popularity rise I guess it would be natural for ajv to implement support for BigInt's.

Getting to the current needs — consider I have an API that returns JSON documents with very big numbers, e.g.:

{ "number": 123, "big": 9123456789123456789 }

The 9123456789123456789 value parsed as Number will give you this: 9123456789123457000, notice the precision loss.

Is there a way that I can actually parse such big numbers as BigInt's with ajv right now? Ideally, I should be able to mark specific values in schema as BigInt's and they must be parsed using new BigInt() constructor instead of Number. Can you please steer me in the right direction so I can implement this in my project? I've looked into the code, but I can't figure out how to add this without rewriting the original source code.

Thanks!

[melroy89]

Given a limited interest I do not see it yet as a valuable feature...

I also need BIgInt now, my backend database allows bigger values but I'm now stuck with JS limits of 64-bit number. So the interest and the need is here...

EDIT: Actually I found out I actually need something like BigDecimal in JS. In my app I will use decimal.js for now.