uint64 support (and bigger integers)

[bettio]

A complete integer bignum implementation, with arbitrary integers, might take quite some flash space, so we may rather investigate support for "bigger integers".

Right now our biggest supported integer type is signed int64, that means we support up to UINT63_MAX, leaving out bigger positive integers such as values that can be represented with an uint64.

Rationale: uint64 is quite widespread in some serialization formats, such as protobuf and others, so it might tricky to handle that kind of values.

On the BEAM boxed integers already have a bit available for sign (001s BIGNUM with sign bit):

1. should we use it for extending support up to 64 + sign bit?
2. supporting wider integers such as 128 bit or 256 bit integers seems a reasonable amount of work. Should we support them (and keep one of them as new limit)?
3. what is a reasonable limit for wider integers? (e.g. 512 bit integers seems quite uncommon)
4. should we rather support uint64 as a wider signed integer?
5. also, let's assume we are going to support a 256 bit integer, should we keep the sign bit out of it?

Also it is worth to be noted we don't need to support multiple bigger integer types, such as both 128 and 256 bit, we can just promote any >= 65 bit to the biggest one (such as the 256 bit integer).

This might be a good feature for v0.7.

[UncleGrumpy]

If we can easily extend support to 64-bit signed, that seems like a nice addition. Perhaps some testing on various supported hardware could help determine the largest BIGNUM to support. 512 does seem a bit much for microcontrollers.

Personally I think it would be better to support signed BIGNUM over size. If it comes down to 256 unsigned or 128 signed, I would opt for 128-bit signed.

[jakub-gonet]

1. I would appreciate "limitations" section in the docs that documents system int limits, changed x register numbers, etc https://www.atomvm.net/doc/main/programmers-guide.html#limitations
2. Would you accept a patch for externalterm to have better error reporting when it can't decode serialized data?

AVM communicate well enough if value is out of bounds in runtime but does pretty poor error reporting if loaded BEAM file has it as a literal:

AtomVM/src/libAtomVM/externalterm.c

Lines 700 to 702 in 23b0781

	if (UNLIKELY(num_bytes > 8 || remaining < (SMALL_BIG_EXT_BASE_SIZE + num_bytes))) {
	return INVALID_TERM_SIZE;
	}

and then aborts the VM (module_load_literal).

Example code:

-module(hello_world).
-export([
    start/0,
    is_bignum/1
]).

start() ->
    erlang:display(?MODULE:is_bignum(1)),
    ok.

% crashes the VM
is_bignum(N) ->
    if
        N >= 1 bsl 128 ->
            big;
        N =< -(1 bsl 128) ->
            smol;
        true ->
            ok
    end.

%% dumps VM state
% is_bignum(N) ->
%     Big = rt_bsl(1, 128),
%     Smol = -rt_bsl(1, 128),
%     if
%         N >= Big ->
%             big;
%         N =< Smol ->
%             smol;
%         true ->
%             ok
%     end.

% rt_bsl(N, 0) -> N;
% rt_bsl(N, Shift) -> rt_bsl(2 * N, Shift - 1).