Add env.json

[leighmcculloch]

What

Add env.json and use it for generating the macro_rule table that is used for generating the environment host interface and interface implementations.

Why

There are a lot of host functions.

There are so many that we use macro_rules to code generate most of the code relating to them in our Rust crates.

Unfortunately macro_rules doesn't help us code generate things that aren't Rust, like documentation, or non-Rust SDKs, or tools.

If we have any hope of developers building alternative non-Rust SDKs, we need to provide some things like the functions that SDKs need to extern reference in a way that will allow them to code generate. In a similar way we code generate today in the Rust crates.

JSON is accessible to everyone, and there are plenty of tools for it.

I considered a couple approaches to this change:

1. Generating a .json file from the macro_rules.
2. Generating the macro_rules from a .json file. (this PR)
3. Doing all interface codegen using proc-macros, which would be a massive departure from our current code infrastructure, so I did not consider it seriously.

I initially wanted to do (1), however macro_rules cannot do things like write files. Only proc-macros can. So it was more practical to do (2) and generate the existing macro_rules from a .json file.

This makes the JSON file the source of truth. If we're adding new host functions, we should add them to the JSON file and rebuild.

This proc-macro generates the existing macro_rule, the same as what used to be there. Here is a snapshot of the generated code:

#[doc(hidden)] #[macro_export] macro_rules!
_call_macro_with_all_host_functions
{
    { $macro_to_call_back : ident } =>
    {
        $macro_to_call_back!
        {
            mod context "x"
            {
                #[doc =
                "This one variant of logging does not take a format string and is live in both Env=Guest and Env=Host configurations."]
                { "_", fn log_value(v : RawVal) -> RawVal }
                #[doc =
                "Get the contractID `Bytes` of the contract which invoked the running contract. Traps if the running contract was not invoked by a contract."]
                { "0", fn get_invoking_contract() -> Object } #[doc = ""]
                { "1", fn obj_cmp(a : RawVal, b : RawVal) -> i64 }
                #[doc =
                "Records a contract event. `topics` is expected to be a `SCVec` with length <= 4 that cannot contain `Vec`, `Map`, or `Bytes` with length > 32 On success, returns an `SCStatus::Ok`."]
                {
                    "2", fn contract_event(topics : Object, data : RawVal) ->
                    RawVal
                }
                #[doc =
                "Get the contractID `Bytes` of the contract which invoked the running contract. Traps if the running contract was not invoked by a contract."]
                { "3", fn get_current_contract() -> Object }
                #[doc =
                "Return the protocol version of the current ledger as a u32."]
                { "4", fn get_ledger_version() -> RawVal }
                #[doc =
                "Return the sequence number of the current ledger as a u32."]
                { "5", fn get_ledger_sequence() -> RawVal }
                #[doc =
                "Return the timestamp number of the current ledger as a u64."]
                { "6", fn get_ledger_timestamp() -> Object }
                #[doc =
                "Return the network passphrase of the current ledger as `Bytes`."]
                { "7", fn get_ledger_network_passphrase() -> Object }
                #[doc =
                "Returns the full call stack from the first contract call to the current one as a vector of vectors, where the inside vector contains the contract id as Hash, and a function as a Symbol."]
                { "8", fn get_current_call_stack() -> Object }
                #[doc =
                "Causes the currently executing contract to fail immediately with a provided status code, which must be of error-type `ScStatusType::ContractError`. Does not actually return."]
                { "9", fn fail_with_status(status : Status) -> RawVal }
                #[doc =
                "Record a debug event. Fmt must be a Bytes. Args must be a Vec. Void is returned."]
                {
                    "a", fn log_fmt_values(fmt : Object, args : Object) ->
                    RawVal
                }
                #[doc =
                "Get whether the contract invocation is from an account or another contract. Returns 0 for account, 1 for contract."]
                { "b", fn get_invoker_type() -> u64 }
                #[doc =
                "Get the AccountID object type of the account which invoked the running contract. Traps if the running contract was not invoked by an account."]
                { "c", fn get_invoking_account() -> Object }
                #[doc =
                "Return the network id (sha256 hash of network passphrase) of the current ledger as `Bytes`. The value is always 32 bytes in length."]
                { "d", fn get_ledger_network_id() -> Object }
            } mod i64 "i"
            {
                #[doc = "Convert a u64 to an object containing a u64."]
                { "_", fn obj_from_u64(v : u64) -> Object }
                #[doc = "Convert an object containing a i64 to a u64."]
                { "0", fn obj_to_u64(obj : Object) -> u64 }
                #[doc = "Convert an i64 to an object containing an i64."]
                { "1", fn obj_from_i64(v : i64) -> Object }
                #[doc = "Convert an object containing an i64 to an i64."]
                { "2", fn obj_to_i64(obj : Object) -> i64 }
                #[doc =
                "Convert the low and high 64-bit words of a u128 to an object containing a u128."]
                { "5", fn obj_from_u128_pieces(lo : u64, hi : u64) -> Object }
                #[doc =
                "Extract the low 64 bits from an object containing a u128."]
                { "6", fn obj_to_u128_lo64(obj : Object) -> u64 }
                #[doc =
                "Extract the high 64 bits from an object containing a u128."]
                { "7", fn obj_to_u128_hi64(obj : Object) -> u64 }
                #[doc =
                "Convert the lo and hi 64-bit words of an i128 to an object containing an i128."]
                { "8", fn obj_from_i128_pieces(lo : u64, hi : u64) -> Object }
                #[doc =
                "Extract the low 64 bits from an object containing an i128."]
                { "9", fn obj_to_i128_lo64(obj : Object) -> u64 }
                #[doc =
                "Extract the high 64 bits from an object containing an i128."]
                { "a", fn obj_to_i128_hi64(obj : Object) -> u64 }
            } mod map "m"
            {
                #[doc = "Create an empty new map."]
                { "_", fn map_new() -> Object }
                #[doc =
                "Insert a key/value mapping into an existing map, and return the map object handle. If the map already has a mapping for the given key, the previous value is overwritten."]
                {
                    "0", fn map_put(m : Object, k : RawVal, v : RawVal) ->
                    Object
                }
                #[doc =
                "Get the value for a key from a map. Traps if key is not found."]
                { "1", fn map_get(m : Object, k : RawVal) -> RawVal }
                #[doc =
                "Remove a key/value mapping from a map if it exists, traps if doesn't."]
                { "2", fn map_del(m : Object, k : RawVal) -> Object }
                #[doc = "Get the size of a map."]
                { "3", fn map_len(m : Object) -> RawVal }
                #[doc =
                "Test for the presence of a key in a map. Returns (SCStatic) TRUE/FALSE."]
                { "4", fn map_has(m : Object, k : RawVal) -> RawVal }
                #[doc =
                "Given a key, find the first key less than itself in the map's sorted order. If such a key does not exist, return an SCStatus containing the error code (TBD)."]
                { "5", fn map_prev_key(m : Object, k : RawVal) -> RawVal }
                #[doc =
                "Given a key, find the first key greater than itself in the map's sorted order. If such a key does not exist, return an SCStatus containing the error code (TBD)."]
                { "6", fn map_next_key(m : Object, k : RawVal) -> RawVal }
                #[doc =
                "Find the minimum key from a map. If the map is empty, return an SCStatus containing the error code (TBD)."]
                { "7", fn map_min_key(m : Object) -> RawVal }
                #[doc =
                "Find the maximum key from a map. If the map is empty, return an SCStatus containing the error code (TBD)."]
                { "8", fn map_max_key(m : Object) -> RawVal }
                #[doc =
                "Return a new vector containing all the keys in a map. The new vector is ordered in the original map's key-sorted order."]
                { "9", fn map_keys(m : Object) -> Object }
                #[doc =
                "Return a new vector containing all the values in a map. The new vector is ordered in the original map's key-sorted order."]
                { "A", fn map_values(m : Object) -> Object }
            } mod vec "v"
            {
                #[doc =
                "Creates a new vector with an optional capacity hint `c`. If `c` is `ScStatic::Void`, no hint is assumed and the new vector is empty. Otherwise, `c` is parsed as an `u32` that represents the initial capacity of the new vector."]
                { "_", fn vec_new(c : RawVal) -> Object }
                #[doc =
                "Update the value at index `i` in the vector. Return the new vector. Trap if the index is out of bounds."]
                {
                    "0", fn vec_put(v : Object, i : RawVal, x : RawVal) ->
                    Object
                }
                #[doc =
                "Returns the element at index `i` of the vector. Traps if the index is out of bound."]
                { "1", fn vec_get(v : Object, i : RawVal) -> RawVal }
                #[doc =
                "Delete an element in a vector at index `i`, shifting all elements after it to the left. Return the new vector. Traps if the index is out of bound."]
                { "2", fn vec_del(v : Object, i : RawVal) -> Object }
                #[doc = "Returns length of the vector."]
                { "3", fn vec_len(v : Object) -> RawVal }
                #[doc = "Push a value to the front of a vector."]
                { "4", fn vec_push_front(v : Object, x : RawVal) -> Object }
                #[doc =
                "Removes the first element from the vector and returns the new vector. Traps if original vector is empty."]
                { "5", fn vec_pop_front(v : Object) -> Object }
                #[doc = "Appends an element to the back of the vector."]
                { "6", fn vec_push_back(v : Object, x : RawVal) -> Object }
                #[doc =
                "Removes the last element from the vector and returns the new vector. Traps if original vector is empty."]
                { "7", fn vec_pop_back(v : Object) -> Object }
                #[doc =
                "Return the first element in the vector. Traps if the vector is empty"]
                { "8", fn vec_front(v : Object) -> RawVal }
                #[doc =
                "Return the last element in the vector. Traps if the vector is empty"]
                { "9", fn vec_back(v : Object) -> RawVal }
                #[doc =
                "Inserts an element at index `i` within the vector, shifting all elements after it to the right. Traps if the index is out of bound"]
                {
                    "A", fn vec_insert(v : Object, i : RawVal, x : RawVal) ->
                    Object
                }
                #[doc =
                "Clone the vector `v1`, then moves all the elements of vector `v2` into it. Return the new vector. Traps if number of elements in the vector overflows a u32."]
                { "B", fn vec_append(v1 : Object, v2 : Object) -> Object }
                #[doc =
                "Copy the elements from `start` index until `end` index, exclusive, in the vector and create a new vector from it. Return the new vector. Traps if the index is out of bound."]
                {
                    "C", fn vec_slice(v : Object, start : RawVal, end : RawVal)
                    -> Object
                }
                #[doc =
                "Get the index of the first occurrence of a given element in the vector. Returns the u32 index of the value if it's there. Otherwise, it returns `ScStatic::Void`."]
                {
                    "D", fn vec_first_index_of(v : Object, x : RawVal) -> RawVal
                }
                #[doc =
                "Get the index of the last occurrence of a given element in the vector. Returns the u32 index of the value if it's there. Otherwise, it returns `ScStatic::Void`."]
                {
                    "E", fn vec_last_index_of(v : Object, x : RawVal) -> RawVal
                }
                #[doc =
                "Binary search a sorted vector for a given element. If it exists, the high-32 bits of the return value is 0x0001 and the low-32 bits contain the u32 index of the element. If it does not exist, the high-32 bits of the return value is 0x0000 and the low-32 bits contain the u32 index at which the element would need to be inserted into the vector to maintain sorted order."]
                { "F", fn vec_binary_search(v : Object, x : RawVal) -> u64 }
            } mod ledger "l"
            {
                #[doc = ""]
                {
                    "_", fn put_contract_data(k : RawVal, v : RawVal) -> RawVal
                } #[doc = ""]
                { "0", fn has_contract_data(k : RawVal) -> RawVal }
                #[doc = ""]
                { "1", fn get_contract_data(k : RawVal) -> RawVal }
                #[doc = ""]
                { "2", fn del_contract_data(k : RawVal) -> RawVal }
                #[doc =
                "Deploys a contract from the current contract. `wasm_hash` must be a hash of the contract code that has already been installed on this network. `salt` is used to create a unique contract id."]
                {
                    "3", fn
                    create_contract_from_contract(wasm_hash : Object, salt :
                    Object) -> Object
                }
                #[doc =
                "Deploys a built-in token contract from the current contract. `salt` is used to create a unique contract id for the token."]
                {
                    "4", fn create_token_from_contract(salt : Object) -> Object
                }
            } mod call "d"
            {
                #[doc =
                "Calls a function in another contract with arguments contained in vector `args`. If the call is successful, forwards the result of the called function. Traps otherwise."]
                {
                    "_", fn
                    call(contract : Object, func : Symbol, args : Object) ->
                    RawVal
                }
                #[doc =
                "Calls a function in another contract with arguments contained in vector `args`. Returns: - if successful, result of the called function. - otherwise, an `SCStatus` containing the error status code."]
                {
                    "0", fn
                    try_call(contract : Object, func : Symbol, args : Object) ->
                    RawVal
                }
            } mod bytes "b"
            {
                #[doc =
                "Serializes an (SC)Val into XDR opaque `Bytes` object."]
                { "_", fn serialize_to_bytes(v : RawVal) -> Object }
                #[doc =
                "Deserialize a `Bytes` object to get back the (SC)Val."]
                { "0", fn deserialize_from_bytes(b : Object) -> RawVal }
                #[doc =
                "Copies a slice of bytes from a `Bytes` object specified at offset `b_pos` with length `len` into the linear memory at position `lm_pos`. Traps if either the `Bytes` object or the linear memory doesn't have enough bytes."]
                {
                    "1", fn
                    bytes_copy_to_linear_memory(b : Object, b_pos : RawVal,
                    lm_pos : RawVal, len : RawVal) -> RawVal
                }
                #[doc =
                "Copies a segment of the linear memory specified at position `lm_pos` with length `len`, into a `Bytes` object at offset `b_pos`. The `Bytes` object may grow in size to accommodate the new bytes. Traps if the linear memory doesn't have enough bytes."]
                {
                    "2", fn
                    bytes_copy_from_linear_memory(b : Object, b_pos : RawVal,
                    lm_pos : RawVal, len : RawVal) -> Object
                }
                #[doc =
                "Constructs a new `Bytes` object initialized with bytes copied from a linear memory slice specified at position `lm_pos` with length `len`."]
                {
                    "3", fn
                    bytes_new_from_linear_memory(lm_pos : RawVal, len : RawVal)
                    -> Object
                } #[doc = "Create an empty new `Bytes` object."]
                { "4", fn bytes_new() -> Object }
                #[doc =
                "Update the value at index `i` in the `Bytes` object. Return the new `Bytes`. Trap if the index is out of bounds."]
                {
                    "5", fn bytes_put(b : Object, i : RawVal, u : RawVal) ->
                    Object
                }
                #[doc =
                "Returns the element at index `i` of the `Bytes` object. Traps if the index is out of bound."]
                { "6", fn bytes_get(b : Object, i : RawVal) -> RawVal }
                #[doc =
                "Delete an element in a `Bytes` object at index `i`, shifting all elements after it to the left. Return the new `Bytes`. Traps if the index is out of bound."]
                { "7", fn bytes_del(b : Object, i : RawVal) -> Object }
                #[doc = "Returns length of the `Bytes` object."]
                { "8", fn bytes_len(b : Object) -> RawVal }
                #[doc =
                "Appends an element to the back of the `Bytes` object."]
                { "9", fn bytes_push(b : Object, u : RawVal) -> Object }
                #[doc =
                "Removes the last element from the `Bytes` object and returns the new `Bytes`. Traps if original `Bytes` is empty."]
                { "A", fn bytes_pop(b : Object) -> Object }
                #[doc =
                "Return the first element in the `Bytes` object. Traps if the `Bytes` is empty"]
                { "B", fn bytes_front(b : Object) -> RawVal }
                #[doc =
                "Return the last element in the `Bytes` object. Traps if the `Bytes` is empty"]
                { "C", fn bytes_back(b : Object) -> RawVal }
                #[doc =
                "Inserts an element at index `i` within the `Bytes` object, shifting all elements after it to the right. Traps if the index is out of bound"]
                {
                    "D", fn bytes_insert(b : Object, i : RawVal, u : RawVal) ->
                    Object
                }
                #[doc =
                "Clone the `Bytes` object `b1`, then moves all the elements of `Bytes` object `b2` into it. Return the new `Bytes`. Traps if its length overflows a u32."]
                { "E", fn bytes_append(b1 : Object, b2 : Object) -> Object }
                #[doc =
                "Copies the elements from `start` index until `end` index, exclusive, in the `Bytes` object and creates a new `Bytes` from it. Returns the new `Bytes`. Traps if the index is out of bound."]
                {
                    "F", fn
                    bytes_slice(b : Object, start : RawVal, end : RawVal) ->
                    Object
                }
            } mod hash "h"
            {
                #[doc = ""] { "_", fn hash_from_bytes(x : Object) -> Object }
                #[doc = ""] { "0", fn hash_to_bytes(x : Object) -> Object }
            } mod key "k"
            {
                #[doc = ""]
                { "_", fn public_key_from_bytes(x : Object) -> Object }
                #[doc = ""]
                { "0", fn public_key_to_bytes(x : Object) -> Object }
            } mod crypto "c"
            {
                #[doc = ""]
                { "_", fn compute_hash_sha256(x : Object) -> Object }
                #[doc = ""]
                {
                    "0", fn
                    verify_sig_ed25519(x : Object, k : Object, s : Object) ->
                    RawVal
                }
            } mod account "a"
            {
                #[doc =
                "Get the low threshold for the account with ID `a` (`a` is `AccountId`). Traps if no such account exists."]
                { "_", fn account_get_low_threshold(a : Object) -> RawVal }
                #[doc =
                "Get the medium threshold for the account with ID `a` (`a` is `AccountId`). Traps if no such account exists."]
                { "0", fn account_get_medium_threshold(a : Object) -> RawVal }
                #[doc =
                "Get the high threshold for the account with ID `a` (`a` is `AccountId`). Traps if no such account exists."]
                { "1", fn account_get_high_threshold(a : Object) -> RawVal }
                #[doc =
                "Get the signer weight for the signer with ed25519 public key `s` (`s` is `Bytes`) on the account with ID `a` (`a` is `AccountId`). Returns the master weight if the signer is the master, and returns 0 if no such signer exists. Traps if no such account exists."]
                {
                    "2", fn account_get_signer_weight(a : Object, s : Object) ->
                    RawVal
                }
                #[doc =
                "Given an ID `a` (`a` is `AccountId`) of an account, check if it exists. Returns (SCStatic) TRUE/FALSE."]
                { "3", fn account_exists(a : Object) -> RawVal }
            } mod test "t"
            {
                #[doc =
                "A dummy function taking 0 arguments and performs no-op. This function is for test purpose only, for measuring the roundtrip cost of invoking a host function, i.e. host->Vm->host."]
                { "_", fn dummy0() -> RawVal }
            }
        }
    } ;
}

There are some downside to loading files in proc-macros.

The Rust compiler will not always determine if the files need to trigger a rebuild of the proc-macro. Because of this you might edit the JSON file and not see a change. Running cargo clean will always resolve this. This is a bit inconvenient, but given that we don't add host functions everyday, I think this is an okay tradeoff.

[dmkozh]

That's really nice for the SDK interop, thanks! The only thing that comes to mind is that given that env.json is meant to be used across SDKs it could be a better idea to store it in a similar repo (which is one more repo to maintain, but makes a lot more sense for the SDK developers). I don't think the decision on that should be tied to this PR though.

[leighmcculloch]

@dmkozh :nod: That's a great point.

I guess this is somewhat similar to XDR. For the longest time XDR lived in the stellar-core repo, and for the most part that was fine, but it was inconvenient at times.

I'm inclined to keep the file here in the short term, because there are advantages – process optimizations mostly – to keeping things that change together, together. But over time it will be even more important that this file is discoverable, and versionable really clearly. It's that last aspect of versioning that I think was our big win with moving XDR into its own repo, and I expect it'll be similar here.

I'm happy to move it into a new repo now if folks think that's a good idea now, or we can wait a little while and do it down the track.

cc @graydon @sisuresh @jayz22 @Soneso ?

[dmkozh]

I'm inclined to keep the file here in the short term

+1 on that, I would be inclined to move it away from env closer to the release, as the env interface should've stabilized enough at that point.