Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization and encoding/decoding #4769

Closed
5 tasks done
IGI-111 opened this issue Jul 6, 2023 · 3 comments
Closed
5 tasks done

Serialization and encoding/decoding #4769

IGI-111 opened this issue Jul 6, 2023 · 3 comments
Assignees
Labels
ABI Everything to do the ABI, especially the JSON representation compiler General compiler. Should eventually become more specific as the issue is triaged epic An epic is a high-level master issue for large pieces of work. lib: std Standard library

Comments

@IGI-111
Copy link
Contributor

IGI-111 commented Jul 6, 2023

Binary encoding of arbitrary sway values is a necessary feature, notably to return values/pass values to contracts in a predictable, portable fashion.

This is tricky and solidity has had to go through multiple versions of it but we can borrow from their encoding scheme.

This should take versioning into account, as we'll likely want to make alterations to this scheme in the future.

We should also consider options for packing the data (such as Solidity's strict mode).

We should also consider the ergonomics of how to use it in the language, a trait based approach seems natural but we don't have derive macroes so having people implement a trait could prove tedious. We may want to rely on intrinsics to make the interface nicer.

@IGI-111 IGI-111 added compiler General compiler. Should eventually become more specific as the issue is triaged lib: std Standard library ABI Everything to do the ABI, especially the JSON representation labels Jul 6, 2023
@IGI-111
Copy link
Contributor Author

IGI-111 commented Jul 17, 2023

After some investigation, it looks like the current way our encoding works is different for input and output.

Input values are directly usable and pointer values are populated with correct memory offsets so they can be directly used.

Output values are fully dereferenced and contain no indirection, which means it's hard/impossible to nest them. Any dynamically sized return value will have to be named last and can't have siblings because it will consume the entire rest of the buffer when decoding.

Note that these two encodings are not compatible with each other, which we'll definitely want to solve.

What we should do instead is similar to Solidity's encoding scheme: split the encoded buffer into two regions, one for inline bytes and the other for dynamic data, and have all pointers values be encoded inline with pointer values that are offsets of the beginning of the data region.

Then, in Sway, for both inputs and outputs, we should parse these into proper pointers for inputs, and generate proper data section offsets for outputs.

@IGI-111
Copy link
Contributor Author

IGI-111 commented Jul 17, 2023

There is a practical issue here in that the test frameworks for sway and fuels-rs are both dependent on each other, and changing this serialization will require a breaking change of both. We'll have to figure out how to choreograph this. Likely by bypassing one of the test suites after we've manually confirmed the first change works on the updated version of the other repository.

@Voxelot
Copy link
Member

Voxelot commented Aug 11, 2023

blocked by: #4929

@IGI-111 IGI-111 assigned xunilrj and unassigned IGI-111 Nov 8, 2023
tritao pushed a commit that referenced this issue Jan 15, 2024
## Description

This PR changes the output of `__log` to encoded values (see
#4769). A quick example of what
that means is

```sway
struct S {
    a: u64,
    b: u32,
    c: u16,
    d: u8,
    e: Vec<u64>,
    f: str,
    g: u256
}

let mut e = Vec::new();
e.push(1);
e.push(2);
e.push(3);

__log(S{
    a: 1,
    b: 2,
    c: 3,
    d: 4,
    e,
    f: "sway",
    g: u256::max()
});
```

will output

```
 [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 3, 4, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4,    115,   119,    97,   121, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255]
  ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^  ^^^^  ^  ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^     ^^^    ^^^     ^^    ^^^  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     s.a         s.b   s.c s.d              s.e.len()                  s.e[0]                  s.e[1]                  s.e[2]               s.f.len()  s.f[0] s.f[1] s.f[2] s.f[3]  s.g
```

This works in two steps:

1 - `__log(s)` is desugared into `__log(encode(s))`;
2 - call `encode`. Its impl can be found at
https://github.com/FuelLabs/sway/pull/5306/files#diff-ee5cebc963e841e8af05f3986de17dd266ee6e9b49dbe089a5eb64764f3b802eR307

It simply creates an append-only buffer and call `abi_encode` from a
special trait named `AbiEncode`.

To be encodable, a type must implement `AbiEncode`. In the example
above, `S` is auto-implemented by the compiler because all its fields
are `AbiEncode`. But we can also have custom impl like `Vec` here:
https://github.com/FuelLabs/sway/pull/5306/files#diff-b5d9688741fea479477f26ca44cd1d1ecbd2f003f3875292abb23df7fad85c58

All this is behind a compiler flag:

```
> forc build -h
Compile the current or target project

USAGE:
    forc build [OPTIONS]

OPTIONS:
...
        --experimental-new-encoding
            Experimental flags for the "new encoding" feature
```

The same flag is available for the `e2e` tests:

```
> cargo r -p test --release -- should_pass/language/logging --verbose --experimental-new-encoding
```

## Limitations

1 - Now that __log demands a `fn` called `encode`, when something is
compiled with `implicit-std: false`, the intrinsic function `__log` does
not work out-of-box anymore. A function called `encode` must "visible"
and the worst part is that it needs to be functional to log anything.

2 - Arrays, string arrays and tuples will have limited implementations.
Currently up to five items.

## Checklist

- [x] I have linked to any relevant issues.
- [x] I have commented my code, particularly in hard-to-understand
areas.
- [x] I have updated the documentation where relevant (API docs, the
reference, and the Sway book).
- [x] I have added tests that prove my fix is effective or that my
feature works.
- [x] I have added (or requested a maintainer to add) the necessary
`Breaking*` or `New Feature` labels where relevant.
- [x] I have done my best to ensure that my PR adheres to [the Fuel Labs
Code Review
Standards](https://github.com/FuelLabs/rfcs/blob/master/text/code-standards/external-contributors.md).
- [x] I have requested a review from the relevant team or maintainers.

---------

Co-authored-by: Joshua Batty <joshpbatty@gmail.com>
@kamyar-tm kamyar-tm added the epic An epic is a high-level master issue for large pieces of work. label Feb 22, 2024
@IGI-111 IGI-111 mentioned this issue Mar 13, 2024
28 tasks
@IGI-111 IGI-111 closed this as completed Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ABI Everything to do the ABI, especially the JSON representation compiler General compiler. Should eventually become more specific as the issue is triaged epic An epic is a high-level master issue for large pieces of work. lib: std Standard library
Projects
None yet
Development

No branches or pull requests

4 participants