feat!: adding `abi` transpiler #2856

arboleya · 2024-07-29T16:27:08Z

Closes Adapt ABI format to new specification #2313

Release notes

In this release, we:

Upgraded forc to v0.63.0 & v0.63.1
Added support for the new ABI spec (#596, #599)

Summary

Given the amount of structural changes we had in the new ABI and the impact it will have on the codebase, and provided the fact we have a lot of technical debt to clear regarding ABI coder and typegen packages (#1795, #2346, #2686), this proposal aim at releasing the pressure of the moment by transpiling the old format into the new one, while still:

Supporting the new ABI spec
Introducing the least amount of changes possible
Keeping all tests and coverage intact

Transpilation occurs internally between public and private APIs.

Transpilation is an internal detail on the Interface level
Typegen includes the new abi format in its generated code but doesn't transpile it
All public-facing APIs use the new spec ABI under JsonAbi, and the old one is available as JsonAbiOld
AbiCoder is no longer exported, and its methods moved to Interface via encodeType/decodeType (ee734b9)
We removed the Interface.findTypeById method in 157bdba because the concept of a type in the new ABI differs from the old one. Whoever wants to get a specific concrete or metadata type can do so directly from the ABI they have on the Interface.jsonAbi property

Special thanks ❤️

Huge shoutout to @nedsalk, who was crucial for this work and ensured everything around this was working as expected everywhere and finalized all the heavy lifting around public vs private APIs via:

chore!: internalize old JsonAbi format #2862

Also, to @petertonysmith94, for the thorough reviews and additional bug-fixing via:

fix: option coder decoding incorrect value #2870

Breaking Changes

New ABI spec

The SDK now adheres to the new specs introduced via:

Check these out to understand all its changes.

`AbiCoder` is no longer available to import

The class AbiCoder is no longer exported, and the way to do encoding and decoding of specific types is now via the Interface.encodeType and Interface.decodeType methods:

// before
const abi = yourAbi;
const functionArg = abi.functions.inputs[0];

const encoded = AbiCoder.encode(abi, functionArg, valueToEncode);
const decoded = AbiCoder.decode(abi, functionArg, valueToDecode, 0);

// after
import { Interface } from 'fuels';

const abi = yourAbi;
const functionArg = abi.functions.inputs[0];

const abiInterface = new Interface(abi);

const encoded = abiInterface.encodeType(functionArg.concreteTypeId, valueToEncode);
const decoded = abiInterface.decodeType(functionArg.concreteTypeId, valueToDecode);

`Interface.findTypeById` has been removed

Previously, you could get a type from the ABI via the Interface.findTypeById. This method was removed after introducing the new abi specification because the concept of a type has been split into concrete and metadata types. If you want a specific type, you can get it directly from the ABI.

// before
const abiInterface = new Interface(abi);

// internally this method searched the abi types:
// abi.types.find(t => t.typeId === id);
const type = abiInterface.findTypeById(id);

// after
import { Interface } from 'fuels';

// search the types on the abi directly
const concreteType = abi.concreteTypes.find(ct => ct.concreteTypeId === id);
const metadataType = abiInterface.jsonAbi.metadataTypes.find(mt => mt.metadataTypeId === id);

`JsonAbiArgument` is no longer exported

The JsonAbiArgument type isn't part of the new ABI spec (#596, #599) as such so we stopped exporting it. Its closest equivalent now would be a concrete type because it fully defines a type.

// before
const arg: JsonAbiArgument = {...};

// after
import { Interface } from 'fuels';

type ConcreteType = JsonAbi["concreteTypes"][number]
const arg: ConcreteType = {...};

Checklist

I added — tests to prove my changes
I updated — all the necessary docs
I reviewed — the entire PR myself, using the GitHub UI
I described — all breaking changes and the Migration Guide

internal/forc/lib/shared.js

petertonysmith94

🚀 🚢

danielbate

github-actions · 2024-08-19T22:18:19Z

Coverage Report:

Lines	Branches	Functions	Statements
79.43%(+0.22%)	71.89%(+0.51%)	77.77%(+0.4%)	79.56%(+0.22%)

Changed Files:

Ok	File (✨=New File)	Lines	Branches	Functions	Statements
🔴	packages/abi-coder/src/AbiCoder.ts	100% (+25%)	0% (+0%)	100% (+33.34%)	100% (+25%)
🔴	packages/abi-coder/src/FunctionFragment.ts	63.15% (+0.99%)	37.5% (+0%)	69.23% (+2.57%)	63.41% (+1.88%)
🔴	packages/abi-coder/src/Interface.ts	76% (+4.58%)	63.63% (+0%)	75% (+2.28%)	75% (+4.17%)
🔴	packages/abi-coder/src/encoding/strategies/getCoderV1.ts	91.83% (+0.17%)	88.57% (+1.48%)	100% (+0%)	92% (+0.17%)
🔴	packages/abi-coder/src/utils/constants.ts	97.56% (+0.06%)	100% (+0%)	0% (+0%)	97.56% (+0.06%)
🔴	✨ packages/abi-coder/src/utils/transpile-abi.ts	97.36% (+97.36%)	87.5% (+87.5%)	94.11% (+94.11%)	95.45% (+95.45%)
✅	packages/abi-typegen/src/templates/utils/formatEnums.ts	100% (+0%)	100% (+50%)	100% (+0%)	100% (+0%)
✅	packages/abi-typegen/src/templates/utils/formatStructs.ts	100% (+0%)	100% (+50%)	100% (+0%)	100% (+0%)
🔴	✨ packages/abi-typegen/src/utils/transpile-abi.ts	100% (+100%)	95.83% (+95.83%)	94.11% (+94.11%)	97.72% (+97.72%)
🔴	packages/account/src/providers/transaction-request/transaction-request.ts	88.48% (+0%)	79.71% (+1.45%)	84% (+0%)	88.73% (+0%)

sdankel · 2024-08-20T23:32:13Z

packages/abi-coder/src/types/JsonAbiNew.ts

+  readonly type: string;
+  readonly metadataTypeId: number;
+  readonly components?: readonly Component[];
+  readonly typeParameters?: readonly number[];


Is this an array of concrete type indexes or metadata type indexes? It's hard to tell which array this refers to.

It is an array of metadata type ids:

"typeParameters": an array of type metadata declaration ID of the type parameters of the type, if the type is generic, otherwise nonexistent. Each type parameter is a type declaration and is represented as described in Generic Type Parameter.

IDs as in the hashes or the array indexes? Everywhere else, it looks like the hashes are stored as strings. Are these hashes stored as numbers instead or are these array indexes?

The spec itself is unclear on this.

Metadata type ids are always numbers. On the topic of array indexes, I wouldn't use array indexes for anything; they might correlate with the metadataTypeId of an entry in the metadataTypes array, but the spec doesn't define that relationship.

sdankel · 2024-08-20T23:45:34Z

packages/abi-coder/src/types/JsonAbiNew.ts

+export interface ConcreteType {
+  readonly type: string;
+  readonly concreteTypeId: string;
+  readonly metadataTypeId?: number;


Per the spec, this should be a string (hash). It would help reduce complexity if this were consistent.

The link you gave us points to this same field the comment is on...

Regardless, the spec says this for concreteType.metadataTypeId:

"metadataTypeId": the type metadata declaration ID of this type, if the type metadata has components or is generic, otherwise nonexistent.

And it defines metadataType.metadataTypeId as this:

"metadataTypeId": a unique integer ID.

So this conforms with the specification.

Sorry, updated the link. It also says this:

- _type metadata declaration_: the declaration or definition of a type which can be generic. `struct Foo { .. }` and `struct Bar<T> { .. }` in the example above are both type declarations. The metadata declaration contains component details about the type. And a single _type metadata declaration_ is generated per type, even for generic types.

Which makes it sound like "type metadata declaration ID" is the string hash, not an array index.

The string hashes are exclusively related to concrete types (hashes of their type property). The spec doesn't mention array indexes in relation to metadataTypeIds, it just defines them as a field of the object. I wouldn't rely on that correlation to build anything as it might change: a metadata type with id 12 might be put onto index 24 and it would still be spec-compliant.

sdankel · 2024-08-20T23:53:01Z

packages/abi-typegen/src/types/interfaces/JsonAbiNew.ts

+  readonly type: string;
+  readonly concreteTypeId: string;


Is type just the type name, or is it the full type string? i.e. MyStruct or path::to::MyStruct<T>

It's the full type string. You can find examples lower down in the spec.

sdankel · 2024-08-21T00:43:40Z

packages/abi-typegen/src/types/interfaces/JsonAbiNew.ts

+export interface AbiFunction {
+  readonly name: string;
+  readonly inputs: readonly AbiFunctionInput[];
+  readonly output: string;


Is this string the concrete type hash or the decoded full type string?

It's the concrete type hash. From the spec:

"output": the type concrete declaration hash based ID of the type being returned by the function.

packages/abi-typegen/package.json

* Upgrading forc to temp branch * Adding ABI transpiler and integrating it with related areas * Updating typegen fixtures * Fixing broken typegen tests; updating outdated IDs * Re-instating the order * Loading forc projects the standard and proper way * Updating coder’s types, regexes, and fixing type matching everywhere * Updating test using callpath-based naming for Option * Fixing test, intercepting ABI loading so it can be transpiled * Adding changeset * Adjusting changeset * Excluding sway-repo source files in `forc:check` script * Ensuring branches are up to date before building * Building internal packages in `--release` mode * Fixing directory path for new build target * Merging duplicated variables, standardizing terminologies * Improving validation to handle strings / non-objects as well * fix: option coder decoding incorrect value (#2870) * Revert "Improving validation to handle strings / non-objects as well" This reverts commit 45f3ef2. * chore: fixing types on OptionCoder * fix: match full qualifying type name with regex * chore: added test groups Co-authored-by: Daniel Bate <djbate23@gmail.com> * chore: revert StructCoder name change --------- Co-authored-by: Daniel Bate <djbate23@gmail.com> * Standardizing timeout limits across workflows * Ignoring `sway-repo`in links check when building things from source * Fixing ignore path * Stop deleting `concreteTypeId`, used in public API * Update .changeset/odd-horses-cheer.md * Ignoring also `fuel-core-repo` for the same reasons as `sway-repo` * Update .changeset/odd-horses-cheer.md * Updating more fixtures * Updating snippet * Fixing random lint warning * Incrasing test timeout limit * Fixing ignored paths * chore!: internalize old `JsonAbi` format (#2862) * use `transpileAbi` in `Interface` * add new interfaces to typegen * no need for `debug` build anymore * fix `getTypegenForcProject` return type * added `as JsonAbi` type assertions to fix type errors * fix typegen tests * rename variable * stop exporting `JsonAbi` from typegen * revert changes that should be in another PR * contttinue exporting * rename to match `master` * fix encode-and-decode test * Renamed `JsonAbiNew` to `JsonAbi` and the old format to `JsonAbiOld` * fix compilation errors * Stop exporting `AbiCoder` and update `encode-and-decode` example * Removed `Interface.findTypeById` method * stop casting `as unknown as JsonAbi` * use `argument` directly * fix import * fix: `Interface.encodeType/decodeType` * Update packages/abi-coder/src/types/JsonAbi.ts Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Update packages/abi-coder/src/types/JsonAbiNew.ts Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Update packages/abi-typegen/src/types/interfaces/JsonAbi.ts Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Update packages/account/test/fuel-wallet-connector.test.ts Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Update packages/abi-typegen/src/utils/transpile-abi.ts Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * copy/paste transpile-abi * fix compilation error * eslint disable todo type * use `{ transpile: T }` params object * update `encode-and-decode.md` docs * fix: spellcheck * Update apps/docs/src/guide/encoding/encode-and-decode.md Co-authored-by: Anderson Arboleya <anderson@arboleya.me> * Update apps/docs/src/guide/encoding/encode-and-decode.md Co-authored-by: Anderson Arboleya <anderson@arboleya.me> * Update apps/docs/src/guide/encoding/encode-and-decode.md Co-authored-by: Anderson Arboleya <anderson@arboleya.me> --------- Co-authored-by: Anderson Arboleya <anderson@arboleya.me> Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Getting around non-null assertions * Adjusting changeset * Adjusting scripts * Updating template fixtures/snapshots * Triggering CI * The branch is gone - temporarily switching to `master` * Temporarily patching sway std lib * Temporarily skipping problematic sway projects and related tests * Lintfix - deprecations, types mismatch, and broken imports * DRYing test setup/teardown * Fixing broken tests * Patching base library individually * Unskipping problematic workspace members * Adjusting predicate data handling * Undoing undesired changed * Adjuting more projects * Revert "Temporarily skipping problematic sway projects and related tests" This reverts commit dc054ff. * Formatting * Revert "Temporarily patching sway std lib" This reverts commit 48b0ab3. * Revert "Patching base library individually" This reverts commit b97c944. * Replacing sway `branch` by `0.63.0` version * Update .changeset/odd-horses-cheer.md Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> * Updating forc version --------- Co-authored-by: Peter Smith <peter@blueoceancomputing.co.uk> Co-authored-by: Nedim Salkić <nedim.salkic@fuel.sh> Co-authored-by: Daniel Bate <djbate23@gmail.com>

arboleya added 10 commits July 29, 2024 12:55

Upgrading forc to temp branch

7585782

Adding ABI transpiler and integrating it with related areas

bb3a168

Updating typegen fixtures

d08b334

Fixing broken typegen tests; updating outdated IDs

81ec4cc

Re-instating the order

3caff6d

Loading forc projects the standard and proper way

5816eaf

Updating coder’s types, regexes, and fixing type matching everywhere

f04bc3f

Updating test using callpath-based naming for Option

a34f676

Fixing test, intercepting ABI loading so it can be transpiled

bf760a2

Adding changeset

27a1e61

arboleya added the feat Issue is a feature label Jul 29, 2024

arboleya self-assigned this Jul 29, 2024

Adjusting changeset

05c45be

vercel bot had a problem deploying to Preview July 29, 2024 16:51 Failure

Excluding sway-repo source files in forc:check script

2a00ce3

vercel bot had a problem deploying to Preview July 29, 2024 18:25 Failure

arboleya added 2 commits July 29, 2024 15:32

Ensuring branches are up to date before building

ddb3cfe

Building internal packages in --release mode

bdcf220

vercel bot had a problem deploying to Preview July 29, 2024 18:36 Failure

github-advanced-security bot found potential problems Jul 29, 2024

View reviewed changes

internal/forc/lib/shared.js Dismissed Show dismissed Hide dismissed

internal/forc/lib/shared.js Dismissed Show dismissed Hide dismissed

internal/forc/lib/shared.js Fixed Show fixed Hide fixed

Fixing directory path for new build target

045adc8

github-advanced-security bot found potential problems Jul 29, 2024

View reviewed changes

internal/forc/lib/shared.js Dismissed Show dismissed Hide dismissed

vercel bot had a problem deploying to Preview July 29, 2024 19:54 Failure

nedsalk mentioned this pull request Jul 30, 2024

chore!: support new encoding schema #2826

Closed

4 tasks

Merge branch 'master' into aa/feat/adding-abi-transpiler

7de6c06

vercel bot had a problem deploying to Preview July 30, 2024 10:58 Failure

arboleya marked this pull request as ready for review July 30, 2024 11:10

arboleya requested review from digorithm, Torres-ssf and Dhaiwat10 as code owners July 30, 2024 11:10

arboleya dismissed nedsalk’s stale review via 28734a9 August 19, 2024 17:46

arboleya requested review from nedsalk and petertonysmith94 August 19, 2024 17:46

vercel bot had a problem deploying to Preview August 19, 2024 17:48 Failure

petertonysmith94 previously approved these changes Aug 19, 2024

View reviewed changes

Torres-ssf previously approved these changes Aug 19, 2024

View reviewed changes

danielbate previously approved these changes Aug 19, 2024

View reviewed changes

arboleya added 2 commits August 19, 2024 17:46

Merge branch 'master' into aa/feat/adding-abi-transpiler

7ef4e88

Updating forc version

f019cba

arboleya dismissed stale reviews from danielbate, Torres-ssf, and petertonysmith94 via f019cba August 19, 2024 21:50

arboleya requested review from Torres-ssf, petertonysmith94 and danielbate August 19, 2024 21:53

vercel bot deployed to Preview August 19, 2024 21:53 View deployment

danielbate approved these changes Aug 19, 2024

View reviewed changes

Torres-ssf approved these changes Aug 19, 2024

View reviewed changes

arboleya merged commit 4c653d0 into master Aug 19, 2024
20 checks passed

arboleya deleted the aa/feat/adding-abi-transpiler branch August 19, 2024 22:22

fuel-service-user mentioned this pull request Aug 19, 2024

ci(release): v0.94.0 @ master #2863

Merged

arboleya mentioned this pull request Aug 20, 2024

Flaky tests #2967

Closed

sdankel reviewed Aug 20, 2024

View reviewed changes

sdankel reviewed Aug 21, 2024

View reviewed changes

danielbate reviewed Aug 21, 2024

View reviewed changes

packages/abi-typegen/package.json Show resolved Hide resolved

arboleya mentioned this pull request Aug 21, 2024

fix: stop building typegen on pretest script #2990

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: adding `abi` transpiler #2856

feat!: adding `abi` transpiler #2856

arboleya commented Jul 29, 2024 •

edited

Loading

petertonysmith94 left a comment

danielbate left a comment

github-actions bot commented Aug 19, 2024

sdankel Aug 20, 2024

nedsalk Aug 21, 2024

sdankel Aug 21, 2024

nedsalk Aug 22, 2024

sdankel Aug 20, 2024 •

edited

Loading

nedsalk Aug 21, 2024

sdankel Aug 21, 2024

nedsalk Aug 22, 2024

sdankel Aug 20, 2024

nedsalk Aug 21, 2024

sdankel Aug 21, 2024

nedsalk Aug 21, 2024 •

edited

Loading

feat!: adding abi transpiler #2856

feat!: adding abi transpiler #2856

Conversation

arboleya commented Jul 29, 2024 • edited Loading

Release notes

Summary

Special thanks ❤️

Breaking Changes

New ABI spec

AbiCoder is no longer available to import

Interface.findTypeById has been removed

JsonAbiArgument is no longer exported

Checklist

petertonysmith94 left a comment

Choose a reason for hiding this comment

danielbate left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sdankel Aug 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nedsalk Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

feat!: adding `abi` transpiler #2856

feat!: adding `abi` transpiler #2856

arboleya commented Jul 29, 2024 •

edited

Loading

`AbiCoder` is no longer available to import

`Interface.findTypeById` has been removed

`JsonAbiArgument` is no longer exported

sdankel Aug 20, 2024 •

edited

Loading

nedsalk Aug 21, 2024 •

edited

Loading