Skip to content

Commit

Permalink
feat: implement "stateless" modules (#3663)
Browse files Browse the repository at this point in the history
this commit implements support for "stateless" modules in vyper.

this is the first major step in implementing vyper's module system.
it redesigns the language's import system, allows calling internal
functions from imported modules, allows for limited use of types from
imported modules, and introduces support for `.vyi` interface files.

note that the following features are left for future, follow-up work:
- modules with variables (constants, immutables or storage variables)
- full support for imported events (in that they do not get exported
  in the ABI)
- a system for exporting imported functions in the external interface
  of a contract

this commit first and foremost changes how imports are handled in vyper.

previously, an imported file was assumed to be an interface file. some
very limited validation was performed in `InterfaceT.from_ast`, but not
fully typechecked, and no code was generated for it.

now, when a file is imported, it is checked whether it is
  1. a `.vy` file
  2. a `.vyi` file
  3. a `.json` file

the `.json` pathway remains more or less unchanged. the `.vyi` pathway
is new, but it is fairly straightforward and is basically a "simple"
path through the `.vy` pathway which piggy-backs off the `.vy` analysis
to produce an `InterfaceT` object.

the `.vy` pathway now does full typechecking and analysis of the
imported module. some changes were made to support this:
- a new ImportGraph data structure tracks the position in the import
  graph and detects (and bands) import cycles
- InputBundles now implement a `_normalize_path()` method.
  this method normalizes the path so that source IDs are stable no
  matter how a file is accessed in the filesystem (i.e., no matter
  what the search path was at the time `load_file()` was called).
- CompilerInput now has a distinction between `resolved_path` and `path`
  (the original path that was asked for). this allows us to maintain UX
  considerations (showing unresolved paths etc) while still having a
  1:1:1 correspondence between source id, filepath and filesystem.

these changes were needed in order to stabilize notions like "which file
are we looking at?" no matter the way the file was accessed or how it
was imported. this is important so that types imported transitively
can resolve as expected no matter how they are imported - for instance,
`x.SomeType` and `a.x.SomeType` resolving to the same type.

the other changes needed to support code generation and analysis for
imported functions were fairly simple, and mostly involved generalizing
the analysis/code generation to type-based dispatch instead of AST-based
dispatch.

other changes to the language and compiler API include:
- import restrictions are more relaxed - `import x` is allowed now
  (previously, `import x as x` was required)
- change function labels in IR
  function labels are changed to disambiguate functions of the same name
  (but whose parent module are different). this was done by computing a
  unique function_id for every function and using that function_id when
  constructing its IR identifier.
- add compile_from_file_input which accepts a FileInput instead of a
  string. this is now the new preferred entry point into the compiler.
  its usage simplifies several internal APIs which expect to have
  `source_id` and `path` in addition to the raw source code.
- change compile_code api from contract_name= to contract_path=

additional changes to internal APIs and passes include:
- remove `remove_unused_statements()`
  the "unused statements" are now important to keep around for imports!
  in general, it is planned to remove both the AST expansion and
  constant folding passes as copying around the AST results in both
  performance and correctness problems
- abstract out a common exception rewriting pattern.
  instead of raising `exception.with_annotation(node)` -- just catch-all
  in the parent implementation and then don't have to worry about it at
  the exception site.
- rename "type" metadata key on most top-level declarators to more
  specific names (e.g. "func_type", "getter_type", etc).
- remove dead package pytest-rerunfailures
  use of `--reruns` was removed in c913b2d
- refactor: move `parse_*` functions, remove vyper.ast.annotation
  move `parse_*` functions into new module vyper.ast.parse and merge in
  vyper.ast.annotation
- rename the old `GlobalContext` class to `ModuleT`
- refactor: move InterfaceT into `vyper/semantics/types/module.py`
  it makes more sense here since it is closely coupled with `ModuleT`.
  • Loading branch information
charles-cooper authored Dec 16, 2023
1 parent 919080e commit c6f457a
Show file tree
Hide file tree
Showing 75 changed files with 2,546 additions and 1,397 deletions.
1 change: 0 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@
"pytest-instafail>=0.4,<1.0",
"pytest-xdist>=2.5,<3.0",
"pytest-split>=0.7.0,<1.0",
"pytest-rerunfailures>=10.2,<11",
"eth-tester[py-evm]>=0.9.0b1,<0.10",
"py-evm>=0.7.0a1,<0.8",
"web3==6.0.0",
Expand Down
10 changes: 9 additions & 1 deletion tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
from vyper import compiler
from vyper.ast.grammar import parse_vyper_source
from vyper.codegen.ir_node import IRnode
from vyper.compiler.input_bundle import FilesystemInputBundle
from vyper.compiler.input_bundle import FilesystemInputBundle, InputBundle
from vyper.compiler.settings import OptimizationLevel, Settings, _set_debug_mode
from vyper.ir import compile_ir, optimizer

Expand Down Expand Up @@ -103,6 +103,12 @@ def fn(sources_dict):
return fn


# for tests which just need an input bundle, doesn't matter what it is
@pytest.fixture
def dummy_input_bundle():
return InputBundle([])


# TODO: remove me, this is just string.encode("utf-8").ljust()
# only used in test_logging.py.
@pytest.fixture
Expand Down Expand Up @@ -255,9 +261,11 @@ def ir_compiler(ir, *args, **kwargs):
ir = IRnode.from_list(ir)
if optimize != OptimizationLevel.NONE:
ir = optimizer.optimize(ir)

bytecode, _ = compile_ir.assembly_to_evm(
compile_ir.compile_to_assembly(ir, optimize=optimize)
)

abi = kwargs.get("abi") or []
c = w3.eth.contract(abi=abi, bytecode=bytecode)
deploy_transaction = c.constructor()
Expand Down
2 changes: 1 addition & 1 deletion tests/functional/codegen/test_call_graph_stability.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ def foo():

# check the .called_functions data structure on foo() directly
foo = t.vyper_module_folded.get_children(vy_ast.FunctionDef, filters={"name": "foo"})[0]
foo_t = foo._metadata["type"]
foo_t = foo._metadata["func_type"]
assert [f.name for f in foo_t.called_functions] == func_names

# now for sanity, ensure the order that the function definitions appear
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
from vyper.compiler import compile_code
from vyper.exceptions import (
ArgumentException,
DuplicateImport,
InterfaceViolation,
NamespaceCollision,
StructureException,
)


Expand All @@ -31,7 +31,7 @@ def allowance(_owner: address, _spender: address) -> (uint256, uint256):

out = compile_code(code, output_formats=["interface"])
out = out["interface"]
code_pass = "\n".join(code.split("\n")[:-2] + [" pass"]) # replace with a pass statement.
code_pass = "\n".join(code.split("\n")[:-2] + [" ..."]) # replace with a pass statement.

assert code_pass.strip() == out.strip()

Expand Down Expand Up @@ -60,7 +60,7 @@ def allowance(_owner: address, _spender: address) -> (uint256, uint256): view
def test(_owner: address): nonpayable
"""

out = compile_code(code, contract_name="One.vy", output_formats=["external_interface"])[
out = compile_code(code, contract_path="One.vy", output_formats=["external_interface"])[
"external_interface"
]

Expand All @@ -85,14 +85,14 @@ def test_external_interface_parsing(make_input_bundle, assert_compile_failed):
interface_code = """
@external
def foo() -> uint256:
pass
...
@external
def bar() -> uint256:
pass
...
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

code = """
import a as FooBarInterface
Expand Down Expand Up @@ -121,9 +121,8 @@ def foo() -> uint256:
"""

assert_compile_failed(
lambda: compile_code(not_implemented_code, input_bundle=input_bundle), InterfaceViolation
)
with pytest.raises(InterfaceViolation):
compile_code(not_implemented_code, input_bundle=input_bundle)


def test_missing_event(make_input_bundle, assert_compile_failed):
Expand All @@ -132,7 +131,7 @@ def test_missing_event(make_input_bundle, assert_compile_failed):
a: uint256
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

not_implemented_code = """
import a as FooBarInterface
Expand All @@ -156,7 +155,7 @@ def test_malformed_event(make_input_bundle, assert_compile_failed):
a: uint256
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

not_implemented_code = """
import a as FooBarInterface
Expand All @@ -183,7 +182,7 @@ def test_malformed_events_indexed(make_input_bundle, assert_compile_failed):
a: uint256
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

not_implemented_code = """
import a as FooBarInterface
Expand Down Expand Up @@ -211,7 +210,7 @@ def test_malformed_events_indexed2(make_input_bundle, assert_compile_failed):
a: indexed(uint256)
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

not_implemented_code = """
import a as FooBarInterface
Expand All @@ -234,13 +233,13 @@ def bar() -> uint256:

VALID_IMPORT_CODE = [
# import statement, import path without suffix
("import a as Foo", "a.vy"),
("import b.a as Foo", "b/a.vy"),
("import Foo as Foo", "Foo.vy"),
("from a import Foo", "a/Foo.vy"),
("from b.a import Foo", "b/a/Foo.vy"),
("from .a import Foo", "./a/Foo.vy"),
("from ..a import Foo", "../a/Foo.vy"),
("import a as Foo", "a.vyi"),
("import b.a as Foo", "b/a.vyi"),
("import Foo as Foo", "Foo.vyi"),
("from a import Foo", "a/Foo.vyi"),
("from b.a import Foo", "b/a/Foo.vyi"),
("from .a import Foo", "./a/Foo.vyi"),
("from ..a import Foo", "../a/Foo.vyi"),
]


Expand All @@ -252,61 +251,79 @@ def test_extract_file_interface_imports(code, filename, make_input_bundle):


BAD_IMPORT_CODE = [
("import a", StructureException), # must alias absolute imports
("import a as A\nimport a as A", NamespaceCollision),
("import a as A\nimport a as A", DuplicateImport),
("import a as A\nimport a as a", DuplicateImport),
("from . import a\nimport a as a", DuplicateImport),
("import a as a\nfrom . import a", DuplicateImport),
("from b import a\nfrom . import a", NamespaceCollision),
("from . import a\nimport a as a", NamespaceCollision),
("import a as a\nfrom . import a", NamespaceCollision),
("import a\nimport c as a", NamespaceCollision),
]


@pytest.mark.parametrize("code,exception_type", BAD_IMPORT_CODE)
def test_extract_file_interface_imports_raises(
code, exception_type, assert_compile_failed, make_input_bundle
):
input_bundle = make_input_bundle({"a.vy": "", "b/a.vy": ""}) # dummy
assert_compile_failed(lambda: compile_code(code, input_bundle=input_bundle), exception_type)
input_bundle = make_input_bundle({"a.vyi": "", "b/a.vyi": "", "c.vyi": ""})
with pytest.raises(exception_type):
compile_code(code, input_bundle=input_bundle)


def test_external_call_to_interface(w3, get_contract, make_input_bundle):
token_interface = """
@view
@external
def balanceOf(addr: address) -> uint256:
...
@external
def transfer(to: address, amount: uint256):
...
"""

token_code = """
import itoken as IToken
implements: IToken
balanceOf: public(HashMap[address, uint256])
@external
def transfer(to: address, _value: uint256):
self.balanceOf[to] += _value
def transfer(to: address, amount: uint256):
self.balanceOf[to] += amount
"""

input_bundle = make_input_bundle({"one.vy": token_code})
input_bundle = make_input_bundle({"token.vy": token_code, "itoken.vyi": token_interface})

code = """
import one as TokenCode
import itoken as IToken
interface EPI:
def test() -> uint256: view
token_address: TokenCode
token_address: IToken
@external
def __init__(_token_address: address):
self.token_address = TokenCode(_token_address)
self.token_address = IToken(_token_address)
@external
def test():
self.token_address.transfer(msg.sender, 1000)
"""

erc20 = get_contract(token_code)
test_c = get_contract(code, *[erc20.address], input_bundle=input_bundle)
token = get_contract(token_code, input_bundle=input_bundle)

test_c = get_contract(code, *[token.address], input_bundle=input_bundle)

sender = w3.eth.accounts[0]
assert erc20.balanceOf(sender) == 0
assert token.balanceOf(sender) == 0

test_c.test(transact={})
assert erc20.balanceOf(sender) == 1000
assert token.balanceOf(sender) == 1000


@pytest.mark.parametrize(
Expand All @@ -320,26 +337,36 @@ def test():
],
)
def test_external_call_to_interface_kwarg(get_contract, kwarg, typ, expected, make_input_bundle):
code_a = f"""
interface_code = f"""
@external
@view
def foo(_max: {typ} = {kwarg}) -> {typ}:
...
"""
code1 = f"""
import one as IContract
implements: IContract
@external
@view
def foo(_max: {typ} = {kwarg}) -> {typ}:
return _max
"""

input_bundle = make_input_bundle({"one.vy": code_a})
input_bundle = make_input_bundle({"one.vyi": interface_code})

code_b = f"""
import one as ContractA
code2 = f"""
import one as IContract
@external
@view
def bar(a_address: address) -> {typ}:
return ContractA(a_address).foo()
return IContract(a_address).foo()
"""

contract_a = get_contract(code_a)
contract_b = get_contract(code_b, *[contract_a.address], input_bundle=input_bundle)
contract_a = get_contract(code1, input_bundle=input_bundle)
contract_b = get_contract(code2, *[contract_a.address], input_bundle=input_bundle)

assert contract_b.bar(contract_a.address) == expected

Expand All @@ -349,8 +376,8 @@ def test_external_call_to_builtin_interface(w3, get_contract):
balanceOf: public(HashMap[address, uint256])
@external
def transfer(to: address, _value: uint256) -> bool:
self.balanceOf[to] += _value
def transfer(to: address, amount: uint256) -> bool:
self.balanceOf[to] += amount
return True
"""

Expand Down Expand Up @@ -510,14 +537,14 @@ def returns_Bytes3() -> Bytes[3]:
"""

should_not_compile = """
import BadJSONInterface as BadJSONInterface
import BadJSONInterface
@external
def foo(x: BadJSONInterface) -> Bytes[2]:
return slice(x.returns_Bytes3(), 0, 2)
"""

code = """
import BadJSONInterface as BadJSONInterface
import BadJSONInterface
foo: BadJSONInterface
Expand Down Expand Up @@ -578,10 +605,10 @@ def balanceOf(owner: address) -> uint256:
@external
@view
def balanceOf(owner: address) -> uint256:
pass
...
"""

input_bundle = make_input_bundle({"balanceof.vy": interface_code})
input_bundle = make_input_bundle({"balanceof.vyi": interface_code})

c = get_contract(code, input_bundle=input_bundle)

Expand All @@ -592,7 +619,7 @@ def test_simple_implements(make_input_bundle):
interface_code = """
@external
def foo() -> uint256:
pass
...
"""

code = """
Expand All @@ -605,7 +632,7 @@ def foo() -> uint256:
return 1
"""

input_bundle = make_input_bundle({"a.vy": interface_code})
input_bundle = make_input_bundle({"a.vyi": interface_code})

assert compile_code(code, input_bundle=input_bundle) is not None

Expand Down
2 changes: 1 addition & 1 deletion tests/functional/codegen/test_selector_table_stability.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def test_dense_jumptable_stability():

# test that the selector table data is stable across different runs
# (tox should provide different PYTHONHASHSEEDs).
expected_asm = """{ DATA _sym_BUCKET_HEADERS b'\\x0bB' _sym_bucket_0 b'\\n' b'+\\x8d' _sym_bucket_1 b'\\x0c' b'\\x00\\x85' _sym_bucket_2 b'\\x08' } { DATA _sym_bucket_1 b'\\xd8\\xee\\xa1\\xe8' _sym_external_foo6___3639517672 b'\\x05' b'\\xd2\\x9e\\xe0\\xf9' _sym_external_foo0___3533627641 b'\\x05' b'\\x05\\xf1\\xe0_' _sym_external_foo2___99737695 b'\\x05' b'\\x91\\t\\xb4{' _sym_external_foo23___2433332347 b'\\x05' b'np3\\x7f' _sym_external_foo11___1852846975 b'\\x05' b'&\\xf5\\x96\\xf9' _sym_external_foo13___653629177 b'\\x05' b'\\x04ga\\xeb' _sym_external_foo14___73884139 b'\\x05' b'\\x89\\x06\\xad\\xc6' _sym_external_foo17___2298916294 b'\\x05' b'\\xe4%\\xac\\xd1' _sym_external_foo4___3827674321 b'\\x05' b'yj\\x01\\xac' _sym_external_foo7___2036990380 b'\\x05' b'\\xf1\\xe6K\\xe5' _sym_external_foo29___4058401765 b'\\x05' b'\\xd2\\x89X\\xb8' _sym_external_foo3___3532216504 b'\\x05' } { DATA _sym_bucket_2 b'\\x06p\\xffj' _sym_external_foo25___108068714 b'\\x05' b'\\x964\\x99I' _sym_external_foo24___2520029513 b'\\x05' b's\\x81\\xe7\\xc1' _sym_external_foo10___1937893313 b'\\x05' b'\\x85\\xad\\xc11' _sym_external_foo28___2242756913 b'\\x05' b'\\xfa"\\xb1\\xed' _sym_external_foo5___4196577773 b'\\x05' b'A\\xe7[\\x05' _sym_external_foo22___1105681157 b'\\x05' b'\\xd3\\x89U\\xe8' _sym_external_foo1___3548993000 b'\\x05' b'hL\\xf8\\xf3' _sym_external_foo20___1749874931 b'\\x05' } { DATA _sym_bucket_0 b'\\xee\\xd9\\x1d\\xe3' _sym_external_foo9___4007206371 b'\\x05' b'a\\xbc\\x1ch' _sym_external_foo16___1639717992 b'\\x05' b'\\xd3*\\xa7\\x0c' _sym_external_foo21___3542787852 b'\\x05' b'\\x18iG\\xd9' _sym_external_foo19___409552857 b'\\x05' b'\\n\\xf1\\xf9\\x7f' _sym_external_foo18___183630207 b'\\x05' b')\\xda\\xd7`' _sym_external_foo27___702207840 b'\\x05' b'2\\xf6\\xaa\\xda' _sym_external_foo12___855026394 b'\\x05' b'\\xbe\\xb5\\x05\\xf5' _sym_external_foo15___3199534581 b'\\x05' b'\\xfc\\xa7_\\xe6' _sym_external_foo8___4238827494 b'\\x05' b'\\x1b\\x12C8' _sym_external_foo26___454181688 b'\\x05' } }""" # noqa: E501
expected_asm = """{ DATA _sym_BUCKET_HEADERS b\'\\x0bB\' _sym_bucket_0 b\'\\n\' b\'+\\x8d\' _sym_bucket_1 b\'\\x0c\' b\'\\x00\\x85\' _sym_bucket_2 b\'\\x08\' } { DATA _sym_bucket_1 b\'\\xd8\\xee\\xa1\\xe8\' _sym_external 6 foo6()3639517672 b\'\\x05\' b\'\\xd2\\x9e\\xe0\\xf9\' _sym_external 0 foo0()3533627641 b\'\\x05\' b\'\\x05\\xf1\\xe0_\' _sym_external 2 foo2()99737695 b\'\\x05\' b\'\\x91\\t\\xb4{\' _sym_external 23 foo23()2433332347 b\'\\x05\' b\'np3\\x7f\' _sym_external 11 foo11()1852846975 b\'\\x05\' b\'&\\xf5\\x96\\xf9\' _sym_external 13 foo13()653629177 b\'\\x05\' b\'\\x04ga\\xeb\' _sym_external 14 foo14()73884139 b\'\\x05\' b\'\\x89\\x06\\xad\\xc6\' _sym_external 17 foo17()2298916294 b\'\\x05\' b\'\\xe4%\\xac\\xd1\' _sym_external 4 foo4()3827674321 b\'\\x05\' b\'yj\\x01\\xac\' _sym_external 7 foo7()2036990380 b\'\\x05\' b\'\\xf1\\xe6K\\xe5\' _sym_external 29 foo29()4058401765 b\'\\x05\' b\'\\xd2\\x89X\\xb8\' _sym_external 3 foo3()3532216504 b\'\\x05\' } { DATA _sym_bucket_2 b\'\\x06p\\xffj\' _sym_external 25 foo25()108068714 b\'\\x05\' b\'\\x964\\x99I\' _sym_external 24 foo24()2520029513 b\'\\x05\' b\'s\\x81\\xe7\\xc1\' _sym_external 10 foo10()1937893313 b\'\\x05\' b\'\\x85\\xad\\xc11\' _sym_external 28 foo28()2242756913 b\'\\x05\' b\'\\xfa"\\xb1\\xed\' _sym_external 5 foo5()4196577773 b\'\\x05\' b\'A\\xe7[\\x05\' _sym_external 22 foo22()1105681157 b\'\\x05\' b\'\\xd3\\x89U\\xe8\' _sym_external 1 foo1()3548993000 b\'\\x05\' b\'hL\\xf8\\xf3\' _sym_external 20 foo20()1749874931 b\'\\x05\' } { DATA _sym_bucket_0 b\'\\xee\\xd9\\x1d\\xe3\' _sym_external 9 foo9()4007206371 b\'\\x05\' b\'a\\xbc\\x1ch\' _sym_external 16 foo16()1639717992 b\'\\x05\' b\'\\xd3*\\xa7\\x0c\' _sym_external 21 foo21()3542787852 b\'\\x05\' b\'\\x18iG\\xd9\' _sym_external 19 foo19()409552857 b\'\\x05\' b\'\\n\\xf1\\xf9\\x7f\' _sym_external 18 foo18()183630207 b\'\\x05\' b\')\\xda\\xd7`\' _sym_external 27 foo27()702207840 b\'\\x05\' b\'2\\xf6\\xaa\\xda\' _sym_external 12 foo12()855026394 b\'\\x05\' b\'\\xbe\\xb5\\x05\\xf5\' _sym_external 15 foo15()3199534581 b\'\\x05\' b\'\\xfc\\xa7_\\xe6\' _sym_external 8 foo8()4238827494 b\'\\x05\' b\'\\x1b\\x12C8\' _sym_external 26 foo26()454181688 b\'\\x05\' } }""" # noqa: E501
assert expected_asm in output["asm"]


Expand Down
Loading

0 comments on commit c6f457a

Please sign in to comment.