Pass-by-Ref / Dynamic Scratch Variables via the loads and stores opcodes

[tzaffi]

Dynamic Scratch Variables

New Capabilities

This PR aims to allow dynamically-indexed scratch variables. For example here are some nice-to-have examples:

Pass-by-Reference Semantics

You can see the resulting teal (including my comments) here.

@Subroutine(TealType.none)
def swap(x: ScratchVar, y: ScratchVar):
    z = ScratchVar(TealType.anytype)
    return Seq(
        z.store(x.load()),
        x.store(y.load()),
        y.store(z.load()),
    )

Use a Dynamic ScratchVar to "iterate" over other ScratchVar's

You can see the resulting teal with comments here.

def wilt_the_stilt():
    player_score = DynamicScratchVar(TealType.uint64)

    wilt = ScratchVar(TealType.uint64, 129)
    kobe = ScratchVar(TealType.uint64)
    dt = ScratchVar(TealType.uint64, 131)

    return Seq(
        player_score.set_index(wilt),
        player_score.store(Int(100)),
        player_score.set_index(kobe),
        player_score.store(Int(81)),
        player_score.set_index(dt),
        player_score.store(Int(73)),
        Assert(player_score.load() == Int(73)),
        Assert(player_score.index() == Int(131)),
        player_score.set_index(wilt),
        Assert(player_score.load() == Int(100)),
        Assert(player_score.index() == Int(129)),
        Int(100),
    )

Other improvements

I've also introduced another way to run end-to-end tests. The script tests/pass_by_ref_test.py works as follows:

• iterates over a list of PyTEAL approval functions defined in the file
• compiles the program and saves into tests/teal with a file name that is dynamically generated from the approval program function name
• loads pre-compiled companion TEAL with an extra _expected suffix in the filename
• compares the just-compiled TEAL against the expected TEAL line by line. The expected TEAL can have extra information -such as comments- but they have to start out exactly the same for the test to pass

Testing

• Unit tests have been added to:
  • pyteal/ast/scratch_test.py
  • pyteal/ast/scratchvar_test.py
  • pyteal/ast/subroutine_test.py
• The end to end test tests/pass_by_ref_test.py has been added
• Temporary PR DO NOT MERGE: Test the current compiler against the future compilations #203 runs all the backwards compatible tests in tests/pass_by_ref_test.py to ensure that the generated TEAL hasn't changes due to this PR

Summary of Changes

• Various __init__*'s
  • To bubble up the visibility of the new DynamicScratchVar and make mypy happy
  • Also, modified scripts/generate_init.py now sorts and uniquifies __all__
• Resolve mypy complaints:
  • pyteal/ast/multi_test.py
• pyteal/ast/scratch.py:
  • ability to query a slot's index at runtime via the new ScratchIndex class and various index() methods
• pyteal/ast/scratchvar.py - new DynamicScratchIndex class that allows subroutines to access and modify external ScratchVar's and also allows it to act as an "iterator" over pre-existing ScratchVar's. It uses the loads and stores op codes in order to achieve its capabilities.
• pyteal/ast/subroutine.py - modify the to allow pass-by-reference semantics
  • SubroutineDefinition now:
    • has a by_ref_args member set so one can tell if an argument is meant to be by-reference or by-value
    • allows return types to be any subtype of Expr, not just Expr
    • allows argument types to be ScratchVar's in addition to the previous Expr
    • has a new arguments() method
  • SubroutineCall now can handle by-ref arguments, by detecting such arguments and calling their new .index() method to pass their index on the stack, instead of their value
  • SubroutineDeclaration now can handle by-ref arguments as well. THIS IS THE TRICKIEST PART OF THE PR. The code changes are deceptively small. There are some new comments which attempt to explain the goings on.
• New end-to-end test tests/pass_by_ref_test.py
  • Various expected outputs of compilation tests/teal/*_expected.py

[tzaffi]

This file changed mostly as a result of getting the generator to sort and uniquify __all__

[tzaffi]

sorting and uniquifying

[jasonpaulos]

Why is deduplicating necessary? I'd argue that's an actual problem with our exports and this script should error if a duplicate is detected.

[tzaffi]

Looking closer, it turns out no de-duping actually occurs. So I'll change the code to make an assertion instead.

[jasonpaulos]

Implementation looks good, I just have lingering questions

[jasonpaulos]

Why get rid of this test?

[tzaffi]

I moved this test case to test_subroutine_definition_invalid() because we are no longer allowing Expr subclasses for annotation return types.

[tzaffi]

I also renamed the function fnWithMixedAnns4AndBytesReturn (it was wrongly named before)

[jasonpaulos]

Is this print statement intentional?

[tzaffi]

Yes. Instead of catching the exception and re-raising with the case as I was doing before, now I'm just printing out each case. So if you run pytest -vv you can see the last case=XYZ that got printed and infer the broken case.

[jasonpaulos]

LGTM