Download smart contract into local chain

[ixje]

TLDR
Add support to download smart contracts from a remote server (e.g. MainNet) into your local private network for development purposes

Background story:
-I've discussed the idea with @roman-khimov on Discord, moving it here with more details-

One of the things we (COZ) have been running into a lot in the past few months is developing contracts that call other contracts (on MainNet) that are not under our control or even open sourced, and wanting to test the functionality. We did a PoC that clones the remote contract + full storage (using findstates), commits the contract state and storage KV pairs locally and then generate an SDK for the contract (similar to neo-go's contract generate-wrapper) to work with it. All good and turns out to be really useful, so we want to make it available for others. I've recently been working with Harry to get it included in Neo Express and I thought it would be great to see this supported in neo-go as well.

Considerations
During our PoC and Neoxpress implementation we ran into a couple of things I feel worth sharing (and perhaps we can align the implementations on how to deal with them). Not all might apply depending on how neo-go's storage is implemented, but that's for you to decide. I'll describe my experience from a NEO C# point of view :-)

1. How to deal with remote contract id's (and possible conflicts with local ones)
2. How to deal with contract overwriting
3. How to deal with contract child dependencies (e.g. remote contract internally calls another contract we don't know off)
4. How to deal with contract functionality tied to account storage

1. How to deal with remote contract id's

We want to make sure that the management contract Deploy() function keeps working, which tracks a last used contract id and takes the next value on deploy. That gives us these scenarios (and possible solutions)

1. remote contract id is <= to the last used local contract id -> Set the new contract id to the next available id according to the management contract
2. remote contract id is > to the last used local contract id -> New contract can be stored 1:1 + update the storage that tracks last used contract id to that of the remote contract id. Note; this means it is possible that contract id's are not continuous.

2. How to deal with contract overwriting

Again some scenario's to think about

1. Do we allow overwriting native contracts, just storage for native contracts or not at all? The PolicyContract is one of those contracts you'd like to be able to sync storage for e.g. if you're writing a contract that has max gas consumption requirements. Others likely do more damage than good (e.g. ManagementContract would destroy the contract list or NeoToken makes you lose development funds).
2. A remote contract is updated and the update might or might not affect storage. You'll likely want to have a choice whether to pull just the new contractstate, just the storage or both. The contract id is static if updated through the management contract Update() function. If we follow the approach of point 1.1 above we end up with a new id and possibly a duplicate contract in storage.

3. How to deal with contract child dependencies

This could be something to solve at a later stage, but a pulled contract might call System.Contract.Call itself and if we do not have the child dependencies we'll error.

My first thought was doing basic static analysis of the contract script searching for these calls. It quickly became obvious that the arguments to these syscalls are not always the last 4 things pushed onto the eval stack so we'd have to track stacks. That approach also quickly fails when you have contracts that dynamically set the external contracts they call like in the BurgerNEO contract linked here.

The last resort would be to detect the missing contract during runtime and download on the fly. Now the question becomes how do we differentiate between contracts that should be missing and those that are actual child dependencies and should be there. I currently don't think there is a way we can.

4. How to deal with contract functionality tied to account storage

Some contracts might expose functions like computeX that does not require account state or getY that is tied to account state you do not necessarily need control over (e.g. Token.Decimals()). Other functions are tied to account state that you do wish to control e.g. to do a token transfer.

Ideally we'd have a storage scheme for the contract so we can modify it as needed. I don't see this ending up in NEO core any time soon or ever. A 3rd party service (e.g. on Dora) could be used to accept (and share) a storage scheme for a contract if they can prove to be the owner. I think that we'll have to accept that there are scenario's we simply can't solve

[ixje]

If there are any concerns that I've missed please discuss here, even if you encounter them during implementation (assuming you will implement). I'll track this issue so I can potentially migrate things to the neoxpress side. Thanks!

[roman-khimov]

Our primary concern is state (account) mangling, but even without it there are use cases where it can useful. As for contract call dependencies, I don't think it could be solved 100%, but again it's not a problem for us, we know contracts we care about. All of the other things seem to be reasonably manageable.

[ixje]

Something I just realized (and don't have a solution for yet) that I thought was worth adding to the discussion is

1. needs to also work in multi-node setup (doable)

2. how will clients connected to the consensus node(s) know the storage updated? 🤔

   In an isolated development environment where you just send invoke transactions to the consensus node and listen to notifications (or rpc requests + response) you probably don't really care. However, if you're also developing some middle ware, e.g. by using a modified client, then you need to have the same local state as the consensus node. If the state of the consensus node is changed by directly modifying a snapshot and committing that, then connected clients won't know as there is no tx/block relayed that tells the client. At first sight I don't see a way to create a tx that can do the modifications because only the smart contract itself can modify it's own storage in a transaction.script (or we'd have a bigger problem in the sense of a security issue).

[roman-khimov]

1. I think we'll have to stop all nodes before doing anything to the storage. If we stop them, it's not an issue.
2. In this mode the chain itself becomes somewhat irrelevant, one can't get CN node state by replaying it, so maybe that's the way it is. But I can throw in a completely crazy idea wrt this --- chain replication with address replacement. Instead of taking the state you take a block with transactions in it and execute it normally except you change all addresses (signers/hash160 function inputs) according to some map. This map becomes a part of "protocol configuration" then, if every other node is configured the same it could then synchronize blocks normally (with all transactions/events) and get the same mangled state.

[ixje]

I slept on it and I think we might be able to solve the chain replication with a P2P plugin. We can put the contractstate + storage in a custom payload and wrap that in an ExtensiblePayload. If neo-go has a similar plugin system then all the non-consensus clients need is the plugin to process the custom payload.

[roman-khimov]

We have complete P2P state synchronization as an extension, but that is tied to MPT and MPT is constructed wrt to "normal" changes done to the system. But we can recalculate it of course if the node is down.