Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Storage.Add Interop (Neo 3) #814

Closed
igormcoelho opened this issue Jun 11, 2019 · 22 comments
Closed

Create Storage.Add Interop (Neo 3) #814

igormcoelho opened this issue Jun 11, 2019 · 22 comments
Labels
Discussion Initial issue state - proposed but not yet accepted

Comments

@igormcoelho
Copy link
Contributor

igormcoelho commented Jun 11, 2019

[EDIT:] The old proposal required another Storage flag, but now it doesn't need anything, except an interop method Storage.Add. Using it, and mempool value caches, verification transfers can be performed safely.

Example:

// .... during verification
Storage.Add("myaddress", -100);
Storage.Add("youraddress", +100);

This is supposed to work on verification, because values can be cached on mempool, and when Block is finally proposed, the "correct" value will be calculated following to tx deterministic order. So, if Tx A transfered something as network fee, and Tx B tries to do the same, this "double spending" won't be allowed on mempool time (not need to wait for block processing).

[EDIT:] old explanation using flags.
There's a nice solution for Neo native and user tokens, if we create the concept of Storage for balances. Usually, a Storage can hold any kind of information, and it could be read-only or read-write, depending on an attribute. We propose another attribute isBalance, that can also be used together with the const attribute.
The behavior of isBalance key is the following: it only accepts non-negative bignumbers; every change creates a an special notification that can be aggregated after contract execution.

Example:

// creating a storage balance
BigInteger funds = 0;
Storage.PutEx("myscripthash", funds, StorageAttribute.IsBalance);
// This automatically generates a notification `contract_hash + special balance flag + key + value_changed_up +funds`
// ...... NEP-5 transfer example 
BigInteger myfunds = Storage.Get("myaddress");
BigInteger yourfunds = Storage.Get("youraddress");
Storage.Put("myaddress", myfunds - value_transfer);
Storage.Put("youraddress", yourfunds + value_transfer);

This helps implementing network fees in verification time (will explain better on a longer proposal): neo-project/specification#3

@igormcoelho
Copy link
Contributor Author

This is a draft proposal, but its evolving. More deeply, perhaps this "balance" could be the combination of 3 attributes:

  • integer / cache
  • non-negative (integer property)
  • notify changes

If we build this way, perhaps it could be easier to understand.

The current behavior of storage is like 'volatile' in languages: all the time we need to read the only trusted state. This is needed for things like:

x += "ab"
x += "cd"
both done in parallel, whats the result? x="abcd" or X="cdab" ? We cannot know unless we order them (in block only).

But if we deal with integers only:
x += 3
x += 4

Result is 7, in any order.

So if we constrain a storage to int only, we already gain some things. This allows us to try to emulate non-volatile storage on mempool. So before reading the real value on storage, it may be reduced by other things on mempool (like netfee consuming resources from.a storage key).

Non-negative is a property that allows this storage to never be negative, so mempool caches may never spend more than existing value.

Notify changes is a storage property that just notify every change on that storage.

@vncoelho this int caching could also be used for mempool voting of consensus nodes.

@vncoelho
Copy link
Member

Exactly, @igormcoelho.

The int used for storages pre-balance will be a similar thing to the votes computation!
But we can have a rule to enable votes computation only if network has been stuck for X time in order to avoid this unnecessary calculation.

@igormcoelho
Copy link
Contributor Author

So, one thing we could do,is during Verification just allow storage read and writing on int/cache values. If I spend some network fee on verification, I'm actually spending it already on mempool cache for that key, what affects parallel tx using the same input address (thus preventing double spending on verification). The definitive value is only calculated after block is relayed and persisted, and after that this cache is dropped, and definitive storage values are guaranteed to be deterministic.

Network fees should be the first to be consumed on a block header,before processing the tx applications, as this guarantee that tx netfee will always be paid.

@vncoelho vncoelho added this to the NEO 3.0 milestone Jun 11, 2019
@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 11, 2019

In the case of payable contracts, it is also simple this way. You transfer extra money for it (which is deduced already on verification cache), and it consumes the necessary part and gives you back the change on Application trigger (if necessary). Automatic refunds can be easily done too.
This is useful for token minting,for example, where the contract may be perhaps invoked twice, but funds will be spent once only (the next invocation would fail).

@shargon

@vncoelho vncoelho added the Discussion Initial issue state - proposed but not yet accepted label Jun 11, 2019
@erikzhang
Copy link
Member

What is its usage scenario?

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 12, 2019

What is its usage scenario?

I think we can use it to provide storage-write during Verification @erikzhang , which is useful to guarantee that network fees will be paid, and consumed even if multiple tx are submitted in parallel. Regular storage cannot provide write access over verification, due to non-determinism, but we can solve non-determinism here by allowing caches to be updated on mempool directly.

The application for us is easy: we update Native NEP-5 with this IntCache storage flag, and only allow writes during verification if it has this flag. From verification perspective, what we do is creating a Map, that computes all changes made on mempool to storage keys of type IntCache, and when we read data, we read IntCache before reading the actual storage value. When block is persisted, we will drop these IntCaches (when removing tx from mempool), and persist the correct value on storage, given the tx order from the blocks. I'll try to draft a PR, so we can see this better on code...

@lock9
Copy link
Contributor

lock9 commented Jun 12, 2019

But what will happen in this scenario:
x *= 4
x += 2
?
The result may change if the order is not correct and we are only using integers

@igormcoelho
Copy link
Contributor Author

I get your point, this was just an example I gave to show how it would work for NEP-5 transfers... in fact, this would work for additive markovian functions, so multiplication would not be allowed. Good thing we don't need them on NEP-5 ;)

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 12, 2019

But in fact, you gave me an idea now... I was prototyping something like Storage.SetCache, but now I think it's better to write Storage.IncreaseBy and Storage.DecreaseBy. This way, only simple additive stuff is allowed. More general is: Storage.Add.

@lock9
Copy link
Contributor

lock9 commented Jun 12, 2019

Good idea! Now it looks safer :)

@igormcoelho igormcoelho changed the title Add Storage flag "Balance" (Neo 3) Add Storage.Add Interop Jun 12, 2019
@igormcoelho igormcoelho changed the title Add Storage.Add Interop Create Storage.Add Interop (Neo 3) Jun 12, 2019
@igormcoelho
Copy link
Contributor Author

@erikzhang perhaps the proposal now is more clear... just a Storage.Add interop is enough to do many nice things.

@erikzhang
Copy link
Member

I did not see how useful this is. We can implement network fees in verification perfectly with #791.

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 13, 2019

This allows (theoretically)removing all fee reverifications on.mempool, except by the last one when speaker proposes de block.
This is a complement to the change on #791.

The example I have in mind now is this Erik:

// NeoContract (perhaps NEP-5 Native GAS?)
object Main(string op, object[] args)
{
   if (TriggerType == Verification) // note that this can run on Verification
   if (op == "transfer") {
        byte[] from = args[0];
        byte[] to = args[1];
        BigInteger value = args[2];
        assert( Runtime.CheckWitness(from) );
        if(Storage.Get(from).AsBigInteger()>=value) {
              Storage.Add(from, -value);
              Storage.Add(to, value);
              return true;
        } 
   }
    return false;
};

This code above is supposed to run fine during Verification Trigger... The trick here is that we are not using Put, so theoretically it's possible to cache intermediate Add values on mempool, so that new transactions that arrive already know that Storage.Get will return a lower value than the one actually persisted. When it comes to the Block Persist itself, when tx are finally ordered, this verification takes place one last time and will actually cause a storage change after executed, thus finally consuming resources.

@erikzhang
Copy link
Member

We can optimize verifications without this. Just disallow all state read APIs in verification.

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 14, 2019

We can optimize verifications without this. Just disallow all state read APIs in verification.

I agree, but NEP-5 native gas will be an exception to this rule, right? I was trying to do it in a way that we wouldn't require any exception... If we allow only Storage.Get and Storage.Add we can do this. Mempool is non-deterministic by nature, so it can control itself until it gets ordered and executed deterministically on a block.

I don't know if this is necessary now, we can leave such thing for the future, but I'm just saying it's possible to do that.

@shargon
Copy link
Member

shargon commented Jun 14, 2019

I don't like this :S why we need to limit the storage access? We need to be able to read the storage in verification

@vncoelho
Copy link
Member

vncoelho commented Jun 14, 2019

We need to solve this puzzle before NEO 3.

Storage of balances I agree, Shargon.
The main discussion is the scalability of the mempool, we need Auxiliary Data Structures (ADS) for reverifying almost instantaneously. The ADS is what we designed with prebalance and limmitted states.

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 14, 2019

This proposal allows both Storage.Get and Storage.Add on verification @shargon. Only Put will be forbidden. The reason is that Add is order-independent, and Put is not.

@shargon
Copy link
Member

shargon commented Jun 14, 2019

But currently is not allowed put in verification.

@igormcoelho
Copy link
Contributor Author

igormcoelho commented Jun 14, 2019

But currently is not allowed put in verification.

Yes. And I hope it stays that way.. such nondeterministic creature 😂

@erikzhang
Copy link
Member

We don't need this. We can just disable all state dependent interop functions on verification. That's all we need to do. In this way, we don't need to reverify the transactions in mempool, we just reverify the fees, which doesn't need to execute the witness scripts.

@igormcoelho
Copy link
Contributor Author

Not feasible anymore... however, lead to nice improvements, such as garbage collection and dettached TState objects ;)

@erikzhang erikzhang removed this from the NEO 3.0 milestone Jun 21, 2019
Thacryba pushed a commit to simplitech/neo that referenced this issue Feb 17, 2020
Thacryba pushed a commit to simplitech/neo that referenced this issue Feb 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Discussion Initial issue state - proposed but not yet accepted
Projects
None yet
Development

No branches or pull requests

5 participants