Interoperability Design Patterns

[MicroProofs]

Design Patterns For an Interoperable Ecosystem

Primer Section on What Double Satisfaction is

TODO: For now here's a link. https://github.com/input-output-hk/plutus/blob/master/doc/read-the-docs-site/reference/writing-scripts/common-weaknesses/double-satisfaction.rst

Let's Start with the batching model

The batching model typically has a batch action to apply user orders to a pool. There is some level of interoperability where you can make a order that goes to some script address with a datum hash. But right now Dexes are limited to one pool validator with one pool input per transaction.

So how do we change that model to allow not just any number of pools, but also any number of defi apps also executing in the same transaction while also protecting every user from double satisfaction. The easiest way I would tackle it is ensure one output per input. Right but how do we make sure we aren't accidentally validating an output that is used by another contract?

The answer is to first find a unique output that is unique to our validator. And then
More TODO here. Going to come back to this. But the main idea is to use the script ref field as an additional guarantee of uniqueness without having to restrict datums.

(Script Ref trick works only in PlutusV3)

Onchain steps to get a unique script ref hash:
First take the hash of the spending output reference for uniqueness

Next shove that into a script via bytearray concat.

Given a 32 byte array which can always be made by doing serialize_data |> blake256hash
you add 5828010000488120 to the front and 0001 to the back of that bytearray to make a valid script.

(Also can be made 4 bytes smaller by using blake224 hash instead. I'll update that here later.)

Next hash that script using the new Blake224 builtin.

Then check for the specified output index that should contain a Some with the same hash.

[fallen-icarus]

You can use an NFT for this. Every pool gets a unique NFT identifier (Pool ID) that must be stored with the pool. The NFT name can be the hash of the output reference for the pool's UTxO - this guarantees uniqueness of the Pool IDs. Creating a new pool and minting the Pool ID can be done in a single step. Now, when a pool UTxO is consumed, the contract can just look for the output with the corresponding Pool ID. Since the Pool ID is guaranteed to be unique, double satisfaction is impossible. Each dApp can have its own Pool ID minting policy which further guarantees uniqueness of Pool IDs. Using Pool IDs, these pools can freely be composed together.

I use this technique extensively. IMO this is a better approach than using reference scripts since reference scripts will likely be larger than a single NFT. If composability is the goal, smaller UTxOs are better.

[fallen-icarus]

(Apologies for the long response but I care a lot about this topic.)

Why does Bob need to include a datum with the sale input reference in a datum when sending to Alice? Is that for uniqueness or could it be simplified to just a unit datum for example?

It is for uniqueness during the actual sale phase. A unit datum would not work.

...you could use the script ref trick instead allowing for Bob to potentially send to an Alice specified address/other dApp with a datum attached

You can already do this while still using the datums like in the previous example as long as the dApp is using a distributed architecture. There is actually no requirement that Bob must send the proceeds to Alice's marketplace address. Alice can specify in the Sale UTxO terms that the proceeds must go to another address. This address can be any address: pubkey, native script, or plutus script. Alice is explicitly consenting that the receiving address can accept the required datum. If accidental locking is too much of a concern, you can require users to use a personal proxy plutus script that can accept any datum and just delegates spending to the staking credential. Since the staking credential can be anything, including a plutus script, the proxy script can have arbitrary logic. If the receiving address is a pubkey address, the datum can already be anything.

From Bob's perspective, since he is directly interacting with the Sale UTxO without having to go through a middleman contract, he can trivially compose his output with other dApps in the same transaction. For example, Bob can swap DJED -> USDC using Carol's Swap UTxO, use that USDC to buy Option Contract A from Mike on the secondary market, execute the option contract for Asset X -> Asset Y at Sarah's Option's address, and finally borrow a loan using Asset Y as collateral. This can all occur in a single transaction since Bob does not need to wait for "payouts" after each step. It looks like this:

flowchart TD
    1[Bob's DJED] -- Swap DJED for USDC --> 2{Bob's USDC};
    1 --> 3{Pay Carol DJED};
    2 -- Buy Option A with USDC --> 4{Bob's Option Contract A};
    2 --> 5{Pay Mike USDC};
    4 -- Execute Option A to Receive Asset Y --> 6{Bob's Asset Y used as collateral};
    4 --> 7{Pay Sarah Asset X};

Loading

The above diagram is the desired interoperability, is it not?

For Alice's perspective, TBH I think the idea of having Bob send the proceeds directly to Alice's other dApp address with a certain datum creates a terrible UX. These are economic dApps and so terms are time dependent. Consider a Sale UTxO where Alice is asking for the proceeds to be sent directly to a loan address to be used as an Offer UTxO to Mike. The Offer UTxO has terms for how much collateral Mike needs to put up to borrow from Alice. Alice creates the Sale UTxO on Monday when the collateral asset's market rate is X. The sale does not go through until Wednesday but by then, the collateral asset's market rate has jumped 20%. Alice's Offer UTxO needs to quickly be changed before it is accepted in order to reflect the collateral's change in value. If Alice is too slow in changing the terms, she may be locked into a loan she actually doesn't want to be in. Basically, by having Bob pay directly to Alice's other dApp address, Alice needs to guess when the sale will actually go through and set the Offer terms according to what she believes the market rates will be at that time. This creates a lot of unnecessary risk for Alice. It is a much better UX to have Bob pay Alice directly when the Sale goes through, whenever it goes through, and Alice can then create the Offer UTxO with the terms that reflect the market rates at that time. This extra step (Alice must convert the proceeds into the Offer) only costs Alice 0.25 ADA in transaction fees (distributed dApps are about an order of magnitude cheaper than concentrated dApps since there is no batcher fee tacked on). So we are talking about increasing the risk for Alice just to save 0.25 ADA...

If it's for uniqueness you could use the script ref trick instead...

Using a reference script instead would make the situation worse. (I have not actually tried this but these are the problems I foresee.) Consider the following example, using the datum as previously described, where Bob purchases 3 NFTs from three different users: Alice, Carol, and Sarah. In this scenario, imagine all Sale UTxO terms have the proceeds going to the same address.
Tx Inputs:

Alice Sale UTxO(output_ref=123): NFT A for 10 ADA to address abc    
Carol Sale UTxO(output_ref=456): NFT B for 10 ADA to address abc    
Sarah Sale UTxO(output_ref=789): NFT C for 10 ADA to address abc    

Tx Outputs:

UTxO at address abc with 10 ADA and datum(output_ref=123)    
UTxO at address abc with 10 ADA and datum(output_ref=789)    

In this example, it is trivial for the validator to see that Carol's sale is unsatisfied (ie, it never found an output with the proper datum). Everything the validator needs to check the validity is present in the Sale UTxO currently being validated: the output reference, the receiving address (in datum), and required payment (in datum). In other words, the validation only requires local state.

Now imagine the same scenario where reference scripts were used for uniqueness. An important thing to point out is that a plutus script can only see a script hash, not the actual contents of the reference script. The plutus script can only go off of the hash to determine uniqueness. The outputs this time are:

UTxO at address abc with 10 ADA and scriptHash(randomHash1)    
UTxO at address abc with 10 ADA and scriptHash(randomHash2)    

How would the validator know that this transaction should fail while only using local state? I don't think it can since the script hash says absolutely nothing about what input corresponds to this output. Instead, the validator needs to know all of the sales in the transaction and make sure there is an output corresponding to all of them. If there are 3 inputs that each require 10 ADA at address abc, there needs to be three outputs to address abc with 10 ADA; any reference script attached to the outputs are irrelevant. In other words, the validator is now forced to used global state. You might be able to mitigate some of this by mapping input to reference script with redeemers but you still need to make sure each mapping is unique. So this approach still means each execution needs more information than just the input UTxO. And since validators are executed once per UTxO consumed, this expensive global logic will be redundantly executed. This expensive logic would absolutely kill performance of the protocol (ie, how many Sale UTxOs can be used in a single transaction). Likewise, it would kill how many protocols can be composed together.

Final Comment

AFAICT for concentrated dApps, the datums cannot be used like in the examples because the datums are used to track ownership of the proceeds for the payout phase. Since there is no payout phase with distributed dApps, the datum is free to be anything. Since interoperability is the topic of this discussion, I think the discussion is better aimed at distributed dApps vs concentrated dApps. Using the datum just isn't an issue for distributed dApps and using the datum is an extremely efficient way to prevent double satisfaction.

[MicroProofs]

Ok I think you are misunderstanding what I mean by script ref trick so let me explain and I believe it allows for unlimited datum flexibility while adding minimal script execution and size. Sorry I didn't explain earlier. I realized I never added it to the original post.

Script Ref trick works only in PlutusV3

Onchain steps
First take the hash of the spending output reference for uniqueness

Next shove that into a script via:

Given a 32 byte array which can always be made by doing serialize_data |> blake256hash
you add 5828010000488120 to the front and 0001 to the back of that bytearray to make a valid script.

(Also can be made 4 bytes smaller by using blake224 hash instead. I'll update that here later.)

Next hash that script using the new Blake224 builtin.

Then check for the specified output index that should contain a Some with the same hash.

Using this the datum no longer needs to be filled with uniqueness output ref which I find to be clunkier in some cases than allowing the datum to be anything and using script ref for uniqueness. There is a small trade off in cost but also the benefit of allowing the datum to be anything rather than requiring the datum to have a field with an output ref in it. This also saves you from having to cast the Datum from Data to some type.

An example where this could be useful is where Bob pays to the address that is specified by Alice and that happens to be the MinSwap script with the correct datum Alice chose in advance. A batcher can then pick that up and immediately spend it without worry about datum shape. I imagine the best would be supporting either datum or script ref uniqueness which would allow for maximum interoperability even with currently existing dApps.

[fallen-icarus]

Oh, interesting. I did misunderstand you since I didn't know that was going to be possible with PlutusV3.

An example where this could be useful is where Bob pays to the address that is specified by Alice and that happens to be the MinSwap script with the correct datum Alice chose in advance.

I agree that this would then become possible. There would be no need to use a proxy script. I am curious what that would do to throughput though since the datum would effectively be twice as large (a datum within a datum). Even without having to cast the internal datum to some type, that size alone could mean far fewer UTxOs would fit in the tx. I'd be curious to test it.

A batcher can then pick that up and immediately spend it without worry about datum shape. I imagine the best would be supporting either datum or script ref uniqueness which would allow for maximum interoperability even with currently existing dApps.

I'm skeptical about this part. Think back to my flow chart of Bob composing a dex, a secondary market, an options trading protocol, and a lending/borrowing protocol. This composition is trivial and trustless with distributed dApps. How would Bob accomplish the same thing using currently existing concentrated dApps even with the datum freed up? Since Bob cannot interact with the UTxOs directly (he has to go through batchers), how would Bob even convey that he wants this composition? Furthermore, since Bob will not sign (or even see) the ultimate transaction that composes them, how could this composition occur in a trust-less fashion? Is there even a solution for this? If yes, would it scale to all possible users? I am definitely biased since I exclusively use the distributed dApp architecture but I don't see how concentrated dApps can ever come close to the level of trust-less interoperability possible with distributed dApps. I don't think the datum issue is the only thing stopping interoperability of concentrated dApps.

[MicroProofs]

For that first example I know MinSwap allows for datum hash which means basically any size datum.

[fallen-icarus]

I didn't think of that... Thanks!

In case others are reading this and don't know how (I think) it works:

1. Alice can specify a datum hash - possibly with a flag to say whether it should be inlined.
2. When Bob goes to spend Alice's UTxO, he can look up the hash using something like koios to get the actual datum that corresponds to that hash.