CIP-0110? | Plutus v1 compatible script references

[Quantumplation]

Plutus v1 scripts are unable to use the CIP-0031 / CIP-0033 reference inputs / script references to reduce transaction size. As a result, between 80% and 90% of every block is occupied by repeating the same script bytes over and over.

This problem will not go away even as new protocols that do support Plutus v2 are released: these protocols exist forever, and in many cases, will hold hundreds of thousands in liquidity practically forever. As long as that is the case, they will represent an arbitrage opportunity that someone somewhere will want to take advantage of.

This proposal suggests a very simple (albeit perhaps aesthetically uncouth), backwards compatible way to support script references for Plutus v1 scripts.

Rendered

[Crypto2099]

Loving the intent here, but not sure if it remains optional if it's still a viable solution as a malicious user can always deploy and use/abuse a "traditional" Plutus v1 script to cause chain congestion and bloat the ledger size (although granted there is a cost to doing so and would be similar to spamming transactions with 15kb of gibberish in the metadata).

Obviously I don't know the impact this would have to the core node and ledger teams to make it all work, but I do agree that it is absolutely critical to solve this one and respect you bringing it up here as a topic of discussion. Hopefully we can hear from some of the core architects that can chime in on potential solutions.

[Quantumplation]

Loving the intent here, but not sure if it remains optional if it's still a viable solution as a malicious user can always deploy and use/abuse a "traditional" Plutus v1 script to cause chain congestion and bloat the ledger size (although granted there is a cost to doing so and would be similar to spamming transactions with 15kb of gibberish in the metadata).

The difference is that v1 versions of the protocol have a financial incentive to them that will create a very long tail of people swapping against them.

Obviously I don't know the impact this would have to the core node and ledger teams to make it all work, but I do agree that it is absolutely critical to solve this one and respect you bringing it up here as a topic of discussion. Hopefully we can hear from some of the core architects that can chime in on potential solutions.

Yes, of course if there is a concrete reason not to do this, we'll have to look for alternatives, but I've never been able to get an answer besides "someone somewhere said it would be dangerous" and "we wanted to apply a policy about the script context consistently"

[MicroProofs]

I also like the fact this CIP makes an effort to separate referenced scripts as inputs into the witness.

I've found it bizarre that the place for reference inputs' values and datums is also where we provide script refs that essentially act as witnesses. The script ref inputs almost always take up extra space in reference inputs and are ignored by plutus execution.

So it has uses for v2 and v3 going forward.

[Quantumplation]

@WhatisRT A birdie told me you in particular might be interested in this, and might be able to help me navigate the relevant updates to the ledger spec :D

[Quantumplation]

I think what would be required for the formal spec is:

1. Add txrefscripts (or maybe txwitrefscripts?) to the definition of TxWitness, with type ScriptHash -> TxIn
   
2. Extend refScripts to resolve the txwitrefscripts as well, something like replacing the inner clause here with utxo (spendInputs tx ∪ refInputs tx ∪ txwitrefscripts tx)
   

(Note: I'm not 100% sure whether txwitscripts and txscripts are intended to be same thing; I couldn't find where txwitscripts was defined)

[Quantumplation]

Our old friend @JaredCorduan helped me track down (roughly) where in the code this change would be made (at least for babbage):

https://github.com/IntersectMBO/cardano-ledger/blob/c1b7f597f750e7684ca73f96943820019b23ef17/eras/babbage/impl/src/Cardano/Ledger/Babbage/UTxO.hs#L137-L141

with the conway implementation currently just forwarding to the babbage implementation:

https://github.com/IntersectMBO/cardano-ledger/blob/c1b7f597f750e7684ca73f96943820019b23ef17/eras/conway/impl/src/Cardano/Ledger/Conway/UTxO.hs#L86C29-L86C29

[perturbing]

For brainstorming purposes: I wonder how this would be costed?

By introducing a way to reference from the witness set, node have to do extra calculation of retrieving the data.

Now for reference inputs, you pay for this, but you also gain that the full UTxO becomes available (so the address, datum, and value). Would your proposed solution be costed the same? And in both cases, how would this possible separation of cases impact the code?

[WhatisRT]

I have a bunch of things to say about this:

1. There is no big difference between this proposal and simply allowing reference inputs to be used in transactions that use V1 scripts and not showing them. Those two options only behave differently in transactions that use both V1 and V2 scripts. Allowing reference inputs in transactions with V1 scripts is trivial, and doesn't even require CDDL changes, so it can be done with an intra-era HF.
2. We have considered this when introducing Babbage, and decided against it. Here are some facts:
   • You can make a really stupid V1 script that can tell if it's being run via reference input: make a pretty large script, and then check if the fees are 'too low'.
   • This means we cannot guarantee that script behaviour of V1 scripts will be exactly the same with this feature as before. Note: I believe it's unlikely that there is a script out there that would actually behave differently in a meaningful way, but there certainly could be.

This means that introducing such a feature carries a certain risk of breaking existing scripts. At the time, we weren't willing to take that risk, and I stand by that decision. If however the entire Cardano community decided that they would be willing to take that risk I see no issue with implementing such a feature.

To address some other questions:

• @Quantumplation txwitscripts are the scripts that are included in txwits, while txscripts is defined in that figure. It looks like the Babbage spec is missing a note that txwitscripts is what used to be called txscripts in earlier specs.
• @perturbing It wouldn't be costed any different than costing reference inputs. We have recently discussed that in a different context and there will be some small changes to the fee structure because of that.

[Quantumplation]

I have a bunch of things to say about this:

1. There is no big difference between this proposal and simply allowing reference inputs to be used in transactions that use V1 scripts and not showing them. Those two options only behave differently in transactions that use both V1 and V2 scripts. Allowing reference inputs in transactions with V1 scripts is trivial, and doesn't even require CDDL changes, so it can be done with an intra-era HF.

I'd be totally fine with this, but my understanding was that we felt materially worse about "hiding" reference inputs, which may also include reference datums etc. Including them in the witness set was a way to morally satisfy the policy of not hiding things in the transaction body from the script context.

Personally, I feel like having them in the witness set is a lot cleaner; they're being included for the purpose of witnessing which script needs to be run. And to @MicroProofs's point above it has the extra benefit of allowing additional optimizations: when my plutus v2 script is scanning through the reference inputs looking for a particular datum, it isn't tripping over the script references.

That being said, I'm 100% ok with a tweak to this CIP that just allows reference inputs on a v1 transaction and is even simpler to adopt.

2. We have considered this when introducing Babbage, and decided against it. Here are some facts:

   • You can make a really stupid V1 script that can tell if it's being run via reference input: make a pretty large script, and then check if the fees are 'too low'.

How would this work in practice? The plutus v1 script has access to the fee paid, but not the minimum fee; and it doesn't have access to the witness set, so it can't calculate what the fee "should have been"; so the only way I can think to achieve this would be a script with some hard-coded heuristic about what the fee should be. Something like if feePaid < 13000000 + 12000 * length inputs.

• This means we cannot guarantee that script behaviour of V1 scripts will be exactly the same with this feature as before. Note: I believe it's unlikely that there is a script out there that would actually behave differently in a meaningful way, but there certainly could be.

This means that introducing such a feature carries a certain risk of breaking existing scripts. At the time, we weren't willing to take that risk, and I stand by that decision. If however the entire Cardano community decided that they would be willing to take that risk I see no issue with implementing such a feature.

To think through the possible outcomes here, there are only three possible outcomes to this change: a script that should be spendable becomes unspendable, and a script that should be unspendable becomes spendable, and a script maintains the same behavior, but with different execution units.

The first one isn't really a concern: if for some bizarre reason this was the case, users could continue to provide the script in the witness, and the semantics would remain unchanged, and they could continue to spend the UTXO like normal. Obviously not ideal, since the point is to reduce tx size, but also not impacting the vast majority of scripts (if any at all), so there's still a huge benefit to this change.

The second one is worth more careful thought, as releasing funds that should stay locked is bad... but I legitimately can't come up with an even semi-reasonable construct where this would happen. Someone would have to be checking explicitly if the fee was lower than some threshold, and using that to execute a different code path. Like you said, certainly possible, but it seems like it'd almost have to be designed with this change in mind 😅

The third one is not really worth worrying about. The outcome is the same, but the fee changes slightly.

If we're really worried about this, a messy solution would be to add the size of any plutusv1 scripts to the size of the transaction, regardless of where they're sourced from. The reduction in fees from saved space is way less important than the actual reduction in space.

To address some other questions:

• @Quantumplation txwitscripts are the scripts that are included in txwits, while txscripts is defined in that figure. It looks like the Babbage spec is missing a note that txwitscripts is what used to be called txscripts in earlier specs.

Ok, that's what I thought, thanks :)

Overally, it's hard to express how valuable some change like this would be (either the version I suggested, or the version you suggested), so I appreciate you having an open mind and talking through the different implications! :)

[AndrewWestberg]

Why not just change the on disk storage so it does a by-hash lookup. The bloat on the wire doesn't really matter then. Also does not require a hard fork.

We might see the same scripts in every block, but save them to disk once.

[AndrewWestberg]

I think that this cip should also be implemented, I just think the storage fix is a quick win.

[Quantumplation]

Why not just change the on disk storage so it does a by-hash lookup. The bloat on the wire doesn't really matter then. Also does not require a hard fork.

We might see the same scripts in every block, but save them to disk once.

I think that's an orthogonal and less urgent concern.

60gb of disk space a year is unpleasant, but tolerable. Certainly not the disk-guzzler that other chains are.

The reason it's so painful is because of the wire bloat. The block propagation times are super sensitive to block size because of the data sent over the wire.

So the point is to cut out the wire bloat. It's equivalent to raising our block size to 125kb without impacting propagation times, or something like that.

Also, the change to the on-disk format may not actually be easier. IIUC the immutable db stuff is some pretty gnarly haskell code, and I could see this ultimately being a huge change.

Ultimately I think that would be worthwhile as well, but the bulk of the benefit, and the minority of the effort, surprisingly.

[michaelpj]

I think:

• The policy of not omitting information when constructing the script context is a good one, and the risks from violating it are real, and not the sort of thing we should generally be doing with a system like this that promises very strong backwards compatibility.
• But the benefits in this case are very high.

So overall I think we should do it. Time has shown that we were wrong about how big a deal this would be.

[michaelpj]

I think we can do it more simply. We can just remove the existing ledger restriction that fails transactions that have reference inputs and V1 scripts (with a protocol version guard, obviously), and tweak the script context translation to omit them. No need to touch the transaction format at all.

This is a trivial change and we should definitely be able to do it for Conway.

(EDIT: I see I'm just repeating @WhatisRT :) )

[michaelpj]

Perhaps worth adding as a historical note: we thought when designing PlutusV2 that it was attractive enough that most traffic would move over one way or another. So in that respect we were simply wrong, and that's a good reason to change things.

[Quantumplation]

I think:

• The policy of not omitting information when constructing the script context is a good one, and the risks from violating it are real, and not the sort of thing we should generally be doing with a system like this that promises very strong backwards compatibility.
• But the benefits in this case are very high.

So overall I think we should do it. Time has shown that we were wrong about how big a deal this would be.

For sure! If I'm being honest, I think I have a bit of (hopefully understandable) frustration born of dealing with the impacts of this over the last year plus, and that frustration bleeds a little into the CIP 😅. But I've also always tried to say that I wasn't in the hot-seat and ultimately responsible for the decision, so there's a very big "benefit of the doubt" blanket that I apply. It's frustration with the state of affairs, not frustration with you or anyone on the team personally.

I agree that having a very strong policy of backwards compatibility is good, sensible, and even essential. I agree that omitting things from the transaction definition can have very real risks, and should be weighed carefully, and that the burden of justification is heavily on the proposer. So long as it doesn't turn into a blanket policy, applied unilaterally regardless of the magnitude of the real world benefits or the slimness of the risks on a case by case basis, then I'm happy.

With regards to the simpler approach you and WhatIsRT suggest, I'd be thrilled with that approach. My only reason for proposing this one is because I thought it was off the table, given the previous commitment to the aforementioned policy and discussions we've had in the past.

I'll make a new revision of this CIP today with the suggested updates.

[Quantumplation]

@michaelpj let me know if the latest draft is sufficient; it includes the historical note and a reworked rationale section that outlines the risks as well as the benefits. The specification section is pretty bare, as I'm not familiar with the exact details (ex: which parts of the spec would need to be updated, etc.), and I could use some recommendations on the "Implementation Plan" if you have a moment!

[lehins]

@Quantumplation Here is a relevant Ledger ticket, which contains some interesting analysis: IntersectMBO/cardano-ledger#3965

which parts of the spec would need to be updated

It would have to be Agda spec for Conway that would need to be updated. @WhatisRT is the person that can answer this. From what I know Alonzo related functionality has been merged recently, so it might be in a state that we can actually implement it in the spec. Although since Babbage functionality (that introduced reference scripts) is still not implemented we might not be able to reflect it in the spec yet.

Implementation Plan

Implementation for this is surprisingly simple. All we need to do is enable Reference scripts for PlutusV1. For example, this is the only thing that is preventing reference scripts being used in Babbage for PlutusV1:

  when (isSJust (txOut ^. referenceScriptTxOutL)) $
    Left $
      inject $
        ReferenceScriptsNotSupported txOutSource

https://github.com/IntersectMBO/cardano-ledger/blob/fdc366df654fc02b1668012342732d41eaa099fe/eras/babbage/impl/src/Cardano/Ledger/Babbage/TxInfo.hs#L94-L97

[rphair]

@Quantumplation you're all set for CIP number 110... please rename your CIP directory as well 🎉

[Quantumplation]

@rphair Thanks! Sorry, I didn't realize the meeting was yesterday or I would have tried to attend heh

[Quantumplation]

As per this comment by @lehins, implementation of this CIP has been officially added to the Conway hard-fork backlog!

IntersectMBO/cardano-ledger#3965 (comment)

[Ryun1]

This is a well written proposal with strong discussion from stakeholders, happy to see it merged.

[rphair]

Based on the importance of this change, identified in CIP meetings, as well as robust peer review I think this is ready to merge at the next meeting (in about 2 days' time) unless any major objections presented at that meeting (or here).

[lehins]

My review contains some questions about 60% number that I don't quite understand and the decision to only partial enabling of reference scripts.

[lehins]

My analysis showed that PlutusV1 scripts account for 41.3% and only 0.5% for PlutusV2 of the total size of blockchain since beginning of Alonzo, which means it is even lower that when taking Byron through Mary into consideration. So, 60% is definitely an overestimation. That being said 40% is still a lot.

Suggested change

	Despite making up less than half the transactions on Cardano, Plutus v1 scripts make up 90% of the space dedicated to scripts (around 60% of the total block space). Increasing the space available to blocks is risky, as it impacts the block propagation time. This proposal puts forth a simple way to reduce this strain.
	Despite making up less than half the transactions on Cardano, Plutus v1 scripts make up 90% of the space dedicated to scripts (around 40% of the total block space). Increasing the space available to blocks is risky, as it impacts the block propagation time. This proposal puts forth a simple way to reduce this strain.

[lehins]

"bytes taken up by constantly re-publishing the same Plutus v1 scripts is nearly 60% of each block"

I am probably misunderstanding this, but this does not make sense to me. How can PlutusV1 take up 60% of a block if in total it takes up 40% of the blockchain size since PlutusV1 was introduced?

[Quantumplation]

It's 40% of the full chain (i.e. of the 151gb of on-disk storage for a fully sync'd node, at the time this was taken, 60gb of that was redundant scripts).

However, looking at an average over the last X time window (I think 1 month?), SMAUG found that:

• considering only the space taken up by scripts, 90% of it was devoted to plutus v1 scripts
• considering the whole block size, around 60% was devoted to the plutus scripts.

So it's exactly as you said: considering the full chain history, it's around 40%, because of the time before smart contracts were even a thing; but on average, on a continuing basis, you can expect around 60% of each block to be taken up by a plutus v1 script.

These are just rough motivating numbers, and it may be closer to 50% or 70%, depending on which time frame you measure in; The point was to express the severity of the problem, and that it's not going away, not to give a high precision measurement.

[lehins]

However, looking at an average over the last X time window (I think 1 month?), SMAUG found that

So, that is the statistics over 1 month. That should either not be present in this CIP or made it absolutely clear that it is just a statistic from an arbitrary period of in the past.

it's around 40%, because of the time before smart contracts were even a thing; but on average, on a continuing basis, you can expect around 60% of each block to be taken up by a plutus v1 script.

First of all it is 41.3% since smart contracts where a thing. See my analysis.

Second of all, this math makes no sense. "40% of blockchain consists of PlutusV1 scripts" literally means on average every block will be 40% PlutusV1 scripts. You can't cherry pick a small period with higher usage and use it as an average.

So, could you please remove that 60% statistic from the CIP, since it is not only confusing but also does not serve as a good representative of the overall usage pattern.

[Quantumplation]

So, what are we trying to achieve here?

Your analysis mentioned that 60gb of overhead was due to redundant plutus v1 scripts; The current on-disk storage for all eras was 151gb (I don't have an updated view). 60gb is 39.7% of 151gb, or about 40%.

It feels like you're picking a fight over me using the phrase "about 40%", instead of an exact 41.3%? What does the extra significant digits there help achieve? Does it materially change the point, or the motivation for adopting the CIP?

As for the 60%, my wording clearly says end of 2023; I'm happy to make it clearer, but I think a statistic that talks about the existing, ongoing usage of the chain, which normalizes for recent trends, is an important view of the picture. The point being that the problem isn't going away: for example, it would be one thing if the first 3 months of Alonzo produced 59gb, and then in all the time since, we've produced an additional gigabyte.

I'm not cherry-picking a small period, I just picked a recent period, and the implication that I'm being deceptive or dishonest is insulting, when your own analysis supports the same conclusion: that this is a problem, and needs to be addressed.

[Quantumplation]

I reworked the explanations to hopefully make it clearer.

[lehins]

It feels like you're picking a fight over me using the phrase "about 40%", instead of an exact 41.3%?

No, not at all. ~40% is fine. Also I am not trying to pick a fight. I am trying to make sure we have an accurate picture here.

As for the 60%, my wording clearly says end of 2023

It is like saying: "Last December the average temperature was -10C, so we can conclude that for the next few years that should be the average temperature every month."
So, what I am saying is the 60% average is wrong. There are no numbers backing up this "average" usage statistic, while there are clear numbers showing that the number is approximately 40%. I hope it is more clear now what I am trying to say.

[lehins]

implication that I'm being deceptive or dishonest is insulting

Also, I am not implying that you are being deceptive. There is no need to get defensive, because no one is trying to attack you here. We are all on the same team. All I am trying to do is point out a flaw in the CIP.

So, just to make it really clear, I am trying to say that using a small sample to infer a bigger picture is wrong, especially when there is a clear and accurate analysis available. That's all.

[lehins]

Why do we need to fail on reference scripts in regular spending inputs?
Whenever a regular input includes a reference script it constitutes a usable reference script in that transaction, so it feels strange and inconsistent to disable just that functionality. Especially considering that the user will have to pay extra for spending that output starting with Conway, since it contains a reference script.

I would suggest we just translate the TxOut while omitting translation of the reference script. In other words it would look like a regular TxOut to PlutusV1 script.

I do agree about the inline datums, we should still fail translation on their presence in the inputs.

@michaelpj, @zliu41 and @Quantumplation any objections to allowing reference scripts in spending inputs?

[Quantumplation]

I'm fine with any relaxing of these rules; This part was written when I was unsure whether even the most gentle version of this was going to be accepted.

When I've raised this in the past, the response I got was that any hiding of information about the transaction when constructing the script context was considered dangerous and wouldn't be supported.

Hence, my original proposal was about including reference scripts in the witness set, which isn't directly visible to the script context.

It was quite a pleasant surprise when MPJ suggested that we just allow reference inputs like normal and leave them off the script context.

So, I have no objection to further relaxations if the ledger team themselves are ok with it, just understand that I was writing this from a posture of "what has the best chance of getting the ledger teams approval, and thus makes the minimal amount of semantic changes to plutus v1" 😅

[Quantumplation]

It's worth noting that I'm also fine with not relaxing this, as I can't think of any scenario where this buys us dApp developers any benefit. i.e. it's largely immaterial to me. Whereas the ability to satisfy a script witness via a reference input is hugely beneficial.

[lehins]

what has the best chance of getting the ledger teams approval

😄

You got our full approval on this CIP.

I personally would prefer to have consistency. So, if we are enabling reference scripts for PlutusV1, then they should work in the same way as they do for PlutusV2 and PlutusV3

[michaelpj]

Yeah, I appreciate the effort by @Quantumplation to make the minimal relaxation possible to allow the key usecase. Generally I do think that's the way to go.

However, I agree with @lehins that if we're going to allow reference scripts to be used in V1 we might as well put in the "whole feature". It's not much worse and otherwise it's a weird corner case.