Review and response to gas impacts of FIP-0045

[anorth]

The implementation of FIP-0045—decoupling the verified registry from markets—will activate on the network in nv17, on Nov 30th at epoch 2383680. This upgrade brings many near- and long-term benefits to storage providers, the storage market, and to the network. But one less desirable effect is an increase in the gas costs associated with Filecoin Plus verified deals.

As now documented in FIP-0045, the increase is about 70M gas per deal when publishing, and a little less when subsequently submitting aggregated sector proofs or replica update (or, borne by the network cron, when submitting single-sector proofs). This is an increase of ~80% over the present gas costs of publishing a verified deal, or 30% over a replica update. The relative increase is larger for batch submissions, the per-sector cost being about constant.

Higher gas costs were not a goal of this work. These changes lay foundations for more gas-efficient markets in the future, but present tough economic conditions understandably cause storage providers to be very cost conscious now. We under-estimated the gas increase during development and thus did not devote sufficient attention to reducing it.

This is ultimately my responsibility as the lead fo these changes. I and the engineers involved have recently conducted a post-mortem on the events leading to this. A summary of notes is presented here, along with plans to both address these specific gas costs and prevent similar occurrences in the future.

Timeline

A brief timeline of the development of FIP-0045. Note that dates are generally +/- 1 day due to timezone differences between contributors.

• August 15: The initial FIP draft notes some gas increase for the new on-chain accounting but it’s not quantified.
• Aug through Sep: FIP-0045 code and spec developed by @anorth and @ZenGround0.
• August 19: initial discussions of potential gas impact; @anorth notes it’s very difficult to quantity until implementation is complete.
• August 21: Built-in actors issue #566 issue opened for gas analysis.
• September 9: FIP-0045 submitted for Last Call.
• September 19: Additional discussion among Lotus team.
• September 23: Last Call ends - FIP is accepted.
• September 28: FIP 0045 implementation basically complete, merged into master.
• October 3: Discussion about what method to use to measure gas impact.
• October 6: FIP-0045 state migration is implemented (though some bug fixes remained).
• October 16: @Kubuxu takes the gas analysis task.
• October 21: @Kubuxu did the first pass of tooling setup but ran into bugs in the state migration that ultimately required Calibration net reset.
• October 24–November 4: PL Lab Weeks events usurp most engineering work.
• November 7: @geoff-vball takes over gas analysis and starts work.
• November 15: Gas measurements published into FIP-0045.

On this timeline, and with current FVM tooling, measuring the gas impact of these changes became possible on October 6, when the migration implementation was completed. The method we do have for measurement involves performing a mock upgrade on mainnet state in order to execute the new code (see below).

With somewhat better tooling, we could have performed a fair analysis at September 28, when the implementation was complete. But with better tooling, we could have additionally gained data points and directed optimization efforts sooner still.

Discussion

Aggregated notes from post-mortem discussion between @anorth, @ZenGround0, @arajasek, @Kubuxu, @jennijuju.

• FVM tooling for gas analysis is significantly weaker than that previously available
  • With the prior fake VM, execution was not metered, which made the problem a little easier. Actors developers could instrument all of the gas-charging actions during development with fake VMs and test harnesses, with no external dependencies (example).
  • Lotus also had great tooling to produce execution traces of mainnet messages, with down to the line of code and call-stack where each gas charge happened.
  • This tooling was useful for performing optimization and tuning parameters.
  • That tooling didn’t evolve with the introduction of FVM in nv16. FVM did not provide similar instrumentation and tracing functionality for Lotus, nor ergonomic harnesses for actor developers.
  • This loss of developer functionality for M1 was acknowledged at the time, but restoring tooling took a lower priority to shorter timelines and a focus on third-party developer adoption (FEVM).
  • Some discussions about this and related topics include:
    • March 30: FVM #514 a harness for benchmarking gas usage
    • April 2: FVM #439 to capture WASM traces for gas
    • August 16: Requests for testing/debugging sandbox to be prioritized for FVM Native Developer Toolbox (PL-internal document)
    • September 20: Built-in actors #678: How to do integration testing
• At present, our only methods for measuring gas involve using Lotus. These weren’t possible to execute until after the full migration was implemented.
  • If Lotus had better migration testing tooling, we might have had the migration integration done sooner, which would have enabled gas analysis a bit sooner as well.
• Uncertainty about governance means implementers often want to have FIP accepted, or close to it, before completing the implementation. Acceptance of a FIP provides the mandate to implementers to do the work.
  • In this case we developed both in parallel since we expected the implementation to provide information relevant to the specification.
  • Writing all the code for a complex change without some governance indication is a difficult proposition for developers who are very busy.
  • SnapDeals is another example here: there was months of work to do after FIP acceptance.
  • The FIP process is unclear here. FIP-0001 states that acceptance pushes a FIP over to core devs to implement and deploy. But part of the FIP template does suggest linking to an implementation.
• For FIPs with clear gas implications, it might be reasonable for analysis of the impacts to be a hurdle to FIP acceptance. That would mean waiting until implementation is complete and integrated for final acceptance.
  • Stakeholders may need to see a actual numbers to make an assessment
  • In this case, such a hurdle would probably have made batching optimizations a must-have, and potentially prompted additional optimization work.
  • This would introduce more uncertainty and delay to implementation timelines, since the need for additional work might not be discovered until implementation is otherwise complete (which is often at the last possible moment to meet planned upgrade timelines).
• With appropriate tooling, estimating costs with constructed scenarios (hand-crafted tests) would provide a lot of information and inform optimizations during development. But it would still be necessary to confirm with mainnet state afterwards, which requires implementation to be complete, along with any migrations.
• This might have in fact been good workflow. If we had published gas numbers earlier, the controversy of a gas increase could have derailed the FIP. We roughly understood that there would be some gas increase and that it was worth the price for this functionality. There was not a world where we would have wanted to block this FIP, though maybe we would have worked harder on optimization.

Actions

A set of actions we intend to undertake to mitigate this issue and prevent its recurrence.

• Land batching optimizations contained in #722. Note that we don’t know the the gas impact of this, due to aforementioned lack of tooling.
• Open a discussion with the Filecoin Core Devs about the sequencing of FIP drafting and acceptance with implementation efforts, how to account for potential gas impacts while retaining developer confidence about the governance acceptance of major changes.
• We will build the FVM tooling that we need to keep developing major changes to built-in actors. It seems that many of the pieces exist, and what’s missing is ergonomic APIs, a bit of plumbing, documentation, and examples or templates to guide an actor developer. We (actors developers) will request the FVM team’s support, but take ownership.
• After appropriate tooling exists, we will analyse and optimise the gas costs associated with publishing and maintaining deals, and hope to deliver further optimisations either in nv18 or shortly thereafter.

To be clear, we have no intentions of halting the network v17 upgrade or its inclusion of FIP-0045. While we wish the gas impacts were smaller, they do not overpower the network progress embodied in these changes. Will will improve it in a subsequent upgrade.

Closing remarks

I would like to acknowledge @jennijuju who early and repeatedly agitated for gas analysis throughout this development. You were on the right path. I wish we had had the tooling to provide this sooner.

@swift-mx contributed #722 toward the end of the nv17 development window, which would likely have reduced costs for batched calls to PublishStorageDeals. This change was in review but deprioritized in favour of simplicity and sticking to our development and network release timelines. We will of course include this in the next network upgrade.

[SBudo]

@anorth
I'm seeing a huge increase (2.8x) in the publish verified deals gas at the moment :-(
Publishing 6 deals two days ago - Gas Used: 314,100,791
Publishing 6 deals today - Gas Used: 877,116,402

[anorth]

The relative increase is larger for batch submissions, the per-sector cost being about constant.

Yes, because the cost is roughly constant per-deal, it becomes more obvious in batched publications where some of the rest of the costs are amortized over multiple deals. Landing batched datacap allocations will help bring the relative increases in line across different batch sizes.

[anorth]

The batching of datacap transfers from the same verified client has been implemented in filecoin-project/builtin-actors#722. This is expected to remove most of the per-deal overheads in batched cases so that batches (with the same client) area again much cheaper, for the next upgrade (nv18).

There has been some solid progress on tooling for instrumentation. Results and follow-up optimisations will be tracked in filecoin-project/builtin-actors#1020. They will probably land later if, as seems likely, they involve reorganising the state of existing built-in actors.