How opinionated will commonware libraries be?

[patrick-ogrady]

[patrick-ogrady]

Thanks for raising this...you highlighted a number of the design minutia we've been contemplating over the last few weeks.

Before answering, I wanted to callout for anyone following along that you can check out a very WIP version of our the first consensus dialect (commonware-consensus::authority) here: #111.

This initial pass includes a basic demo example called commonware-clock that employs commonware-runtime::tokio, commonware-cryptography::ed25519, commonware-p2p::authenticated, and commonware-consensus::authority to enable a set of participants (of which f may be byzantine) to agree on the current time: https://github.com/commonwarexyz/monorepo/blob/0fd49df39a84b2c8d42e56fca064c7cfb1707317/examples/clock/README.md

[patrick-ogrady]

Quick question on how flexible we can expect the consensus primitives in commonware-consensus to be. Totally understand this is all in relatively early-phase development but wanted to ask these questions in case these considerations weren't already on your roadmap.

We're currently exploring the right abstraction for integration with the application layer now. Thus far, we've been using the following traits to build reusable testing capability: https://github.com/commonwarexyz/monorepo/blob/0fd49df39a84b2c8d42e56fca064c7cfb1707317/consensus/src/lib.rs

I'm not yet convinced it is possible (and don't expect) that all dialects in commonware-consensus::* adhere to the exact same interface for applications and instead think it is more important to ensure they use similar primitive types when they expose unique capability. At a high-level, I find structured but unique interaction manageable but unstructured and unique interaction very difficult to reason about.

All in all, it is important to keep in mind this is a library and not a rigid framework (so some primitive dialects may only selectively support what other dialects do).

[patrick-ogrady]

For example, if I'd like to build a chain in a single proposer setting (e.g. PBFT, Tendermint, etc) or instead a chain where all nodes contribute in each round (Narwhal+X, other DAG-based algorithms).

At first (not to say we won’t explore a different path in the future), we're focusing on the single proposer setting with the ability to scale throughput using concurrent, reliable broadcast (think a Motorway-like construction: https://arxiv.org/abs/2401.10369).

This decision follows from the commonware-cryptography::threshold work, which can be made most useful (to provide links and seeds) when there is a single block per height that may reference an arbitrary number of "sub-blocks" (https://x.com/_patrickogrady/status/1828797790371352673).

[patrick-ogrady]

Related: How flexible is voting? Is something a la vote extensions (implemented in CometBFT: https://docs.cometbft.com/v1.0/tutorials/forum-application/7.vote-extension) intended to be possible? Or is the expectation that we should be directly hooking into p2p for much of the more "specialized" behavior?

Unlike other consensus constructions I’ve looked into, the consensus construction we are working on externalizes all activity from consensus (both expected and faults) for external use. This makes it a great fit for integration with an AVS or some external verifier (where staking funds might be kept). In the application layer, this “activity” can be incorporated back into the block to provide staking rewards/penalties. This interface is described and tested in the above PR.

On the vote extension side, I’d consider that something that would be injected into the block but not part of the consensus engine. To provide some assurance of timeliness, such interactions could be restricted to only be allowed at a certain view/height. We don’t yet have an example of doing something like this but expect we will in the coming months.

Hope this helps!

[patrick-ogrady]

[patrick-ogrady]

Funny to hear you mention Motorway (now, I suppose, renamed "Autobahn"). That was exactly the project I was setting out to build, which reminded me about your work here from the inital Twitter posts 🙂

Yeah, I saw that lolol. I suppose it now has the flair any good research paper needs to capture mindshare.

I suppose it's fair to say that certain abstraction efforts may be preemptive and/or unergonomic before seeing how a prototype of non-leader-based algorithms could stand to benefit from "higher-level commonware abstractions"

I think the practical benefit would be that you could use the surrounding commonware layers (below and above agreement) to reduce the amount of work you'd have to do to bring something like that to life at a producition-ready level. You'll see specifically what that looks like (hopefully) soon.

A small side-note: Would love to see how/if you are planning on utilizing commonware-runtime::deterministic to employ some sort of determinstic simulation testing approach, if any.

We are using it extensively to test p2p (https://github.com/commonwarexyz/monorepo/blob/main/p2p/src/authenticated/mod.rs) and to test a very early prototype of consensus (#111). We'll look to break it out into some sort of harness that can run for days/hours at a time for longer simulations but just CI-ready for now.