Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Switch from devp2p to libp2p #268

Merged
merged 35 commits into from
Jul 15, 2018
Merged

Switch from devp2p to libp2p #268

merged 35 commits into from
Jul 15, 2018

Conversation

tomaka
Copy link
Contributor

@tomaka tomaka commented Jun 30, 2018

Close #187
Follow-up of #263

Removes the ethcore-network-devp2p crate and integrates a new substrate-libp2p crate.
The API of this crate is exactly the same as devp2p, so the replacement is quite straight-forward.
Once this is merged I would like to refactor a bit the networking API in order to make it less error-prone.

What the code does on the wire is it performs a random Kademlia FIND_NODE request every 30 seconds, and tries to open the /substrate/dot/0 protocol with each of the resulting nodes (until we reach min_peers, after which it only listens to connections).
Consequently, in practice the bootstrap nodes should be used only for Kademlia and not for substrate/dot, unless there are not enough nodes available.

The format of a peer address in libp2p is not enode://...@..., but /ip4/1.2.3.4/tcp/10000/p2p/Qm... (or /ip6/::1/tcp/10000/p2p/Qm...) where the Qm part is the base56 encoding of a peer ID.

NAT traversal "works" in the sense that nodes have knowledge of their external IP and report it. However there is no such thing as UDP hole punching implemented yet.

Note that the code can be greatly simplified by having the network immediately start when we create the service, and by passing a list of protocols to the constructor instead of registering them afterwards.

I will only do these changes after this PR has been merged, in order to not introduce merge conflicts.

I will continue to push commits on this branch in order to improve readability and reliability, but you can can start playing with it if you want. Since the change surface is really low, it should be easily rebasable on any other branch.

@tomaka tomaka mentioned this pull request Jun 30, 2018
Closed
@jacogr jacogr added the A3-in_progress Pull request is in progress. No review needed at this stage. label Jun 30, 2018
@tomaka tomaka changed the title [WIP] Switch from devp2p to libp2p Switch from devp2p to libp2p Jul 3, 2018
@tomaka
Copy link
Contributor Author

tomaka commented Jul 3, 2018

I'm still tweaking/cleaning the code, but there's nothing preventing it from being merged.

@tomaka tomaka added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Jul 3, 2018
gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated

/// Sets information about a peer.
pub fn set_peer_info(&self, node_id: PeerstorePeerId, endpoint: Endpoint, client_version: String,
local_addr: Multiaddr, remote_addr: Multiaddr) -> Result<PeerId, IoError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code style for long arg lists in functions is:

fn function_name(
	arg1: SomeType,
	arg2: SomeType,
	...
) -> SomeType {
	...
}

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated

/// Returns true if we should open a new outgoing connection to a peer.
/// This takes into account the number of active peers.
pub fn should_open_outgoing_connecs(&self) -> bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

connecs?

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated
ErrorKind::Io(IoError::new(IoErrorKind::Other, err))
})?;
Ok(())

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

superfluous empty line

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated
debug_assert_eq!(old, Some(peer_id));
peer_info
} else {
return;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

superfluous ;

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated
}
Err(err) => {
warn!(target: "sub-libp2p", "Failed to flush changes to JSON \
peer store: {}", err);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

way too many tabs, it seems (carry-on lines should be one-indent more than original)

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated
//
// The function only accepts already-locked structs, so that we don't risk any deadlock.
fn accept_connection(connections: &mut Connections, next_peer_id: &atomic::AtomicUsize,
node_id: PeerstorePeerId, endpoint: Endpoint)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent style

gavofyork
gavofyork reviewed Jul 3, 2018
View reviewed changes
substrate/network-libp2p/src/network_state.rs Outdated
}

// Obtains or generates the local private key using the configuration.
fn obtain_private_key(config: &NetworkConfiguration) -> Result<secio::SecioKeyPair, IoError> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a monster function - probably better expressed as two or three auxilliaries.

@gavofyork
Copy link
Member

@rphmeier bump

@gavofyork
Copy link
Member

or @arkpar ?

@rphmeier
Copy link
Contributor

rphmeier commented Jul 13, 2018 •
edited
Loading

@gavofyork taking a look today. have been busy past few days.

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/custom_proto.rs Outdated
type Future = future::FutureResult<(Self::Output, Self::MultiaddrFuture), IoError>;

fn upgrade(self, socket: C, protocol_version: Self::UpgradeIdentifier,
_: Endpoint, remote_addr: Maf) -> Self::Future {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: all-or-nothing

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/custom_proto.rs Outdated
let framed = AsyncRead::framed(socket, VarintCodec::default());
let msg_rx = msg_rx.map(Message::SendReq)
.chain(stream::once(Ok(Message::Finished)))
.map_err(|()| unreachable!());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assumptions around this kind of thing can change quickly. Usually I would use an explicit panic or unreachable with message to indicate why it's impossible. then the reader can check it while reading.

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/custom_proto.rs Outdated

#[inline]
fn upgrade(self, socket: C, upgrade_identifier: Self::UpgradeIdentifier,
endpoint: Endpoint, remote_addr: Maf) -> Self::Future {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: all-or-nothing

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs
let shared = self.shared.clone();
let join_handle = thread::spawn(move || {
// Tokio core that is going to run everything in this thread.
let mut core = match Core::new() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be easily replaced with a tokio::current_thread::Runtime?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Libp2p-rs uses the old tokio-core thing, because I admit I'm a bit lost in tokio versions, so no.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there plans to upgrade it? we're mostly using new-tokio in polkadot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that the new tokio v0.1 thing can compile to emscripten, may have to check if it works before upgrading.

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs Outdated
debug_assert!(self.inner.protocols.read().has_protocol(protocol),
"invalid protocol id requested in the API of the libp2p networking");
// TODO: restore
//debug_assert!(packet_id < self.inner.protocols.read().iter().find(|p| p.id == protocol).unwrap().packet_count,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does it need restoring?

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs
},

FinalUpgrade::Identify(IdentifyOutput::RemoteInfo { .. }, _) =>
unreachable!("We are never dialing with the identify protocol"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if someone dials us?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone dials us, we enter the FinalUpgrade::Identify(IdentifyOutput::Sender { sender }, original_addr) block.

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs Outdated

FinalUpgrade::Identify(IdentifyOutput::Sender { sender }, original_addr) => {
trace!(target: "sub-libp2p", "Sending back identification info");
/*let protocols_to_report = ConnectionUpgrade::<_, Box<Future<Item = Multiaddr, Error = IoError>>>::protocol_names(&listener_upgrade)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented-out code here

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs
info: &IdentifyTransportOutcome,
original_addr: Multiaddr,
endpoint: Endpoint,
transport: &impl Transport
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please just use a generic parameter. IMO impl Trait should only be used where we can't name the type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the function doesn't know whether the caller can name the type 😕
I would argue that using impl Trait in parameters is fine for private functions, because we can always change their signature later without breaking much.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the function doesn't know whether the caller can name the type

I don't get it. If I write fn foo<F: FnOnce>(f: F) then I can call that with an unnameable type. It doesn't matter if the caller can name it.

Copy link
Member

@gavofyork gavofyork Jul 14, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree with @tomaka

the impl syntax is more concise and readable - i'd prefer it over the clumsier generic syntax unless there's a reason to believe that it'll be called with explicit type args. in this case, it's private so there's no reason to believe that.

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/service.rs Outdated
/// in a lazy way. This is what this wrapper does.
#[derive(Clone)]
struct DelayedProtosList(Arc<Shared>);
impl<C, Maf> ConnectionUpgrade<C, Maf> for DelayedProtosList
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is Maf short for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MultiaddressFuture. Should I add a comment?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that would be helpful

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/src/timeouts.rs Outdated
core: Handle,
timeouts_rx: impl Stream<Item = (Instant, T), Error = ()>
) -> impl Stream<Item = T, Error = IoError> {
let next_timeout = next_in_timeouts_stream(timeouts_rx);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

next_in_timeouts_stream will panic if timeouts_rx throws an error. But the type of timeouts_rx isn't known. Can you prove this assumption?

rphmeier
rphmeier reviewed Jul 13, 2018
View reviewed changes
substrate/network-libp2p/tests/tests.rs Outdated
@@ -0,0 +1,149 @@
// Copyright 2015-2017 Parity Technologies (UK) Ltd.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you started this earlier than I thought :)

rphmeier
rphmeier previously requested changes Jul 13, 2018
View reviewed changes
Copy link
Contributor

@rphmeier rphmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some small grumbles. could be a bit more rigorous with panickers.

@gavofyork
Copy link
Member

looks like all grumbles from @rphmeier have been addressed.

@gavofyork gavofyork dismissed rphmeier’s stale review July 14, 2018 14:19

grumbles addressed

@gavofyork gavofyork merged commit 41d9507 into master Jul 15, 2018
@gavofyork gavofyork deleted the tka-libp2p branch July 15, 2018 09:45
lamafab pushed a commit to lamafab/substrate that referenced this pull request Jun 16, 2020
@gui1117 @gavofyork
Update substrate polkadot-master branch (paritytech#268)
ed98381 
* update substrate

* update lock
liuchengxu pushed a commit to chainx-org/substrate that referenced this pull request Aug 23, 2021
update substrate with redis fix (paritytech#268)
4624736 
update substrate with redis fix
adoerr added a commit that referenced this pull request Sep 22, 2021
@adoerr
bump Substrate and Deps (#268)
ccc148c
ghost pushed a commit that referenced this pull request Sep 23, 2021
@adoerr @andresilva
@tomusdrw @dependabot-preview @HCastano @tomaka @dependabot @ascjones @bkchr @drewstone @ordian @tolak @JoshOrndorff @gilescope
Integrate BEEFY (#9833)
65ac6f3 
* Initial project setup and skeleton (#4)

* initial project setup for beefy gadget client

* update editorconfig

* update gitignore

* add initial skeleton for beefy gadget worker

* add skeleton for gossip processing

* add app crypto

* move around some code

* add basic flow for voting

* add logic for picking blocks to sign

* add rustfmt config

* add example node with beefy gadget

* use u32::next_power_of_two

* make maximum periodicity configurable

* add copyright header

* rename max_periodicity to min_interval

* CI stuff (#5)

* CI stuff.

* Fix workspace.

* cargo fmt --all

* Add license for beefy-gadget

* One toolchain to rule them all.

* Clippy.

* Fix clippy.

* Clippy in the runtime.

* Fix clippy grumbles.

* cargo fmt --all

* Primitives & Light Client examples (#8)

* Primitives.

* Docs.

* Document primitives.

* Simple tests.

* Light client examples.

* Fix stuff.

* cargo fmt --all

* Add a bunch of tests for imports.

* Add more examples.

* cargo fmt --all

* Fix clippy.

* cargo fmt --all

* Apply suggestions from code review

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Add GRANDPA / FG clarifications.

* Fix min number of signatures.

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update to substrate master (#22)

* update to substrate master

* update dependencies

* fix clippy issues

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Add beefy pallet (#25)

* move beefy application crypto to primitives

* make primitives compile under no_std

* add beefy pallet that maintains authority set

* add beefy pallet to node example runtime

* tabify node-example cargo.toml files

* use double quotes in Cargo.toml files

* add missing hex-literal dependency

* add runtime api to fetch BEEFY authorities

* fix clippy warnings

* rename beefy-pallet to pallet-beefy

* sort dependencies in node-example/runtime/Cargo.toml

* Signed commitments rpc pubsub (#26)

* move beefy application crypto to primitives

* make primitives compile under no_std

* add beefy pallet that maintains authority set

* add beefy pallet to node example runtime

* tabify node-example cargo.toml files

* use double quotes in Cargo.toml files

* add missing hex-literal dependency

* add runtime api to fetch BEEFY authorities

* fix clippy warnings

* gadget: use commitment and signedcommitment

* gadget: send notifications for signed commitments

* gadget: add rpc pubsub for signed commitments

* node-example: enable beefy rpc

* gadget: fix clippy warnings

* rename beefy-pallet to pallet-beefy

* sort dependencies in node-example/runtime/Cargo.toml

* gadget: add documentation on SignedCommitment rpc wrapper type

* gadget: add todos about dummy beefy commitments

* gadget: remove redundant closure

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Integrate MMR and deposit root into the digest. (#24)

* Add basic MMR.

* Deposit digest item.

* cargo fmt --all

* Merge with primitives.

* cargo fmt --all

* Fix extra spaces.

* cargo fmt --all

* Switch branch.

* remove stray whitespace

* update to latest td-mmr commit

* fix clippy error

Co-authored-by: André Silva <andrerfosilva@gmail.com>

* use new mmr root as commitment payload (#27)

* use new mmr root as commitment payload

* fix mmr root codec index

* warn on MMR root digest not found

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* add type alias for MMR root hash

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Bump serde_json from 1.0.59 to 1.0.60 (#28)

* Update to latest substrate. (#32)

* Update to latest substrate.

* Fix tests.

* cargo fmt --all

* Switch to master.

* Bump serde from 1.0.117 to 1.0.118 (#29)

* Bump serde from 1.0.117 to 1.0.118

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.117 to 1.0.118.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.117...v1.0.118)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Bump arc-swap.

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Remove transition flag (#35)

* Get rid of is_set_transition_flag

* Fix tests.

* cargo fmt --all

* Bump futures from 0.3.9 to 0.3.12 (#50)

* Bump log from 0.4.11 to 0.4.13 (#52)

* Bump Substrate and Deps (#57)

* Update README (#58)

* Update README

* Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* address review comments

* missed a typo

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Add validator set to the pallet. (#65)

* Bump Substrate and Deps (#71)

* Bump Substrate and Deps

* pin serde and syn

* bump Substrate again for '__Nonexhaustive' fix

* add cargo deny ignore

* Beefy pallet test (#74)

* setup mock

* test session change

* silence beefy

* clippy still

* no change - no log

* clippy again

* Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* code review changes, added additional test

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Beefy node cleanup (#75)

* bump serde

* bump substrate, scale-codec 2.0.0

* we need a proper beefy node

* rename primitives as well

* Sort members.

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Migrate beefy-pallet to FRAMEv2 (#76)

* migrate beefy-pallet to FRAMEv2

* Code review

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>

* Run BEEFY worker as non-validator (#77)

* run BEEFY worker as non-validator

* don't check for roloe.is_authority

* change enum type name

* Bump Substrate and Deps (#79)

* Add BEEFY gadget as extra peer set (#80)

* Add BEEFY gadget as extra peer set

* use BEEFY protocol

* Add ValidatorSetId to BEEFY digest (#85)

* add ValidatorSetId to BEEFY digest

* apply review changes

* Bump Substrate and Deps (#91)

* Bump Substrate and Deps

* Bump Substrate again in order to include a hot-fix

* redo again

* use CryptoStore issue

* cargo fmt

* Bump serde_json from 1.0.63 to 1.0.64 (#93)

* Track BEEFY validator set (#94)

* Track BEEFY validator set

* Add validator_set_id to BeefyWorker

* Make validattor_set_id optional

* Ad 92 (#97)

* sign_commitment()

* Error handling todo

* Add error type (#99)

* Add error type

* Address review

* Extract worker and round logic (#104)

* Bump serde from 1.0.123 to 1.0.124 (#106)

* Rework BeefyAPI (#110)

* Initialize BeefyWorker with current validator set (#111)

* Update toolchain (#115)

* Use nightly toolchain

* dongradde to latest clippy stable

* GH workflow trail and error

* next try

* use stable for clippy

* update wasm builder

* yet another try

* fun with CI

* no env var

* and one more

* allow from_over_into bco contruct_runtime

* back to start

* well ...

* full circle

* old version was still used

* Bump Substrate and Deps (#117)

* Bump Substrate and Deps

* cargo fmt should enforce uniform imports

* merge some imports

* Delayed BEEFY worker initialization (#121)

* lifecycle state

* add Client convenience trait

* rework trait identifiers

* WIP

* rework BeefyWorker::new() signature

* Delayed BEEFY gadget initialization

* address review

* Bump substrate. (#123)

* Bump substrate.

* Fix tests.

* Lower log-level for a missing validator set (#124)

* lower log-level for a missing validator set

* move best_finalized_block initialization

* Setup Prometheus metrics (#125)

* setup Prometheus metrics

* expose validator set id

* cargo fmt

* Update beefy-gadget/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* add vote messages gossiped metric

* track authorities change, before checking for MMR root digest

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Make Client convenience trait public (#126)

* Bump serde from 1.0.124 to 1.0.125 (#131)

* Reset rounds on new validator set. (#133)

* Re-set rounds on new validator set.

* Fix docs.

* Bump Substrate and Deps (#134)

* beefy: authority set changes fixes (#139)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type (#140)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* gadget: reduce gossip spam (#141)

* node: fix grandpa peers set config

* gadget: update best finalized number only when finalized with beefy

* gadget: process authorities changes regardless of vote status

* gadget: remove superfluous signature type

* gadget: only gossip last 5 rounds

* gadget: note round to gossip validator before gossiping message

* gadget: fix clippy warnings

* gadget: update docs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* gadget: verify SignedCommitment message signature (#142)

* gadget: verify SignedCommitment message signature

* gadget: log messages with bad sigs

* gadget: move todo comment

* Bump futures from 0.3.13 to 0.3.14 (#145)

* Milestone 1 (#144)

* use best_finalized, prevent race

* make best_finalized_block an Option, should_vote_on bails on None

* Bump futures from 0.3.13 to 0.3.14

* Revert futures bump

* Revert "Revert futures bump"

This reverts commit a1b5e7e9bac526f2897ebfdfee7f02dd29a13ac5.

* Revert "Bump futures from 0.3.13 to 0.3.14"

This reverts commit a4e508b118ad2c4b52909d24143c284073961458.

* debug msg if the bail voting

* validator_set()

* local_id()

* get rid of worker state

* Apply review suggestions

* fix should_vote_on()

* Extract BeefyGossipValidator (#147)

* Extract BeefyGossipValidator

* Apply review suggestions

* Add block_delta parameter to start_beefy_gadget (#151)

* Add block_delta parameter

* rename to min_block_delta

* Add additional metrics (#152)

* Add additional metrics

* add skipped session metric

* add some comment for temp metric

* don't log under info for every concluded round (#156)

* don't log error on missing validator keys (#157)

* don't log error on missing validator keys

* remove unused import

* Fix validator set change handling (#158)

* reduce some logs from debug to trace

* fix validator set changes handling

* rename validator module to gossip

* run rustfmt

* Fix should_vote_on() (#160)

* Fix should_vote_on()

* by the textbook

* fix the algorithm

* Apply review suggestions

* don't use NumberFor in vote_target

Co-authored-by: André Silva <andrerfosilva@gmail.com>

* Make KeyStore optional (#173)

* Use builder pattern for NonDefaultSetConfig (#178)

Co-authored-by: adoerr <0xad@gmx.net>

* Append SignedCommitment to block justifications (#177)

* Append SignedCommitment

* add BeefyParams

* add WorkerParams

* use warn

* versioned variant for SignedCommitment

* Bump serde from 1.0.125 to 1.0.126 (#184)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.125 to 1.0.126.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.125...v1.0.126)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump strum from 0.20.0 to 0.21.0 (#195)

* Bump strum from 0.20.0 to 0.21.0

Bumps [strum](https://github.com/Peternator7/strum) from 0.20.0 to 0.21.0.
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/commits)

---
updated-dependencies:
- dependency-name: strum
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* use dervie feature for strum; clippy and deny housekeeping

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Make concluded round an info log (#200)

* Remove external crypto trait bounds (#207)

* BeefyKeystore newtype

* WIP

* remove mod ecdsa

* WIP

* fix tests

* some polishing

* Rename AuthorityId to BeefyId to avoid type conflict in UI (#211)

* Add trace points; Reduce MAX_LIVE_GOSSIP_ROUNDS (#210)

* Add trace points; Reduce MAX_LIVE_GOSSIP_ROUNDS

* log local authority id

* Additional initial authority id's (#217)

* Scratch concluded rounds

* adjust testnet doc

* fix authority key typo

* We don't want no scratches

* address review comments

* Fix note_round() (#219)

* rename BeefyGossipValidator

* Fix note_round()

* use const for assert

* put message trace points back in

* test case note_same_round_twice()

* address review comments

* remove redundant check

* Use LocalKeystore for tests (#224)

* private_keys()

* Use LocalKeystore for tests

* Use keystore helper

* Address review

* some reformatting

* Cache known votes in gossip (#227)

* Implement known messages cache.

* Add tests.

* Appease clippy.

* More clippy

Co-authored-by: adoerr <0xad@gmx.net>

* Some key store sanity checks (#232)

* verify vote message

* verify_validator_set()

* rework logging

* some rework

* Tone down warnings.

* Add signature verification.

* Tone down more.

* Fix clippy

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Use Binary Merkle Tree instead of a trie (#225)

* Binary tree merkle root.

* Add proofs and verification.

* Clean up debug.

* Use BEEFY addresses instead of pubkeys.

* Use new merkle tree.

* Optimize allocations.

* Add test for larger trees.

* Add tests for larger cases.

* Appease clippy

* Appease clippy2.

* Fix proof generation & verification.

* Add more test data.

* Fix CLI.

* Update README

* Bump version.

* Update docs.

* Rename beefy-merkle-root to beefy-merkle-tree

Co-authored-by: adoerr <0xad@gmx.net>

* Bump Substrate and Deps (#235)

* BEEFY+MMR pallet (#236)

* Add MMR leaf format to primitives.

* Fix tests

* Initial work on the BEEFY-MMR pallet.

* Add tests to MMR pallet.

* Use eth addresses.

* Use binary merkle tree.

* Bump libsecp256k1

* Fix compilation.

* Bump deps.

* Appease cargo deny.

* Re-format.

* Module-level docs.

* no-std fix.

* update README

Co-authored-by: adoerr <0xad@gmx.net>

* Fix noting rounds for non-authorities (#238)

* Bump env_logger from 0.8.4 to 0.9.0 (#242)

Bumps [env_logger](https://github.com/env-logger-rs/env_logger) from 0.8.4 to 0.9.0.
- [Release notes](https://github.com/env-logger-rs/env_logger/releases)
- [Changelog](https://github.com/env-logger-rs/env_logger/blob/main/CHANGELOG.md)
- [Commits](rust-cli/env_logger@v0.8.4...v0.9.0)

---
updated-dependencies:
- dependency-name: env_logger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* gadget: add global timeout for rebroadcasting messages (#243)

* gadget: add global timeout for rebroadcasting messages

* update rustfmt.toml

* make message_allowed() a debug trace

Co-authored-by: adoerr <0xad@gmx.net>

* Bump Substrate and Deps (#245)

* Bump Substrate and Deps

* Bump Substrate again

* Bump futures from 0.3.15 to 0.3.16 (#247)

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.15 to 0.3.16.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](rust-lang/futures-rs@0.3.15...0.3.16)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump libsecp256k1 from 0.5.0 to 0.6.0 (#249)

* Bump libsecp256k1 from 0.5.0 to 0.6.0

Bumps [libsecp256k1](https://github.com/paritytech/libsecp256k1) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/paritytech/libsecp256k1/releases)
- [Changelog](https://github.com/paritytech/libsecp256k1/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/libsecp256k1/commits)

---
updated-dependencies:
- dependency-name: libsecp256k1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* use correct crate name

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Derive `scale_info::TypeInfo` for types used in polkadot (#218)

* Add scale-info TypeInfo derives

* Update scale-info

* Add crates.io patches

* Use substrate aj-metadata-vnext branch

* Revert master branch substrate deps

* Add scale-info to beefy-pallet

* scale-info v0.9.0

* Remove github dependencies and patches

* More TypeInfo derives

* Update scale-info to 0.10.0

* Add missing scale-info dependency

* Add missing TypeInfo derive

* Hide TypeInfo under a feature.

Co-authored-by: Tomasz Drwięga <tomasz@parity.io>

* Bump serde from 1.0.126 to 1.0.127 (#260)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.126 to 1.0.127.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.126...v1.0.127)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump Substrate and Deps (#262)

* Update jsonrpc (#265)

* Update jsonrpc

* Update Substrate

* bump Substrate and Deps (#268)

* Bump serde from 1.0.127 to 1.0.128 (#272)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.127 to 1.0.128.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.127...v1.0.128)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix spelling (#271)

* Bump serde from 1.0.128 to 1.0.130 (#276)

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.128 to 1.0.130.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](serde-rs/serde@v1.0.128...v1.0.130)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scale-info from 0.10.0 to 0.12.0 (#275)

Bumps [scale-info](https://github.com/paritytech/scale-info) from 0.10.0 to 0.12.0.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/commits)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Update to scale-info 1.0 (#278)

* bump substrate (#282)

* bump Substrate and Deps

* cargo fmt

Co-authored-by: Wenfeng Wang <kalot.wang@gmail.com>

* Update worker.rs (#287)

* Bump anyhow from 1.0.43 to 1.0.44 (#290)

* Bump anyhow from 1.0.43 to 1.0.44

Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.43 to 1.0.44.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](dtolnay/anyhow@1.0.43...1.0.44)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* derive Default

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* Remove optional `scale-info` feature (#292)

* Make scale-info dependency non-optional

* Remove feature gated TypeInfo derives

* Import TypeInfo

* Update substrate

* Fix up runtime

* prune .git suffix (#294)

* remove unused deps (#295)

* remove unused deps

* update lock file

* Bump libsecp256k1 from 0.6.0 to 0.7.0 (#296)

* Bump libsecp256k1 from 0.6.0 to 0.7.0

Bumps [libsecp256k1](https://github.com/paritytech/libsecp256k1) from 0.6.0 to 0.7.0.
- [Release notes](https://github.com/paritytech/libsecp256k1/releases)
- [Changelog](https://github.com/paritytech/libsecp256k1/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/libsecp256k1/commits)

---
updated-dependencies:
- dependency-name: libsecp256k1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update sec advisories

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: adoerr <0xad@gmx.net>

* clean compile

* use path dependencies

* beefy-gadget license header

* pallet-beefy license header

* pallet-beefy-mmr license header

* beefy-primitves license header

* carg fmt

* more formatting

* shorten line

* downgrade parity-scale-codec to 2.2.0

* use path dependency for Prometheus endpoint

* remove clippy annotations

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
Co-authored-by: André Silva <andrerfosilva@gmail.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Jones <ascjones@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: drewstone <drewstone329@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Wenfeng Wang <kalot.wang@gmail.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
liuchengxu added a commit to autonomys/substrate that referenced this pull request Jun 3, 2022
@liuchengxu
Merge pull request paritytech#268 from subspace/cirrus-test-and-state…
c502487 
…-mismatch-fix

Introduce cirrus test framework and fix the state mismatch between executor authority and full node
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@rphmeier rphmeier rphmeier left review comments

@gavofyork gavofyork gavofyork approved these changes

Assignees
No one assigned
Labels
A0-please_review Pull request needs code review.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tomaka @gavofyork @rphmeier @jacogr