Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

client/authority-discovery: Introduce AuthorityDiscoveryService #6760

Merged
16 commits merged into from
Aug 12, 2020

Conversation

mxinden
Copy link
Contributor

@mxinden mxinden commented Jul 29, 2020

Add a basic AuthorityDiscoveryService implementation which enables
callers to get the addresses for a given AuthorityId and the AuthorityId
for a given PeerId from the local cache.

impl Service {
        /// Get the addresses for the given [`AuthorityId`] from the local address cache.
	pub async fn get_addresses_by_authority_id(&mut self, authority: AuthorityId) -> Option<Vec<Multiaddr>> {
		// ...
	}

	/// Get the [`AuthorityId`] for the given [`PeerId`] from the local address cache.
	pub async fn get_authority_id_by_peer_id(&mut self, peer_id: PeerId) -> Option<AuthorityId> {
                // ...
	}
}

To achieve the above this pull request also

  • Restructures the crate into Service and Worker (e9a12e8).

  • Updates the peerset priority group on an interval instead of on each DHT value found (ad1a711).


Fixes #6595.

polkadot companion: paritytech/polkadot#1563

Add a basic `AuthorityDiscoveryService` implementation which enables
callers to get the addresses for a given `AuthorityId` from the local
cache.
@mxinden mxinden added A3-in_progress Pull request is in progress. No review needed at this stage. B3-apinoteworthy C1-low PR touches the given topic and has a low impact on builders. labels Jul 29, 2020
Move `Service` and `Worker` to their own Rust modules resulting in the
following file structure.

├── build.rs
├── Cargo.toml
└── src
    ├── error.rs
    ├── lib.rs
    ├── service.rs
    ├── tests.rs
    ├── worker
    │   ├── addr_cache.rs
    │   ├── schema
    │   │   └── dht.proto
    │   └── tests.rs
    └── worker.rs
Instead of updating the authority discovery peerset priority group each
time a new DHT value is found, update it regularly on an interval.

This removes the need for deterministic random selection. Instead of
trying to return a random stable set of `Multiaddr`s, the `AddrCache`
now returns a random set on each call.
@mxinden mxinden added A0-please_review Pull request needs code review. A7-needspolkadotpr and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Aug 10, 2020
@mxinden
Copy link
Contributor Author

mxinden commented Aug 10, 2020

While still missing a simple Polkadot companion, this pull request is ready for a first review @tomaka @rphmeier.

e9a12e8 moves around most of the code resulting in a large overall diff (+1,732 −1,402). You can either look at the whole change at once, or, commit by commit ignoring most of e9a12e8.

mxinden added a commit to paritytech/polkadot that referenced this pull request Aug 10, 2020
paritytech/substrate#6760 introduces the concept
of an authority discovery `Service` allowing one to communicate with an
authority discovery `Worker`, e.g. to learn the `Multiaddr`s for a given
`AuthorityId`.

Along with the new `Service` structure it also alters the authority
discovery constructor to return both a worker and a service. This
commit adjusts the callside of the constructor, ignoring the `Service`
for now.
Copy link
Contributor

@tomaka tomaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weren't we also supposed to remove the automatic update of the peerset?
This can be done later.

Comment on lines 40 to 41
/// Get the addresses for the given [`AuthorityId`] from the local address cache.
pub async fn get_addresses_by_authority_id(&mut self, authority: AuthorityId) -> Option<Vec<Multiaddr>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you document that the addresses are always of the form .../p2p/...?
And also document that None means that the authority isn't in the cache.

rx.await.ok().flatten()
}

/// Get the [`AuthorityId`] for the given [`PeerId`] from the local address cache.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same remark: None means "not in cache".

@mxinden
Copy link
Contributor Author

mxinden commented Aug 11, 2020

Weren't we also supposed to remove the automatic update of the peerset?

I would prefer to keep it in place until upper layer components are capable of handling it themselves.

Could you document that the addresses are always of the form .../p2p/...?

Done in 30613ce.

And also document that None means that the authority isn't in the cache.

Documented in 05664d7. Right now, the case of connection failure between Service and Worker is also modeled as None. Let me know @tomaka in case you would prefer that to be a separate Result wrapped around the Option<Vec<Multiaddr>>.

@tomaka
Copy link
Contributor

tomaka commented Aug 11, 2020

Documented in 05664d7. Right now, the case of connection failure between Service and Worker is also modeled as None. Let me know @tomaka in case you would prefer that to be a separate Result wrapped around the Option<Vec>.

Since None means "not-in-cache/unknown", then it covers the situation where the worker is down.

@tomaka
Copy link
Contributor

tomaka commented Aug 11, 2020

I think this is ready for merge, just waiting to see if @rphmeier wants to review or not.

@rphmeier
Copy link
Contributor

@tomaka If the APIs match what is in the description and the code seems OK to you, it's fine by me, I don't need to review

@mxinden
Copy link
Contributor Author

mxinden commented Aug 12, 2020

Thanks for the reviews!

@ordian would you mind taking another look?

Copy link
Member

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@tomaka
Copy link
Contributor

tomaka commented Aug 12, 2020

bot merge

@ghost
Copy link

ghost commented Aug 12, 2020

Waiting for commit status.

@ghost
Copy link

ghost commented Aug 12, 2020

Checks failed; merge aborted.

@ordian
Copy link
Member

ordian commented Aug 12, 2020

bot merge

@ghost
Copy link

ghost commented Aug 12, 2020

Trying merge.

@ghost ghost merged commit 473a23f into paritytech:master Aug 12, 2020
ghost pushed a commit to paritytech/polkadot that referenced this pull request Aug 12, 2020
* service/src/lib: Update authority discovery construction

paritytech/substrate#6760 introduces the concept
of an authority discovery `Service` allowing one to communicate with an
authority discovery `Worker`, e.g. to learn the `Multiaddr`s for a given
`AuthorityId`.

Along with the new `Service` structure it also alters the authority
discovery constructor to return both a worker and a service. This
commit adjusts the callside of the constructor, ignoring the `Service`
for now.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 04fb79c.

* Update Substrate

Co-authored-by: parity-processbot <>
weichweich added a commit to KILTprotocol/substrate that referenced this pull request Aug 18, 2020
commit f8c83bd
Author: Roman Borschel <romanb@users.noreply.github.com>
Date:   Tue Aug 18 07:59:32 2020 +0200

    Add support for sourced metrics. (paritytech#6895)

    * Add support for sourced metrics.

    A sourced metric is a metric that obtains its values
    from an existing source, rather than the values being
    independently recorded. It thus allows collecting
    metrics from existing counters or gauges without
    having to duplicate them in a dedicated prometheus
    counter or gauge (and hence another atomic value).

    The first use-case is to feed the bandwidth counters
    from libp2p directly into prometheus.

    * Tabs, not spaces.

    * Tweak bandwidth counter registration.

    * Add debug assertion for variable labels and values.

    * Document monotonicity requirement for sourced counters.

    * CI

    * Update client/network/src/service.rs

    Co-authored-by: Max Inden <mail@max-inden.de>

    Co-authored-by: Max Inden <mail@max-inden.de>

commit 8e1ed7d
Author: Shawn Tabrizi <shawntabrizi@gmail.com>
Date:   Mon Aug 17 22:59:23 2020 +0200

    WeightInfo for System, Timestamp, and Utility (paritytech#6868)

    * initial updates to system

    * fix compile

    * Update writer.rs

    * update weights

    * finish system weights

    * timestamp weights

    * utility weight

    * Fix overflow in weight calculations

    * add back weight notes

    * Update for whitelisted benchmarks

    * add trait bounds

    * Revert "add trait bounds"

    This reverts commit 12b08b7.

    * Update weights for unaccounted for read

commit 399421a
Author: Wei Tang <wei@that.world>
Date:   Mon Aug 17 21:07:30 2020 +0200

    Derive Clone for AlwaysCanAuthor, NeverCanAuthor, CanAuthorWithNativeVersion (paritytech#6906)

commit 287ecc2
Author: Wei Tang <wei@that.world>
Date:   Mon Aug 17 19:36:29 2020 +0200

    pow: add access to pre-digest for algorithm verifiers (paritytech#6900)

    * pow: fetch pre-runtime digest to verifier

    * Add Other error type

    * Fix log target and change docs to refer to pre_runtime

commit 488b7c7
Author: Wei Tang <wei@that.world>
Date:   Mon Aug 17 13:41:09 2020 +0200

    babe, aura, pow: only call check_inherents if authoring version is compatible (paritytech#6862)

    * pow: check can_author_with before calling check_inherents

    * babe: check can_author_with before calling check_inherents

    * aura: check can_author_with before calling check_inherents

    * Fix node and node template compile

    * Add missing comma

    * Put each parameter on its own line

    * Add debug print

    * Fix line width too long

    * Fix pow line width issue

commit fc743da
Author: Pierre Krieger <pierre.krieger1708@gmail.com>
Date:   Mon Aug 17 11:19:16 2020 +0200

    Add a DirectedGossip struct (paritytech#6803)

    * Add a DirectedGossip struct

    * Move protocol from prototype::new to biuld

    * More traits impls

    * Explain ordering

    * Apply suggestions from code review

    Co-authored-by: Toralf Wittner <tw@dtex.org>

    * Address concerns

    * Add basic test

    * Concerns

    * More concerns

    * Remove QueueSenderPrototype

    * Rename

    * Apply suggestions from code review

    Co-authored-by: Max Inden <mail@max-inden.de>

    Co-authored-by: Toralf Wittner <tw@dtex.org>
    Co-authored-by: parity-processbot <>
    Co-authored-by: Max Inden <mail@max-inden.de>

commit 0079140
Author: Bastian Köcher <bkchr@users.noreply.github.com>
Date:   Sun Aug 16 00:05:36 2020 +0200

    Don't take the origin in `can_set_code` (paritytech#6899)

    It makes no sense that `can_set_code` takes the origin for checking it.
    Everybody reusing this function is only interested in the other checks
    that are done by this function. The origin should be checked by every
    dispatchable individually.

commit cd3b62b
Author: Seun Lanlege <seunlanlege@gmail.com>
Date:   Sat Aug 15 10:08:31 2020 +0100

    RpcHandlers Refactorings (paritytech#6846)

    * allow access to the underlying Pubsub instance from RpcHandlers

    * bump Cargo.lock

    * no more Arc<RpcHandlers>

    * bump Cargo.lock

    * Debug,.

    * Arc<RpcHandlers>

    * RpcHandler

    * RpcHandlers::io_handler

    * remove chain spec from cli

    * address pr comments

    * remove stray newline

    Co-authored-by: Ashley <ashley.ruglys@gmail.com>

    Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
    Co-authored-by: Ashley <ashley.ruglys@gmail.com>

commit eec7d71
Author: Max Inden <mail@max-inden.de>
Date:   Fri Aug 14 18:15:45 2020 +0200

    client/authority-discovery: Revert query interval change (paritytech#6897)

    Revert the accidental query interval change from every one minute back
    to every 10 minutes.

commit 13b0650
Author: Roman Borschel <romanb@users.noreply.github.com>
Date:   Fri Aug 14 10:41:47 2020 +0200

    Update to libp2p-0.23. (paritytech#6870)

    * Update to libp2p-0.23.

    Thereby incorporate bandwidth measurement along the
    lines previously done by libp2p itself.

    * Tweak dependencies for wasm32 compilation.

    For wasm32 we need to enable unstable features to
    make `task::Builder::local` available.

    * Simplify dependencies.

    * Simplify.

    Leave the calculation of bytes sent/received per second
    to the outer layers of the code, subject to their own
    individual update intervals.

    * Cleanup

    * Re-add lost dev dependency.

    * Avoid division by zero.

    * Remove redundant metric.

    * Enable sending of noise legacy handshakes.

    * Add comment about monotonic gauge.

    * CI

commit 0e703a5
Author: Alan Sapede <alan.sapede@gmail.com>
Date:   Fri Aug 14 04:15:59 2020 -0400

    Adds debug logs to EVM frame (paritytech#6887)

commit f16cbc1
Author: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Date:   Thu Aug 13 23:30:22 2020 +0200

    More renaming to move away from phragmen. (paritytech#6886)

commit 8993a75
Author: André Silva <123550+andresilva@users.noreply.github.com>
Date:   Thu Aug 13 19:38:14 2020 +0100

    network: don't log re-discovered addresses (paritytech#6881)

    * network: move LruHashSet to network crate utils

    * network: don't log re-discovered external addresses

    * Update client/network/src/utils.rs

    Co-authored-by: mattrutherford <44339188+mattrutherford@users.noreply.github.com>

    Co-authored-by: mattrutherford <44339188+mattrutherford@users.noreply.github.com>

commit 4d3c948
Author: Alexander Popiak <alexander.popiak@parity.io>
Date:   Thu Aug 13 18:54:05 2020 +0200

    add runtime migrations to release notes/changelog (paritytech#6875)

commit d019a66
Author: Wei Tang <wei@that.world>
Date:   Thu Aug 13 14:53:42 2020 +0200

    pallet-evm: avoid double fee payment (paritytech#6858)

    * pallet-evm: avoid double fee payment

    * Only skip fee payment for successful calls

commit ed4f7a1
Author: Bastian Köcher <bkchr@users.noreply.github.com>
Date:   Wed Aug 12 21:35:10 2020 +0200

    Make `HexDisplay` useable in `no_std` (paritytech#6883)

    Actually I use this quite often when debugging some WASM bugs and there
    is no harm in enabling it by default. Before I just always copied it
    everytime I needed it.

commit 473a23f
Author: Max Inden <mail@max-inden.de>
Date:   Wed Aug 12 16:16:40 2020 +0200

    client/authority-discovery: Introduce AuthorityDiscoveryService (paritytech#6760)

    * client/authority-discovery: Rename AuthorityDiscovery to XXXWorker

    * client/authority-discovery: Introduce AuthorityDiscoveryService

    Add a basic `AuthorityDiscoveryService` implementation which enables
    callers to get the addresses for a given `AuthorityId` from the local
    cache.

    * client/authority-discovery: Split into worker and service mod

    Move `Service` and `Worker` to their own Rust modules resulting in the
    following file structure.

    ├── build.rs
    ├── Cargo.toml
    └── src
        ├── error.rs
        ├── lib.rs
        ├── service.rs
        ├── tests.rs
        ├── worker
        │   ├── addr_cache.rs
        │   ├── schema
        │   │   └── dht.proto
        │   └── tests.rs
        └── worker.rs

    * client/authority-discovery: Cache PeerId -> AuthorityId mapping

    * client/authority-discovery: Update priority group on interval

    Instead of updating the authority discovery peerset priority group each
    time a new DHT value is found, update it regularly on an interval.

    This removes the need for deterministic random selection. Instead of
    trying to return a random stable set of `Multiaddr`s, the `AddrCache`
    now returns a random set on each call.

    * client/authority-discovery: Implement Service::get_authority_id

    * client/authority-discovery: Use HashMap instead of BTreeMap

    * client/authority-discovery: Rework priority group interval

    * client/authority-discovery: Fix comment

    * bin/node/cli: Update authority discovery constructor

    * client/authority-discovery: Fuse from_service receiver

    * client/authority-discovery: Remove Rng import

    * client/authority-discovery: Ignore Multiaddr without PeerId

    * client/authority-discovery/service: Add note on returned None

    * client/authority-discovery/addr_cache: Replace double clone with deref

commit c495f89
Author: Cecile Tonglet <cecile@parity.io>
Date:   Wed Aug 12 16:07:11 2020 +0200

    Add async test helper to timeout and provide a task_executor automatically (paritytech#6651)

    * Initial commit

    Forked at: 60e3a69
    Parent branch: origin/master

    * Add async test helper to timeout and provide a task_executor automatically

    * simplify error message to avoid difference between CI and locally

    * forgot env var

    * Use runtime env var instead of build env var

    * Rename variable to SUBSTRATE_TEST_TIMEOUT

    * CLEANUP

    Forked at: 60e3a69
    Parent branch: origin/master

    * Apply suggestions from code review

    Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

    * Re-export from test-utils

    * Default value to 120

    * fix wrong crate in ci

    * Revert "Default value to 120"

    This reverts commit 8e45871.

    * Fix version

    * WIP

    Forked at: 60e3a69
    Parent branch: origin/master

    * WIP

    Forked at: 60e3a69
    Parent branch: origin/master

    * WIP

    Forked at: 60e3a69
    Parent branch: origin/master

    * remove feature flag

    * fix missing dependency

    * CLEANUP

    Forked at: 60e3a69
    Parent branch: origin/master

    * fix test

    * Removed autotests=false

    * Some doc...

    * Apply suggestions from code review

    Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

    * WIP

    Forked at: 60e3a69
    Parent branch: origin/master

    * WIP

    Forked at: 60e3a69
    Parent branch: origin/master

    * Update test-utils/src/lib.rs

    Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

commit 0c3cdf1
Author: mattrutherford <44339188+mattrutherford@users.noreply.github.com>
Date:   Wed Aug 12 12:53:21 2020 +0100

    Implement tracing::Event handling & parent_id for spans and events (paritytech#6672)

    * implement events handling, implement parent_id for spans & events

    * add events to sp_io::storage

    * update test

    * add tests

    * adjust limit

    * let tracing crate handle parent_ids

    * re-enable current-id tracking

    * add test for threads with CurrentSpan

    * fix log level

    * remove redundant check for non wasm traces

    * remove duplicate definition in test

    * Adding conditional events API

    * prefer explicit parent_id over current,

    enhance test

    * limit changes to client::tracing event implementation

    * remove From impl due to fallback required on parent_id

    * implement SPAN_LIMIT

    change event log output

    * change version of tracing-core

    * update dependancies

    * revert limit

    * remove duplicate dependency

    * Apply suggestions from code review

    Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

    Co-authored-by: Matt Rutherford <mattrutherford@users.noreply.github.com>
    Co-authored-by: Benjamin Kampmann <ben@parity.io>
    Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

commit 5b809d2
Author: Wei Tang <wei@that.world>
Date:   Wed Aug 12 12:46:28 2020 +0200

    pallet-evm: fix wrong logic in mutate_account_basic (paritytech#6786)

    * pallet-evm: fix wrong logic in mutate_account_basic

    * Add test for mutate account

commit f6d66db
Author: Pierre Krieger <pierre.krieger1708@gmail.com>
Date:   Wed Aug 12 11:58:01 2020 +0200

    Add a warning if users pass --sentry or --sentry-nodes (paritytech#6779)

    * Add a warning if users pass --sentry or --sentry-nodes

    * Apply suggestions from code review

    Co-authored-by: Max Inden <mail@max-inden.de>

    * Fix text

    Co-authored-by: parity-processbot <>
    Co-authored-by: Max Inden <mail@max-inden.de>

commit d7979d0
Author: Shaopeng Wang <spxwang@gmail.com>
Date:   Wed Aug 12 21:21:36 2020 +1200

    Implement 'transactional' annotation for runtime functions. (paritytech#6763)

    * Implement 'transactional' annotation for runtime functions.

    * Allow function attributes for dispatchable calls in decl_module.

    * decl_module docs: add transactional function example.

    * decl_module docs: add function attributes notes.

    * Fix license header.

commit d4efdf0
Author: Pierre Krieger <pierre.krieger1708@gmail.com>
Date:   Wed Aug 12 10:58:16 2020 +0200

    Fuse the import queue receiver (paritytech#6876)

    * Fix the import queue receiver

    * Add logging

commit a20fbd5
Author: André Silva <123550+andresilva@users.noreply.github.com>
Date:   Tue Aug 11 22:21:45 2020 +0100

    docs: fix references to code of conduct document (paritytech#6879)

commit e7cc595
Author: h4x3rotab <h4x3rotab@gmail.com>
Date:   Wed Aug 12 04:12:34 2020 +0800

    Add Phala Network SS58 address type (paritytech#6758)

commit fe3fc04
Author: André Silva <123550+andresilva@users.noreply.github.com>
Date:   Tue Aug 11 20:55:15 2020 +0100

    docs: convert code of conduct to markdown (paritytech#6878)

commit 72addfa
Author: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Date:   Tue Aug 11 17:07:17 2020 +0200

    Fix wrong staking doc about transaction payment. (paritytech#6873)

    * Fx paritytech#4616

    * Fix paritytech#4616

commit 4064378
Author: André Silva <123550+andresilva@users.noreply.github.com>
Date:   Tue Aug 11 16:05:59 2020 +0100

    grandpa: change some logging from trace to debug (paritytech#6872)

    * grandpa: change some logging from trace to debug

    * grandpa: cleanup unused import

commit 6f57582
Author: Nikolay Volf <nikvolf@gmail.com>
Date:   Tue Aug 11 18:05:31 2020 +0300

    Move to upstream wasmtime, refactor globals snapshot  (paritytech#6759)

    * refactor globals snapshot

    * ignore test

    * update pwasm-utils ref

    * line width

    * add doc comment for internal struct

    * add explanation for iteration

    * Demote rustdoc to a comment

    * use 0.14

    Co-authored-by: Sergei Shulepov <sergei@parity.io>

commit a362997
Author: Arkadiy Paronyan <arkady.paronyan@gmail.com>
Date:   Tue Aug 11 16:12:30 2020 +0200

    Block packet size limit (paritytech#6398)

    * Block packet size limit

    * Update client/network/src/protocol.rs

    Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

    * Add block response limit

    Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. C1-low PR touches the given topic and has a low impact on builders.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

client/authority-discovery: Add option to specify relevant authority set
4 participants