Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kad: Providers part 7: better types and public API, public addresses & known providers #246

Merged
merged 43 commits into from
Sep 30, 2024
Merged

kad: Providers part 7: better types and public API, public addresses & known providers #246

merged 43 commits into from
Sep 30, 2024

Conversation

dmitry-markin
Copy link
Collaborator

@dmitry-markin dmitry-markin commented Sep 18, 2024 •
edited
Loading

This PR introduces the following changes:

  1. Introduces a better type ContentProvider without a burden of KademliaPeer.
  2. Simplifies public litep2p content providers API.
  3. Uses PublicAddresses API when advertising local providers.
  4. Adds locally known providers to discovered providers when performing GET_PROVIDERS request.
  5. Emits IncomingProvider event when remote node registers as a provider via ADD_PROVIDER request.

It may be easier to review this PR through individual commits. The commits of interest are:

  1. Introduce ContentProvider type
  2. Use PublicAddresses API
  3. Use locally known providers when performing `KademliaHandle::get_providers() request
  4. Emit IncomingProvider event

All other commits are merge commits from upstream branches.

Builds upon #245.

@dmitry-markin
Introduce ADD_PROVIDER query
1048320
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-provider-queries
2ab84ae
@dmitry-markin
Execute ADD_PROVIDER query
778078b
@dmitry-markin
Introduce local providers in MemoryStore
323a26f
@dmitry-markin
Add provider refresh interval to Kademlia config
8946809
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-republish-providers
474a58a
@dmitry-markin
Move FuturesStream to a separate file
d02acee
@dmitry-markin
Refresh providers: dry-run without network queries
5432c9b
@dmitry-markin
Remove try_get_record() and other try_...() non-async methods
feb493d
@dmitry-markin
Move query ID generation from KademliaHandle to Kademlia
982c73d
@dmitry-markin
Republish providers
68c7a87
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-republish-providers
f9ed3e7
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-add-providers
764a3f2
@dmitry-markin
Use getter TransportService::local_peer_id() instead of accessing d…
5e97484 
…irectly
@dmitry-markin
Use getter TransportService::local_peer_id() instead of accessing d…
7caf290 
…irectly
@dmitry-markin
Introduce GET_PROVIDERS query
60b8fb6
@dmitry-markin
Update get_providers() to receive query ID via oneshot channel
d340c35
@dmitry-markin
Make lines fit into 100 characters
10b96db
@dmitry-markin
Introduce GetProvidersContext & GetProvidersConfig
6146540
@dmitry-markin
Implement GET_PROVIDERS query
e167419
@dmitry-markin
Merge returned provider addresses and deduplicate records
eb6eee8
@dmitry-markin
Merge branch 'dm-add-providers' into dm-republish-providers
8314d95
@dmitry-markin
minor: fix log target
974eac5
@dmitry-markin
Revert "Remove try_get_record() and other try_...() non-async met…
e895a44 
…hods"

This reverts commit feb493d.
@dmitry-markin
Revert "Move query ID generation from KademliaHandle to Kademlia"
ce18594 
This reverts commit 982c73d.
@dmitry-markin
Use AtomicUsize to generate QueryId in both KademliaHandle and …
0fcd621 
…`Kademlia`
@dmitry-markin
Merge branch 'dm-republish-providers' into dm-get-providers
5f26161
@dmitry-markin
Use open_substream_or_dial() to add provider records to peers
57f17ce
@dmitry-markin
Merge remote-tracking branch 'origin/master' into dm-add-providers
8594103
@dmitry-markin
Fix refresh when we are the only provider
4861959
@dmitry-markin dmitry-markin added the enhancement New feature or request label Sep 18, 2024
lexnv
lexnv approved these changes Sep 20, 2024
View reviewed changes
Copy link
Collaborator

@lexnv lexnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Will probably need to take a quick look at the first PR in the series after all others are merged into it 🙏

@dmitry-markin
Copy link
Collaborator Author

Will probably need to take a quick look at the first PR in the series after all others are merged into it 🙏

My idea was to merge PRs one-by-one into master. When the first PR is merged, GitHub will automatically set the target branch for the second PR to master, and so on.

@lexnv
Copy link
Collaborator

lexnv commented Sep 23, 2024

Sounds like a plan! 👍

@dmitry-markin dmitry-markin mentioned this pull request Sep 27, 2024
Merged
Base automatically changed from dm-stop-providing to master September 30, 2024 14:44
@dmitry-markin dmitry-markin merged commit 1a0325c into master Sep 30, 2024
8 checks passed
@dmitry-markin dmitry-markin deleted the dm-improve-providers-api branch September 30, 2024 17:02
@lexnv lexnv mentioned this pull request Nov 1, 2024
Merged
lexnv added a commit that referenced this pull request Nov 4, 2024
@lexnv @dmitry-markin
chore: Release litep2p version 0.8.0 (#280)
af0535c 
## [0.8.0] - 2024-11-01

This release adds support for content provider advertisement and
discovery to Kademlia protocol implementation (see libp2p
[spec](https://github.com/libp2p/specs/blob/master/kad-dht/README.md#content-provider-advertisement-and-discovery)).
Additionally, the release includes several improvements and memory leak
fixes to enhance the stability and performance of the litep2p library.

### Added

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](#246))
- kad: Providers part 6: stop providing
([#245](#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](#236))
- kad: Providers part 4: refresh local providers
([#235](#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](#234))

### Changed

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](#255))
- kad/executor: Add timeout for writting frames
([#277](#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](#233))
- peer_state: Robust state machine transitions
([#251](#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](#250))
- kad: Remove unused serde cfg
([#262](#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](#254))

### Fixed

- tcp/websocket/quic: Fix cancel memory leak
([#272](#272))
- transport: Fix pending dials memory leak
([#271](#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](#273))
- kad: Fix not retrieving local records
([#221](#221))

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
@lexnv lexnv mentioned this pull request Nov 4, 2024
Merged
github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this pull request Nov 5, 2024
@lexnv @dmitry-markin
litep2p: Update litep2p to v0.8.0 (#6353)
94389a9 
This PR updates litep2p to the latest release.

- `KademliaEvent::PutRecordSucess` is renamed to fix word typo
- `KademliaEvent::GetProvidersSuccess` and
`KademliaEvent::IncomingProvider` are needed for bootnodes on DHT work
and will be utilized later


### Added

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](paritytech/litep2p#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](paritytech/litep2p#246))
- kad: Providers part 6: stop providing
([#245](paritytech/litep2p#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](paritytech/litep2p#236))
- kad: Providers part 4: refresh local providers
([#235](paritytech/litep2p#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](paritytech/litep2p#234))

### Changed

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](paritytech/litep2p#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](paritytech/litep2p#255))
- kad/executor: Add timeout for writting frames
([#277](paritytech/litep2p#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](paritytech/litep2p#233))
- peer_state: Robust state machine transitions
([#251](paritytech/litep2p#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](paritytech/litep2p#250))
- kad: Remove unused serde cfg
([#262](paritytech/litep2p#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](paritytech/litep2p#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](paritytech/litep2p#254))

### Fixed

- tcp/websocket/quic: Fix cancel memory leak
([#272](paritytech/litep2p#272))
- transport: Fix pending dials memory leak
([#271](paritytech/litep2p#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](paritytech/litep2p#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](paritytech/litep2p#273))
- kad: Fix not retrieving local records
([#221](paritytech/litep2p#221))

See release changelog for more details:
https://github.com/paritytech/litep2p/releases/tag/v0.8.0

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
lexnv added a commit to paritytech/polkadot-sdk that referenced this pull request Nov 15, 2024
@lexnv @dmitry-markin
litep2p: Update litep2p to v0.8.0 (#6353)
1cac380 
This PR updates litep2p to the latest release.

- `KademliaEvent::PutRecordSucess` is renamed to fix word typo
- `KademliaEvent::GetProvidersSuccess` and
`KademliaEvent::IncomingProvider` are needed for bootnodes on DHT work
and will be utilized later

- kad: Providers part 8: unit, e2e, and `libp2p` conformance tests
([#258](paritytech/litep2p#258))
- kad: Providers part 7: better types and public API, public addresses &
known providers ([#246](paritytech/litep2p#246))
- kad: Providers part 6: stop providing
([#245](paritytech/litep2p#245))
- kad: Providers part 5: `GET_PROVIDERS` query
([#236](paritytech/litep2p#236))
- kad: Providers part 4: refresh local providers
([#235](paritytech/litep2p#235))
- kad: Providers part 3: publish provider records (start providing)
([#234](paritytech/litep2p#234))

- transport_service: Improve connection stability by downgrading
connections on substream inactivity
([#260](paritytech/litep2p#260))
- transport: Abort canceled dial attempts for TCP, WebSocket and Quic
([#255](paritytech/litep2p#255))
- kad/executor: Add timeout for writting frames
([#277](paritytech/litep2p#277))
- kad: Avoid cloning the `KademliaMessage` and use reference for
`RoutingTable::closest`
([#233](paritytech/litep2p#233))
- peer_state: Robust state machine transitions
([#251](paritytech/litep2p#251))
- address_store: Improve address tracking and add eviction algorithm
([#250](paritytech/litep2p#250))
- kad: Remove unused serde cfg
([#262](paritytech/litep2p#262))
- req-resp: Refactor to move functionality to dedicated methods
([#244](paritytech/litep2p#244))
- transport_service: Improve logs and move code from tokio::select macro
([#254](paritytech/litep2p#254))

- tcp/websocket/quic: Fix cancel memory leak
([#272](paritytech/litep2p#272))
- transport: Fix pending dials memory leak
([#271](paritytech/litep2p#271))
- ping: Fix memory leak of unremoved `pending_opens`
([#274](paritytech/litep2p#274))
- identify: Fix memory leak of unused `pending_opens`
([#273](paritytech/litep2p#273))
- kad: Fix not retrieving local records
([#221](paritytech/litep2p#221))

See release changelog for more details:
https://github.com/paritytech/litep2p/releases/tag/v0.8.0

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lexnv lexnv lexnv approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Networking
Status: Blocked ⛔️
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dmitry-markin @lexnv