feat: memory based connection limits #4281

hanabi1224 · 2023-08-01T13:10:05Z

Description

Implements memory-based connection limits where the user can specify an absolute or a relative limit of the process' memory usage in relation to the available system memory.

Related: #4252.

TODO list:

Refactor connection_limits:Behaviour to take a generic ConnectionLimits type
Rename ConnectionLimits to StaticConnectionLimits
Implement MemoryBasedConnectionLimits
Disable the feature on unsupported platforms
Code docs and examples
Tests

Notes & open questions

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

mxinden · 2023-08-01T13:13:42Z

Thank you for testing this out right-away @hanabi1224. Let me know once you would like a review.

hanabi1224 · 2023-08-01T13:15:06Z

@mxinden It will be great if you can take a quick look and let me know if it's on the right path. The code is done, while docs and tests are still missing

thomaseizinger

Thank you for the contribution!

I'd like to discuss whether we should extend the existing libp2p-connection-limits here or create a new crate: libp2p-memory-connection-limits.

I think there a several arguments for a new crate:

It means that we have more design freedom in regards to the API and can still release it as a patch-release because it is just additions.
There isn't actually any shared code between those two. For example, the memory limits don't care about the connection kind at all. Tthe "put together what changes together" mantra also suggests that this should be separate.
A dedicated NetworkBehaviour will be able to make use of the poll function to register a timer for refreshing the memory stats. Alternatively, we could simply refresh them on every connection attempt (how expensive is gathering those stats?)

misc/connection-limits/src/connection_limits/memory.rs

hanabi1224 · 2023-08-01T13:40:35Z

@thomaseizinger Thanks for your feedback, I can make it a new crate.

Explicitly set `libp2p-kad` `Kademlia::set_mode` to `Mode::Server` to reduce complexity of example, i.e. not having to explicitly set external addresses. Pull-Request: libp2p#4197.

`OutboundQueryCompleted` hasn't been fully replaced by `OutboundQueryProgressed` in kad doc. Pull-Request: libp2p#4257.

`libp2p-relay` crate specifies the `libp2p-swarm/async-std` feature, which causes an `async-std` dependency to always be introduced. Pull-Request: libp2p#4283.

Pull-Request: libp2p#3013.

Pull-Request: libp2p#4288.

move

…tion-limits

thomaseizinger

Thanks! Some suggestions :)

misc/memory-connection-limits/CHANGELOG.md

misc/memory-connection-limits/src/lib.rs

misc/memory-connection-limits/src/connection_limits.rs

misc/memory-connection-limits/src/lib.rs

Co-authored-by: Thomas Eizinger <thomas@eizinger.io>

hanabi1224 · 2023-08-03T16:14:03Z

Thanks! Some suggestions :)

@thomaseizinger Thanks! I've just added 2 unit tests for max_bytes and max_percentage scenarios, please let me know if that's sufficient.

thomaseizinger

Great progress! I've added some more comments :)

misc/memory-connection-limits/src/lib.rs

misc/memory-connection-limits/tests/util.rs

misc/memory-connection-limits/tests/max_percentage.rs

misc/memory-connection-limits/tests/max_bytes.rs

misc/memory-connection-limits/src/error.rs

Co-authored-by: Thomas Eizinger <thomas@eizinger.io>

thomaseizinger

Great work! Two more comments :)

misc/memory-connection-limits/src/lib.rs

…ection-limits

thomaseizinger · 2023-08-07T12:54:34Z

I pushed some follow-up commits, hope you don't mind :)

thomaseizinger

This is ready from my PoV but I've also pushed some commits to this so I'll defer to @mxinden for a final review :)

thomaseizinger · 2023-08-07T12:59:10Z

One more thing: We should add this to the libp2p meta crate and re-export it from there. Plus linking to it from the main changelog would be great! :)

hanabi1224 · 2023-08-07T13:38:42Z

One more thing: We should add this to the libp2p meta crate and re-export it from there. Plus linking to it from the main changelog would be great! :)

Done!

libp2p/CHANGELOG.md

libp2p/Cargo.toml

mxinden

Thank you for working on this @hanabi1224 and @thomaseizinger!

misc/memory-connection-limits/tests/util.rs

Approvals have been dismissed because the PR was updated after the send-it label was applied.

thomaseizinger · 2023-08-08T15:34:37Z

@hanabi1224 No need to manually update the PR once it is queued (i.e. send-it label has been applied). See https://github.com/libp2p/rust-libp2p/blob/master/CONTRIBUTING.md#merging-of-prs-is-automated.

Implements memory-based connection limits where the user can specify an absolute or a relative limit of the process' memory usage in relation to the available system memory. Related: #4252. Pull-Request: #4281.

hanabi1224 added 2 commits August 1, 2023 20:43

feat: memory based connection limits

90cd7d6

sets system ram

5014500

code golf

e5519d7

thomaseizinger reviewed Aug 1, 2023

View reviewed changes

misc/connection-limits/src/connection_limits/memory.rs Outdated Show resolved Hide resolved

misc/connection-limits/src/connection_limits/memory.rs Outdated Show resolved Hide resolved

mxinden and others added 9 commits August 3, 2023 20:02

fix(examples/file-sharing): set Kademlia Mode::Server

024da42

Explicitly set `libp2p-kad` `Kademlia::set_mode` to `Mode::Server` to reduce complexity of example, i.e. not having to explicitly set external addresses. Pull-Request: libp2p#4197.

docs: fix last OutboundQueryCompleted in kad doc

065efb1

`OutboundQueryCompleted` hasn't been fully replaced by `OutboundQueryProgressed` in kad doc. Pull-Request: libp2p#4257.

fix(relay): remove unconditional async-std dependency

2a8e371

`libp2p-relay` crate specifies the `libp2p-swarm/async-std` feature, which causes an `async-std` dependency to always be introduced. Pull-Request: libp2p#4283.

deps(yamux): update yamux to v0.12

94dc11b

Pull-Request: libp2p#3013.

deps: bump Swatinem/rust-cache from 2.5.1 to 2.6.0

51a91b7

Pull-Request: libp2p#4288.

move to new crate

6581822

move

Merge remote-tracking branch 'origin/master' into memory-based-connec…

1106ad7

…tion-limits

doc

6066b2e

lock

1fa32f0

thomaseizinger reviewed Aug 3, 2023

View reviewed changes

hanabi1224 and others added 7 commits August 3, 2023 23:21

doc and tests

05cbf5e

Merge branch 'master' into memory-based-connection-limits

e308be7

Apply suggestions from code review

7d05ee5

Co-authored-by: Thomas Eizinger <thomas@eizinger.io>

hide MemoryUsageBasedConnectionLimits

9ba799e

update MemoryUsageLimitExceeded fmt as suggested

b8a9916

fix clippy

f43fed5

fix rustdoc

9a66866

thomaseizinger reviewed Aug 4, 2023

View reviewed changes

hanabi1224 and others added 3 commits August 4, 2023 16:13

Update misc/memory-connection-limits/src/lib.rs

7990be6

Co-authored-by: Thomas Eizinger <thomas@eizinger.io>

resolve comments

07e30bb

Make MemoryUsageLimitExceeded fields private

6adf21f

thomaseizinger reviewed Aug 6, 2023

View reviewed changes

misc/memory-connection-limits/src/lib.rs Outdated Show resolved Hide resolved

misc/memory-connection-limits/src/lib.rs Outdated Show resolved Hide resolved

thomaseizinger and others added 8 commits August 7, 2023 14:41

Remove refresh-interval field

1b7fb24

Use early return to reduce indentation

280d13a

Only set timestamp if we actually refreshed the data

aadad16

Use early return instead of else

2875817

Make tests for stable

800d501

Add docs to constant

7331b8f

Sort functions are per trait definition

a1d0b55

Merge branch 'memory-connection-limits-thomas' into memory-based-conn…

9da8e1e

…ection-limits

hanabi1224 marked this pull request as ready for review August 7, 2023 12:55

thomaseizinger previously approved these changes Aug 7, 2023

View reviewed changes

Re-export from libp2p meta crate

b24fb86

Merge branch 'master' into memory-based-connection-limits

14f0866

thomaseizinger reviewed Aug 7, 2023

View reviewed changes

libp2p/CHANGELOG.md Outdated Show resolved Hide resolved

Update libp2p/CHANGELOG.md

9eeee38

thomaseizinger reviewed Aug 7, 2023

View reviewed changes

libp2p/Cargo.toml Show resolved Hide resolved

mxinden previously approved these changes Aug 7, 2023

View reviewed changes

misc/memory-connection-limits/tests/util.rs Show resolved Hide resolved

thomaseizinger added send-it and removed send-it labels Aug 7, 2023

Merge branch 'master' into memory-based-connection-limits

13f09a8

thomaseizinger approved these changes Aug 8, 2023

View reviewed changes

mergify bot merged commit b5d9932 into libp2p:master Aug 8, 2023
66 of 67 checks passed

hanabi1224 deleted the memory-based-connection-limits branch August 9, 2023 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: memory based connection limits #4281

feat: memory based connection limits #4281

hanabi1224 commented Aug 1, 2023 •

edited

Loading

mxinden commented Aug 1, 2023

hanabi1224 commented Aug 1, 2023

thomaseizinger left a comment

hanabi1224 commented Aug 1, 2023

thomaseizinger left a comment

hanabi1224 commented Aug 3, 2023

thomaseizinger left a comment

thomaseizinger left a comment

thomaseizinger commented Aug 7, 2023

thomaseizinger left a comment

thomaseizinger commented Aug 7, 2023

hanabi1224 commented Aug 7, 2023

mxinden left a comment

thomaseizinger commented Aug 8, 2023

feat: memory based connection limits #4281

feat: memory based connection limits #4281

Conversation

hanabi1224 commented Aug 1, 2023 • edited Loading

Description

TODO list:

Notes & open questions

Change checklist

mxinden commented Aug 1, 2023

hanabi1224 commented Aug 1, 2023

thomaseizinger left a comment

Choose a reason for hiding this comment

hanabi1224 commented Aug 1, 2023

thomaseizinger left a comment

Choose a reason for hiding this comment

hanabi1224 commented Aug 3, 2023

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Aug 7, 2023

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Aug 7, 2023

hanabi1224 commented Aug 7, 2023

mxinden left a comment

Choose a reason for hiding this comment

thomaseizinger commented Aug 8, 2023

hanabi1224 commented Aug 1, 2023 •

edited

Loading