Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netlink: add ExecuteFunc method to process large responses with low mem overhead #214

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bobby-stripe
Copy link

One of the great things you can do with netlink is query for detailed stats on all open TCP connections, including e.g. retransmits, how much data is in socket buffers, the computed BBR bandwidth estimates, etc. Unfortunately, if you ask the kernel for this info on a host with a lot of connections, the []netlink.Message slice returned by netlink/Conn.Execute can take 50+ MB of space, when the slice will be (likely) be thrown away after its iterated over once.

This PR adds a new method ExecuteFunc where a user can pass in a callback function that is called once per message. Becuase of how Conn.receive/the netlink recvmsg API works in batches/streaming, this dramatically reduces the peak RSS/memory overhead required to process a large stream of netlink message responses.

To try to minimize duplication, I refactored receive into receiveEach which operates on a callback, changed lockedReceive to call receiveEach, and then added a new lockedReceiveEach which is callled by ExecuteFunc (I started with the Each suffix, but then saw that a bunch of the callback variants in the Go stdlib have a Func suffix. LMK if you prefer one or the other or something else and I can standardize the naming).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant