Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement configurable requesting of packed chunks from peers #633

Merged
merged 2 commits into from
Oct 22, 2024

Conversation

shizzard
Copy link
Collaborator

@JamesPiechota WIP version implemented

@@ -267,3 +267,10 @@ sync_range({Start, End, Peer, TargetStoreID, RetryCount} = Args) ->
end
end
end.

get_target_packing(StoreID) ->
{ok, Config} = application:get_env(arweave, config),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR looks good! I'd suggest loading the config once during init and storing it in the #state. I'm not sure how significant an impact the get_env call will have (may be minor), but during heavy syncing we might expect this call to happen hundreds of times a second (and billions of times overall) so it may add up

Copy link
Collaborator Author

@shizzard shizzard Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual ar_storage_module:get_packing(StoreID) is O(n) as it traverses the list. Is this will become a problem, this is potential candidate for optimization.

@shizzard shizzard changed the title Implement configurable requesting packed chunks from peers Implement configurable requesting of packed chunks from peers Oct 20, 2024
Copy link
Collaborator

@JamesPiechota JamesPiechota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@shizzard
Copy link
Collaborator Author

shizzard commented Oct 22, 2024

The way to configure the distributed packing.
Vocabulary:

  • miner node: the node collecting the packed chunks;
  • packer node: the node packing chunks for the miner node.

Miner node must have the following configuration options:

  • local_peer <packer-node-host>:<packer-node-port>: should be added for each packer node available;
  • sync_from_local_peers_only: only allows miner node to sync chunks only from the list of local peers;
  • data_sync_request_packed_chunks: forces requesting the already packed chunks.

Packer node must have the following configuration options:

  • enable pack_served_chunks: enables packer node to pack the requested chunks with the address and composite depth requested by the peer.

After starting the packer nodes with the option described above, reconfigure the miner node.
You can verify the actual requested packing by sniffing the traffic on the pacler node, looking for the GET /chunk2/<chink-id> requests and the x-packing header:

GET /chunk2/4566605654 HTTP/1.1
host: 127.0.0.1:1985
X-Block-Format: 3
X-Network: arweave.N.1
X-Version: 8
x-bucket-based-offset: true
x-p2p-port: 1995
x-packing: composite_P6SbdirPgsRBnSz8SBMQd5R1kQaZDoLbwisb9I1wqzk.2  <--
x-release: 72

@shizzard shizzard merged commit 93e2dbf into master Oct 22, 2024
67 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants