Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Change from ipld traversal to direct link access #2931

Conversation

fredcarle
Copy link
Collaborator

Relevant issue(s)

Resolves #2930

Description

This PR removes the use of the ipld traversal function in favor of direct link access. This makes the sync process a bit more efficient and will make it easier to initiate encryption key exchange.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

make test

Specify the platform(s) on which this was tested:

  • MacOS

@fredcarle fredcarle added the area/p2p Related to the p2p networking system label Aug 20, 2024
@fredcarle fredcarle requested a review from a team August 20, 2024 01:14
@fredcarle fredcarle self-assigned this Aug 20, 2024
Copy link

codecov bot commented Aug 20, 2024

Codecov Report

Attention: Patch coverage is 54.83871% with 14 lines in your changes missing coverage. Please review.

Project coverage is 79.34%. Comparing base (fdf4047) to head (36ad742).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
net/sync_dag.go 54.84% 9 Missing and 5 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2931      +/-   ##
===========================================
- Coverage    79.38%   79.34%   -0.04%     
===========================================
  Files          326      326              
  Lines        24769    24778       +9     
===========================================
- Hits         19661    19659       -2     
- Misses        3696     3704       +8     
- Partials      1412     1415       +3     
Flag Coverage Δ
all-tests 79.34% <54.84%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
net/sync_dag.go 58.14% <54.84%> (-24.21%) ⬇️

... and 15 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdf4047...36ad742. Read the comment docs.

Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :)

@fredcarle fredcarle force-pushed the fredcarle/chore/2930-direct-link-access branch from 6b05fa7 to 4901737 Compare August 20, 2024 14:44
Copy link
Contributor

@islamaliev islamaliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Now it's straightforward to comprehend. I don't understand why we used that convoluted selector/builder/walker approach.

if ctx.Err() != nil {
return
}
nd, err := lsys.Load(linking.LinkContext{Ctx: ctx}, lnk, coreblock.SchemaPrototype)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: with this change doesn't it mean we can read directly from datastore.Blockstore in mergeProcessor.loadComposites instead of pulling it from block service?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loadComposites doesn't use a block service. It already reads directly from the blockstore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright, I see

asyncErrOnce.Do(func() { setAsyncErr(err) })
return
}
err = loadBlockLinks(ctx, lsys, linkBlock)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

questions: are we supposed to traverse the whole tree every time an update happens? Shouldn't we stop once we read an existing block?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benefit for the moment is that partial syncs can be completed at a later time. If we stop when we have a known block, then completing partial syncs won't be possible until we implement some tracking logic either for the completely synced or partially synced branches. The implementation before this PR did a complete traversal as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay. Thanks for clarification.

Copy link
Member

@shahzadlone shahzadlone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a question

net/sync_dag.go Outdated Show resolved Hide resolved
@fredcarle fredcarle force-pushed the fredcarle/chore/2930-direct-link-access branch from 4901737 to 36ad742 Compare September 1, 2024 14:09
@fredcarle fredcarle merged commit 9256357 into sourcenetwork:develop Sep 1, 2024
41 of 42 checks passed
@fredcarle fredcarle deleted the fredcarle/chore/2930-direct-link-access branch September 1, 2024 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/p2p Related to the p2p networking system
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch from ipld schema traversal to direct link access.
4 participants