Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Simplify Merkle CRDT workflow #2111

Merged

Conversation

fredcarle
Copy link
Collaborator

Relevant issue(s)

Resolves #2110
Possibly Resolves #917

Description

This PR aims to simplify the CRDT packages ahead of the new CRDT types.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

make test

Specify the platform(s) on which this was tested:

  • MacOS

@fredcarle fredcarle added area/crdt Related to the (Merkle) CRDT system refactor This issue specific to or requires *notable* refactoring of existing codebases and components labels Dec 5, 2023
@fredcarle fredcarle added this to the DefraDB v0.9 milestone Dec 5, 2023
@fredcarle fredcarle requested a review from a team December 5, 2023 21:05
@fredcarle fredcarle self-assigned this Dec 5, 2023
@fredcarle fredcarle changed the title refactor: Simplify Merkle CRDT workflow. refactor: Simplify Merkle CRDT workflow Dec 5, 2023
Copy link

codecov bot commented Dec 5, 2023

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (eda5b6b) 73.91% compared to head (ba0975c) 73.97%.

❗ Current head ba0975c differs from pull request most recent head 4f69bc3. Consider uploading reports for the commit 4f69bc3 to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2111      +/-   ##
===========================================
+ Coverage    73.91%   73.97%   +0.05%     
===========================================
  Files          249      247       -2     
  Lines        24859    24672     -187     
===========================================
- Hits         18374    18249     -125     
+ Misses        5223     5178      -45     
+ Partials      1262     1245      -17     
Flag Coverage Δ
all-tests 73.97% <88.61%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
core/crdt/base.go 75.00% <100.00%> (+1.92%) ⬆️
core/crdt/composite.go 70.00% <100.00%> (-1.65%) ⬇️
core/crdt/lwwreg.go 65.88% <100.00%> (-3.91%) ⬇️
db/collection_delete.go 33.15% <100.00%> (-0.37%) ⬇️
db/db.go 63.50% <ø> (-0.78%) ⬇️
db/errors.go 71.74% <ø> (ø)
db/fetcher/versioned.go 61.17% <100.00%> (+1.38%) ⬆️
merkle/crdt/composite.go 83.33% <100.00%> (+4.02%) ⬆️
merkle/crdt/lwwreg.go 100.00% <100.00%> (+15.79%) ⬆️
net/process.go 82.76% <100.00%> (+0.87%) ⬆️
... and 3 more

... and 10 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eda5b6b...4f69bc3. Read the comment docs.

Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this Fred! Changes look good, it should be much easier to follow now, just added a couple of extra suggestions/minor questions but am happy for this to be merged regardless of whether they are actioned.

I suggest closing #917 - the removal of the factory stuff in particular will make this much easier to follow, and we can always create a new issue if it remains a problem error instead of cluttering the backlog.

client/errors.go Outdated Show resolved Hide resolved
core/crdt/composite.go Show resolved Hide resolved
core/crdt/composite.go Show resolved Hide resolved
merkle/crdt/composite.go Show resolved Hide resolved
@shahzadlone
Copy link
Member

Wow 👌 much cleaner. Not very well versed with this part of the code but LGTM from what I can see.

Copy link
Contributor

@islamaliev islamaliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Just added couple minor suggestions.

core/crdt/base.go Outdated Show resolved Hide resolved
db/collection.go Show resolved Hide resolved
merkle/crdt/merklecrdt.go Show resolved Hide resolved
net/process.go Show resolved Hide resolved
Copy link
Member

@jsimnz jsimnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone over this twice, and will prob go over it again two more times 😂 as there is a few subtle changes I want to make sure don't cause any issues with current and future CRDT implementations.

Two current suggestions/todos noted.

}

// Publishes the delta to state.
func (base *baseMerkleCRDT) Publish(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: I am hesitant to remove this Publish function as it keeps a cleaner seperation of concerns when writting MerkleCRDT delta-mutator functions.

The recipe is simple at the moment. Call the core crdt equivalent mutator, then call Publish(delta).

Even if Publish is just calling clock.AddDAGNode under the hood, it feels like its needlessly bleeding abstractions, and lose on some of the purity from before.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I saw it is that when we called merkleCRDT.Set(...) and there was a call to Publish, we had to go to Publish only to find out that it was only calling clock.AddDAGNode and I found it obstructive more than anything. The way it was been changed makes the flow very clear. We set the CRDT delta object and then send that object to the clock for insertion into the DAG. Can't get purer than that if you ask me.

Copy link
Contributor

@AndrewSisley AndrewSisley Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that I agree with Fred here - I find the new, less-abstract, flow easier to read.

merkle/crdt/composite.go Outdated Show resolved Hide resolved
@fredcarle fredcarle force-pushed the fredcarle/refactor/merkle-crdt branch 3 times, most recently from a560a63 to ba0975c Compare December 6, 2023 20:34
@fredcarle fredcarle force-pushed the fredcarle/refactor/merkle-crdt branch from ba0975c to 4f69bc3 Compare December 8, 2023 01:49
@fredcarle fredcarle merged commit 48d0c24 into sourcenetwork:develop Dec 8, 2023
27 of 28 checks passed
@fredcarle fredcarle deleted the fredcarle/refactor/merkle-crdt branch December 8, 2023 02:20
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Jan 22, 2024
## Relevant issue(s)

Resolves sourcenetwork#2110 
Possibly Resolves sourcenetwork#917

## Description

This PR aims to simplify the CRDT packages ahead of the new CRDT types.
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
## Relevant issue(s)

Resolves sourcenetwork#2110 
Possibly Resolves sourcenetwork#917

## Description

This PR aims to simplify the CRDT packages ahead of the new CRDT types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/crdt Related to the (Merkle) CRDT system refactor This issue specific to or requires *notable* refactoring of existing codebases and components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Simplify Merkle CRDT workflow. Write logic seems far harder to read than it needs to be
5 participants