Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add PN Counter CRDT type #2119

Merged

Conversation

fredcarle
Copy link
Collaborator

Relevant issue(s)

Resolves #2115

Description

This PR adds the PN Counter CRDT type. This type enforces changes that are incrementing or decrementing the value. A value cannot be set directly.

The PR also simplifies the CRDT flow a bit further. For example, the Data field on the composite CRDT has been removed since the type links to the changed fields. This change will removes the data leak that would be present once we implement field level access control.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

make test with added integrations tests

Specify the platform(s) on which this was tested:

  • MacOS

@fredcarle fredcarle added feature New feature or request area/crdt Related to the (Merkle) CRDT system labels Dec 8, 2023
@fredcarle fredcarle added this to the DefraDB v0.9 milestone Dec 8, 2023
@fredcarle fredcarle requested a review from a team December 8, 2023 17:39
@fredcarle fredcarle self-assigned this Dec 8, 2023
Copy link

codecov bot commented Dec 8, 2023

Codecov Report

Attention: 71 lines in your changes are missing coverage. Please review.

Comparison is base (9241b4c) 74.29% compared to head (4beaa19) 74.23%.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2119      +/-   ##
===========================================
- Coverage    74.29%   74.23%   -0.06%     
===========================================
  Files          252      256       +4     
  Lines        25242    25366     +124     
===========================================
+ Hits         18752    18829      +77     
- Misses        5217     5261      +44     
- Partials      1273     1276       +3     
Flag Coverage Δ
all-tests 74.23% <76.17%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
cli/utils.go 46.27% <ø> (+1.98%) ⬆️
client/document.go 64.07% <100.00%> (-2.38%) ⬇️
client/errors.go 62.71% <100.00%> (+5.85%) ⬆️
config/config.go 74.13% <100.00%> (+0.18%) ⬆️
core/crdt/composite.go 69.37% <100.00%> (-0.63%) ⬇️
core/crdt/lwwreg.go 69.23% <100.00%> (-0.18%) ⬇️
db/base/collection_keys.go 80.00% <100.00%> (ø)
db/collection_delete.go 32.77% <ø> (-0.38%) ⬇️
db/errors.go 67.48% <ø> (-0.67%) ⬇️
db/fetcher/indexer_iterators.go 82.06% <100.00%> (ø)
... and 14 more

... and 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9241b4c...4beaa19. Read the comment docs.

Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - thanks Fred. I'm approving now with a handful of small comments open, assuming that they will all be resolved sensibly.

I might have more comments/questions depending on the documentation added in db/collection.go RE todo: Please document why LLWRs should not do this, but as it is a very local issue and I'm going away please don't block the merging of this PR until I'm back if I don't get back to you in the meantime.

client/ctype.go Outdated Show resolved Hide resolved
client/ctype.go Show resolved Hide resolved
client/value.go Show resolved Hide resolved
core/crdt/composite.go Outdated Show resolved Hide resolved
core/crdt/pncounter.go Outdated Show resolved Hide resolved
tests/integration/schema/crdt_type_test.go Show resolved Hide resolved
tests/integration/schema/crdt_type_test.go Outdated Show resolved Hide resolved
jsimnz
jsimnz previously requested changes Dec 11, 2023
Copy link
Member

@jsimnz jsimnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only part way through, but wanted to submit this now.

core/crdt/pncounter.go Outdated Show resolved Hide resolved
core/crdt/pncounter.go Outdated Show resolved Hide resolved
client/ctype.go Outdated Show resolved Hide resolved
merkle/crdt/pncounter.go Outdated Show resolved Hide resolved
core/crdt/pncounter.go Outdated Show resolved Hide resolved
Copy link
Contributor

@islamaliev islamaliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great so far. I just have few questions and minor requests.

config/config.go Outdated Show resolved Hide resolved
core/crdt/pncounter.go Show resolved Hide resolved
db/collection.go Outdated Show resolved Hide resolved
tests/integration/mutation/create/crdt/pncounter_test.go Outdated Show resolved Hide resolved
tests/integration/mutation/update/crdt/pncounter_test.go Outdated Show resolved Hide resolved
tests/integration/schema/crdt_type_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@islamaliev islamaliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@fredcarle fredcarle force-pushed the fredcarle/feat/crdt-pn-counter branch 2 times, most recently from 01d9b7a to f62b2b9 Compare January 6, 2024 19:17
@fredcarle fredcarle force-pushed the fredcarle/feat/crdt-pn-counter branch 7 times, most recently from 4f4ff6f to 63e011a Compare January 8, 2024 18:47
`cfg.LoadRootDirFromFlagOrDefault` is already called by `cfg.LoadWithRootdir`.
added a `WithCustomValues` method on `Config` to persis changes on the viper config. This will ensure that when the cli is executed, it uses the manually assigned values and not just the default.
This commit applies all the received feedback plus adds support for Float kind in for the PN Counter.
@fredcarle fredcarle force-pushed the fredcarle/feat/crdt-pn-counter branch from 8d3bd46 to f78e33b Compare January 8, 2024 19:19
@fredcarle fredcarle dismissed jsimnz’s stale review January 8, 2024 20:10

Discussed in standup that we will revisit if concerns are validated in the future

@fredcarle fredcarle merged commit f7a4166 into sourcenetwork:develop Jan 8, 2024
30 of 31 checks passed
@fredcarle fredcarle deleted the fredcarle/feat/crdt-pn-counter branch January 8, 2024 20:10
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Jan 22, 2024
## Relevant issue(s)

Resolves sourcenetwork#2115

## Description

This PR adds the PN Counter CRDT type. This type enforces changes that
are incrementing or decrementing the value. A value cannot be set
directly.

The PR also simplifies the CRDT flow a bit further. For example, the
`Data` field on the composite CRDT has been removed since the type links
to the changed fields. This change will removes the data leak that would
be present once we implement field level access control.
shahzadlone pushed a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
## Relevant issue(s)

Resolves sourcenetwork#2115

## Description

This PR adds the PN Counter CRDT type. This type enforces changes that
are incrementing or decrementing the value. A value cannot be set
directly.

The PR also simplifies the CRDT flow a bit further. For example, the
`Data` field on the composite CRDT has been removed since the type links
to the changed fields. This change will removes the data leak that would
be present once we implement field level access control.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/crdt Related to the (Merkle) CRDT system feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

PN Counter CRDT
5 participants