Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add size retention policy #2093

Merged
merged 15 commits into from
Sep 30, 2023
Merged

Conversation

ABresting
Copy link
Contributor

@ABresting ABresting commented Sep 29, 2023

Description

Size retention Policy has been added. Based on the size limit provided in mb or gb. Once the size limit overflows by messages, then the 20% of the outdated messages are dropped and a vacuum of database for defragmentation is triggered.

Changes

  • Size related retention policy
  • Vacuum after the deletion of database entries

@ABresting ABresting changed the title Chore add size retention policy chore: add size retention policy Sep 29, 2023
@github-actions
Copy link

github-actions bot commented Sep 29, 2023

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2093

Built from 8a8e107

Copy link
Contributor

@SionoiS SionoiS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thank you!

See my comments for minor changes.

tests/waku_archive/test_retention_policy Outdated Show resolved Hide resolved
tests/waku_archive/test_retention_policy.nim Outdated Show resolved Hide resolved
waku/waku_archive/driver.nim Outdated Show resolved Hide resolved
waku/waku_archive/driver/queue_driver/queue_driver.nim Outdated Show resolved Hide resolved
@@ -51,5 +52,38 @@ proc new*(T: type RetentionPolicy,
let retPolicy: RetentionPolicy = CapacityRetentionPolicy.init(retentionCapacity)
return ok(some(retPolicy))

elif policy == "size":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated but why is this not a enum instead of a string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overkill for limited parameters?

Comment on lines +69 to +71
var numMessagesRes = await driver.getMessagesCount()
if numMessagesRes.isErr():
return err("failed to get messages count: " & numMessagesRes.error)
var numMessages = numMessagesRes.value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Suggested change
var numMessagesRes = await driver.getMessagesCount()
if numMessagesRes.isErr():
return err("failed to get messages count: " & numMessagesRes.error)
var numMessages = numMessagesRes.value
var numMessages = await driver.getMessagesCount().valueOr:
return err("failed to get messages count: " & error)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idk what if there is a 0 message condition, and someone triggers the policy check on an empty database?

@ABresting ABresting requested a review from SionoiS September 29, 2023 17:43
ABresting and others added 15 commits September 30, 2023 10:25
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* Add test aggregator to all directories.
* Implement coverage script.
Also use absolute path to load Groovy script.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
…tially concurrent, add version (#2080)

* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version

* add more metrics, refactor how most metrics are calculated

* rework metrics table fillup

* reset connErr to make sure we honour successful reconnection
…2079)

* Adding cpp example that integrates the `libwaku`

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
@ABresting ABresting force-pushed the chore-add-size-retention-policy branch from b539a43 to 806c7b9 Compare September 30, 2023 04:57
@github-actions
Copy link

This PR may contain changes to configuration options of one of the apps.

If you are introducing a breaking change (i.e. the set of options in latest release would no longer be applicable) make sure the original option is preserved with a deprecation note for 2 following releases before it is actually removed.

Please also make sure the label release-notes is added to make sure any changes to the user interface are properly announced in changelog and release notes.

@ABresting ABresting merged commit 8897ae1 into master Sep 30, 2023
6 of 10 checks passed
@ABresting ABresting deleted the chore-add-size-retention-policy branch September 30, 2023 05:40
jm-clius added a commit that referenced this pull request Oct 2, 2023
This reverts commit 8897ae1.

(cherry picked from commit b213b2c385b0534481448cd6e30af18e183d0504)
jm-clius added a commit that referenced this pull request Oct 2, 2023
This reverts commit 8897ae1.

(cherry picked from commit b213b2c385b0534481448cd6e30af18e183d0504)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants