chore(archive): archive and drivers refactor #2761

SionoiS · 2024-06-03T14:33:24Z

This is a feature branch used to coordinate a big refactor of archive and all drivers to clean/optimize specifically for Store v3.

This work is required before we start measuring performance of Store v3.

@gabrielmer & @Ivansete-status already reviewed all the PRs but if you want to double check go ahead.

What's new are non-PR related commits.

Related PRs;

Closes:

github-actions · 2024-06-05T11:24:29Z

This PR may contain changes to database schema of one of the drivers.

If you are introducing any changes to the schema, make sure the upgrade from the latest release to this change passes without any errors/issues.

Please make sure the label release-notes is added to make sure upgrade instructions properly highlight this change.

SionoiS · 2024-06-05T11:49:37Z

~~@SionoiS #2756 (comment)~~ done

github-actions · 2024-06-06T13:00:09Z

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2761-rln-v1

Built from 3ffa8f5

github-actions · 2024-06-06T13:00:23Z

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2761-rln-v2

Built from 3ffa8f5

SionoiS · 2024-06-06T14:04:22Z

I have some more issues to fix before review.

SionoiS · 2024-06-07T15:39:41Z

CI is red but I can't reproduce anything...

Ivansete-status

Thanks for this PR! This is a very complex task and great work so far!
I'm adding some comments that I hope you find useful.
Let's merge when we revisit all the comments and all CI are green (but the js-waku ones)

Ivansete-status · 2024-06-10T09:45:30Z

migrations/message_store_postgres/content_script_version_6.nim

@@ -0,0 +1,72 @@
+const ContentScriptVersion_6* =
+  """
+ALTER TABLE IF EXISTS messages_backup RENAME TO messages;


We cannot follow the approach of creating a temporary table because there is normally enough disk space to duplicate the current one. Considering that, we need to perform the database changes directly into the partitioned table, i.e. messages table. That will likely apply the change to the underlying partitions.

You mean all changes have to be done in-place? Can this be done even for a primary key? Seams complicated with partitions too. I've been looking into it but I can't find a way to do this. We can't even know the name of the partitions in advance.

edit: i did find a way but it doesn't solve the problem of disk space overhead when migrating.

Ivansete-status · 2024-06-10T09:45:43Z

migrations/message_store_postgres/content_script_version_6.nim

+   version INTEGER NOT NULL,
+   timestamp BIGINT NOT NULL,
+   meta VARCHAR,
+   id VARCHAR,


Would it be possible to also remove the id field?

I was thinking that we keep the old values for now so that we are still compatible with v2. Would that not work?

tests/all_tests_waku.nim

tests/waku_archive/test_driver_queue_index.nim

vendor/negentropy

SionoiS · 2024-06-12T19:48:06Z

Latest error in CI

ERR 2024-06-12 19:30:49.752+00:00 error creating ArchiveDriver               tid=8120 file=postgres_legacy.nim:14 error="postgres health check error: connRes.isErr in query: pool is not live"

I can't reproduce locally.

Ivansete-status · 2024-06-20T14:10:31Z

Just for our reference, the following PR is waiting for this #2761 one to be merged: status-im/status-go#5123

Once merged we will need to ping @richard-ramos and then we will test this in status.staging

( cc @SionoiS )

github-actions · 2024-07-02T15:27:12Z

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2761

Built from 8dc0454

renaming to pubsub

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* posgres legacy: stop using the storedAt field * migration script 6: we still need the id column The id column is needed because it contains the message digest which is used in store v2, and we need to keep support to store v2 for a while * legacy archive: set target migration version to 6 * waku_node: try to use wakuLegacyArchive if wakuArchive is nil * node_factory, waku_node: mount legacy and future store simultaneously We want the nwaku node to simultaneously support store-v2 requests and store-v3 requests. Only the legacy archive is in charge of archiving messages, and the archived information is suitable to fulfill both store-v2 and store-v3 needs. * postgres_driver: adding temporary code until store-v2 is removed --------- Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>

Ivansete-status · 2024-07-12T16:21:55Z

Thank you so much guys for the patience and energy invested in this PR 🙌
I think it looks nice! ( until we find another bug ;P )

Ivansete-status · 2024-07-12T17:26:48Z

I've tested that the schema migration works well in a 121GB database ( in wakudev host .)

github-actions bot assigned SionoiS Jun 3, 2024

SionoiS changed the title ~~chore(archive) archive and drivers refactor~~ chore(archive): archive and drivers refactor Jun 3, 2024

SionoiS mentioned this pull request Jun 5, 2024

chore(archive): tests refactor #2756

Merged

SionoiS force-pushed the chore--archive-drivers-refactor branch from 254a419 to 9f90434 Compare June 6, 2024 11:37

SionoiS marked this pull request as ready for review June 6, 2024 12:41

SionoiS requested review from gabrielmer and Ivansete-status June 6, 2024 12:49

SionoiS removed request for gabrielmer and Ivansete-status June 6, 2024 14:03

SionoiS marked this pull request as draft June 6, 2024 14:03

SionoiS marked this pull request as ready for review June 7, 2024 15:39

SionoiS force-pushed the chore--archive-drivers-refactor branch from 6f17946 to 35a4412 Compare June 7, 2024 15:50

SionoiS requested review from gabrielmer and Ivansete-status June 7, 2024 19:02

Ivansete-status reviewed Jun 11, 2024

View reviewed changes

SionoiS added this to the v0.31.0 milestone Jun 11, 2024

SionoiS force-pushed the chore--archive-drivers-refactor branch from 35a4412 to 937c247 Compare June 12, 2024 19:17

SionoiS force-pushed the chore--archive-drivers-refactor branch from 937c247 to ba56d60 Compare July 2, 2024 15:22

SionoiS force-pushed the chore--archive-drivers-refactor branch from ba56d60 to d3af693 Compare July 4, 2024 11:34

Ivansete-status mentioned this pull request Jul 5, 2024

chore: simple PR to enhance postgres and retention policy logs #2884

Merged

SionoiS removed the request for review from gabrielmer July 8, 2024 11:57

SionoiS and others added 20 commits July 12, 2024 15:23

chore(archive): archive refactor (#2752)

d5a2ac9

chore(archive): sqlite driver refactor (#2754)

ab23f16

chore(archive): postgres driver refactor (#2755)

2cab4db

chore(archive): tests refactor (#2756)

951da1f

renaming to pubsub

chore(archive): renaming & copies (#2751)

85fc6c0

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

misc. fixes

f37db9a

remove merge duplicate

8fe3937

fix testlib not use legacy archive

cd2140c

migration script & db output fix

c1d9349

adding back a test

cce89ed

new improved migration script

ecbd2a4

rebase fixes & clean-up

5723841

new multistep migration script

ea4d1be

rebase fixes

66cfea5

more fixes

171e9f3

one more fix

e6161f1

fix storedat -> timestamp

36c2082

disabling legacy archive tests

70499fa

added back auto migration script

fc49d82

Ivansete-status force-pushed the chore--archive-drivers-refactor branch from 16b692c to 423b4d8 Compare July 12, 2024 13:24

sqlite_driver/queries: remove wrong code result of rebase with master

cc7b0a6

Ivansete-status approved these changes Jul 12, 2024

View reviewed changes

gabrielmer approved these changes Jul 12, 2024

View reviewed changes

Ivansete-status merged commit f54ba10 into master Jul 12, 2024
8 of 10 checks passed

Ivansete-status deleted the chore--archive-drivers-refactor branch July 12, 2024 16:19

This was referenced Jul 12, 2024

bug(storev3): can't advance cursor #2745

Closed

chore(storev3): only select the messageHash column when using a store query with include_data: false #2637

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(archive): archive and drivers refactor #2761

chore(archive): archive and drivers refactor #2761

SionoiS commented Jun 3, 2024 •

edited

Loading

github-actions bot commented Jun 5, 2024

SionoiS commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

SionoiS commented Jun 6, 2024 •

edited

Loading

SionoiS commented Jun 7, 2024

Ivansete-status left a comment

Ivansete-status Jun 10, 2024

SionoiS Jun 11, 2024 •

edited

Loading

Ivansete-status Jun 10, 2024

SionoiS Jun 11, 2024

SionoiS commented Jun 12, 2024

Ivansete-status commented Jun 20, 2024

github-actions bot commented Jul 2, 2024 •

edited

Loading

Ivansete-status commented Jul 12, 2024

Ivansete-status commented Jul 12, 2024

chore(archive): archive and drivers refactor #2761

chore(archive): archive and drivers refactor #2761

Conversation

SionoiS commented Jun 3, 2024 • edited Loading

github-actions bot commented Jun 5, 2024

SionoiS commented Jun 5, 2024 • edited Loading

github-actions bot commented Jun 6, 2024 • edited Loading

github-actions bot commented Jun 6, 2024 • edited Loading

SionoiS commented Jun 6, 2024 • edited Loading

SionoiS commented Jun 7, 2024

Ivansete-status left a comment

Choose a reason for hiding this comment

Ivansete-status Jun 10, 2024

Choose a reason for hiding this comment

SionoiS Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Ivansete-status Jun 10, 2024

Choose a reason for hiding this comment

SionoiS Jun 11, 2024

Choose a reason for hiding this comment

SionoiS commented Jun 12, 2024

Ivansete-status commented Jun 20, 2024

github-actions bot commented Jul 2, 2024 • edited Loading

Ivansete-status commented Jul 12, 2024

Ivansete-status commented Jul 12, 2024

SionoiS commented Jun 3, 2024 •

edited

Loading

SionoiS commented Jun 5, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

SionoiS commented Jun 6, 2024 •

edited

Loading

SionoiS Jun 11, 2024 •

edited

Loading

github-actions bot commented Jul 2, 2024 •

edited

Loading