Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(rest): improved message cache logic #2221

Merged
merged 3 commits into from
Nov 28, 2023

Conversation

SionoiS
Copy link
Contributor

@SionoiS SionoiS commented Nov 15, 2023

Description

I refactored the message cache with the intent to allow fetching messages by pubsub OR content topic. Adding a message now add the missing topic either pubsub or content (still need to be subscribed to either). I also added a check to prevent duplicate messages.

The capacity is now TOTAL instead of PER topic.

Changes

  • new message cache
  • updated APIs
  • new tests for new behaviour

Tracking: #2201
Fix: #2192

Copy link

github-actions bot commented Nov 15, 2023

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2221

Built from d3d4bf1

@SionoiS SionoiS force-pushed the chore--improve-message-cache-logic branch from 0c20ae1 to e778198 Compare November 15, 2023 15:51
@SionoiS SionoiS self-assigned this Nov 15, 2023
@SionoiS SionoiS added the E:1.2: Autosharding for autoscaling See https://github.com/waku-org/pm/issues/65 for details label Nov 15, 2023
@SionoiS SionoiS marked this pull request as ready for review November 15, 2023 16:24
@SionoiS SionoiS force-pushed the chore--improve-message-cache-logic branch from e778198 to b75945b Compare November 15, 2023 20:51
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
@SionoiS SionoiS force-pushed the chore--improve-message-cache-logic branch 2 times, most recently from f138828 to 2acc66b Compare November 16, 2023 21:23
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Outdated Show resolved Hide resolved
waku/waku_api/message_cache.nim Show resolved Hide resolved
@SionoiS SionoiS marked this pull request as draft November 17, 2023 16:41
@SionoiS SionoiS removed the request for review from NagyZoltanPeter November 17, 2023 16:42
@SionoiS SionoiS marked this pull request as ready for review November 21, 2023 16:06
@SionoiS SionoiS force-pushed the chore--improve-message-cache-logic branch from d58821a to 99da23b Compare November 21, 2023 16:06
test fixes

Fix legacy handler

Fixes and tests
Copy link
Contributor

@jm-clius jm-clius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM. Since the cache here is generally very small, I would have imagined that a simple, less efficient structure (e.g. just iterating through the cache to get and remove messages based on a filter) would be acceptable, but there may be some complexity that I'm missing.

@SionoiS
Copy link
Contributor Author

SionoiS commented Nov 27, 2023

Generally LGTM. Since the cache here is generally very small, I would have imagined that a simple, less efficient structure (e.g. just iterating through the cache to get and remove messages based on a filter) would be acceptable, but there may be some complexity that I'm missing.

I discussed the impl. at length with @gabrielmer and yes, not my best work. It's done, it works, I don't want to spend even more time on this.

@jm-clius
Copy link
Contributor

I don't want to spend even more time on this

Fair enough 😄

Copy link
Contributor

@gabrielmer gabrielmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks so much! 🤩

@SionoiS SionoiS merged commit bebaa59 into master Nov 28, 2023
9 of 10 checks passed
@SionoiS SionoiS deleted the chore--improve-message-cache-logic branch November 28, 2023 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
E:1.2: Autosharding for autoscaling See https://github.com/waku-org/pm/issues/65 for details
Projects
None yet
3 participants