Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message deduplication #854

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Message deduplication #854

wants to merge 11 commits into from

Conversation

snichme
Copy link
Member

@snichme snichme commented Nov 19, 2024

WHAT is this pull request doing?

Implementation for message deduplication on exchanges.
It works for any exchange type and is enabled by setting the argument x-message-deduplication = true on any exchange.

Features

  • Deduplication on Exchanges
  • Deduplication cache storage in memory
  • TTL for each message
  • Default TTL
  • Deduplication on queues

Fixes #833

HOW can this pull request be tested?

Specs.

@snichme snichme requested a review from a team as a code owner November 19, 2024 21:22
@snichme
Copy link
Member Author

snichme commented Nov 19, 2024

TODO: Expose the dedup_count as prometheus metric as well.

@snichme snichme force-pushed the message-deduplication branch from b42efc9 to 3b7828d Compare November 20, 2024 08:37
@snichme
Copy link
Member Author

snichme commented Nov 20, 2024

Documentation for the website: https://gist.github.com/snichme/40a33df7ea9a02c9a7d77a5c81d1f52d

src/lavinmq/amqp/queue/queue.cr Show resolved Hide resolved
src/lavinmq/exchange/dedup_ext.cr Outdated Show resolved Hide resolved
src/lavinmq/exchange/dedup_ext.cr Outdated Show resolved Hide resolved
src/lavinmq/exchange/dedup_ext.cr Outdated Show resolved Hide resolved
@snichme snichme force-pushed the message-deduplication branch from 2ee7ba3 to cbd08d4 Compare November 27, 2024 08:47
src/lavinmq/deduplication.cr Outdated Show resolved Hide resolved
src/lavinmq/deduplication.cr Outdated Show resolved Hide resolved
src/lavinmq/deduplication.cr Outdated Show resolved Hide resolved
@spuun
Copy link
Member

spuun commented Dec 13, 2024

I think the cache needs to be thread-safe?

@snichme snichme force-pushed the message-deduplication branch from 43e7c26 to fb2fb4b Compare January 17, 2025 09:06
@snichme
Copy link
Member Author

snichme commented Jan 17, 2025

I think the cache needs to be thread-safe?

@spuun For it to work properly now or to support MT later on?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Message deduplication
5 participants