Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Message deduplication for exchanges #853

Closed
wants to merge 175 commits into from
Closed

Message deduplication for exchanges #853

wants to merge 175 commits into from

Conversation

snichme
Copy link
Member

@snichme snichme commented Nov 19, 2024

WHAT is this pull request doing?

Implementation for message deduplication on exchanges.
It works for any exchange type and is enabled by setting the argument x-message-deduplication = true on any exchange.

Features

  • Deduplication on Exchanges
  • Deduplication cache storage in memory
  • TTL for each message
  • Default TTL
  • Deduplication on queues
  • Deduplication cache storage on disk
  • Apply deduplication using policy

Fixes #833

HOW can this pull request be tested?

Specs.

kickster97 and others added 30 commits November 13, 2024 14:02
@kickster97
beginning of mqtt-poc, ping
60cab17
@kickster97
add send/recieve oct_count, make prometheus controller compile
3462c8e
@kickster97
client stores client_id and quick solution for UI
7e20f25
@kickster97
amqp publish
eece84f
@snichme @kickster97
mqtt integration spec: ping
20660fd
@kickster97
fixup! amqp publish
7c99bfb
@snichme @kickster97
int-specs
afa1f90
@snichme @kickster97
cleanup
76f7520
@snichme @kickster97
connect specs (pending)
690455d
@snichme @kickster97
all integration specs (pending)
6b89644
@snichme @kickster97
Synchronizer -> channel
a2e50f1
@snichme @kickster97
spec: publish mqtt ends up in amqp queue
d10dec4
@kickster97
send puback after publish if qos > 0
f735cdb
@kickster97
publish will to session WIP
bd2885a
@snichme @kickster97
publish returns PubAck
84c7701
@kickster97
cleanup will preparation
1326ee0
@snichme @kickster97
Got first working subscribe
14d7d5a 
But a lot of things missing :)
@snichme @kickster97
subscribe specs
99a52f5
@snichme @kickster97
packet_id stuff
bd84fc9
@kickster97
pass more connect specs
6d260ee
@kickster97
work on subscriber specs
c81c15e
@kickster97
start working on broker for clients
9deb2af
@kickster97
wrap vhost in broker for mqtt::client
da8ef99
@kickster97
pass more specs and handle session_present
ef792f5
@kickster97
all connect specs working except replace client with new connection
f825a44
@kickster97
fix logs in client and dot expect connack in specs
9ca561b
@kickster97
add sessions to broker, and handle incoming subscribe with session
ce14c4e
@kickster97
fixup! add sessions to broker, and handle incoming subscribe with ses…
cdb2b81 
…sion
@kickster97
fixup! fixup! add sessions to broker, and handle incoming subscribe w…
537fc53 
…ith session
@kickster97
subscribe and bind a session
7ef8ad8
kickster97 and others added 27 commits November 13, 2024 14:02
@kickster97
satisfy ameba for spec files
7c66aed
@kickster97
rename specfile with suffix
5348b77
@kickster97
fix flaky wait_for
aefc93b
@spuun @kickster97
Use enum for protocol to get compile time validation
4341545
@spuun @kickster97
Fix specs to use protocol enum
d1cb30e
@kickster97
use short block notation
13ae093
@snichme
fix mqtt exchange routing spec
f01d21e
@snichme
remove comment
a21fa05
@snichme
scope fix
6ef6527
@spuun
Multi-vhost support
571b7fb
@kickster97
expand details tuple for consumer UI
7f06362
@kickster97
no need to convert routing key
0a73f6e
@kickster97
format
e32c154
@kickster97
handle unexpected close from client
37d13a8
@snichme
connection_at for mqtt connections
bdc39bc
@snichme
deliver packet not msg from session (#843)
d256cd1 
* deliver packet not msg from session

* it's already a topic, no convert needed

* fixes
@kickster97
truncate the previous content before you retain a message
d82a762
@kickster97
safely overwrite retained messages
8d47862
@spuun
Cant use constant as key in NamedTuple
4189531
@kickster97
merge solutions for retain store
5e7cc4b
@kickster97
general fixup after comments
a3202d1
@kickster97
format
43ec295
@kickster97
set flaky spec to pending
d358a60
@kickster97
just a test
303b81e
@kickster97
just a test
393a53b
@baelter
tmp: debug clustering_spec
3f0c1ab
@snichme
message dedup for exchange
72e00f7
@snichme snichme requested a review from a team as a code owner November 19, 2024 21:21
@snichme snichme closed this Nov 19, 2024
@snichme snichme deleted the message-dedup-exchange branch November 19, 2024 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Message deduplication
4 participants
@snichme @kickster97 @spuun @baelter