Code

Message deduplication for exchanges#853

Code

Closed

snichme wants to merge 175 commits intomaincloudamqp/lavinmq:mainfrom message-dedup-exchangecloudamqp/lavinmq:message-dedup-exchangeCopy head branch name to clipboard

+3,878-90

Commits

Commits on Nov 13, 2024

• beginning of mqtt-poc, ping

  kickster97

  committed
  60cab17

• add send/recieve oct_count, make prometheus controller compile

  kickster97

  committed
  3462c8e

• client stores client_id and quick solution for UI

  kickster97

  committed
  7e20f25

• amqp publish

  kickster97

  committed
  eece84f

• mqtt integration spec: ping

  

  snichme

  authored and

  kickster97

  committed
  20660fd

• fixup! amqp publish

  kickster97

  committed
  7c99bfb

• int-specs

  

  snichme

  authored and

  kickster97

  committed
  afa1f90

• cleanup

  

  snichme

  authored and

  kickster97

  committed
  76f7520

• connect specs (pending)

  

  snichme

  authored and

  kickster97

  committed
  690455d

• all integration specs (pending)

  

  snichme

  authored and

  kickster97

  committed
  6b89644

• Synchronizer -> channel

  

  snichme

  authored and

  kickster97

  committed
  a2e50f1

• spec: publish mqtt ends up in amqp queue

  

  snichme

  authored and

  kickster97

  committed
  d10dec4

• send puback after publish if qos > 0

  kickster97

  committed
  f735cdb

• publish will to session WIP

  kickster97

  committed
  bd2885a

• publish returns PubAck

  

  snichme

  authored and

  kickster97

  committed
  84c7701

• cleanup will preparation

  kickster97

  committed
  1326ee0

• Got first working subscribe

  Show description for 14d7d5a
  

  snichme

  authored and

  kickster97

  committed
  14d7d5a

• subscribe specs

  

  snichme

  authored and

  kickster97

  committed
  99a52f5

• packet_id stuff

  

  snichme

  authored and

  kickster97

  committed
  bd84fc9

• pass more connect specs

  kickster97

  committed
  6d260ee

• work on subscriber specs

  kickster97

  committed
  c81c15e

• start working on broker for clients

  kickster97

  committed
  9deb2af

• wrap vhost in broker for mqtt::client

  kickster97

  committed
  da8ef99

• pass more specs and handle session_present

  kickster97

  committed
  ef792f5

• all connect specs working except replace client with new connection

  kickster97

  committed
  f825a44

• fix logs in client and dot expect connack in specs

  kickster97

  committed
  9ca561b

• add sessions to broker, and handle incoming subscribe with session

  kickster97

  committed
  ce14c4e

• fixup! add sessions to broker, and handle incoming subscribe with session

  kickster97

  committed
  cdb2b81

• fixup! fixup! add sessions to broker, and handle incoming subscribe with session

  kickster97

  committed
  537fc53

• subscribe and bind a session

  kickster97

  committed
  7ef8ad8

• temp

  kickster97

  committed
  1ba53bd

• exchange.publish

  

  snichme

  authored and

  kickster97

  committed
  32a82b2

• mqtt exchange

  

  snichme

  authored and

  kickster97

  committed
  a2cf9f1

• refactor mqtt session

  kickster97

  committed
  d01cc16

• add get method to prepare for qos 1

  kickster97

  committed
  bd4179a

• feel like this got ugly, but subscribe specs are now passing

  kickster97

  committed
  4a1be89

• cleanup old code and use session instead of queue

  kickster97

  committed
  01c1fa1

• refctoring consumer handling, and handle msg acks better

  kickster97

  committed
  f698fc3

• move consumers deliver_loop to session, specs do not pass

  kickster97

  committed
  289b7ee

• cleanup

  kickster97

  committed
  4109e56

• SubscriptionTree

  

  snichme

  authored and

  kickster97

  committed
  1d92ceb

• rebase main

  kickster97

  committed
  be2bf24

• format

  kickster97

  committed
  4405f8d

• override method

  

  snichme

  authored and

  kickster97

  committed
  052469b

• string token specs

  

  snichme

  authored and

  kickster97

  committed
  e3f7d6c

• crystal 1.14 fixes

  

  snichme

  authored and

  kickster97

  committed
  de0c11a

• subscription tree specs

  

  snichme

  authored and

  kickster97

  committed
  1ea3759

• beginning of puback

  kickster97

  committed
  1d196cb

• subscription tree specs

  

  snichme

  authored and

  kickster97

  committed
  77f79e8

• remove puts

  

  snichme

  authored and

  kickster97

  committed
  2db7b92

• Use monkey patched IO::Buffered#peek(n) to ensure data exists

  

  spuun

  authored and

  kickster97

  committed
  bd03f2b

• It should return slice

  

  spuun

  authored and

  kickster97

  committed
  9fedf5e

• Add spec to verify positive size check

  

  spuun

  authored and

  kickster97

  committed
  12d2c99

• Rename stuff in an attempt to make things more readable

  

  spuun

  authored and

  kickster97

  committed
  831265b

• Refactor specs

  

  spuun

  authored and

  kickster97

  committed
  8685963

• Cleaner specs

  

  spuun

  authored and

  kickster97

  committed
  8959b1f

• Update spec description

  

  spuun

  authored and

  kickster97

  committed
  ab38399

• Fix condition when checking if enough data exists

  

  spuun

  authored and

  kickster97

  committed
  750c325

• Move data in existing buffer if needed

  

  spuun

  authored and

  kickster97

  committed
  922a32c

• Return what we got instead of rasing in read fails

  

  spuun

  authored and

  kickster97

  committed
  8cd324f

• Remove unused code

  

  spuun

  authored and

  kickster97

  committed
  598aa73

• merge jons pr and fix sleep problem

  kickster97

  committed
  ac04cbb

• spec fix for subtree

  

  snichme

  authored and

  kickster97

  committed
  14f9771

• set dup value when delivering a msg

  kickster97

  committed
  7ac2018

• cleanup

  kickster97

  committed
  e077058

• retain_store and topic_tree

  kickster97

  committed
  d876c00

• Restructure retain-store and topic_tree for better fit in LavinMQ

  Show description for d2e0987
  

  kickster97

  and

  spuun

  committed
  d2e0987

• publish retained message

  kickster97

  committed
  6c1113d

• Update src/lavinmq/mqtt/retain_store.cr

  Show description for b0c93d5
  

  kickster97

  and

  spuun

  committed
  b0c93d5

• retained messages spec pass

  kickster97

  committed
  652a4de

• Improve handling of non-clean sesssions

  Show description for 2c0c7c6
  

  spuun

  authored and

  kickster97

  committed
  2c0c7c6

• pass will specs

  kickster97

  committed
  792fcb5

• cleanup

  kickster97

  committed
  e4b1f6b

• pass duplicate message specs

  kickster97

  committed
  c91dadc

• format

  kickster97

  committed
  af6a2b3

• fix publish

  kickster97

  committed
  84d9294

• remove obsolete header

  kickster97

  committed
  5738995

• cleanup

  kickster97

  committed
  5ec6576

• raise io error for invalid package_id

  kickster97

  committed
  94bee1b

• handle raise for double connect

  kickster97

  committed
  449ac25

• revert double connect rescue

  kickster97

  committed
  0ba7db3

• Remove unused variable

  

  spuun

  authored and

  kickster97

  committed
  65b8d25

• Use method overloading instead of type check

  

  spuun

  authored and

  kickster97

  committed
  e8b8e90

• Use lowercase log source

  

  spuun

  authored and

  kickster97

  committed
  c0df423

• Use mqtt.session as log source for Session

  

  spuun

  authored and

  kickster97

  committed
  12297ce

• Add spec to test publisher->subscriber flow

  

  spuun

  authored and

  kickster97

  committed
  c35077e

• Add spec to verify that session is restored properly

  

  spuun

  authored and

  kickster97

  committed
  51cb7c6

• Use getter instead of instance variable

  

  spuun

  authored and

  kickster97

  committed
  71425ee

• beginning of max_inflight

  kickster97

  committed
  61ab96b

• send in topic to subscription tree, pass specs

  kickster97

  committed
  8aaa93a

• clean up

  kickster97

  committed
  0eb4429

• create publish packet in client instead of accepting will in #broker:publish

  kickster97

  committed
  609e997

• Be consistent with typing

  

  spuun

  authored and

  kickster97

  committed
  0cfc711

• Add routing specs

  

  spuun

  authored and

  kickster97

  committed
  d9654f8

• Lint

  

  spuun

  authored and

  kickster97

  committed
  ec5147f

• Return nil and let spec assert

  

  spuun

  authored and

  kickster97

  committed
  ef14b1c

• Add a will spec (and some clean up)

  

  spuun

  authored and

  kickster97

  committed
  e59131e

• publish will if PacketDecode exception

  kickster97

  committed
  e4f3f3c

• move vhost logic from client into broker

  kickster97

  committed
  5742831

• Improve logging

  

  spuun

  authored and

  kickster97

  committed
  2e8b8a4

• add retain_store specs for .retain and .each

  kickster97

  committed
  d697e5e

• No need to prefix class, use namespace

  

  spuun

  authored and

  kickster97

  committed
  40db582

• Update src/lavinmq/mqtt/client.cr

  Show description for abedfb5
  

  kickster97

  and

  spuun

  committed
  abedfb5

• remove unnessecary socket.close

  kickster97

  committed
  56de5f2

• log warning instead of raise

  kickster97

  committed
  956a40f

• fetch max_inflight_messages form config

  kickster97

  committed
  0ab3b28

• Convert Publish to Message in exchange

  

  spuun

  authored and

  kickster97

  committed
  32f04ae

• Less aggressive logging

  

  spuun

  authored and

  kickster97

  committed
  52ff9de

• Suspend fiber while waiting for msg or consumer

  Show description for 2d34399
  

  spuun

  authored and

  kickster97

  committed
  2d34399

• add specs for handling connect packets with empty client_ids

  kickster97

  committed
  5c4bd06

• handle connect packets with empty client_id strings

  kickster97

  committed
  8a77827

• fixup! Suspend fiber while waiting for msg or consumer

  

  spuun

  authored and

  kickster97

  committed
  e824c61

• Move Sessions to separate file

  

  spuun

  authored and

  kickster97

  committed
  c834d87

• Dont convert topic to routing key, and use topic all the way through

  kickster97

  committed
  3f8f8b5

• prefix sessions with mqtt. and do not let amqp queues create queues that start with .mqtt

  kickster97

  committed
  7d7c245

• move validation to queue_factory and return preconditioned fail for amqp queues with .mqtt prefix

  kickster97

  committed
  5930625

• prefix_validation wip

  kickster97

  committed
  de3a6c9

• delete old cherry-picked code, replaced with #818

  kickster97

  committed
  3a9f774

• mqtt exchange receives MQTT::Publish but publish AMQP::Message to queue(session)

  

  snichme

  authored and

  kickster97

  committed
  64e9c35

• remove obsolete spec

  kickster97

  committed
  44dd545

• format

  kickster97

  committed
  7e0392f

• rebase in abstrace queue

  kickster97

  committed
  25d8e9a

• adapt for queue abstraction

  kickster97

  committed
  36f78d1

• repain broken amqp specs

  kickster97

  committed
  580b510

• rename prefix validator to namevalidator and move valid_entity_name into it

  kickster97

  committed
  d9b49d9

• remove unnessecary allocation in #NameValidator.valid_prefix

  kickster97

  committed
  4768353

• cleanup ordering in connections js

  kickster97

  committed
  da37d78

• remove sessions from vhost, redundant

  kickster97

  committed
  ef48059

• use default random instead of secure for client_id

  kickster97

  committed
  9a8a154

• delete unreferences messages in retain store when building index

  kickster97

  committed
  9f42ddc

• move exchange into the mqtt namespace

  kickster97

  committed
  45cd64f

• use mqtt namespace MqttBindingKey->BindingKey

  kickster97

  committed
  969a408

• remove redundant return value

  kickster97

  committed
  b5c8804

• update name validator

  kickster97

  committed
  121d50c

• move unacked_messagesapi logic to queue and overload method in session

  kickster97

  committed
  b73353c

• format

  kickster97

  committed
  735512d

• update logs for name validation failures

  kickster97

  committed
  df8291d

• don't have risk of overwriting retain store msg files

  kickster97

  committed
  805f417

• ensure to remove file

  kickster97

  committed
  4914577

• format

  kickster97

  committed
  0cee5e3

• replicate retain store, wip

  kickster97

  committed
  7ce9348

• add mqtts config + listener

  kickster97

  committed
  346950a

• format

  kickster97

  committed
  c97832d

• replication spec works correctly

  kickster97

  committed
  c710523

• add mqtt_proxy for clustering client

  kickster97

  committed
  715d8f5

• r+ needs file to exist before open

  kickster97

  committed
  391cdb3

• format

  kickster97

  committed
  b266f13

• fix ameba failures

  kickster97

  committed
  b141229

• satisfy ameba for spec files

  kickster97

  committed
  7c66aed

• rename specfile with suffix

  kickster97

  committed
  5348b77

• fix flaky wait_for

  kickster97

  committed
  aefc93b

• Use enum for protocol to get compile time validation

  

  spuun

  authored and

  kickster97

  committed
  4341545

• Fix specs to use protocol enum

  

  spuun

  authored and

  kickster97

  committed
  d1cb30e

• use short block notation

  kickster97

  committed
  13ae093

Commits on Nov 14, 2024

• fix mqtt exchange routing spec

  snichme

  committed
  f01d21e

• remove comment

  snichme

  committed
  a21fa05

• scope fix

  snichme

  committed
  6ef6527

• Multi-vhost support

  spuun

  committed
  571b7fb

• expand details tuple for consumer UI

  kickster97

  committed
  7f06362

• no need to convert routing key

  kickster97

  committed
  0a73f6e

• format

  kickster97

  committed
  e32c154

• handle unexpected close from client

  kickster97

  committed
  37d13a8

• connection_at for mqtt connections

  snichme

  committed
  bdc39bc

• deliver packet not msg from session (#843)

  Show description for d256cd1
  snichme

  authored
  d256cd1

Commits on Nov 15, 2024

• truncate the previous content before you retain a message

  kickster97

  committed
  d82a762

• safely overwrite retained messages

  kickster97

  committed
  8d47862

• Cant use constant as key in NamedTuple

  spuun

  committed
  4189531

• merge solutions for retain store

  kickster97

  committed
  5e7cc4b

Commits on Nov 18, 2024

• general fixup after comments

  kickster97

  committed
  a3202d1

• format

  kickster97

  committed
  43ec295

• set flaky spec to pending

  kickster97

  committed
  d358a60

• just a test

  kickster97

  committed
  303b81e

• just a test

  kickster97

  committed
  393a53b

Commits on Nov 19, 2024

• tmp: debug clustering_spec

  baelter

  committed
  3f0c1ab

• message dedup for exchange

  snichme

  committed
  72e00f7