Add internal metadata fields for tracking ACKs and NACKs

[lucasoares]

Description

During a conversation with a colleague we discussed the need for some internal metadata fields to track ACKs (acknowledgments) and NACKs (negative acknowledgments) in Deckard. This issue proposes adding these fields to the metadata map of our messages.

Background

We were exploring the best approach to limit message retries (using NACK + LockMs) to a maximum of N attempts. The two options we considered were by using metadata or breakpoint field, which we were uncertain about its usage.

After discussing the matter, it became clear that for now the breakpoint field would serve as a suitable workaround, because the metadata field shuold not be used since it is not updated in NACKs or ACKs.

The "breakpoint" field is commonly used when a message is used to process a large number of elements, enabling the worker applications to track the progress and resume from where it left off. For example, if we have a social media page with 10,000 posts and need to process them in batches of 10, the "breakpoint" field allows the application to store the last processed position, ensuring it can continue from there in subsequent iterations on different workers.

Proposition

It would be beneficial to include some standard metadata fields in Deckard's pull responses. These fields would provide information on the number of ACKs/NACKs a message has received and the consecutive number of ACKs/NACKs. This information could be useful in the case described in this issue. It also would be useful, for example, in dynamically adjusting the time lock for a message based on the number of consecutive NACKs. All this controlled by the client application, which would be good to increase Deckard's ability to help users in different use cases without relying on some additional database.

The proposed metadata fields to be added are as follows:

{
 "_deckard.total_nacks" : "0",
 "_deckard.total_acks" : "321",
 "_deckard.consecutive_acks" : "12",
 "_deckard.consecutive_nacks" : "0"
}

Where all our internal metadata would be prefixed by _deckard to allow users distinguish between their own metadata.

We could add helpers for each client to get these numbers, since the metadata field is a map[string]string and I don't like adding each field individually in our message's struct. Any suggestion?

[lucasoares]

Maybe will be better adding a new field on the message struct called diagnostic or something like that, where we can put diagnostic information using either a structured message.

[cezar-tech]

I think this is an awsome idea,

Recently @lucasoares has told me of an "eternal" message in our private enterprise, this is something that could also have helped me out. We can track down useless/broken elements that for some reason we can't work with and can just discard them with a removal_ack.

Can further lead down the road to even some sort of deadlettering architecture... who knows.

Since the best every event sourcing system of the galaxy (google pubsub) also delivers this so called diagnostic fields so I would say lets just mirror what works and this is pretty much it.

I back up this idea.

[lucasoares]

I will add a new message inside all messas named "Diagnostic" and concentrate this kind of field inside it.