what is the behaviour when no server is available

[camillebourget]

Hello,

I would like to know how the framework behaves when it can't communicate with the grafana loki server?

I haven't found anything about it

Thank you in advance for your help
Camille

[mishamyte]

Hi @camillebourget,

This sink doesn't send events immediately. It batches it in a memory and send with batches.
You could control it by next settings:

• batchPostingLimit - the maximum number of events to post in a single batch. Default value is 1000
• queueLimit - the maximum number of events stored in the queue in memory, waiting to be posted over the network. Default value is infinitely
• period - the time to wait between checking for event batches. Default value is 2 seconds

So if server would be unavailable, sink will add events into a memory queue (main code could be found here) and tries to retry with a exponential backoff.

So when server become available, data would be delivered.

So there are two tricky things:

• QueueLimit - it should be defined according to your requirements and Loki instance availability. Otherwise you could face with OutOfMemoryException of your app instance
• App instance restart - if in time of Loki unavailability your instance would be restarted (for example by K8s), the data would be lost.

In general my previous company and some of guys, who use this sink tested it with a HA Loki instance (like 99.99% uptime) and there were no problems with data loss.

It you expect that your Loki instance or network, or smthng that could affect, would be unavailable for a huge amount of time (hours etc), you should look to a DevOps-first approaches of getting a logs. For example using writing to stdout with Serilog's JSON formatter and Promtail for scrapping this logs from pods.

[camillebourget]

Thank you for your detailed answer and your reactivity I understand better now how it works

[mishamyte]

You are welcome)