Manage sync threads failures

[trueleo]

Fixes #43

Description

This PR makes bunch of changes to ways how local sync and s3 sync threads are managed inside parseable. The goal is to make parseable main thread be aware of unwanted failure inside either of these dedicated sync thread. Additionally we want to have control over life cycle of these threads and have them stop on command, This can be later explored in future if we want to have graceful shutdown an also to guarantee more data consistency in case of failure.

This solution works something like this :-

Sync threads are spawned with pair of one shot channels (named with suffix inbox and outbox) for communication with main thread.

• inbox is sender type which can be used to stop the respective thread by breaking its scheduler loop.
• outbox is receiver type which is polled inside the main thread's looped select. So that whenever we receive a message from thread we stop.

scheduler is ran inside loop which runs pending tasks and polls the channel for any message from main thread ( if any then stop the thread )

All of this is wrapped by a catch unwind. So in case of an unwinding because of panic there is a chance to let main thread know and handle accordingly. This is added as a safeguard for now but later we need to verify that local_sync and s3_sync don't have anything that can panic.

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

[trueleo]

Is it okay to thread sleep here for small amount of time ?

[nitisht]

Why is sleep needed here?