Serilog sink using ZoneTree

[pwelter34]

Hello,

I'm considering using ZoneTree for a Serilog Sink. Would ZoneTree be a good candidate for this?

Requirements

• Fast write performance, low overhead resource usage
• low cost data storage
• date based partitioning, if necessary. most of the queries will include a log date
• querable for log searching
• history management by date, expire old logs. Custom code to trigger acceptable.
• support Azure app services, Azure functions and docker container app
• multiple readers and writers. web app and background app write to same log for better query support.

Any tips to consider when creating a prototype?

thanks

[koculu]

Yes, ZoneTree is a great choice for your Serilog sink, and it covers all your key needs:

1. Fast Write Performance & Low Resource Usage: ZoneTree gives you plenty of options to tweak memory, CPU, and disk usage, making it super efficient for high-volume logging.

2. Flexible Storage: Thanks to the IFileStreamProvider and IFileStream interfaces, you can store the database pretty much anywhere, so it works in all kinds of environments.

3. Partitioning: If you’re doing date-based partitioning, just spin up multiple ZoneTree instances (one for each partition) and you’re good to go!

4. Log Searching: ZoneTree’s FullTextSearch engine handles searches smoothly. For range queries, keep your logs ordered by date in descending order. A simple key structure could look like this:

   public readonly struct LogKey
   {
       public long Ticks { get; }
       // Ensures uniqueness for log entries that have the same timestamp value.
       public short Id { get; }
   
       public LogKey(long ticks, short id)
       {
           Ticks = ticks;
           Id = id;
       }
   }
   // Use Interlocked.Increment(ref nextLogId) during inserts to keep the log keys unique

5. TTL Expiration: ZoneTree supports custom expiration using the IsDeleted delegate. TTL for keys support added with PR

6. Azure & Docker Support: Works perfectly with Azure App Services and Docker. For Azure Functions, you’ll just need a Web API to access ZoneTree.

7. Thread Safety: ZoneTree is already thread-safe, so no worries there unless your custom code requires something special. You can create multiple iterators, but individual ones aren’t thread-safe. For an example in action, check out the ZoneTree.FullTextSearch.Playground project here.

8. Shared log file: ZoneTree cannot share the same storage folder across multiple processes, so if you have a web app and background app both writing to the same log, you’ll need to create a server for ZoneTree. Expose an API on this server, and have other processes log to it via the API to ensure smooth logging and querying from multiple sources.

Let me know if you run into any issues! It looks like you’ve got everything you need to hit your goals, and I’m happy to help if needed.