V1 Roadmap

Fluentd's Roadmap

v0.12 (early August)

Filter & Label

• Goal: Don't rewrite tags!
  • Encourage the use of a tag as an identifier of a data source, NOT as a mechanism to filter/modify data. Use filter_label for the latter.
• Design:
  • See filter_label
• Implementation:
  • See filter_label

v0.13

Odd number is development version

v0.14 (mid August)

Actor

• Goal: Provide a API set to start/stop threads and receive/send data through sockets to plugin developers
• Design:
  • Create new class
  • Initialize/start/stop the class automatically at a base class or mix-in module.
• Necessary refactoring:
  • Add #close method to all plugins to not cause "already closed" exception
    • #shutdown stops emitting new records
    • #close stops receiving new records
    • Actor stops threads at shutdown and closes sockets at #close
• Implementation:
  • Actor should use an IO library which supports JRuby and Windows.
  • Otherwise, Actor can use threads.

v0.15 (TBD)

v1.0

TODO items

Error stream

• Goal: Provide standardized concept and config syntax to handle broken events to users
• Design:
  • Route broken events (which caused a exception in emit of output plugins) to a special built-in label
  • Provide a base class or mix-in for plugin developers
  • Don't use chain any more
• Implementation:
  • EventRouter catches an exception and calls @error_handle_collector.emit
  • Optionally, output plugins can call directly @error_handle_collector.emit.
    • In this case, output plugins should not throw an exception.
    • This is necessary if only part of multiple events in a EventStream are broken.
    • In other words, output plugins need to call handle exceptions appropriately if they have optimization to handle multiple events at once.

ServerEngine

• Purpose: Support Windows and simplify multi-process code
• Design:
  • Replace supervisor and server code with ServerEngine
  • Including daemonize, logging, shutdown/restart, and signal handling.
• Implementation:
  • Implement worker_type=spawn to ServerEngine

SocketManager

• Purpose: Zero-downtime restart
• Design:
  • Add a new class named SocketManager
  • SocketManager listens TCP/UDP sockets in the parent process of ServerEngine
    • Child processes requests the parent process to listen a TCP/UDP socket
    • The parent process listens a new socket and passes its file descriptor to the child process
• Implementation:
  • On Linux: v11/SocketManager
  • On Windows: TODO. Keywords: WSADuplicateSocket, ffi

New Buffer API

• Purpose: Buffer plugins should be easier to create
• Design:
  • Not designed yet
• Implementation:
  • Create a new Buffer class
  • Create a new buffered output class (in Fluentd::plugin namespace?)
  • Create an adaptor code into old buffered output class to use the new Buffer class