Skip to content

Commit

Permalink
filter_lua: Add chunk mode for processing multiple records
Browse files Browse the repository at this point in the history
This commit will introduce a chunk_mode for lua filter.
It can be needed for use cases like parallelization (see lua lanes).

Please note that the lua functions will take only two arguments:

function process_records(tag, records)
  if records and type(records) == "table" then
    for i, record_row in ipairs(records) do
        local timestamp = record_row.timestamp
        local record = record_row.record

        print("Timestamp entry:", timestamp.sec, timestamp.nsec)
        print("Record entry:", record.message)
    end
  else
    print("Error: Invalid 'records' table or nil")
  end
  return records
end

The returned table must be in the same format (table of timestamp and
record pairs).

This mode currently only supports time_as_table by default and does
always emit the returned records. There is no return code to be set.

Signed-off-by: Richard Treu <richard.treu@sap.com>
  • Loading branch information
drbugfinder-work committed Feb 20, 2024
1 parent 9c3978c commit a956e55
Show file tree
Hide file tree
Showing 4 changed files with 287 additions and 102 deletions.
2 changes: 1 addition & 1 deletion include/fluent-bit/flb_lua.h
Original file line number Diff line number Diff line change
Expand Up @@ -100,5 +100,5 @@ void flb_lua_tompack(lua_State *l,
struct flb_lua_l2c_config *l2cc);
void flb_lua_dump_stack(FILE *out, lua_State *l);
int flb_lua_enable_flb_null(lua_State *l);

void flb_lua_bulk_process(lua_State *l, struct flb_time *t, msgpack_object *o, int table_index);
#endif
Loading

0 comments on commit a956e55

Please sign in to comment.