Logging Zeek Output to Kafka

A Zeek log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Zeek.

This software is a part of the Apache Metron project which integrates a variety of open source, big data technologies to offer a platform to detect and respond to cyber threats at-scale.

Installation
Activation
Settings
Kerberos
Contributing

Installation

`zkg` Installation

zkg is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to activate and configure the plugin after your installation.

Install librdkafka, a native client library for Kafka. This plugin has been tested against librdkafka v1.4.2.

In order to use this plugin within a kerberized Kafka environment, you will also need libsasl2 installed and will need to pass --enable-sasl to the configure script.
```
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install
```
Configure zkg by following the quickstart guide here.

Install the plugin using zkg install.

$ zkg install apache/metron-bro-plugin-kafka --version master
The following packages will be INSTALLED:
  zeek/apache/metron-bro-plugin-kafka (master)

Verify the following REQUIRED external dependencies:
(Ensure their installation on all relevant systems before proceeding):
  from zeek/apache/metron-bro-plugin-kafka (master):
    librdkafka ~1.4.2

Proceed? [Y/n]
zeek/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib]
Saved answers to config file: /home/jonzeolla/.zkg/config
Running unit tests for "zeek/apache/metron-bro-plugin-kafka"
all 10 tests successful


Installing "zeek/apache/metron-bro-plugin-kafka"........
Installed "zeek/apache/metron-bro-plugin-kafka" (master)
Loaded "zeek/apache/metron-bro-plugin-kafka"

Run the following command to ensure that the plugin was installed successfully.

$ zeek -N Apache::Kafka
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)

Manual Installation

Manually installing the plugin should only occur in situations where installing and configuring zkg is not reasonable. If you are running zeek in an environment where you do not have Internet connectivity, investigate bundles or creating an internal package source.

These instructions could also be helpful if you were interested in distributing this as a package (such as a deb or rpm).

Install librdkafka, a native client library for Kafka. This plugin has been tested against librdkafka v1.4.2.

In order to use this plugin within a kerberized Kafka environment, you will also need libsasl2 installed and will need to pass --enable-sasl to the configure script.
```
$ curl -L https://github.com/edenhill/librdkafka/archive/v1.4.2.tar.gz | tar xvz
$ cd librdkafka-1.4.2/
$ ./configure --enable-sasl
$ make
$ sudo make install
```

Build the plugin using the following commands.

$ ./configure --with-librdkafka=$librdkafka_root
$ make
$ sudo make install

Run the following command to ensure that the plugin was installed successfully.

$ zeek -N Apache::Kafka
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)

Activation

The following examples highlight different ways that the plugin can be used. Simply add the Zeek script language to your local.zeek file (for example, /usr/share/zeek/site/local.zeek) as shown to demonstrate the example.

In addition to activating the plugin, when running Zeek in a cluster it is highly recommended to leverage one or more Zeek loggers as shown here to separate logging activities from the manager thread.

Example 1 - Send a list of logs to kafka

The goal in this example is to send all HTTP and DNS records to a Kafka topic named zeek.

Any configuration value accepted by librdkafka can be added to the kafka_conf configuration table.
The topic_name will default to send all records to a single Kafka topic called 'zeek'.
Defining logs_to_send will send the HTTP and DNS records to the brokers specified in your Kafka::kafka_conf.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "server1.example.com:9092,server2.example.com:9092"
);

Example 2 - Send all active logs

This plugin has the ability send all active logs to the "zeek" kafka topic with the following configuration.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::send_all_active_logs = T;
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:9092"
);

Example 3 - Send all active logs with exclusions

You can also specify a blacklist of zeek logs to ensure they aren't being sent to kafka regardless of the Kafka::send_all_active_logs and Kafka::logs_to_send configurations. In this example, we will send all of the enabled logs except for the Conn log.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::send_all_active_logs = T;
redef Kafka::logs_to_exclude = set(Conn::LOG);
redef Kafka::topic_name = "zeek";
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:9092"
);

Example 4 - Send each zeek log to a unique topic

It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named http and all DNS records to a separate Kafka topic named dns.

The topic_name value must be set to an empty string.
The $path value of Zeek's Log Writer mechanism is used to define the topic name.
Any configuration value accepted by librdkafka can be added to the $config configuration table.
Each log writer accepts a separate configuration table.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::topic_name = "";
redef Kafka::tag_json = T;

event zeek_init() &priority=-10
{
    # handles HTTP
    local http_filter: Log::Filter = [
        $name = "kafka-http",
        $writer = Log::WRITER_KAFKAWRITER,
        $config = table(
                ["metadata.broker.list"] = "localhost:9092"
        ),
        $path = "http"
    ];
    Log::add_filter(HTTP::LOG, http_filter);

    # handles DNS
    local dns_filter: Log::Filter = [
        $name = "kafka-dns",
        $writer = Log::WRITER_KAFKAWRITER,
        $config = table(
                ["metadata.broker.list"] = "localhost:9092"
        ),
        $path = "dns"
    ];
    Log::add_filter(DNS::LOG, dns_filter);
}

Example 5 - Zeek log filtering

You may want to configure zeek to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Apache Metron currently doesn't support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are multiple ways to approach this). In this example we will do that that, and are assuming a somewhat standard zeek kafka plugin configuration, such that:

All zeek logs are sent to the default zeek topic.
Each JSON message is tagged with the appropriate log type (such as http, dns, or conn), by setting Kafka::tag_json to true.
If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::tag_json = T;

event zeek_init() &priority=-10
{
    # handles HTTP
    Log::add_filter(HTTP::LOG, [
        $name = "kafka-http",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);

    # handles DNS
    Log::add_filter(DNS::LOG, [
        $name = "kafka-dns",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);

    # handles Conn
    Log::add_filter(Conn::LOG, [
        $name = "kafka-conn",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);
}

Notes

logs_to_send is mutually exclusive with $pred, thus for each log you want to set $pred on, you must individually setup a Log::add_filter and refrain from including that log in logs_to_send.
The is_v6_addr() function can also be used in your $pred to identify if an IP address is IPv6.
Alternatively, if you are using Apache Metron to pull from the specified kafka topic, you could filter the IPv6 logs using Stellar. In that case Stellar would filter the logs out and a $pred would not be necessary. The benefit to this approach is that kafka would receive an unfiltered set of logs.

Example 6 - Sending a log to multiple topics

You are able to send a single zeek log to multiple different kafka topics in the same kafka cluster by overriding the default topic (configured with Kafka::topic_name) by creating a custom zeek Log::Filter. In this example, the DHCP, RADIUS, and DNS logs are sent to the "zeek" topic; the RADIUS log is duplicated to the "shew_zeek_radius" topic; and the DHCP log is duplicated to the "shew_zeek_dhcp" topic.

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
redef Kafka::topic_name = "zeek";
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "server1.example.com:9092,server2.example.com:9092"
);
redef Kafka::tag_json = T;

event zeek_init() &priority=-10
{
    # Send RADIUS to the shew_zeek_radius topic
    local shew_radius_filter: Log::Filter = [
        $name = "kafka-radius-shew",
        $writer = Log::WRITER_KAFKAWRITER,
        $path = "shew_zeek_radius"
        $config = table(["topic_name"] = "shew_zeek_radius")
    ];
    Log::add_filter(RADIUS::LOG, shew_radius_filter);

    # Send DHCP to the shew_zeek_dhcp topic
    local shew_dhcp_filter: Log::Filter = [
        $name = "kafka-dhcp-shew",
        $writer = Log::WRITER_KAFKAWRITER,
        $path = "shew_zeek_dhcp"
        $config = table(["topic_name"] = "shew_zeek_dhcp")
    ];
    Log::add_filter(DHCP::LOG, shew_dhcp_filter);
}

Note: Because Kafka::tag_json is set to True in this example, the value of $path is used as the tag for each Log::Filter. If you were to add a log filter with the same $path as an existing filter, Zeek will append "-N", where N is an integer starting at 2, to the end of the log path so that each filter has its own unique log path. For instance, the second instance of conn would become conn-2.

Example 7 - Add static values to each outgoing Kafka message

It is possible to define name value pairs and have them added to each outgoing Kafka json message when tagged_json is set to true. Each will be added to the root json object. * the Kafka::additional_message_values table can be configured with each name and value * based on the following configuration, each outgoing message will have "FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": "SECOND_STATIC_VALUE" added.

@load packages
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
redef Kafka::topic_name = "zeek";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table(["metadata.broker.list"] = "kafka-1:9092,kafka-2:9092");
redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = "FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");
redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);
redef Known::cert_tracking = ALL_HOSTS;
redef Software::asset_tracking = ALL_HOSTS;

Settings

`logs_to_send`

A set of logs to send to kafka.

redef Kafka::logs_to_send = set(Conn::LOG, DHCP::LOG);

`send_all_active_logs`

If true, all active logs will be sent to kafka other than those specified in logs_to_exclude.

redef Kafka::send_all_active_logs = T;

`logs_to_exclude`

A set of logs to exclude from being sent to kafka.

redef Kafka::logs_to_exclude = set(Conn::LOG, DNS::LOG);

`topic_name`

The name of the topic in Kafka where all Zeek logs will be sent to.

redef Kafka::topic_name = "zeek";

`kafka_conf`

The global configuration settings for Kafka. These values are passed through directly to librdkafka. Any valid librdkafka settings can be defined in this table. The full set of valid librdkafka settings are available here.

redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "localhost:9092",
    ["client.id"] = "zeek"
);

`additonal_message_values`

A table of of name value pairs. Each item in this table will be added to each outgoing message at the root level if tag_json is set to T.

redef Kafka::additional_message_values = table(
    ["FIRST_STATIC_NAME"] = "FIRST_STATIC_VALUE",
    ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE"
);

`tag_json`

If true, a log stream identifier is appended to each JSON-formatted message. For example, a Conn::LOG message will look like { 'conn' : { ... }}.

redef Kafka::tag_json = T;

`json_timestamps`

Uses Ascii log writer for timestamp format. Default is JSON::TS_EPOCH. Other options are JSON::TS_MILLIS and JSON::TS_ISO8601.

redef Kafka::json_timestamps = JSON::TS_ISO8601;

`max_wait_on_shutdown`

The maximum number of milliseconds that the plugin will wait for any backlog of queued messages to be sent to Kafka before forced shutdown.

redef Kafka::max_wait_on_shutdown = 3000;

`debug`

A comma separated list of debug contexts in librdkafka which you want to enable. The available contexts are:

all
generic
broker
topic
metadata
feature
queue
msg
protocol
cgrp
security
fetch
feature
interceptor
plugin
consumer
admin

Kerberos

This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set.

SASL

If you are using SASL as a security protocol for kafka, then you must have libsasl or libsasl2 installed. You can tell if sasl is enabled by running the following from the directory in which you have build librdkafka:

examples/rdkafka_example -X builtin.features
builtin.features = gzip,snappy,ssl,sasl,regex

Producer Config

As stated above, you can configure the producer kafka configs in ${ZEEK_HOME}/share/zeek/site/local.zeek. There are a few configs necessary to set, which are described here. For an environment where the following is true:

The broker is node1:6667
This kafka is using SASL_PLAINTEXT as the security protocol
The keytab used is the metron keytab
The service principal for metron is metron@EXAMPLE.COM

The kafka topic zeek has been given permission for the metron user to write:

# login using the metron user
kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic zeek

The following is how the ${ZEEK_HOME}/share/zeek/site/local.zeek looks:

@load packages/metron-bro-plugin-kafka/Apache/Kafka
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "zeek";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667"
                               , ["security.protocol"] = "SASL_PLAINTEXT"
                               , ["sasl.kerberos.keytab"] = "/etc/security/keytabs/metron.headless.keytab"
                               , ["sasl.kerberos.principal"] = "metron@EXAMPLE.COM"
                               );

Contributing

If you are interested in contributing to this plugin, please see the Apache Metron CONTRIBUTING.md.

Name		Name	Last commit message	Last commit date
Latest commit History 48 Commits
.github		.github
cmake		cmake
dev_utilities/release-utils		dev_utilities/release-utils
docker		docker
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
CHANGES		CHANGES
CMakeLists.txt		CMakeLists.txt
COPYING		COPYING
MAINTAINER		MAINTAINER
Makefile		Makefile
NOTICE		NOTICE
README.md		README.md
VERSION		VERSION
configure		configure
configure.plugin		configure.plugin
zkg.meta		zkg.meta

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Logging Zeek Output to Kafka

Installation

`zkg` Installation

Manual Installation

Activation

Example 1 - Send a list of logs to kafka

Example 2 - Send all active logs

Example 3 - Send all active logs with exclusions

Example 4 - Send each zeek log to a unique topic

Example 5 - Zeek log filtering

Notes

Example 6 - Sending a log to multiple topics

Example 7 - Add static values to each outgoing Kafka message

Settings

`logs_to_send`

`send_all_active_logs`

`logs_to_exclude`

`topic_name`

`kafka_conf`

`additonal_message_values`

`tag_json`

`json_timestamps`

`max_wait_on_shutdown`

`debug`

Kerberos

SASL

Producer Config

Contributing

About

Releases

Packages

Contributors 7

Languages

License

apache/metron-bro-plugin-kafka

Folders and files

Latest commit

History

Repository files navigation

Logging Zeek Output to Kafka

Installation

zkg Installation

Manual Installation

Activation

Example 1 - Send a list of logs to kafka

Example 2 - Send all active logs

Example 3 - Send all active logs with exclusions

Example 4 - Send each zeek log to a unique topic

Example 5 - Zeek log filtering

Notes

Example 6 - Sending a log to multiple topics

Example 7 - Add static values to each outgoing Kafka message

Settings

logs_to_send

send_all_active_logs

logs_to_exclude

topic_name

kafka_conf

additonal_message_values

tag_json

json_timestamps

max_wait_on_shutdown

debug

Kerberos

SASL

Producer Config

Contributing

About

Topics

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

Packages 0

Contributors 7

Languages

`zkg` Installation

`logs_to_send`

`send_all_active_logs`

`logs_to_exclude`

`topic_name`

`kafka_conf`

`additonal_message_values`

`tag_json`

`json_timestamps`

`max_wait_on_shutdown`

`debug`

Packages