[Bro] Bro Logs Sending to Kafka

Lewis, David [USA] Lewis_David at bah.com
Fri Nov 24 13:39:40 PST 2017


Hi all,

Apologies if this is not the place to post this question. If it is not, please re-direct me to the correct location.
Question:
I’ve added the following code (below – host:port says an actual host and port number) to local.bro in order to send 2 different logs (DNS and CONN) to 3 different Kafka topics.
However, the conn logs being sent to one of the kafka topics is sending messages very slowly (~single digit messages per batch) while the other conn logs are sending to the other kafka topic very quickly (~2 thousand message per batch). Why is this happening?

To follow up, is there an optimal batch size to max.ms in order to send thousands of batched message to different Kafka topics. I’ve been able to max out at around 2-3K per batch even if I increase the max ms to 30000.

To run: sudo bro –i lo /path/to/local.bro

Code:

@load Bro/Kafka/logs-to-kafka.bro
#redef Kafka::logs_to_send = set(Conn::LOG);
#redef LogAscii::include_meta T;
#redef LogAscii::separator ",";
redef Kafka::topic_name = "";
#redef Kafka::tag_json = T;
#redef Kafka::max_wait_on_shutdown = 3000;
redef Kafka::kafka_conf = table(
    ["metadata.broker.list"] = "host:port", #host:port
    ["debug"] = "generic, broker, topic, msg", #debugging lines
    ["queue.buffering.max.messages"] = "5000", #send in batches of this number
    ["queue.buffering.max.ms"] = "3500" #wait this many ms until sending to topic
);


#init function to name kafka topic (path), set KAFKAWRITER, and receive only LOG files
event bro_init()
{
        #Conn logs sent straight to
        Log::add_filter(Conn::LOG, [
                $name = "kafka-conn",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "conn-logs-batches"
        ]);
        #Conn logs sent to
        Log::add_filter(Conn::LOG, [
                $name = "kafka-conn-kate",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "conn-logs-kate"
        ]);
        #DNS logs to DGA
        Log::add_filter(DNS::LOG, [
                $name = "kafka-dns",
                $writer = Log::WRITER_KAFKAWRITER,
                $path = "dns-logs-dga"
        ]);

}


Thank you,
David


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20171124/bb89b456/attachment.html 


More information about the Bro mailing list