[Bro] is there a bro script to ignore duplicated logs?

[MAÁN ABU SHAQRA]

Hi All,


@fatema bannatwala<mailto:fatema.bannatwala at gmail.com> im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3  and kept monitoring one interface only. on conn.log im getting below:


1536746523.249570 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp dns 11.342418 264 88 SF T T 0 Dd 6 432 2 144 (empty) worker-em1-1

on dns.log


1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F


ive also checked the http://try.bro.org/<http://try.bro.org/#/?example=basics-switches-solution-switch-exercise> and ran the exercies , and found the same issue.

Try Bro<http://try.bro.org/#/?example=basics-switches-solution-switch-exercise>
try.bro.org
Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.

the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.


 Log::add_filter(DNS::LOG, [
        $name = "kafka-dns",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: DNS::Info) = { return ! (( |rec$uid| == |rec$uid| )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);

 but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.


thanks



________________________________
From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Jan Grashöfer <jan.grashoefer at gmail.com>
Sent: Thursday, October 4, 2018 4:33 PM
To: bro at bro.org
Subject: Re: [Bro] is there a bro script to ignore duplicated logs?

On 04/10/2018 14:57, fatema bannatwala wrote:
> You can run this script and see if the duplicate connections are happening
> on which workers and go from there:

This can be further automated by using Justins bro-doctor script
available as a package: https://github.com/ncsa/bro-doctor
[https://avatars0.githubusercontent.com/u/7528333?s=400&v=4]<https://github.com/ncsa/bro-doctor>

GitHub - ncsa/bro-doctor<https://github.com/ncsa/bro-doctor>
github.com
Bro Doctor. This plugin provides a "doctor.bro" command for broctl that will help to troubleshoot various common cluster problems. This plugin runs the following checks:




Jan
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20181018/febc9c2e/attachment-0001.html