[Bro] is there a bro script to ignore duplicated logs?

[Azoff, Justin S]

Can you post your node.cfg?  It looks like you may have told bro to load balance, but you are not actually using the af_packet plugin.

as in, you have

interface=em1
lb_method=custom
lb_procs=6

instead of

interface=af_packet::em1
lb_method=custom
lb_procs=6

— 
Justin Azoff

> On Oct 18, 2018, at 3:07 AM, MAÁN ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> Hi All,
> 
> @fatema bannatwala im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3  and kept monitoring one interface only. on conn.log im getting below:
> 
> 1536746523.249570
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52851
> 10.190.129.250 53
> udp dns
> 11.342418 264
> 88 SF
> T T
> 0 Dd
> 6 432
> 2 144
> (empty) worker-em1-1
> on dns.log
> 
> 1536746526.252543
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746526.252634
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746530.283534
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52851 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52852 10.190.129.250
> 53 udp
> 59985 -
> 100.2.116.192.in-addr.arpa 1
> C_INTERNET 12
> PTR -
> - F
> 1536746526.252543
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746526.252634
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283534
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178 52852
> 10.190.129.250 53
> udp 59985
> - 100.2.116.192.in-addr.arpa
> 1 C_INTERNET
> 12 PTR
> - -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 1536746530.283625
> CbQGeN3yuYTKQd6xE
> 10.1.196.178
> 52853
> 10.190.129.250
> 53
> udp
> 59985
> -
> 100.2.116.192.in-addr.arpa
> 1
> C_INTERNET
> 12
> PTR
> -
> -
> F
> 
> ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.  
> Try Bro
> try.bro.org
> Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.
> the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.
> 
>  Log::add_filter(DNS::LOG, [
>         $name = "kafka-dns",
>         $writer = Log::WRITER_KAFKAWRITER,
>         $pred(rec: DNS::Info) = { return ! (( |rec$uid| == 
> |rec$uid| )); },
>         $config = table(
>             ["metadata.broker.list"] = "localhost:9092"
>         )
>     ]);
> 
>  but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.
> 
> thanks
> 
> 
> 
> From: bro-bounces at bro.org <bro-bounces at bro.org> on behalf of Jan Grashöfer <jan.grashoefer at gmail.com>
> Sent: Thursday, October 4, 2018 4:33 PM
> To: bro at bro.org
> Subject: Re: [Bro] is there a bro script to ignore duplicated logs?
>  
> On 04/10/2018 14:57, fatema bannatwala wrote:
> > You can run this script and see if the duplicate connections are happening
> > on which workers and go from there:
> 
> This can be further automated by using Justins bro-doctor script 
> available as a package: https://github.com/ncsa/bro-doctor
> 
> GitHub - ncsa/bro-doctor
> github.com
> Bro Doctor. This plugin provides a "doctor.bro" command for broctl that will help to troubleshoot various common cluster problems. This plugin runs the following checks:
> 
> 
> 
> Jan
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro