[Bro] is there a bro script to ignore duplicated logs?

[Azoff, Justin S]

> On Oct 18, 2018, at 10:05 AM, MAÁN ABU SHAQRA <maanamen at hotmail.com> wrote:
> 
> No I didn’t upload a pcap, the provided pcaps on the website show duplicate dns UIDs. I suspect that it’s a duplicated packets issue as I’ve analyzed some traffic on wireshark and it had no duplicates. 
> 
> I’d appreciate it if anyone can assist with this,
> 
> Thanks
> 
> MA’AN ABUSHAQRA 
> Dubai, UAE
> 

Oh, yes, I see what you are saying now.

The repeated entries i see on the pcap on try.bro.org are from netbios queries that are all broadcast queries for WORKGROUP.  I believe those are actually just repeated broadcasts with the same 5 tuple which is what causes the duplicates.

At this point with your traffic I would stop bro and run a simple tcpdump to generate a pcap file while you generate some known traffic (like a small file download over HTTP) and then inspect the resulting pcap file.

You could also set lb_procs=1 to ensure that there is only one bro process running to rule out any issue with af_packet load balancing.

One weird thing I see is that for your conn.log entry:

1536746523.249570
CbQGeN3yuYTKQd6xE
10.1.196.178 52851
10.190.129.250 53
udp dns
11.342418 264
88 SF
T T
0 Dd
6 432
2 144
(empty) worker-em1-1

The 6 and 2 are orig_pkts and resp_pkts.  For a simple DNS lookup, you would expect orig_pkts and resp_pkts to both be one.. one packet for the query and then one packet for the response.  But your bro worker somehow saw 6 packets for the query and 2 packets for the response.

— 
Justin Azoff