[Bro] [Non-DoD Source] Re: File Extraction Gaps

[Weasel, Gary W Jr CIV DISA RE (US)]

So I just tested running bro in command line mode (i.e. not using broctl), fed it my usual policy files and dumped to pcap.

The extracted_files folder showed the same story, lots of file gaps, all different hashes for the same file.

When I loaded up the pcap in wireshark and extracted all the files, all files hash correctly.

So this tells me, at some point Bro is getting all the data.  Something is just messing up for some reason when it comes to the file analysis and/or file extraction modules.

-----Original Message-----
From: Jon Siwek <jsiwek at corelight.com>
Sent: Tuesday, May 29, 2018 12:52 PM
To: Weasel, Gary W Jr CIV DISA RE (US) <gary.w.weasel2.civ at mail.mil>
Cc: bro at bro.org
Subject: [Non-DoD Source] Re: [Bro] File Extraction Gaps

On Tue, May 29, 2018 at 9:55 AM, Weasel, Gary W Jr CIV DISA RE (US)
<gary.w.weasel2.civ at mail.mil> wrote:

> In my example, I am curling an exe to a server, where that traffic is spanned to my Bro sensor (the exe in question is 1 MB in size).  If I curl repeatedly, Bro sees all the files, but the number of file gap events varies wildly (anywhere from 2 or 3 to over 100).  The part that gets me, if I tcpdump alongside Bro, and pull the files out of pcap, they're all intact and hash correctly, so I know I'm getting all the packets on wire.  Bro and PF_RING report 0 packet loss.

Seems either Bro behaves differently in offline vs. live usage (not
what I'd expect to see in this case, though I can't rule it out for
sure) or it's not actually seeing the same packet input in the live
deployment as it was in offline usage.  Maybe to pursue whether the
later is true you could have Bro write out the packets that it
actually saw with the `-w <pcap file>` command line flag and examine
the resulting pcap to see if it looks like you expect.

- Jon