[Bro] [Non-DoD Source] Re: File Extraction Gaps

[BortolameottiR]

Bro should write in weird.log something like : inflate_failed. This error
should represent (tell me if I am wrong) that bro failed to decompress
(inflate/deflate of http)
the file correctly. By cross-checking the connection ids you should be able
to verify if that was the problem.

It happened to me in different settings (with a manipulated pcap) that
file reconstruction did not work properly. I was not able to fix it though.

R.

On 05/29/2018 07:57 PM, Jon Siwek wrote:
> Would be most helpful to get a pcap, scripts, and command that can
> reproduce what you see.  Or else steps one could follow to try to
> reproduce a pcap that may exhibit the same problem.  Otherwise, only
> guess I have is to check for anything unusual at the TCP-layer -- the
> file analysis/extraction is dependent on the TCP reassembly process,
> so if the sequence of events at the TCP level lead Bro to believe it
> missed part of the TCP stream, that also manifests as a gap event in
> any associated file analysis.  You could go as far as looking at the
> contents/ordering of things in a tcp_packet event handler as a sanity
> check, though there may also be residuals in weird.log that you could
> simply check (I don't recall particular names to look for off the top
> of my head).
>
> - Jon
>
> On Tue, May 29, 2018 at 12:22 PM, Weasel, Gary W Jr CIV DISA RE (US)
> <gary.w.weasel2.civ at mail.mil> wrote:
>> So I just tested running bro in command line mode (i.e. not using broctl), fed it my usual policy files and dumped to pcap.
>>
>> The extracted_files folder showed the same story, lots of file gaps, all different hashes for the same file.
>>
>> When I loaded up the pcap in wireshark and extracted all the files, all files hash correctly.
>>
>> So this tells me, at some point Bro is getting all the data.  Something is just messing up for some reason when it comes to the file analysis and/or file extraction modules.
>>
>> -----Original Message-----
>> From: Jon Siwek <jsiwek at corelight.com>
>> Sent: Tuesday, May 29, 2018 12:52 PM
>> To: Weasel, Gary W Jr CIV DISA RE (US) <gary.w.weasel2.civ at mail.mil>
>> Cc: bro at bro.org
>> Subject: [Non-DoD Source] Re: [Bro] File Extraction Gaps
>>
>> On Tue, May 29, 2018 at 9:55 AM, Weasel, Gary W Jr CIV DISA RE (US)
>> <gary.w.weasel2.civ at mail.mil> wrote:
>>
>>> In my example, I am curling an exe to a server, where that traffic is spanned to my Bro sensor (the exe in question is 1 MB in size).  If I curl repeatedly, Bro sees all the files, but the number of file gap events varies wildly (anywhere from 2 or 3 to over 100).  The part that gets me, if I tcpdump alongside Bro, and pull the files out of pcap, they're all intact and hash correctly, so I know I'm getting all the packets on wire.  Bro and PF_RING report 0 packet loss.
>> Seems either Bro behaves differently in offline vs. live usage (not
>> what I'd expect to see in this case, though I can't rule it out for
>> sure) or it's not actually seeing the same packet input in the live
>> deployment as it was in offline usage.  Maybe to pursue whether the
>> later is true you could have Bro write out the packets that it
>> actually saw with the `-w <pcap file>` command line flag and examine
>> the resulting pcap to see if it looks like you expect.
>>
>> - Jon
>>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro