[Bro] Inconsistent file size during extraction

[Josh Liburdi]

Yup, that clears up some things I forgot. And thanks, happy to be active
again!

On Thu, Feb 1, 2018 at 7:49 PM, Seth Hall <seth at corelight.com> wrote:

> Yep, I was going to comment that that's probably the issue, but I'll give
> some more details on why things may end up that way.
>
> "total_bytes" - is for when the size of the file is known by some
> secondary mechanism, like the file size being transmitted as part of a
> protocol or a file being read off disk.
> "seen_bytes" - represents the number of actual bytes of data that were
> passed into the file analysis framework.
>
> This is another case where small packet loss issues can have outsized
> effects because the following bytes can't be reassembled into the file
> correctly and you don't get anymore data.
>
> Also, nice to see on the mailing list again Josh!
>
> .Seth
>
> On 1 Feb 2018, at 22:07, Josh Liburdi wrote:
>
> Seems that this particular connection may be affected by tapping issues.
>
> On Thu, Feb 1, 2018 at 4:13 PM, Josh Liburdi <liburdi.joshua at gmail.com>
> wrote:
>
>> Hi all,
>>
>> I'm seeing instances where files are being extracted inconsistently with
>> what is reported in files.log. Here is a redacted example:
>>
>> files.log:
>> #fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers
>> mime_type filename duration local_orig is_orig *seen_bytes* *total_bytes*
>> missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256
>> extracted extracted_cutoff extracted_size
>> #types time string set[addr] set[addr] set[string] string count
>> set[string] string string interval bool bool count count count count bool
>> string string string string string bool count
>> 1517528771.042220 Fz2Z2m3zwQcc3gqDS3 x.x.x.x x.x.x.x CpaGD227W0Cy2BA1Tf
>> HTTP 0 EXTRACT application/vnd.openxmlformats
>> -officedocument.spreadsheetml.sheet 0.258350 - F *219414* *12977556* 0 0
>> F - - - - extract-1517528771.04222-HTTP-Fz2Z2m3zwQcc3gqDS3 F -
>>
>> File on disk:
>> *219414* Feb  1 16:04 extract-1517528771.04222-HTTP-Fz2Z2m3zwQcc3gqDS3
>>
>> The file on disk is the same size as the amount of bytes sent to the file
>> analyzer (seen_bytes field) -- it should be the same size as the
>> total_bytes field. I've seen this happen many times (though, relatively
>> speaking, it is rare).
>>
>> Any thoughts on this behavior? I'm seeing this on Bro 2.5.1.
>>
>> Thanks,
>> Josh
>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180204/2a94f087/attachment.html