[Bro-Dev] [JIRA] (BIT-1240) TCP gaps inserted in wrong place

[Jon Siwek (JIRA)]

    [ https://bro-tracker.atlassian.net/browse/BIT-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18001#comment-18001 ] 

Jon Siwek commented on BIT-1240:
--------------------------------

{quote}
However the extracted file has nulls padded at the beginning, but they should be at offset 1448.
{quote}

Or rather the gap should start at byte offset 1146, I think?  The first packet w/ reply content has 1448 bytes of data, but 302 are HTTP-specific stuff (e.g. headers), not related to the "file".  So 1448 - 302 = 1146.

{quote}
Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count.  Should gaps always be signalled in order with the data 
{quote}

Yes, that is how it should work, thanks for the bug report.

{quote}
this also affects users of content_gap, who would receive the data and hole "out of order"?
{quote}

If you literally mean the "content_gap" event, I'm guessing that's unaffected as the issue isn't related to TCP reassembly/analysis.  The source of the problem is specifically w/ how MIME entity data is buffered and passed along to protocol analyzers (and then to file analyzers) in discrete amounts, but gaps are always reported right away.  So either gaps need to be buffered along w/ the data, or the buffering needs to be removed.  I'll see if I can do the later as that may have additional benefits of decoupling HTTP file analysis from some things it's not supposed to be coupled with...

> TCP gaps inserted in wrong place
> --------------------------------
>
>                 Key: BIT-1240
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1240
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: Bro
>    Affects Versions: git/master, 2.3
>         Environment: CentOS 6
>            Reporter: Jimmy Jones
>         Attachments: get-hole1.trace
>
>
> Using attached test file, I tried using the file analysis framework to extract out the payload, which is a copy of one of the bro testcases but with a packet removed.
> However the extracted file has nulls padded at the beginning, but they should be at offset 1448. Looking at what happens, the gap is signalled to File::Gap before the data is received in File::DataIn, which calculates the offset to write the payload by adding the seen bytes to the missing byte count. Should gaps always be signalled in order with the data - this also affects users of content_gap, who would receive the data and hole "out of order"?
> Used the following bro script:
> event file_new(f: fa_file)
> {
>   Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=f$id]);
> }



--
This message was sent by Atlassian JIRA
(v6.4-OD-04-006#64001)